RNN을 이용한 Expressive Talking Head from Speech의 합성

1.1 Related Researches

Research on TH has been done for a long time and various approaches have been proposed.

Hidden Markov Model (HMM) is often used for generation of TH. In the method of Wang et al., TH generation is realized by learning the phoneme sequence and the trajectory of mouth shape using HMM[3]. Since the image obtained as an output is a monochrome image with low resolution, matching between the output image and the high-resolution color image is performed to achieve the corresponding high-resolution image as a realistic output image.

Fan et al. generated TH using a recurrent neural network (RNN) instead of HMM for shape prediction of the face, and gave better results than a method using HMM[2]. RNN has taken the spotlight in recent years as a method that can efficiently learn series signals[4]. For example, it is used for series signals such as machine translation, speech recognition, and video analysis. RNN has a structure in which the hidden layer of the feed-forward neural network (FFNN) has links in the time series direction, so that time series information can be learned. However, neither method trains the relation between phoneme and facial code, so it can only generate TH with fixed facial expression without intonation. In addition, phonemes differ depending on the language, so those TH depends on the language. Moreover, since it cannot be applied to sounds like laughter which cannot be converted to phonemes, there is a problem in using only phonemes as input for generating TH.

Many types of research have been done to add facial expressions to TH in order to realize more natural TH generation. In the sample-based method by Cosatto et. al., a facial part is divided into a forehead, eyebrows, eyes, a jaw, and a mouth, and the shape of each part is adjusted and combined to generate TH with an expression[5]. This is realized by determining the parameters of each part such as the opening width of the mouth and the position of the lips. In [5], the importance of facial expressions during utterance is mentioned, and in addition to the mouth in the example above, by adjusting parameters of other parts such as eyebrows, eyes, chin, and forehead, various expressions can be achieved. However, their method is limited to randomly adding facial expressions to the face generated from phonemes. Furthermore, in order to determine the parameters of each part, facial parts such as eyes and mouth should be detected by image processing. Since for the data set, data such as the size of mouth opening should be collected, the process is weak against noise and detection error. It is a disadvantage that data collection is difficult in this method.

Wan et al. succeeded in adding arbitrary facial expressions by learning the generation of TH based on text input and the expression of TH independently[6]. By adjusting parameters manually, facial expressions of six kinds of emotions such as neutral, kindness, anger, joy, fear, and sadness can be added correspondingly to TH. However, with this method, since it is necessary to adjust the expression parameters manually corresponding to each emotion at the time of generation, it is not possible to generate varying emotion during an utterance.

These TH generation methods cannot generate expressive facial shapes without manual parameters setting. Moreover, intonation cannot be expressed. This is because these methods use phonemes as inputs and information on facial expressions is not used. When speech signals are converted to phonemes, information on expressions and intonation will be lost. For this reason, in these studies, they can only generate TH with a fixed facial expressions unless information on facial expressions is not given manually. In addition, since phonemes are dependent on languages, it is necessary to redesign them all when it is applied to other languages. Even in the same language, facial expressions based on voice, from which phonemes cannot be extracted, such as screams, cannot be reproduced.

[21] attempted to generate an animation of the adjacent area of lip from the voice signal directly. However, it does not generate TH of the whole face, and the evaluation of the generated image is only quantitative reconstruction error.

Similar to the proposed method in this paper, [22] attempts to generate a direct TH from speech using a neural network. However, this does not focus on emotions and intonation included in speech but reports evaluation results of phoneme recognition by human subjects experiment.

1.2 Research Aim

In this research, instead of converting speech signals into phonemes, we propose a method of converting speech to low-level acoustic features instead and generating TH with facial expression and intonation. By using low level acoustic features including information on facial expressions and intonation, it is possible to generate TH with facial expressions and intonation.

The main contribution of this research is to be able to generate TH with facial expression and intonation. Since the low-level acoustic features of speech do not depend on languages and phonemes, there is no need to adjust for each language, and there is an advantage that TH can be generated from special speech such as laughter which cannot be converted to phonemes. In addition, since it is not necessary to manually label low-level acoustic features and facial codes, it is easy to collect large amounts of data sets. Examples of applications of the proposed method are as follows.

2.1 Dataset

We create datasets with labels for training and evaluation based on the Surrey Audio-Visual Expressed Emotion database (SAVEE)[10], which is a dataset of movies (i.e. videos with audio) of frontal face speaker. SAVEE contains movies of English speech as 60 fps video and 44.1 kHz audio, by 4 native English male speakers. For each speaker, 120 sentences with 7 types of emotion are recorded. Lengths of the movies are about 3 seconds. The 7 emotions are neutral, anger, disgust, fear, happiness, sadness and surprise. There are 15 sentences for each emotion except for neutral that are 30 sentences. In this research, we used 120 movies of 1 speaker out of 4, who is identified as DC in the database.

Note that SAVEE provides blue markers painted on speaker's face for ease of extraction of facial landmark points. However, we do not use them because there are too few markers around the mouth to sufficiently describe its deformation. In particular, there are no markers on the inner or outer boundary of the lip, which are crucial for our modeling. Instead, we use a set of 68 landmarks defined in Multi-PIE face database[11].

In order to use the dataset as input and output for the network, we need to preprocess the dataset. As the preprocess for an audio waveform, we extract short segments of waveform at the same time intervals of video frames (i.e. 60 fps) by sliding window as shown in [Fig. 2] We refer to the segments as audio frames. The window size is 33.3 ms and the stride is 16.6 ms. Note that the window size of windows is determined as to extract low revel audio feature in the range between 20 ms and 40 ms, which is according to human characteristic of auditory perception. In the SAVEE database, 120 movies of subject DC are converted to 27765 pairs of image and audio frame in total. We describe feature extraction for these frames in section 3.

Fig. 2

Separate utterance videos into frames

2.2 Sequential regression by RNN

Speech movie is a pair of sequential frames of audio and video that are tied and temporally synchronized. Thus, the relationship between audio features and facial codes can be modeled as regression, which is the main idea of our proposed method.

Speech movie has a nature that the mouth moves prior to the occurrence of the vocal sound. In addition, at the end of the utterance, the mouth is moving to close after the stop of the voice. Therefore, audio features and facial codes in speech movie have dependencies not only on the forward direction of time but also on the reverse direction of time. In order to predict facial codes from audio features, the information in both forward and reverse time directions should be captured. Thus, we employ BLSTM as the regression model whose input is audio feature and output is facial code. By training the regression model, we predict facial codes only from audio signals and then synthesize THs.

In this research, we train and compare 3 variants of 3 layers network. Each network consists of a composition of BLSTM layer and feed-forward layer, which we refer to them as B and F. In particular, we use FBF, BFF, and BBF as shown in [Fig. 3]. We apply ReLU activation function after F layer except for the last F layer, since the output (i.e. facial code) is valued on real vector. The loss function is defined as the mean squared error between predicted outputs and labels. We minimize the loss by the stochastic gradient descent.

Fig. 3

Three types of networks used in this research

3.1 Audio feature

Previous work uses phoneme as audio feature, which is converted from raw audio signal. However, using phoneme leads to discard some information about expression or accent. On the other hand, we employ other low-level audio feature which retains expressive information more than phoneme.

Emotion recognition from voice has a long history and there are comparison studies about audio features that are suited to capture expression or intonation[12,13]. These researches developed low-level audio features that well represent expression in voices. Note that, the information that represents expression or intonation, is information such as pitch or loudness of voice.

Mouth shape is mainly determined by vowels and several consonants. Therefore, audio feature for speech recognition, which must convey information about phonemes, can be considered as correlating to mouth shape. Mel frequency cepstral coefficients (MFCCs) are audio feature that approximates the characteristics of human auditory perception. In particular, MFCC itself and its dynamics (ΔMFCC, Δ²MFCC) are classical features that are universally used for speech recognition[14]. In addition, F0 and energy are the most important features for emotion recognition[12]. F0 represents the perceptual pitch of the voice and energy represents the strength of the voice. Therefore, in this research, we employ MFCC, energy, their dynamics, and F0 as the feature that captures mouth movement and expression.

We extract the feature from all audio frames in the dataset. We use Hidden Markov Model Toolkit (HTK)[15] to extract MFCC, energy and their dynamics, and we use SPTK[16] to extract F0. More precisely, we extract MFCC with 0.97 of pre-emphasis coefficient and 20 channels of Mel filter bank, and we use 12 lower coefficients as MFCC feature. As the result, we obtain 40 dimensional vector of audio feature by concatenating them as (c; e; Δ(c); Δ(e);Δ² (c);Δ² (e);z) , where c∈ ¹² is MFCC, e∈ is energy, z∈ is F0 and Δ(⋅) is an operator that compute regression coefficients.

3.2 Facial code

Directly predicting facial images from audio features by a prediction model is difficult since raw facial images are very high dimensional signal. Thus, we have to describe images by low dimensional codes that can be approximately reconstructed to original images. Active appearance model (AAM) is a well-established technique that can efficiently encode and decode a face image[17]. AAM is a parametric model of deformable objects in a 2D image, which is described by shape parameters and appearance parameters. AAM can represent facial images of a variety of individuals or expressions by shape and appearance parameters, which are weights of the principal component analysis (PCA). In this research, we employ shape and appearance parameters as facial code. In particular, let p∈ 8 be the shape parameters and lambda in λ∈ 8 be the appearance parameters of AMM, then we define our facial code as their concatenation (p;λ).

In order to encode images in the dataset into facial codes, we first construct AAM then encode the videos into the sequences of facial codes by the AAM. Since AAM uses iterative registration and requires a good initial crop of image that is close to the location of the face. Therefore, for the first image in a video, we apply face detection to obtain the location of the face. For the subsequent images the location of the face estimated in the previous frame is used as the initial value.

Since AAM is based on PCA, it requires a lot of manually annotated images as training data to construct a model that appropriately represent a variety of faces. As described in 2.1, we manually annotate facial key points in images in the dataset for training AAM. Lucas-Kanade algorithm[19] is used for fitting. For implementation, we used Menpo library[18].

4.1 Details of experimental setup

4.2 Details of networks and training

We set the equal number of units for each layer in a network. In particular, for the proposed method, the number of units is 312 for FBF, 418 for BFF and 262 for BBF. The number of units are adjusted to match the number of parameters (i.e. about 600,000) among networks. For the comparative methods, the number of units is 624 for FBF, 806 for BFF and 516 for BBF. The number of parameters is about 2,450,000. We applied Dropout[23] with 0.5 of the drop rate to first and second layers. We used RMSprop[20] for optimization with hyperparameter $$ \gamma =0.99,\in =1e-8$$ and the learning rate is 0.1 for the proposed method and 0.001 for the comparative method.

4.3 Quantitative evaluation of the prediction models

We evaluate the performance of prediction of facial code by comparing the proposed method that is based on low-level audio feature, and the comparative method that is based on phoneme and emotion label as described in the 4.1.2. For quantitative evaluation, we define a performance metric CORR, which is an average over correlations between sequences of predicted facial codes and ground truth. It is more interpretable than mean squared error. More precisely, CORR is defined as:M3

Where N_video is the number of videos, D_n is a number of frames of the n-th video, $$ \widehat{{\upsilon }_d^n}$$ and $$ {\upsilon }_d^n$$ are prediction and ground truth of facial codes of d-th frame in the n-th video, corr(·,·) is the normalized correlation. Note that CORR indicates the fitness of predicted facial codes to the ground truth and higher the better, especially maximum and minimum of CORR are 1 and 0. [Table 1] shows the result of quantitative evaluation. The best network in the proposed method was BBF, which CORR was 0.38857. The best network in the comparative method was also BBF, which CORR was 0.17053. The result suggests the proposed method outperformed the comparative method.

Table 1

CORR value of predicted result

4.4 Qualitative evaluation for the synthesized talking heads

We quantitatively evaluated synthesized THs, which are sequences of decoded images from facial codes predicted from audio features by the best prediction models of both proposed method and comparative method. We conducted subjective evaluation test by asking human subjects whether the vocal expression (i.e. emotion or accent) contained in emotional speech is reflected to the synthesized TH as facial expression. We investigated by questionnaire for 62 human subjects. 15 subjects are Europeans and North Americans, and 47 subjects are Asian.

Examples of synthesized THs are shown in [Fig. 4] and [Fig. 5].

Fig. 4

TH images of every twentieth frame. Top images are original, middle images are proposed, and bottom images are phoneme-based TH. The proposed method successfully reproduced the mouth movements such as opening widely or puckering his mouth, which exist in the original video

Fig. 5

TH images of every fifth frames. Top images are original, middle images are proposed, and bottom images are phoneme-based TH. The proposed method produces more dynamic expression than another

The questionnaire items and results are as described below. Note that the subjects do not know which THs come from the proposed method or the comparative method through experiment.

• (ⅰ)  “Watch a TH without voice (i.e. video only), the answer which emotion does the TH express”. See [Fig. 6] for the result.
• (ⅱ)  “Watch two THs with voice (synthesize by proposed and comparative method), then answer which TH is more correctly reproducing the vocal expression”. The result is that 77% of subjects answered the proposed method was better than the comparison method.
• (ⅲ)  “Watch two THs with voice (synthesize by proposed and comparative method), then answer the degree of compatibility between video and voice on a five-point scale”. See [Fig. 7] for the result.
• (ⅳ)  “Watch two THs with voice (synthesize by proposed and comparative method), then answer the naturalness on a five-point scale”. See [Fig. 8] for the result.

Fig. 6

Emotion estimation result

Fig. 7

Degree of compatibility of TH with audio (Five levels evaluation)

Fig. 8

Degree of naturalness of TH (Five levels evaluation)

4.5 Discussion

For quantitative evaluation, BBF performed the best result both in the proposed and comparative method. This result suggests that applying recurrent connection in early layer performs better than in late layer, and multiple recurrent layers also increase prediction accuracy.

For THs synthesized by both methods, we observed that the mouth was moving according to the speech sentence. However, THs of the comparative method lacked dynamics such as opening the mouth wider according to vocal accent. On the other hand, THs of the proposed method successfully reproduced accent or small movement of the mouth.

For qualitative evaluation, human subjects more correctly answered true emotion from TH without voice for the proposed method than the comparative method, as shown in [Fig. 6]. However, the accuracy is indeed not high in both methods. One of the causes of the low accuracy comes from the difficulty of the problem itself. Since facial expression of anger and disgust, or fear and surprise are similar to each other, distinguishing them without voice is difficult even for experienced people.

On the other hand, in terms of compatibility and naturalness for TH with voice, the proposed method achieved better scores.

From these results, we observe that how much vocal accent is appropriately reflected in synthesized TH is important for watcher to make an impression of naturalness. Because in comparative method, true information about emotion is additionally given but information about accent was not used.

In the case treating 7 classes of emotion as categorical information as our proposed or comparative methods, it is feasible to synthesize expressive THs by manually adjusting parameters of emotion. However, it is difficult by hand to adjust appropriate non-symbolic parameters such as accent or intonation. Since our method can incorporate such information in a natural way, THs that further match speech voice can be synthesized.