The Implementation and Analysis of Facial Expression Customization for a Social Robot

2.1 Generation of robot facial expressions

Social robots use various means such as facial expressions^[19-22], sounds^[23-25], colors^[23,24,26], and motions^{[23,24][26,28]} to express emotions. Facial expressions convey a considerable proportion of the emotional information^[14,29]. There are two ways for a robot to interact through robot facial expressions: either imitate the facial expressions of a user, or generate an emotion for the robot through a decision-making process. For social interaction, which is the main function of social robots, it does not suffice to imitate human facial expressions alone. Therefore, studies that explore how robots can express their own specific emotions, distinct from the user emotion, may play an important role in unlocking a richer user experience of social robots^[10]. A number of studies have considered the concept of emotion space to allow a robot to generate its own emotions^[30,31]. A robot using emotion space is required to have access to a process that can generate robot facial expressions that correspond to the robot’s intended emotional expression^[32]. In addition, many attempts have been made to obtain a more natural mode of expression on a robot’s face, including studies that naturally implement emotional expression by applying dynamical processes to the transitions of robot emotion^[33,34]. Park et al. added physiological movement to create more natural and diverse robot facial expressions^[35,36]. Prajapati et al. implemented a process that allowed a user’s facial expression to naturally affect the robot’s responsive facial expression through kinetic emotions that use emotional information extracted from the user’s face^[37].

2.2 Preference of robot facial expression

In general, a particular robot product has a pre-set range of facial expressions, limiting the robot’s ability to satisfy the preferences of various users and to form deeply-connected HRR. The factors that influence the preferences of any particular user are as follows:

The facial expression of the preferred robot varies depending on whether the user’s personality is introverted or extroverted ^[38-40]; The degree of vitality and abundance of the preferred facial expressions depends on the user’s nationality^[41]; Certain people prefer robots with funny expressions^[42]; The preference for an asymmetric facial expression depends on the user’s dominant hand direction (left or right)^[43]. Asymmetric robot facial expression can enhance human-robot interaction, for example, by encouraging communication in interactions between robots and teenagers or by inducing favorable responses among consumers^[17,44]; Cultural interpretations also influence users’ recognition of robot facial expressions. Barrett et al. argued that people are active constructors of the classification of facial movements using culturally-learned emotional concepts^[45,46];

Preferred robot facial expressions depend on factors such as personality, nationality, and culture. It is not straightforward to establish robot facial expressions that will satisfy all users. This problem may be mitigated by an appropriate customization tool that can flexibly change a robot facial expression according to the user’s desire.

2.3 Customization of robot expression

Certain studies have allowed users to change a robot’s expression through customization in a flexible manner. For example, teenagers have been involved in robot customization^[1]. Furthermore, FLEXI, a robot with customizable body shapes and facial expressions, has been designed to meet the needs of various users^[16]. Another example demonstrated that a customized process increased intimacy with robots and assisted with forming social relationships^[47]. However, most studies have failed to implement dynamic changes in facial expression in a robot by customizing only static facial expressions. In addition, no analyses of the customization process and the user experience of generated robot expressions have been conducted in depth. This study addresses both of these factors.

3.1 Prototype robot

The prototype robot shown in [Fig. 1] was used in this study. The robot was used to enable users to interact with a social robot that expressed its emotions through its face. The prototype robot was designed to be simple in order to minimize the impact of certain appearances. The robot had a built-in 4-inch LCD screen and was driven by connecting an HDMI cable to a PC to display the robot facial expression. To blend in with the screen, the overall shape of the prototype was designed using the metaphor of a helmet. The robot facial expression shown in [Fig. 2] is a neutral facial expression with all control points at zero. The robot facial expression included ten control points, each designed to control a specific element of emotional expression. The elements that constitute a robot’s face are its two eyes and mouth, which contain the most information about emotion. The ten control points were denoted CP_{i = 1⋯10} and each control point had a normalized value between -1 and 1. The robot’s eyes included pupils and eyelids. Both pupils always pointed in the same direction. The eyelid structure was designed to accommodate an eyebrow structure as well, using dark colors. These control points allowed the robot to express various facial expressions with a simple underlying structure. The shapes of the eyes were determined by seven control points (CP₁ ~ CP₇);

[Fig. 1]

The prototype robot with 4 inch display (robot size: 21 cm ×18 cm ×12 cm)

[Fig. 2]

Prototype face control point (CP), 1: eye size, 2: pupil size, 3: angle of eyelid, 4: height of left eyelid, 5: height of right eyelid, 6: vertical location of pupil, 7: horizontal location of pupil, 8: mouth width, 9: mouth height, 10: mouth corner height. All CPs are set to zero in this example

• ‧ CP₁ (eye size): This could be adjusted from 80 ~ 120% of the default eye size.
• ‧ CP₂ (pupil size): This could be adjusted from 75 ~ 125% of the default pupil size.
• ‧ CP₃ (angle of eyelid): Relative to the horizontal axis of the screen, eyelids could rotate from -30 ~ +30 degrees. The angles of the left and right eyelids were reversed to make the angles of the two eyelids symmetrical.
• ‧ CP₄, CP₅ (height of left and right eyelids): By adjusting the distance between the eyelid and the center of the eye, the degree of eyelid closure could be controlled (-1: fully closed, 1: fully opened).
• ‧ CP₆, CP₇ (vertical and horizontal positions of pupils): CP₆ determined the y-position and CP₇ determined the x-position. When CP₆ and CP₇ were both equal to 1, the pupil met the edge of the eye in the northeast direction of the eye.

The shape of the mouth was determined by three control points (CP₈ ~ CP₁₀);

• ‧ CP₈ (mouth width): This could be adjusted from 30 ~ 170% of the default mouth width
• ‧ CP₉ (mouth height): This could be adjusted from 0 ~ 200% of the default mouth height.
• ‧ CP₁₀ (mouth corner height): When the value was 0, the corner of the mouth was located in the center of the mouth height. The corners of the mouth moved upward when this value was positive and downward when the value was negative.

3.2 Robot face simulator

As shown in [Fig. 3], the robot face simulator consisted of a frontend and a back-end. JavaScript React was used on the front-end and was used to create the robot facial expression in the browser environment. In the back-end, there was an emotion engine written in Python language. In order to express emotion dynamically, the emotion engine received a stimulus signal as an input and calculated the value of each control point in response. When communicating between the front-end and the back-end, stimulus signals were transmitted from the front-end to the back-end, and control point values were transmitted from the back-end to the front-end. The robot face simulator operated at a sampling frequency of 100 Hz.

[Fig. 3]

Structure of robot face simulator

The operation of the robot face simulator is represented by the following set of equations and their clarifying descriptions. First, we present the stimulus signal vector:

In the front-end, when a user pressed a button corresponding to a certain emotion, a stimulus corresponding to that emotion was generated and transmitted to the back-end. The stimulus signal vector (r) in Eq. (1) consists of a 6-dimensional vector expressed as γ₁ to γ₆. The index i in γ_i represents the emotion number, with the order proceeding through sadness, happiness, anger, disgust, fear, and surprise. For example, when the ‘happiness’ button was pressed by a user, the value of γ₂ was maintained at 1 for about 2 seconds.

The stimulus signal vector (r) received by the back-end was used as an input value for the emotion engine. The emotion engine used in the back-end referenced a mental model^[33] and a linear affect space model^[48]. Second, we present the force vector:

The force vector (s) is calculated by multiplying the stimulus signal vector (r) by K, as expressed in Eq. (2)^[48]. This vector acts as a force applied to an emotion in the emotion dynamics. Next, we present the equation of emotion:

Eq. (3) expresses the second-order equation of emotion used in Miwa’s mental model^[33]. The force vector (s) drives the emotion vector (e) in the emotion dynamics. The emotion vector (e) represents the coordinate position of the emotion in the emotion space. The values of the parameters in Eq. (3) are as follows:

The values of M (inertia), C (viscosity), and K (elasticity) expressed in Eq. (4) determine the degree to which the robot facial expression changes naturally. These values can be adjusted by the user, but in this experiment we fixed the heuristically determined values to focus on the customization of facial expressions. Finally, we present the control point vector:

In Eq. (5), T is a transformation matrix that converts the 6-dimensional emotion vector (e) into the 10-dimensional control point vector (p). The T matrix is a collection of control point values for each of the six emotions obtained when a user completes the customization process. Therefore, if the T matrix changes when the robot expresses the same emotion, the control point vector (p) also changes. Finally, the resulting control point vector (p) is transmitted to the front-end for creating the robot facial expression.

This model renders facial expressions more naturally and allows a dynamic expression of the changes between successive emotional expressions. In addition, if the user has a customized T matrix, it has the advantage that a dynamic robot facial expression may be generated immediately. These functions allow users to customize their own robots without any additional work from the developer when using the customizable robot.

3.3 Customization tool

In order to customize the robot facial expression, the participants were required to adjust the robot control point for each emotion. As shown in [Fig. 4], the customization tool was implemented in the front-end to enable experimentation in the browser environment. Each control point could be adjusted using a slide bar, with values varying from -1 to 1. An emotion button was present in the lower left corner of the monitor. When the emotion button was pressed, ‘selected’ was displayed and highlighted in green.

[Fig. 4]

The customization tool for customizing the robot’s facial expression

4.1 Hypotheses

As discussed in Section 2, one of the aims of the study was to explore how customized robot facial expressions would affect the HRR. An additional aim was to establish how the customization experience was evaluated by the users.

Consequently, our hypotheses were set up as follows:

• ‧ H1 A more positive effect on HRR is achieved when using a customized version rather than a predefined version of robot facial expressions.
• ‧ H2 Users experience different workload and satisfaction levels while they are customizing each emotional expression of the robot.
• ‧ H3 The customization process itself is interesting and valuable to users

4.2 Participants

A total of forty-four participants (twenty-seven males and seventeen females) were recruited. Participants provided age and gender information and consented to participate in the experiment in advance. Participants’ ages ranged from 19 to 40 years, with a mean of 23.7 (SD = 3.33 years). Data were collected from all participants without exception.

4.3 Procedure

As shown in [Fig. 5], the experimental environment included a prototype robot, a monitor, and a PC to run the customization tool. A mouse and keyboard were provided to the participants so that they could use the customization tool displayed on the monitor. The experimental sequence was designed with the assumption that it would be a user’s first use of a customizable social robot. The overall experimental sequence proceeded as follows: First, individual participants interacted with the robot, reflecting the predefined version of robot facial expression. Second, the robot facial expression was customized through the PC. Finally, individual participants interacted with their respective customized versions.

[Fig. 5]

Overview of the experimental environment

4.4 Measures

Measures obtained from users included their responses to the questionnaire and time data measured during customization. After providing sentences for evaluating each index, participants were asked to rate their degree of agreement with these sentences on a Likert 7-point scale. A measure of 1 on the scale meant “strongly disagree” and a measure of 7 meant “strongly agree”. For example, for the topic of difficulty, the participants were given the sentence “Customizing this emotion was difficult.”.

After a user customized the robot facial expression, the difficulty and satisfaction of each emotion were rated. Concurrently, the elapsed time used by the participant to customize the facial expression corresponding to each emotion was measured. Difficulty and elapsed time were used as factors to investigate the workload.

After all the experiments had been completed, the predefined version and the customized versions were evaluated for intimacy, empathy, and enjoyment, respectively. Finally, the participants evaluated the customization tool in terms of interest and value indices. Participants’ opinions were acquired through an interview, which included the following questions:

• ‧ Q1 “If you are actually using this social robot, which robot do you want to use, predefined or customized? What is the reason?”
• ‧ Q2 “What did you like and dislike when customizing the robot facial expressions for each emotion?”

Through the answers to the above questions, detailed opinions that were not covered in the evaluation indices were acquired.

5.1 Comparison of predefined and customized version of robot facial expressions

A paired t-test (student’s t-test) was conducted to compare the predefined version and the customized version in terms of intimacy, empathy, and enjoyment. As shown in [Table 1], the results for the predefined version and the customized version displayed significant differences. The customized version was rated more highly than the predefined version in terms of intimacy (MD = -0.66). The predefined version and customized version exhibited an average difference of MD = -0.75 in the empathy index. In term of enjoyment, the two versions also exhibited a significant difference (MD = -0.55). In summary, the customized version of robot facial expression was rated more highly in terms of intimacy, empathy, and enjoyment than the predefined version.

[Table 1]

Comparison between predefined version and customized version with paired t-test (student’s t-test), P: Predefined version, C: Customized version

5.2 Analysis of workload (difficulty, elapsed time) and satisfaction

Calculations using Repeated Measures ANOVA (RM-ANOVA) were conducted to ascertain the difference in workload and satisfaction associated with the respective emotions. As the sample size was 44 (n > 30), it was assumed that a normal distribution would be satisfied, applying the central limit theorem. For the sphericity test, Mauchly’s test was conducted. We ascertained that Mauchly’s test of sphericity had been violated in all the indices. Therefore, the Greenhouse-Geisser sphericity estimate was used to correct the degrees of freedom for the F-distribution.

5.2.1 Difficulty

As shown in [Table 2], sadness, happiness, anger, and surprise displayed significant differences with respect to disgust and fear. Further, no significant differences between sadness, happiness, anger, and surprise were noted. Similarly, no significant difference was noted between disgust and fear. As displayed in [Fig. 7(a)], the set of emotions {sadness, happiness, anger, surprise} were associated with significantly lower levels of difficulty than {disgust, fear}.

[Table 2]

Difficulty analysis according to emotions using RM-ANOVA

[Fig. 7]

Box plot: (a) Difficulty, (b) Satisfaction, and (c) Elapsed time associated with respective emotion

5.2.2 Satisfaction

[Table 3] clearly reveals the relationships between happiness, anger, disgust, fear, and surprise. There were no significant differences between happiness, anger, and surprise. Similarly, there was no significant difference between disgust and fear. Notably, the comparison between these two groups of emotions, disgust and fear, consistently indicated a significant difference (p < 0.05). The set of emotions {happiness, anger, surprise} were associated with significantly lower levels of difficulty than {disgust, fear}. Sadness could not be clearly ordered. Sadness displayed a significant difference only with respect to anger and fear (p_sad-anger = 0.013, p_sad-fear = 0.011). As shown in [Fig. 7(b)], sadness is located between the two groups {happiness, anger, surprise}, and {disgust, fear} in terms of satisfaction.

[Table 3]

Satisfaction analysis according to emotions using RM-ANOVA

5.2.3 Elapsed time

[Table 4] clearly reveals the relationships between sadness, anger, disgust, fear, and surprise. There were no significant differences between sadness, anger, and surprise. Similarly, there was no significant difference between disgust and fear. The comparison between these two groups of emotions consistently showed a significant difference (p < 0.001). Therefore, the order of emotions is {sadness, anger, surprise} < {disgust, fear} with respect to elapsed time. Happiness could not be clearly ordered. However, happiness did not display a significant difference with respect to disgust, and surprise (p > 0.05), whereas it did display a significant difference with respect to sadness, anger and fear (p < 0.05). In [Fig. 7(c)], happiness is located between the two groups {sadness, anger, surprise}, and {disgust, fear} in terms of elapsed time.

[Table 4]

Elapse time analysis according to emotions using RM-ANOVA

5.3 Analysis of customization process

We obtained data, assessing interest and value for the respective customization processes. Interest was calculated with an average point value of 6.52 and a standard deviation of 0.698. The mean of the value data was 6.41 points and the standard deviation was 0.972. These two mean scores both lay between 6 and 7 on the 7-point Likert scale.

6.1 The impact of a customizable robot

The results of [Table 1] indicate that the customized version was rated higher than the predefined version for all indices, and these differences were found to be statistically significant. This result implies that users felt intimate with the customized version, empathized to a greater degree with the robot facial expressions, and enjoyed interacting with it. The same conclusions could be drawn from the results of the interviews. In response to the question Q1, 30 out of 44 participants chose the customized version. Participants explained their preferences for the customized version by using words including ‘intimacy’ (P24), ‘empathy’ (P17, P22, P27, P35), and ‘enjoyment’ (P18). Eight participants answered: ‘Suitable for me.’. Four participants emphasized the need for a customized version, noting that each person’s emotions and facial expressions are subtly different. Participant P24 expressed that the process of creating the robot facial expression imbued the robot with character and personality. Participant P36 emphasized that the customized version felt like his own personal robot and expressed a feeling of possessiveness towards the robot.

These results confirm hypothesis H1. Users generated customized versions of the robot as a partner with a preferred facial expression, character, and personality through the customization process. Through this process, users experienced feelings of intimacy, empathy, and enjoyment towards the customized version. These are attributes that improve HRR. We may conclude that the availability of a customized version of robot facial expression has a more positive effect on HRR than a predefined version.

6.2 Design guideline for customizing robot facial expressions

As revealed by the results, the more difficult it was to customize the robot facial expression, the longer it took and the less satisfying it was. Furthermore, the experienced workload and satisfaction associated with customizing robot facial expression varied across the respective emotions. Mostly, participants experienced greater difficulty and longer elapsed time in customizing the facial expressions of disgust and fear than for other emotions. We are able to identify the root causes of these results by analyzing the interviews. The causes are classified into three categories. First, when the user’s desired facial expression element (control point) was not available, the evaluations were different for each emotion. Fifteen participants wanted other control points to create the desired expression, mainly in the case of disgust and fear, but several control points were mentioned in the case of sadness and happiness. Examples of control points that were mentioned are ‘round eyelids’, ‘grimacing expression’, ‘distance between pupils’. Another complicating factor is the scenario where different robot facial expressions could be chosen for the same emotion, depending on the situation. Seven participants commented about this scenario and felt that this happened in the case of fear. An example of a participant’s statement is as follows. “I couldn’t think of a representative facial expression because the facial expression was different depending on the situation in which the fear was expressed.” (P24). A final complicating factor occurred when the user was not familiar with making a robot facial expression and did not know how to express it. Eight participants noted that they were not familiar in creating disgust emotions. Nine participants mentioned that it was not easy to imagine how to express the robot’s emotions of disgust and fear.

We conclude that hypothesis H2 was confirmed and that there were particular difficulties associated with expressing disgust and fear. It is necessary to pay attention to dealing with the emotions of disgust and fear in the design of robot facial expressions; The following comments summarize the insights gained from this study: An optimized robot facial expression element needs to be added to adequately express disgust and fear; The process of segmenting the situation according to the particular emotion should be required; An improved customization process is needed, such as providing sample robot facial expressions to enable users to identify robot facial expressions of disgust and fear.

6.3 Implications of customization process

The participants expressed very strongly that they were interested in the customization process and that they found it valuable. A comparative analysis was not conducted, but the participants’ interviews indicated which aspects had a positive impact on the customization process. Examples of comments from the interviews follow. First, with respect to interest: “It was exciting to be able to see the robot facial expression change while customizing the robot.” (P18). “It was fun to create a robot expression the way I wanted while using the customization tool.” (P33). Second, with respect to value: “I think I feel affectionate and familiar with the robot because I customized the robot facial expression.” (P24). “I feel more friendly through the process of creating my expression on the robot.” (P40). “Because I performed the customization process, I felt more empathetic to the robot.” (P41).

We conclude that the participants are interested in changing and generating robot facial expressions. Therefore, the customization tool can be utilized for performing additional interaction with the robot to increase user interest. In addition, the process of creating a specialized robot facial expression for individuals is regarded as valuable. Therefore, the customization tool can not only provide the user with an opportunity to create a customized expression but also play a role in amplifying the effect of the customized expressions. Participants noted that the experience of creating their own facial expressions on the robot made them feel more affectionate, friendly and sympathetic towards the robot. These results are consistent with hypothesis H3.