Database of video images of natural emotional facial expressions: perception of emotions and automated analysis of facial structure

Королькова О.А., Лободинская Е.А. База видеоизображений естественных эмоциональных экспрессий: восприятие эмоций и автоматизированный анализ мимики лица // Оптический журнал. 2022. Т. 89. № 8. С. 97–103. http://doi.org/10.17586/1023-5086-2022-89-08-97-103

Korolkova O.A., Lobodinskaya E.A. Database of video images of natural emotional facial expressions: perception of emotions and automated analysis of facial structure [in Russian] // Opticheskii Zhurnal. 2022. V. 89. № 8. P. 97–103. http://doi.org/10.17586/1023-5086-2022-89-08-97-103

O. A. Korolkova and E. A. Lobodinskaya, "Database of video images of natural emotional facial expressions: perception of emotions and automated analysis of facial structure," Journal of Optical Technology. 89(8), 498-501 (2022). https://doi.org/10.1364/JOT.89.000498

Subject of study. The development of methods for automated recognition of emotions on a face requires the creation of new stimuli sets that comprise dynamic natural emotional expressions. To date, no such databases have been designed or validated on the Russian population. Aim of study. The aims of this study were to develop a database of dynamic recordings of natural emotional facial expressions and to compare the profiles of subjective evaluation of these recordings with the results of an automated analysis of facial gestures. Method. Emotional profiles of video images of facial expressions were based on their evaluation by human observers. Study participants (N=250) rated the intensity of 33 differential emotions on a 5-point scale in each of 210 video fragments containing the natural expressions of 5 models. An automated analysis of facial structure in the video fragments of expressions was performed using OpenFace 2.0 to quantify the dynamic changes on the models’ faces. The emotional profiles of video fragments were compared with the results of automated mapping using representational similarity analysis. We calculated the rank correlation coefficient between matrices that represent the structure of subjective evaluation of expressions and their formal description. Additionally, we performed k-means cluster analysis based on subjective evaluation to identify the categorical structure of perceived emotional states. Main results. The representational similarity analysis demonstrated a significant positive correlation between subjective evaluation of expressions and their description in terms of facial actions. However, the correlation was low (0.214), which suggested a substantial variability of mimic patterns that can be subjectively perceived as similar emotions. A cluster analysis revealed five clusters corresponding to basic emotions: attention, joy, surprise, sadness, and disgust. Practical significance. The developed database of natural emotional expressions will be of interest to researchers in the field of affective computing, particularly, for the development of more effective methods for the recognition of users’ emotional states and more accurate simulation of emotional responses in robotic systems.

The research was supported by RSF within the scientific project No. 18-18-00350-П "Perception within nonverbal communication".