ru/ ru

ISSN: 1023-5086


ISSN: 1023-5086

Scientific and technical

Opticheskii Zhurnal

A full-text English translation of the journal is published by Optica Publishing Group under the title “Journal of Optical Technology”

Article submission Подать статью
Больше информации Back

DOI: 10.17586/1023-5086-2022-89-08-33-42

УДК: 612.84, 528.854

Hardware and software complex for the detection and identification of informative features in images of means of nonverbal communication

For Russian citation (Opticheskii Zhurnal):

Жукова О.В., Шелепин Ю.Е., Нариса Нан Чу, По-Лей Ли, Хао-Тенг Сюй, Пронин С.В., Шелепин Е.Ю., Васильев П.П., Лебедев В.С., Моисеенко Г.А., Морозов С.А. Аппаратно-программный комплекс для поиска и выделения информативных признаков в изображениях средств невербальной коммуникации // Оптический журнал. 2022. Т. 89. № 8. С. 33–42.


Zhukova O.V., Shelepin Yu.E., Chu N.N., Lee P.-L., Hsu H.-T., Pronin S.V., Shelepin E.Yu., Vasiljev P.P., Lebedev V.S., Moiseenko G.A., Morozov S.A. Hardware and software complex for the detection and identification of informative features in images of means of nonverbal communication  [in Russian] // Opticheskii Zhurnal. 2022. V. 89. № 8. P. 33–42.

For citation (Journal of Optical Technology):

O. V. Zhukova, Yu. E. Shelepin, N. N. Chu, P.-L. Lee, H.-T. Hsu, S. V. Pronin, E. Yu. Shelepin, P. P. Vasil’ev, V. S. Lebedev, G. A. Moiseenko, and S. A. Morozov, "Hardware and software complex for the detection and identification of informative features in images of means of nonverbal communication," Journal of Optical Technology. 89(8), 454-460 (2022).


Subject of study. Techniques for identifying informative features in dynamic images (video) of means of nonverbal communication of operators during a joint constructive activity were the subject of this study. Aim of study. This study aimed to develop methods for detecting and identifying significant features of nonverbal communication in the received dynamic (video) images of faces of partners, i.e., key features in the joint activity of partners upon disabling the verbal communication channel. Method. A technique for interaction was developed, with a hardware and software complex designed for identifying communication markers in images of facial expressions and eye movements of partners during a joint play activity, in which nonverbal communication is a key aspect and the speech communication channel is disabled. The joint activity involved one partner searching for a hidden target in the perceived image using a nonverbal clue from the second partner, who views the target position on his screen. Partners were chosen among representatives of two geographically distant cultures—Russian and Chinese. The interactions of the partners during their work with images of the object of activity received via simultaneous video communication in two research centers located in western and eastern Eurasia were evaluated through simultaneously recorded physiological parameters. Main results. A new algorithm for investigating images as a means of nonverbal interaction for ensuring the achievement of a shared goal without the speech communication channel was designed. The effectiveness of eye movements of the partners, which are the key features of nonverbal communication, was demonstrated during the constructive joint activity. The task was presented to Chinese and Russian test subjects in the form of English text. Based on the performed measurements, nonverbal interaction in the task of looking for the target position functions similarly for representatives of both cultures and depends on the roles of the partners and the type of executed task. The correlation between the activities of different neural networks of the brain that ensure the recognition of images during the target search in nonverbal communication was identified. An analysis of the images of eye movement during the communication isolated a classic pattern of eye capture for guided and guiding partners. Constant switching occurs between the assessment of images of partners’ eyes and the target search in the entire image. The obtained results suggest that 1) the cognitive process of recognition of facial expressions is probably the most complex visual cognitive process, and 2) universal behavioral algorithms exist when a common joint activity understandable to both partners is performed. Practical significance. A hardware and software complex for detecting significant features of nonverbal communication in the images of faces was designed. The working capacity of the proposed method for detecting features was demonstrated using clear basic signals of nonverbal communication, thus enabling new methods of training artificial neural networks to ensure intuitive nonverbal communication between machines and humans via the recognition of hidden markers of nonverbal communication.


image processing and synthesis, image perception, video communication, nonverbal communication, image recognition, facial expressions, neural networks, eye movements


The research was supported by RFBR and Ministry of Science and Technology of Taiwan within the scientific project No. 21-515-52004.

OCIS codes: 100.4996, 330.2210, 330.6110


1. H. Schober, “Informationstheorie in optik und fernsehen,” Optik 13, 350–364 (1956).
2. Yu. E. Shelepin, A. K. Kharauzov, O. V. Zhukova, S. V. Pronin, M. S. Kupriyanov, and O. V. Tsvetkov, “Masking and detection of hidden signals in dynamic images,” J. Opt. Technol. 87(10), 624–632 (2020) [Opt. Zh. 87(10), 89–102 (2020)].
3. E. Yu. Malakhova, E. Yu. Shelepin, and R. O. Malashin, “Temporal data processing from webcam eye tracking using artificial neural networks,” J. Opt. Technol. 85(3), 186–188 (2018) [Opt. Zh. 85(3), 77–80 (2018)].
4. I. Wang and J. Ruiz, “Examining the use of nonverbal communication in virtual agents,” Int. J. Hum. Comput. Interact. 37(17), 1648–1673 (2021).
5. G. Buzsaki, Rhythms of the Brain (Oxford University Press, Oxford, 2006).
6. J. Rimmele, J. Gross, S. Molholm, and A. Keitel, Brain Oscillations in Human Communication (Frontiers Media, Lausanne, 2018), pp. 1–4.

7. V. A. Barabanshchikov, Facial Expressions and Perception Thereof (Izdatel’stvo “Institut Psikhologii RAN,” Moscow, 2012).