The common nature of eye-movement algorithms that ensure that genre scenes will be recognized in texts and in images

Shelepin, E.Y., Zhukova, O.V., Pronin, S.V., Zhashchirinskaya, O.V., Shelepin, Y.E.

Full text «Opticheskii Zhurnal»

Full text on elibrary.ru

Publication in Journal of Optical Technology

For Russian citation (Opticheskii Zhurnal):

Шелепин Е.Ю., Жукова О.В., Пронин С.В., Защиринская О.В., Шелепин Ю.Е. Общность алгоритмов движений глаз, обеспечивающих распознавание жанровых сцен в текстах и в изображениях // Оптический журнал. 2019. Т. 86. № 11. С. 79–89. http://doi.org/10.17586/1023-5086-2019-86-11-79-89

Shelepin E.Yu., Zhukova O.V., Pronin S.V., Zhashchirinskaya O.V., Shelepin Yu.E. The common nature of eye-movement algorithms that ensure that genre scenes will be recognized in texts and in images [in Russian] // Opticheskii Zhurnal. 2019. V. 86. № 11. P. 79–89. http://doi.org/10.17586/1023-5086-2019-86-11-79-89

For citation (Journal of Optical Technology):

E. Yu. Shelepin, O. V. Zhukova, S. V. Pronin, O. V. Zhashchirinskaya, and Yu. E. Shelepin, "The common nature of eye-movement algorithms that ensure that genre scenes will be recognized in texts and in images," Journal of Optical Technology. 86(11), 739-747 (2019). https://doi.org/10.1364/JOT.86.000739

Abstract:

This paper discusses the characteristics of eye movement in tasks of recognizing texts and comics with descriptions of dynamic genre scenes. The interconnection of the semantic space of the comics mentally constructed by the observer and the semantic space of test images constructed by the researcher is analyzed by measuring the characteristics of eye movements, which play the role of a distinctive marker that projects onto the test object a subjective algorithm for analyzing the content. The common features inherent to the distribution of saccades and fixations of the gaze in image space when comics and textual images are being experienced probably reflect the algorithm for detecting the semantic, conceptual structure of images common to the analysis of different methods of transferring information in multimedia. This algorithm breaks down under various diseases of the brain. It is assumed that, regardless of the difference of representing information in texts and comics, understanding is provided by common mechanisms for constructing the internal, imaginary, time-varying content.

Keywords:

content, scenes recognition, eye movement, reading, text, scenes imaging, multimedia

Acknowledgements:

This work was carried out with financing under the Program of Fundamental Scientific Research of State Academies in 2013–2020 (GP-14, Section 63), I. P. Pavlov Institute of Physiology. The study of high-school students with different levels of intellectual development was carried out as part of Russian Science Foundation Project No. 14-18-02135 “Psychophysiological and neurolinguistic aspects of the process of recognition of verbal and nonverbal patterns.”

OCIS codes: 100.4996, 330.2210, 330.6110

References:

1. L. V. Moskovchenko, V. A. Tupikov, and É. L. Lysenko, “The concept of the construction of prospective systems of optoelectronic passive rangefinding for surface ships,” Nats. Oborona (9), 60–61 (2012).
2. Yu. E. Shelepin, Introduction to Neuroiconics (Troitski˘ı Most, St. Petersburg, 2017).
3. N. N. Krasil’nikov, Digital Processing of 2D and 3D Images (BKhV-Peterburg, St. Petersburg, 2011).
4. A. M. Lamminpiya, S. V. Pronin, and Yu. E. Shelepin, “Spatial frequency text filtering for local and global analysis,” J. Opt. Technol. 85(8), 476–481 (2018) [Opt. Zh. 85(8), 38–45 (2018)].
5. A. M. Lamminpiya and E. Yu. Malakhova, “The perception of the motion of an actual and an imaginary scene,” in Neurotechnology, Yu. E. Shelepina and V. N. Chikhmana, eds. (St. Petersburg, 2018), pp. 257–268.
6. A. M. Lamminpiya, O. A. Vakhrameeva, D. V. Ra˘ıt, S. V. Pronin, and Yu. E. Shelepin, “The effect of wavelet filtering of images of texts on the characteristics of eye movements during reading,” Sens. Sist. 27(1), 3–9 (2013).
7. A. B. Shipunov, E. M. Baldin, P. A. Volkova, A. I. Korobeynikov, S. A. Nazarova, S. V. Petrov, and V. G. Sufiyanov, Visual Statistics: Use R! (DMK Press, Moscow, 2014).
8. N. Cohn, The Visual Language of Comics: Introduction to the Structure and Cognition of Sequential Images (Bloomsbury, London, UK, 2013).
9. M. Hegarty and M. A. Just, “Constructing mental models of machines from text and diagrams,” J. Mem. Lang. 32(6), 717–742 (1993).
10. K. Rayner, C. M. Rotello, A. J. Stewart, J. Keir, and S. A. Duffy, “Integrating text and pictorial information: eye movements when looking at print advertisements,” J. Exp. Psychol. 7(3), 219–226 (2001).
11. V. A. Barabanshchikov, Facial Expressions and Their Perception (Institut Psikhologii RAN, Moscow, 2012).
12. A. M. Lamminpiya, Yu. E. Shelepin, S. M. Bauér, O. A. Vakhrameeva, and D. V. Ra˘ıt, “The effect of the optical properties of text on the operation of the channels of the human visual system,” in Collection of Abstracts of the Reports of the All-Russia Youth Conference on Neurobiology of the Integrative Functions of the Brain (St. Petersburg, 2013), p. 43.
13. L. N. Gassovski˘ı and N. A. Nikol’skaya, “Eye movement in the process of continuous fixation of a point,” Tr. Gos. Opt. Inst. 15, 112–120 (1941).
14. V. D. Glezer, Vision and Thought (Nauka, Leningrad, 1993).
15. K. Fu, Structural Methods in Pattern Recognition (Mir, Moscow, 1977).
16. V. A. Tupikov, V. A. Pavlova, S. N. Kryukov, M. V. Sozinova, and P. K. Shulzhenko, “Linguistic methods in problems of image recognition,” Izv. YuFU (Tekh. Nauki) 1(162), 142–151 (2015).
17. R. V. Shannon, F. G. Zeng, V. Kamath, J. Wygonski, and M. Ekelid, “Speech recognition with primarily temporal cues,” Science 270, 303–304 (1995).
18. V. V. Lublinskaya, J. Ross, and E. A. Ogorodnikova, “Auditory perception and processing of amplitude modulation in speechlike signals: legacy of the Chistovich–Kozhevnikov group,” in NATO Science Series I: Life and Behavioural Sciences, Vol. 374 of Dynamics of Speech Production and Perception (2006), pp. 87–103.
19. K. Yu. Shelepin, S. V. Pronin, and Yu. E. Shelepin, “Recognizing fragmented images and the appearance of ‘insight’,” J. Opt. Technol. 82(10), 700–706 (2015) [Opt. Zh. 82(10), 70–78 (2015)].
20. K. Yu. Shelepin, G. E. Trufanov, V. A. Fokin, P. P. Vasiliev, and A. V. Sokolov, “Activity of neural networks of the human brain before, during, and after insight in image recognition,” in Neurotechnology, Yu. E. Shelepina and V. N. Chikhmana, eds. (St. Petersburg, 2018), pp. 207–243.
21. K. Y. Shelepin, G. E. Trufanov, V. A. Fokin, P. P. Vasiliev, and A. V. Sokolov, “Digital visualization of the activity of neural networks of the human brain before, during, and after insight when images are being recognized,” J. Opt. Technol. 85(8), 468–475 (2018) [Opt. Zh. 85(8), 29–38 (2018)].
22. A. Ardila, B. Bernal, and M. Rosselli, “Language and visual perception associations: meta-analytic connectivity modeling of Brodmann area 37,” Behav. Neurol. 2015, 565871 (2015).
23. A. Ardila, B. Bernal, and M. Rosselli, “How localized are language brain areas? A review of Brodmann areas involvement in oral language,” Arch. Clin. Neuropsychol. 31, 112–122 (2016).