Optical oculography – a neural network approach to determining operator gaze direction

Shelepin, E.Y., Skuratova, K.A., Lekhnitskaya P.A., Shelepin, K.Y.

For Russian citation (Opticheskii Zhurnal):

Шелепин Е.Ю., Скуратова К.А., Лехницкая П.А., Шелепин К.Ю. Оптическая окулография — нейросетевой подход определения направления взгляда оператора // Оптический журнал. 2025. Т. 92. № 12. С. 84–93. http://doi.org/10.17586/1023-5086-2025-92-12-84-93

Shelepin E.Yu., Skuratova K.A., Lekhnitskaya P.A., Shelepin K.Yu. Optical oculography — a neural network approach to determining operator gaze direction [in Russian] // Opticheskii Zhurnal. 2025. V. 92. № 12. P. 84–93. http://doi.org/10.17586/1023-5086-2025-92-12-84-93

For citation (Journal of Optical Technology):

Abstract:

Subject of the study. Optical digital oculography — determining the operator’s gaze direction. Purpose of the study. Developing a technology for determining the gaze direction of operators working with video terminals. Methods. Optical oculography (eye tracking) and digital image processing using a convolutional neural network were used. PyTorch, a machine learning framework for the Python language, was utilized. Main results. An algorithm (pipeline) was developed for a sequence of optical measurements and processing of the resulting images using a convolutional neural network trained on diverse datasets. Detection of eyes, irises, pupils, and glare in the frame is ensured. The geometric center of the sphere (the eyeball) is used as the gaze source, and gaze direction is defined as a vector drawn from this center through the center of the pupil to the monitor screen observed by the operator. The intersection point of the gaze vector with the screen is represented in monitor coordinates. Practical significance. Gaze detection technology is a key element of hybrid intelligence assistive technologies and is intended for use in ergonomics, medicine, marketing, and transportation.

Keywords:

physiological optics, eye movements, optical oculography, gaze source, cornea, pupil, artificial neural networks, detection, recognition

Acknowledgements:

this work was supported by federal budget funds from two sources. The development of the oculograph and training neural network were carried out using funds received under State Contract № 1021062411653-4-3.1.8 to the I.P. Pavlov Institute of Physiology of the Russian Academy of Sciences. The testing of an oculograph with a trained neural network is described in the State Contract “Hardware and software complex for the diagnosis of emotional and cognitive impairments in stress-related disorders using synchronous recording of video-oculography indicators and other psychophysiological parameters,” Registration number 125013101179-7, from the Institute of Cognitive Sciences and Neurotechnology, V.P. Serbsky National Medical Research Center for Psychiatry and Narcology, Ministry of Health of the Russian Federation, Moscow, Russia

OCIS codes: 3330.2210, 330.5370

References:

1. Барабанщиков В.А., Жегалло А.В. Айтрекинг: методы регистрации движений глаз в психологических исследованиях и практике. М.: Когито-центр, 2014. 128 с.
Barabanshchikov V.A., Zhegallo A.V. Eye-tracking: methods of recording eye movements in psychological research and practice [in Russian]. Moscow: KogitoCenter, 2014. 128 p.
2. Chi J., Liu J., Wang F., Chi Y., Hou Z.-G. 3-D gazeestimation method using a multi-camera-multi-lightsource system // IEEE Transactions on Instrumentation and Measurement. 2020. V. 69. № 12. P. 9695–9708. https://doi.org/10.1109/TIM.2020.3006681
3. Шелепин К.Ю., Шелепин Е.Ю. Ассистивный комплекс «Стерх»: системы видеоокулографии в восстановительной медицине // Биотехнические системы и технологии. 2019. С. 46–50.
Shelepin K.Yu., Shelepin E.Yu. Assistive complex “Sterkh”: video-oculography systems in rehabilitation medicine [in Russian] // Biotechnical Systems and Technologies. 2019. P. 46–50.
4. Sharma K., Giannakos M., Dillenbourg P. Eye-tracking and artificial intelligence to enhance motivation and learning // Smart Learning Environments. 2020. V. 7. P. 1–19. https://doi.org/10.1186/s40561-020-00122-x
5. Яровая Н.П., Аравийская Е.Р., Зуева В.С., Скуратова К.А., Шелепин Е.Ю. Взаимосвязь самоотношения и глазодвигательных паттернов при восприятии женщинами собственного лица // Российский психологический журнал. 2021. Т. 18. №. 1. С. 22–33. https://doi.org/10.21702/rpj.2021.1.2
Yarovaya N.P., Araviyskaya E.R., Zueva V.S., Skuratova K.A., Shelepin E.Yu. Relationship between selfattitude and eye movement patterns in women’s perception of their own face [in Russian] // Russian Psychological Journal. 2021. V. 18. № 1. P. 22–33. https://doi.org/10.21702/rpj.2021.1.2
6. Wang Q., Ma D., Chen H., Ye X. Effects of background complexity on consumer visual processing: An eyetracking study // Journal of Business Research. 2020. V. 111. P. 270–280. https://doi.org/10.1016/j.jbusres.2019.07.018
7. Degen J., Kursat L., Leigh D.D. Seeing is believing: Testing an explicit linking assumption for visual world eye-tracking in psycholinguistics // Proceedings of the Annual Meeting of the Cognitive Science Society. 2021. V. 43. № 43. P. 1500–1506.
8. Sharafi Z., Sharif B., Gueheneuc Y., Begel A., Bednarik R., Crosby M. A practical guide on conducting eye tracking studies in software engineering // Empirical Software Engineering. 2020. V. 25. P. 3128–3174. https://doi.org/10.1007/s10664-020-09829-4
9. Navaneethan S., Sreedhar. S.S., Padmakala S., Senthilkuma C. The human eye pupil detection system using BAT optimized deep learning architecture // Computer Systems Science and Engineering. 2023. V. 46. № 1. P. 125–135. https://doi.org/10.32604/csse.2023.034546
10. Fuhl W., Schneider J., Kasneci E. 1000 pupil segmentations in a second using Haar like features and statistical learning // Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021. P. 3466–3476. https://doi.org/10.48550/arXiv.2102.01921
11. Molina-Cantero A.J., Lebrato C., Castro J., Monge M. A review on visible-light eye-tracking methods based on a low-cost camera // Journal of Ambient Intelligence and Humanized Computing. 2024. V. 15. № 4. P. 2381–2397. https://doi.org/10.1007/s12652-024-04760-8
12. Lee K.I., Jeon J.H., Song B.C. Deep learning-based pupil center detection for fast and accurate eye tracking system // Computer Vision–ECCV 2020: 16th European Conference. Glasgow, UK. August 23–28, 2020. Proceedings. Part XIX 16. Springer International Publishing. 2020. P. 36–52. https://doi.org/10.1007/978-3-030-58529-7_3
13. Romaguera T., Romaguera L., Piñol D., Seisdedos C. Pupil center detection approaches: a comparative analysis // Computación y Sistemas. 2021. V. 25. №. 1. P. 67–81. https://doi.org/10.13053/CyS-25-1-3385
14. Ou W.L., Kuo T.L, Chang C.C., Fan C.P. Deep-learningbased pupil center detection and tracking technology for visible-light wearable gaze tracking devices // Applied Sciences. 2021. V. 11. № 2. P. 851. https://doi.org/10.3390/app11020851
15. Choi J.H., Lee K.I., Song B.C. Eye pupil localization algorithm using convolutional neural networks // Multimedia Tools and Applications. 2020. V. 79. № 43. P. 32563–32574. https://doi.org/10.1007/s11042-020-09711-x
16. Huang X., Lee S., Kim C., Choi S. An automatic screening method for strabismus detection based on image processing // PLoS One. 2021. Aug 3. V. 16(8). P. e0255643. https://doi.org/10.1371/journal.pone.0255643
17. Kanan C., Cottrell G.W. Color-to-grayscale: does the method matter in image recognition? // PloS one. 2012. V. 7. № 1. P. e29740. https://doi.org/10.1371/journal.pone.0029740
18. Saravanan C. Color image to grayscale image conversion // 2010 second international conference on computer engineering and applications // IEEE. 2010. V. 2. P. 196–199. https://doi.org/10.1109/ICCEA.2010.192
19. Hall L.A., Hunt C., Young G., Wolffsohn J. Factors affecting corneoscleral topography // Invest Ophthalmol Vis Sci. 2013. May 1. V. 54(5). P. 3691–701. https://doi.org/10.1167/iovs.13-11657
20. Jesus D.A., Kedzia R., Iskander D.R. Precise measurement of scleral radius using anterior eye profilometry // Contact Lens and Anterior Eye. 2017. V. 40. № 1. P. 47–52. https://doi.org/10.1016/j.clae.2016.11.003
21. Manns F., Fernandez V., Zipper S., Sandadi S., Hamaoui M., Ho A., Parel J.M. Radius of curvature and asphericity of the anterior and posterior surface of human cadaver crystalline lenses // Exp Eye Res. 2004. Jan. V. 78(1). P. 39–51. https://doi.org/10.1016/j.exer.2003.09.025
22. Chi J., Liu J., Wang F., Chi Y. 3D gaze-estimation method using a multi-camera-multi-light-source system // IEEE Transactions on Instrumentation and Measurement. 2020. V. 69. № 12. P. 9695–9708. https://doi.org/10.1109/TIM.2020.3006681
23. Guestrin E.D., Eizenman M. General theory of remote gaze estimation using the pupil center and corneal reflections // IEEE Transactions on biomedical engineering. 2006. V. 53. № 6. P. 1124–1133. https://doi.org/10.1109/TBME.2005.863952
24. Katz M., Kruger P.B. The human eye as an optical system. Chapt. 33. http://www.oculist.net/downaton502/prof/ebook/duanes/pages/contents.html (дата посещения 10.09.2025)
25. Wallerstein A., Ridgway C. Gatinel D., Debellemanière G., Mimouni M., Albert D., Cohen M., Lloyd J., Gauvin M. Angle kappa influence on multifocal IOL outcomes //J. Refract. Surg. 2023. V. 39. P. 840–849. https://doi.org/10.3928/1081597X-20231101-01
26. Cervantes-Coste G., Tapia A., Corredor-Ortega C., Osorio M., Valdez R., Massaro M., Velasco-Barona C., Gonzalez-Salinas R. The influence of angle alpha, angle kappa, and optical aberrations on visual outcomes after the implantation of a high-addition trifocal IOL // J. Clin. Med. 2022. V. 11. P. 896. https://doi.org/10.3390/jcm11030896
27. Sandoval H.P., Potvin R., Solomon K.D. The effects of angle kappa on clinical results and patient-reported outcomes after implantation of a trifocal intraocular lens // Clin. Ophthalmol. 2022. V. 16. P. 1321–1329. https://doi.org/10.2147/OPTH.S363536
28. Marques J.H., Baptista P.M., Ribeiro B., Menéres P., Beirão J.M. Intraocular lens power calculation: angle κ and ocular biomechanics // J. Cataract. Refract. Surg. 2024. V. 50. P. 345–351. https://doi.org/10.1097/j.jcrs.0000000000001362
29. Wallerstein A., Ridgway C., Gauvin M. Comment on: Intraocular lens power calculation: angle K and ocular biomechanics // Journal of Cataract & Refractive Surgery. August 2024. V. 50(8). P. 895. https://doi.org/10.1097/j.jcrs.0000000000001487
30. Ohlendorf A., Wahl S. Positions of the horizontal and vertical centre of rotation in eyes with different refractive errors // Ophthalmic Physiol. Opt. 2022. V. 42. P. 376–383. https://doi.org/10.1111/opo.12940
31. Shelepin E. Eye tracking in expertise assessment case studies // Annual meeting of the Vision Sciences Society. May 17–22. 2024. St. Pete Beach, USA Abstracts. P. 154–155. https://www.visionsciences.org/documents/VSS_2024_Abstracts.pdf (дата посещения 10.09.2025) https://doi.org/10.1167/jov.24.10.1034