ITMO
ru/ ru

ISSN: 1023-5086

ru/

ISSN: 1023-5086

Scientific and technical

Opticheskii Zhurnal

A full-text English translation of the journal is published by Optica Publishing Group under the title “Journal of Optical Technology”

Article submission Подать статью
Больше информации Back

DOI: 10.17586/1023-5086-2023-90-10-48-66

УДК: 004.93'12

Neural network training for thermal image classification based on visible spectrum images

Ermachenkova M.K., Malashin, R.O., Boiko, A.A.

Full text on elibrary.ru

Publication in Journal of Optical Technology

For Russian citation (Opticheskii Zhurnal):

Ермаченкова М.К., Малашин Р.О., Бойко А.А. Обучение нейронных сетей для классификации тепловизионных изображений на основе изображений видимого спектра // Оптический журнал. 2023. Т. 90. № 10. С. 48–66. http://doi.org/10.17586/1023-­5086-­2023­-90­-10­-48-­66

 

Ermachenkova M.K., Malashin R.O., Boiko A.A. Neural network training for thermal image classification based on visible spectrum images [In Russian] // Opticheskii Zhurnal. 2023. V. 90. № 10. P. 48–66. http://doi.org/10.17586/1023­5086­2023­90­10­48­66

 

 

 

For citation (Journal of Optical Technology):

M. K. Ermachenkova, R. O. Malashin, and A. A. Boiko, "Neural network training for thermal image classification based on visible spectrum images," Journal of Optical Technology. 90(10), 590-600 (2023). https://doi.org/10.1364/JOT.90.000590

Abstract:

Subject of the study. Methods of visible spectrum images augmentation in the tasks of thermal images classification were considered. The aim of the study is to investigate the ways to improve the generalization ability of neural networks trained on visible spectrum images to recognize the thermal images. Method. Existing sets of the thermal images have limited size, and obtaining such data requires expensive equipment. At the same time, the classifiers trained on visible spectrum data show low classification accuracy on data of different optical spectra. There are various methods of enriching the thermal datasets to solve the problem of object recognition, for example, the use of synthesized images, however these approaches require the use of thermal images in this or that form, which imposes restriction on the possibilities of their application. Meanwhile, there are artistic methods of modeling far­infrared scenes based on visible spectrum images that allow to achieve visual similarity, for example, by means of contrast correction and transformation of color channel values. We have proposed and investigated a preliminary image transformation method to determine whether the classifying neural network is capable of extracting features from modified visible spectrum images sufficient to generalize to thermal data. Main results. Owing to the developed method of augmentation and preparation of the visible spectrum data, the level of classification errors was reduced from 17% to 6%. Practical Significance. Our study shows that the proposed method of training made it possible to improve the classification accuracy of the thermal imaging data without using the images of the appropriate spectrum in the training sample. This approach can be used as a method of data enrichment, for example, if the available resources for obtaining thermal imagery data are limited.

Keywords:

thermal image classification, data augmentation methods, thermal infrared images, neural network training

OCIS codes: 150.1135, 100.4996

References:
  1. Mittal U., Srivastava S., Chawla P. Object detection and classification from thermal images using region based convolutional neural network // Journal of Computer Science. 2019. V. 15. № 7. P. 961–971. https://doi.org/10.3844/jcssp.2019.961.971
  2. FLIR Systems, Inc. Free Flir Thermal Dataset for Algorithm Training // URL: https://www.flir.com/oem/adas/adas­dataset­agree (accessed 13.06.2022)
  3. Qirat Ashfaq, Usman Akram, Roshaan Zafar. Thermal Image dataset for object classification. Mendeley Data. 2021. Ver 1. https://doi.org/10.17632/btmrycjpbj.1
  4. James Cook. Chips Thermal Face Dataset // URL: https://www.kaggle.com/kagglechip/chips­thermal­face­dataset (accessed 30.04.2022)
  5. Miller D., Boyang Song, Farnsworth M., Divya Tiwari. Pitch­In LBAM Thermal Imaging Dataset // (updated 14.05.2021) URL: https://www.kaggle.com/dbmiller/pitchin­lbam­thermal­imaging­dataset (accessed 30.04.2022)
  6. Farzeen Munir, Shoaib Azam, Muhammd Aasim Rafique, Ahmad Muqeem Sheri, Moongu Jeon, Witold Pedrycz. Exploring thermal images for object detection in underexposure regions for autonomous driving // Applied Soft Computing. May 2022. V. 121. № C. https://doi.org/10.1016/j.asoc.2022.108793.
  7. Zhou Hang, Min Sun, Xiang Ren, Xiuyuan Wang. Visible­thermal image object detection via the combination of illumination conditions and temperature information // Remote Sensing. 2021. V. 13. № 18. P. 3656. https://doi.org/10.3390/rs13183656
  8. Tsurkov V.I, Grinchuk O.V. Cyclic generative neural networks to improve face recognition in non­standard domains // Journal of Computer and Systems Sciences International. 2018. V. 57. № 4. P. 620–625. https://doi.org/10.1134/S1064230718040093
  9. Dai Xuerui, Yuan Xue, Wei Xueye. TIRNet: Object detection in thermal infrared images for autonomous driving // Applied Intelligence. 2021. V. 51. P. 1–18. https://doi.org/10.1007/s10489­020­01882­2
  10. Lee D.­G., Jeon M.­H., Cho Y., Kim A. Edge­guided multi­domain RGB­to­TIR image translation for training vision tasks with challenging labels // In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). May 29, 2023 — Jun 2, 2023. London, UK. P. 8291–8298. https://doi.org/10.1109/ICRA48891.2023.10161210
  11. Akkaya B., Altinel F., Halici U. Self­training guided adversarial domain adaptation for thermal imagery // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), online, June 19–25. 2021. Nashville, TN, USA. P. 4317–4326. https://doi.org/10.1109/cvprw53098.2021.00488
  12. Gundogdu E., Koç A., Alatan A.A. Object classification in infrared images using deep representations // 2016 IEEE International Conference on Image Processing (ICIP). Phoenix. Arizona. USA. 25–28 Sept. 2016. P. 1066–1070. https://doi.org/10.1109/ICIP.2016.7532521
  13. Xu J., Vázquez D., Ramos S., López A.M., Ponsa D. Adapting a pedestrian detector by boosting LDA exemplar classifiers // 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Portland, OR, USA. 2013. P. 688–693. https://doi.org/ 10.1109/CVPRW.2013.104
  14. Feng D., Haase­Schütz C., Rosenbaum L., Hertlein H., Glaeser C., Timm F., Wiesbeck W., Dietmayer K. Deep multi­modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges // IEEE Transactions on Intelligent Transportation Systems. Feb 17, 2020. V. 22 № 3. P. 1341–1360. https://doi.org/ 10.1109/TITS.2020.2972974
  15. Craig Boehman. How to create an infrared effect in Photoshop (published 28.06.2022). URL: https://www.makeuseof.com/how­to­create­an­infrared­effect­in­photoshop/(accessed 01.05.2023).
  16. Shorten C., Khoshgoftaar T.M. A survey on image data augmentation for deep learning // J Big Data. 2019. V. 6. № 60. P. 1–48. https://doi.org/10.1186/s40537­019­0197­0
  17. Haeberli P., Voorhies В. Image processing by linear interpolation and extrapolation // IRIS Universe Magazine. 1994. V. 28. P. 8–9.
  18. Gonzalez R.C., Woods R.E. Digital image processing (4th ed.). New York, NY: Pearson, 2018. P. 138–140.
  19. Leo Levi. Unsharp masking and related image enhancement techniques // Computer Graphics and Image Processing. 1974. V. 3. № 2. P. 163–177.
  20. Wightman R. PyTorch image models. GitHub repository. 2019. URL: https://github.com/rwightman/pytorch­image­models. (accessed 30.04.2023) https://doi.org/10.5281/zenodo.4414861
  21. Deng J., Dong W., Socher R., Li L.­J., Kai Li, Li Fei­Fei. ImageNet: A large­scale hierarchical image database // 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA. 2009. P. 248–255. https:// doi.org/ 10.1109/CVPR.2009.5206848
  22. Malashin R.O., Kadykov A.B. Investigation of the generalizing capabilities of convolutional neural networks in forming rotationinvariant attributes // Journal of Optical Technology. 2015. V. 82. № 8. P. 509–515. https://doi.org/10.1364/JOT.82.000509
  23. He K., Zhang X., Ren S., Sun J. Deep residual learning for image recognition // Proceedings of the IEEE conference on computer vision and pattern recognition. June 27 — June 30. 2016. Las Vegas, Nevada, US. P. 770–778. https://doi.org/10.48550/arXiv.1512.03385
  24. Tan M., Le Q. EfficientNet: Rethinking model scaling for convolutional neural networks // Proceedings of the 36th International Conference on Machine Learning. PMLR. 9–15 June 2019. Long Beach. California. USA. V. 97. P. 6105–6114. https://doi.org/10.48550/arXiv.1905.11946
  25. Murphy Kevin P. Machine learning: a probabilistic perspective. Cambridge, Massachusetts, USA: MIT Press, 2012. 1067 p.
  26. Hinton G. Coursera neural networks for machine learning. Lecture 6. 2018. URL: https://www.coursera.org/learn/neural­networks­deep­learning (accessed 30.04.2023).
  27. Cubuk E.D., Zoph B., Shlens J., Le Q.V. RandАugment: Practical automated data augmentation with a reduced search space // IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle, WA, USA. June 14–19 2020. P. 3008–3017. https://doi.org/10.1109/CVPRW50498.2020.00359
  28. Cubuk E.D., Zoph B., Mané D., Vasudevan V., Le Q.V. AutoAugment: Learning augmentation strategies from data // IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA. June 16–20. 2019. P. 113–123. https://doi.org/10.1109/CVPR.2019.00020
  29. Hendrycks D., Mu N., Cubuk E.D., Zoph B., Gilmer J., Lakshminarayanan B. Augmix: A simple data processing method to improve robustness and uncertainty // arXiv preprint arXiv:1912.02781 (published: 17.02.2020). URL: https://doi.org/10.48550/arXiv.1912.02781 (accessed 30.04.2023).
  30. Zhong Zhun, Liang Zheng, Guoliang Kang, Shaozi Li, Yi Yang. Random erasing data augmentation // Proceedings of the AAAI conference on artificial intelligence. 2020. V. 34. № 7. P. 13001–13008. https://doi.org/10.48550/arXiv.1708.04896