ru
Back
DOI: 10.17586/1023-5086-2024-91-03-62-78
УДК: 535.8 004.93
Neural-network-based methods in digital and computer-generated holography. А review
Черёмхин П.А., Рымов Д.А., Свистунов А.С., Злоказов Е.Ю., Стариков Р.С. Нейросетевые методы в цифровой и компьютерной голографии. Обзор // Оптический журнал. 2024. Т. 91. № 3. С. 62–78. http://doi.org/10.17586/1023-5086-2024-91-03-62-78
Cheremkhin P.A., Rymov D.A., Svistunov A.S., Zlokazov E.Yu., Starikov R.S. Neural-network-based methods in digital and computer-generated holography. А review [in Russian] // Opticheskii Zhurnal. 2024. V. 91. № 3. P. 62–78. http://doi.org/10.17586/1023-5086-2024-91-03-62-78
Subject of study. An overview of modern neural-network-based methods for digital and computer-generated holography. Relevant works on phase and amplitude reconstruction, media characterization, phase unwrapping, computer hologram generation and other topics were discussed. Aim of study. Modern neural-network-based methods for digital and computer-generated holography are investigated. Method. The methods discussed in this review are based on neural-networks and are developed for particular tasks in the field of holography. Training dataset for supervised learning usually contains a set of input images (digital holograms, wrapped phase, 3D scenes, etc.) and a set of target images (reconstructed scenes, unwrapped phase, 3D holograms, etc.) the network is supposed to learn to generate from the input images during training. In case of unsupervised learning, there is no need to prepare a set of target images, the training is based only on the input images and the transformations applied to them. Main results. An overview of the applications of neural networks in the field of holography is provided. The main focus is on the state-of-the-art works. Neural network architectures most commonly used in holography are discussed. Cited works are organized based on the particular applications. The most important results and achievements of neural-network-based methods for digital and computer-generated holography are discussed. Practical significance. This review may interest the researchers specializing in holography or adjacent fields. This review will familiarize the readers with the modern neural-network-based methods used in computer hologram generation, holographic image reconstruction as well as the particulars of their practical application. The data presented in this review demonstrates, that in some cases the use of neural-network-based methods can provide advantages in speed and/or quality when compared against conventional methods.
laser-induced periodic surface structures, laser-induced backward transfer, color laser marking, structural colors
Acknowledgements:this work was supported by the Russian Science Foundation, grant № 23-12-00336
OCIS codes: 090.1995, 110.4280, 230.5160, 090.1760
References:1. Pi D., Liu J., Wang Y. Review of computer-generated hologram algorithms for color dynamic holographic three-dimensional display // Light Sci. Appl. 2022. V. 11. P. 231. https://doi.org/10.1038/s41377-022-00916-3
2. Ouyang W., Xu X., Lu W., et al. Ultrafast 3D nanofabrication via digital holography // Nat. Commun. 2023. V. 14. P. 1716. https://doi.org/10.1038/s41467-023-37163-y
3. Dorrah A.H., Bordoloi P., de Angelis V.S., et al. Light sheets for continuous-depth holography and three-dimensional volumetric displays // Nat. Photon. 2023. V. 17. P. 427–434. https://doi.org/10.1038/s41566-023-01188-y
4. Situ G. Deep holography // Light: Advanced Manufacturing. 2022. V. 3. № 2. P. 278–300. https://doi.org/10.37188/lam.2022.013
5. Zeng T., Zhu Y., Lam E.Y. Deep learning for digital holography: A review // Opt. Exp. 2021. V. 29. № 24. P. 40572–40593. https://doi.org/10.1364/oe.443367
6. Rivenson Y., Zhang Y., Günaydın H., et al. Phase recovery and holographic image reconstruction using deep learning in neural networks // Light Sci. Appl. 2018. V. 7. № 2. P. 17141. https://doi.org/10.1038/lsa.2017.141
7. Chen H., Huang L., Liu T., et al. Fourier imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization // Light Sci. Appl. 2022. V. 11. P. 254. https://doi.org/10.1038/s41377-022-00949-8
8. Рымов Д.А., Черёмхин П.А., Стариков Р.С. Нейросетевая реконструкция сцен с цифровых голограмм на основе извлечения амплитуды и фазы // Оптический журнал. 2022. Т. 89. № 9. С. 11–19. https://doi.org/10.17586/1023-5086-2022-89-09-11-19
Rymov D.A., Starikov R.S., Cheremkhin P.A. Neural-network-enabled holographic image reconstruction via amplitude and phase extraction // J. Opt. Technol. 2022. V. 89. № 9. P. 511–516. https://doi.org/10.1364/JOT.89.000511
9. Ju Y.-G., Choo H.-G., Park J.-H. Learning-based complex field recovery from digital hologram with various depth objects // Opt. Exp. 2022. V. 30. № 15. P. 26149–26168. https://doi.org/10.1364/oe.461782
10. Shao S., Mallery K., Kumar. S.S., et al. Machine learning holography for 3D particle field imaging // Opt. Exp. 2020. V. 28. № 3. P. 2987–2999. https://doi.org/10.1364/oe.379480
11. Wu Y., Wu J., Jin. S., et al. Dense-U-net: Dense encoder-decoder network for holographic imaging of 3D particle fields // Opt. Commun. 2021. V. 493. P. 126970. https://doi.org/10.1016/j.optcom.2021.126970
12. Svistunov A.S., Rymov D.A., Starikov R.S., et al. HoloForkNet: Digital hologram reconstruction via multibranch neural network // Appl. Sci. 2023. V. 13. № 10. P. 6125. https://doi.org/10.3390/app13106125
13. Eybposh M.H., Caira N.W., Atisa M., et al. DeepCGH: 3D computer-generated holography using deep learning // Opt. Exp. 2020. V. 28. № 18. P. 26636–26650. https://doi.org/10.1364/OE.399624
14. Ishii Y., Shimobaba T., Blinder D., et al. Optimization of phase-only holograms calculated with scaled diffraction calculation through deep neural networks // Appl. Phys. B. 2022. V. 128. P. 22. https://doi.org/10.1007/s00340-022-07753-7
15. Kadis A., Mouthaan R., Dong D., et al. Binary-phase computer-generated holography using hardware-in-the-loop feedback // OSA Imaging and App. Opt. Congress 2021 (3D, COSI, DH, ISA, pcAOP). Washington, DC United States. July 19–23, 2021. P. DW5E.1. https://doi.org/10.1364/DH.2021.DW5E.1
16. Yu G., Wang J., Yang H., et al. Asymmetrical neural network for real-time and high-quality computer-generated holography // Opt. Lett. 2023. V. 48. № 20. P. 5351–5354. https://doi.org/10.1364/OL.497518
17. Zhong C., Sang X., Yan B., et al. Real-time high-quality computer-generated hologram using complex-valued convolutional neural network // IEEE Trans. Vis. Comput. Graph. 2023. P. 1–11. https://doi.org/10.1109/TVCG.2023.3239670
18. Zheng H., Peng J., Wang Z., et al. Diffraction model-driven neural network trained using hybrid domain loss for real-time and high-quality computer-generated holography // Opt. Exp. 2023. V. 31. № 12. P. 19931–19944. https://doi.org/10.1364/OE.492129
19. Delli P.M., Memmolo P., Ciaparrone G., et al. Neuroblastoma cells classification through learning approaches by direct analysis of digital holograms // IEEE Sel. Top. Quantum Electron. 2021. V. 27. № 5. P. 1–9. https://doi.org/10.1109/JSTQE.2021.3059532
20. Dudaie M., Barnea I., Nissim N., et al. On-chip label-free cell classification based directly on off-axis holograms and spatial-frequency-invariant deep learning // Sci. Rep. 2023. V. 13. № 1. P. 12370. https://doi.org/10.1038/s41598-023-38160-3
21. Castaneda R., Trujillo C., Doblas A. Video-rate quantitative phase imaging using a digital holographic microscope and a generative adversarial network // Sensors. 2021. V. 21. № 23. P. 8021. https://doi.org/10.3390/s21238021
22. Huang L., Liu T., Yang X., et al. Holographic image reconstruction with phase recovery and autofocusing using recurrent neural networks // ACS Photonics. 2021. V. 8. № 6. P. 1763–1774. https://doi.org/10.1021/acsphotonics.1c00337
23. Zeng T., So H.K.-H., Lam E.Y. RedCap: Residual encoder-decoder capsule network for holographic image reconstruction // Opt. Exp. 2020. V. 28. № 4. P. 4876–4887. https://doi.org/10.1364/oe.383350
24. Di J., Wu J., Wang K., et al. quantitative phase imaging using deep learning-based holographic microscope // Front. Phys. 2021. V. 9. P. 651313. https://doi.org/10.3389/fphy.2021.651313
25. Wang H., Lyu M., Situ G. eHoloNet: A learning-based end-to-end approach for in-line digital holographic reconstruction // Opt. Exp. 2018. V. 26. № 18. P. 22603–22614. https://doi.org/10.1364/oe.26.022603
26. Wang K., Song L., Wang C., et al. On the use of deep learning for phase recovery // arXiv. 2023. P. 2308.00942. https://doi.org/10.48550/arXiv.2308.00942
27. Yin D., Gu Z., Zhang Y., et al. Digital holographic reconstruction based on deep learning framework with unpaired data // IEEE Photonics J. 2020. V. 12. № 2. P. 1–12. https://doi.org/10.1109/JPHOT.2019.2961137
28. Wang S., Jiang X., Guo H., et al. Improved SNR and super-resolution reconstruction of multi-scale digital holography based on deep learning // Opt. Commun. 2023. V. 545. № 4. P. 129634. https://doi.org/10.1016/j.optcom.2023.129634
29. Wu Y., Luo Y., Chaudhari G., et al. Bright-field holography: Cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram // Light Sci. Appl. 2019. V. 8. № 1. P. 25. https://doi.org/10.1038/s41377-019-0139-9
30. Jaferzadeh K., Fevens T. HoloPhaseNet: Fully automated deep-learning-based hologram reconstruction using a conditional generative adversarial model // Biomed. Opt. Exp. 2022. V. 13. № 7. P. 4032–4046. https://doi.org/10.1364/boe.452645
31. Niknam F., Qazvini H., Latifi H. Holographic optical field recovery using a regularized untrained deep decoder network // Sci. Rep. 2021. V. 11. № 1. P. 10903. https://doi.org/10.1038/s41598-021-90312-5
32. Zhang Y., Andreas M.N., Vagovoc P., et al. PhaseGAN: A deep-learning phase-retrieval approach for unpaired datasets // Opt. Exp. 2021. V. 29. № 13. P. 19593–19604. https://doi.org/10.1364/oe.423222
33. Horisaki R., Fujii K., Tanida J. Single-shot and lensless complex-amplitude imaging with incoherent light based on machine learning // Opt. Rev. 2018. V. 25. № 5. P. 593–597. https://doi.org/10.1007/s10043-018-0452-1
34. Xu Z., Zuo S., Lam E.Y. End-to-end learning for digital hologram reconstruction // High-Speed Biomedical Imaging and Spectroscopy III: Toward Big Data Instrumentation and Management. San Francisco, United States. January 27 – February 1, 2018. P. 36. https://doi.org/10.1117/12.2288141
35. Chen B., Li Z., Zhou Y., et al. Deep-learning multiscale digital holographic intensity and phase reconstruction // Appl. Sci. 2023. V. 13. № 17. P. 9806. https://doi.org/10.3390/app13179806
36. Li H., Chen X., Chi Z., et al. Deep DIH: Single-shot digital in-line holography reconstruction by deep learning // IEEE Access. 2020. V. 8. P. 202648–202659. https://doi.org/10.1109/ACCESS.2020.3036380
37. Chen X., Wang H., Razi A., et al. DH-GAN: A physics-driven untrained generative adversarial network for holographic imaging // Opt. Exp. 2023. V. 31. P. 10114–10135. https://doi.org/10.1364/OE.480894
38. Wang K., Kemao Q., Di J., et al. Y4-Net: A deep learning solution to one-shot dual-wavelength digital holographic reconstruction // Opt. Lett. 2020. V. 45. № 15. P. 4220. https://doi.org/10.1364/ol.395445
39. Wang K., Dou J., Kemao Q., et al. Y-Net: A one-to-two deep learning framework for digital holographic reconstruction // Opt. Lett. 2019. V. 44. № 19. P. 4765–4768. https://doi.org/10.1364/ol.44.004765
40. Shimobaba T., Takahashi T., Yamamoto Y., et al. Digital holographic particle volume reconstruction using a deep neural network // Appl. Opt. 2019. V. 58. № 8. P. 1900–1906. https://doi.org/10.1364/AO.58.001900
41. Shao S., Mallery K., Hong J. Machine learning holography for measuring 3D particle distribution // Chem. Eng. Sci. 2020. V. 225. P. 115830. https://doi.org/10.1016/j.ces.2020.115830
42. Horisaki R., Takagi R., Tanida J. Deep-learning-generated holography // Appl. Opt. 2018. V. 57. № 14. P. 3859–3863. https://doi.org/10.1364/AO.57.003859
43. Lee J., Jeong J., Cho J., et al. Deep neural network for multi-depth hologram generation and its training strategy // Opt. Exp. 2020. V. 28. № 18. P. 27137–27154. https://doi.org/10.1364/OE.402317
44. Zheng H., Hu J., Zhou C., et al. Computing 3D phase-type holograms based on deep learning method // Photonics. 2021. V. 8. № 7. P. 280. https://doi.org/10.3390/photonics8070280
45. Khan A., Zhijiang Z., Yu Y., et al. GAN-Holo: Generative adversarial networks-based generated holography using deep learning // Complexity. 2021. V. 2021. P. 1–7. https://doi.org/10.1155/2021/6662161
46. Zhu R., Chen L., Zhang H. Computer holography using deep neural network with Fourier basis // Opt. Lett. 2023. V. 48. № 9. P. 2333–2336. https://doi.org/10.1364/OL.486255
47. Yang D., Seo W., Yu H., et al. Diffraction-engineered holography: Beyond the depth representation limit of holographic displays // Nat. Commun. 2022. V. 13. № 1. P. 6012. https://doi.org/10.1038/s41467-022-33728-5
48. Shi L., Li B., Matusik W. End-to-end learning of 3D phase-only holograms for holographic display // Light Sci. Appl. 2022. V. 11. № 1. P. 247. https://doi.org/10.1038/s41377-022-00894-6
49. Horisaki R., Nishizaki Y., Kitaguchi K., et al. Three-dimensional deeply generated holography // Appl. Opt. 2021. V. 60. № 4. P. A323–A328. https://doi.org/10.1364/AO.404151
50. Shi L., Li B., Kim C., et al. Towards real-time photorealistic 3D holography with deep neural networks // Nature. 2021. V. 591. № 7849. P. 234–239. https://doi.org/10.1038/s41586-020-03152-0
51. Dong Z., Xu C., Ling Y., et al. Fourier-inspired neural module for real-time and high-fidelity computer-generated holography // Opt. Lett. 2023. V. 48. № 3. P. 759–762. https://doi.org/10.1364/OL.477630
52. Liu K., Wu J., He Z., et al. 4K-DMDNet: Diffraction model-driven network for 4K computer-generated holography // Opto-Electronic Adv. 2023. V. 6. № 5. P. 220135 https://doi.org/10.29026/oea.2023.220135
53. Liu Q., Chen J., Qiu B., et al. DCPNet: A dual-channel parallel deep neural network for high quality computer-generated holography // Opt. Exp. 2023. V. 31. № 22. P. 35908–35921. https://doi.org/10.1364/OE.502503
54. Sun F., Zhu L., Wang W., et al. Three-dimensional dynamic optical trapping using non-iterative computer-generated holography // Opt. Lasers Eng. 2023. V. 164. P. 107500. https://doi.org/10.1016/j.optlaseng.2023.107500
55. Faini G., Tanese D., Molinier C., et al. Ultrafast light targeting for high-throughput precise control of neuronal networks // Nat. Commun. 2023. V. 14. № 1. P. 1888. https://doi.org/10.1038/s41467-023-37416-w
56. Peng Y., Choi S., Padmanaban N., et al. Neural holography with camera-in-the-loop training // ACM Trans. Graph. 2020. V. 39. № 6. P. 1–14. https://doi.org/10.1145/3414685.3417802
57. Peng Y., Choi S., Kim J., et al. Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration // Sci. Adv. 2021. V. 7. № 46. P. eabg5040. https://doi.org/10.1126/sciadv.abg5040
58. Gopakumar M., Kim J., Choi S., et al. Unfiltered holography: Optimizing high diffraction orders without optical filtering for compact holographic displays // Opt. Lett. 2021. V. 46. № 23. P. 5822–5825. https://doi.org/10.1364/OL.442851
59. Chao B., Gopakumar M., Choi S., et al. High-brightness holographic projection // Opt. Lett. 2023. V. 48. № 15. P. 4041–4044. https://doi.org/10.1364/OL.489617
60. Pavillon N., Sekhar C.S., Kühn J., et al. Suppression of the zero-order term in off-axis digital holography through nonlinear filtering // Appl. Opt. 2009. V. 48. № 34. P. H186–H195. https://doi.org/10.1364/AO.48.00H186
61. Bianco V., Memmolo P., Leo M., et al. Strategies for reducing speckle noise in digital holography // Light Sci. Appl. 2018. V. 7. № 1. P. 48. https://doi.org/10.1038/s41377-018-0050-9
62. Cheremkhin P.A., Evtikhiev N.N., Krasnov V.V., et al. Shot noise and fixed-pattern noise effects on digital hologram reconstruction // Opt. Lasers Eng. 2021. V. 139. P. 106461. https://doi.org/10.1016/j.optlaseng.2020.106461
63. Leith E.N., Upatnieks J. Wavefront reconstruction with diffused illumination and three-dimensional objects // JOSA. 1964. V. 54. № 11. P. 1295–1301. https://doi.org/10.1364/JOSA.54.001295
64. Yamaguchi I., Zhang T. Phase-shifting digital holography // Opt. Lett. 1997. V. 22. № 16. P. 1268–1270. https://doi.org/10.1364/OL.22.001268
65. Momey F., Denis L., Olivier T., et al. From Fienup’s phase retrieval techniques to regularized inversion for in-line holography: Tutorial // JOSA. A. 2019. V. 36. № 12. P. D62–D80. https://doi.org/10.1364/JOSAA.36.000D62
66. Gerchberg R.W., Saxton W.O. A practical algorithm for the determination of phase from image and diffraction plane pictures // Optik. 1971. V. 2. P. 237–246.
67. Ding B., Qian H., Zhou J. Activation functions and their characteristics in deep neural networks // 2018 Chinese Control and Decision Conf. (CCDC). Shenyang, China. June 9–11, 2018. P. 1836–1841. https://doi.org/10.1109/CCDC.2018.8407425
68. Ronneberger O., Fischer P., Brox T. U-Net: Convolutional networks for biomedical image segmentation // Lecture Notes in Computer Sci. 2015. V. 9351. P. 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
69. Nguyen T., Bui V., Lam V., et al. Automatic phase aberration compensation for digital holographic microscopy based on deep learning background detection // Opt. Exp. 2017. V. 25. № 13. P. 15043–15057. https://doi.org/10.1364/oe.25.015043
70. Wang K., Li Y., Kemao Q., et al. One-step robust deep learning phase unwrapping // Opt. Exp. 2019. V. 27. № 10. P. 15100–15115. https://doi.org/10.1364/oe.27.015100
71. He K., Zhang X., Ren S., et al. Deep residual learning for image recognition // 2016 IEEE Conf. Computer Vision and Pattern Recognition (CVPR). Las Vegas, United States. June 26 – July 1, 2016. P. 770–778. https://doi.org/10.1109/CVPR.2016.90
72. Pan X., Tewari A., Leimkühler T., et al. Drag your GAN: Interactive point-based manipulation on the generative image manifold // Special Interest Group on Computer Graphics and Interactive Techniques Conf. Conf. Proc. Los Angeles, United States. August 6–10, 2023. P. 1–11. https://doi.org/10.1145/3588432.3591500
73. Terbe D., Orzó L., Zarándy Á. Classification of holograms with 3D-CNN // Sensors. 2022. V. 22. № 21. P. 8366. https://doi.org/10.3390/s22218366
74. Huang L., Chen H., Liu T., et al. Self-supervised learning of hologram reconstruction using physics consistency // Nat. Mach. Intell. 2023. V. 5. № 8. P. 895–907. https://doi.org/10.1038/s42256-023-00704-7
75. Lu Z., Cao Y., Liu M., et al. Image-to-image translation for improved digital holographic reconstruction based on a generative adversarial network learning framework // Opt. Laser Technol. 2023. V. 166. P. 109654. https://doi.org/10.1016/j.optlastec.2023.109654
76. Chen H., Huang L., Liu T., et al. EFIN: Enhanced Fourier imager network for generalizable autofocusing and pixel super-resolution in holographic imaging // IEEE Sel. Top. Quantum Electron. 2023. V. 29. № 4. P. 1–10. https://doi.org/10.1109/JSTQE.2023.3248684
77. Verrier N., Atlan M. Off-axis digital hologram reconstruction: Some practical considerations // Appl. Opt. 2011. V. 50. № 34. P. H136–H146. https://doi.org/10.1364/AO.50.00H136
78. Ren Z., Xu Z., Lam E.Y. End-to-end deep learning framework for digital holographic reconstruction // Advanced Photonics. 2019. V. 1. № 1. P. 016004. https://doi.org/10.1117/1.AP.1.1.016004
79. Park S., Kim Y., Moon I. Automated phase unwrapping in digital holography with deep learning // Biomed. Opt. Exp. 2021. V. 12. № 11. P. 7064–7081. https://doi.org/10.1364/boe.440338
80. Park S., Kim Y., Moon I. Fast automated quantitative phase reconstruction in digital holography with unsupervised deep learning // Opt. Lasers Eng. 2023. V. 167. P. 107624. https://doi.org/10.1016/j.optlaseng.2023.107624
81. Huang L., Tang J., Yan L., et al. Wrapped phase aberration compensation using deep learning in digital holographic microscopy // Appl. Phys. Lett. 2023. V. 123. № 14. P. 141109. https://doi.org/10.1063/5.0166210
82. Zhang Y., Lu Y., Wang H., et al. Automatic classification of marine plankton with digital holography using convolutional neural network // Opt. Laser Technol. 2021. V. 139. P. 106979. https://doi.org/10.1016/j.optlastec.2021.106979
83. Kim S.-J., Wang C., Zhao B., et al. Deep transfer learning-based hologram classification for molecular diagnostics // Sci. Rep. 2018. V. 8. № 1. P. 17003. https://doi.org/10.1038/s41598-018-35274-x
84. Yolalmaz A., Yüce E. Comprehensive deep learning model for 3D color holography // Sci. Rep. 2022. V. 12. № 1. P. 2487. https://doi.org/10.1038/s41598-022-06190-y
85. Chen L., Zhu R., Zhang H. Speckle-free compact holographic near-eye display using camera-in-the-loop optimization with phase constraint // Opt. Exp. 2022. V. 30. № 26. P. 46649–46665. https://doi.org/10.1364/OE.475066
86. Cuenat S., Andréoli L., André A.N., et al. Fast autofocusing using tiny transformer networks for digital holographic microscopy // Opt. Exp. 2022. V. 30. № 14. P. 24730–24746. https://doi.org/10.1364/OE.458948
87. Fang Q., Xia H., Song Q., et al. Speckle denoising based on deep learning via a conditional generative adversarial network in digital holographic interferometry // Opt. Exp. 2022. V. 30. № 12. P. 20666–20683. https://doi.org/10.1364/oe.459213
88. Dong Z., Ling Y., Xu C., et al. Gaze-contingent efficient hologram compression for foveated near-eye holographic displays // Displays. 2023. V. 79. P. 102464. https://doi.org/10.1016/j.displa.2023.102464
89. Wang H., Li K., Jiang X., et al. Zero-order term suppression in off-axis holography based on deep learning method // Opt. Commun. 2023. V. 537. P. 129264. https://doi.org/10.1016/j.optcom.2023.129264
90. Yu H., Kim Y., Yang D., et al. Deep learning-based incoherent holographic camera enabling acquisition of real-world holograms for holographic streaming system // Nat. Commun. 2023. V. 14. № 1. P. 3534. https://doi.org/10.1038/s41467-023-39329-0