image: a The basic network architecture including registration, residual-in-residual dense attention block (RRDAB) modules, conv layers, the skip connection, upsampling operation, and discriminator. The full frameworks can be found in Materials and methods. b A commercial two-photon microscope with a fast galvo-resonant scanning system and a slow dual-axis galvo scanning system. BS, beam splitter; CL, collect lens; DM, dichroic mirror; NDD, nondescanned detectors; OB, objective; SL, scan lens; TL, tube lens. view more
Credit: by Binglin Shen, Shaowen Liu, Yanping Li, Ying Pan, Yuan Lu, Rui Hu, Junle Qu, Liwei Liu
Nonlinear optical microscope can provide abundant structural and functional information simultaneously and enable comprehensive and informative analysis of various biochemical phenomena despite a confined field of view (FOV). Many instrument-based methods have shown great accomplishments in enlarging FOV despite additional costly devices and intricate optical path design. As for deep learning-enabled super-resolution restoration, in most cases the low-resolution degraded images are generated from the measured high-contrast images with synthesized noises. Such computational degradation is no guarantee of authenticity because the real situation always has the full statistical complexity.
In a new paper published in Light Science & Application, a team of scientists, led by Professor Liwei Liu from Key Laboratory of Optoelectronic Devices and Systems of Guangdong Province and Ministry of Education, College of Physics and Optoelectronic Engineering, Shenzhen University, China and co-workers have developed a self-alignment attention-guided residual-in-residual dense generative adversarial networks architecture, and applied it to label-free nonlinear optical microscope to close the gap between speed, FOV, and resolution. This inherent trade-off due to the limitations of sophisticated mechanical and optical setups is a Gordian knot for laser scanning microscopes, especially label-free nonlinear optical microscopes which are notoriously slow.
This research team demonstrated deep learning autofluorescence-harmonic microscopy (DLAM) based on the proposed networks, which included a self-alignment pyramid, cascading, and deformable convolutions framework to automatically learn and realize pixelwise alignment between the input and ground truth images to ensure reliable reconstruction. A channel attention mechanism was introduced to explicitly model the feature map interdependencies, and a spatial attention mechanism was integrated to unscramble the interspatial relationship of the feature regions. The attention mechanisms help to identify the crucial features and feature regions to improve super-resolution details and avoid oversmoothness. These scientists summarize the primary competitive advantages of their deep learning microscope:
“We demonstrated the reconstruction capability of proposed networks for label-free large-field multimodal imaging of human ovarian and skin pathological tissues. (1) For the first time, we studied the scanning fringe artifacts resulting from the fast scanning, and stitching lattice artifacts caused by the multi-field stitching in the captured input images. (2) The high enhancements (e.g. 13.3 dB for PSNR, 316% for SSIM) verified the high-quality and high-fidelity reconstruction ability of the networks. (3) We also demonstrated the transfer learning ability of the networks, which suppressed the over smooth, out of focus, and deceptive artifacts arising in other network inferences.”
“With a 24.3-fold speed-up and 2.3-fold spatial resolution enhancement for a large image of 5.4 × 5.4 mm2, deep learning autofluorescence-harmonic microscopy enables fast, large-field, stain-free histopathology of tissue specimens that can possibly supersede surgical frozen section analysis.” they added.
“This technique can also be applied to brain structure and function investigation without genetically encoded calcium indicators such as GCaMP, and has great potential for high-speed super-resolution cell structure analysis, brain 3D in toto observation, and in vivo diagnostic examination, which can help facilitate applications of the optical microscopes in biomedical research and clinical diagnosis.” the scientists forecast.
Journal
Light Science & Applications