image: * view more
Credit: by Waleed Tahir, Hao Wang, and Lei Tian
Light scattering in complex media is a pervasive problem across many areas, such as deep tissue imaging, imaging in a degraded environment, and wavefront shaping. However, to date, there is no simple solution for inverting scattering because first, only a small number of ballistic photons can travel through a complex medium without interacting with any scatterers. For strongly scattering media, unfortunately, these photons propagate only a short distance (for example, ~100 µm in human tissue) without scattering. Thus, it cannot meet the requirements for biomedical imaging, imaging in a degraded environment, etc. Secondly, a major limitation of the existing imaging through scattering media approaches is their high susceptibility to model perturbations. Slight changes of the medium (such as a small movement of the live biological sample) can lead to much-reduced correlations between the speckles measured before and after. This indicates the breakdown of the previous input–output relation and results in rapid degradation of the transferred images.
In recent years, the Horisaki Group from Tokyo University (Opt. Express 2016) and the Barbastathis Group from MIT (Optica 2018) have presented machine-learning-based methods for single-shot imaging through scattering media. Their approaches enable model-free imaging, where they can learn the relationship of objects and speckle images from a captured dataset without accurately measuring the input-output relation to model the complex scattering media.
In 2018, our group first proposed a new network framework that is highly scalable to both medium perturbations and measurement requirements (Optica 2018). To do so, we propose a statistical “one-to-all generalist” deep learning (DL) technique that encapsulates a wide range of statistical variations for the model to be resilient to speckle decorrelations. Specifically, we develop a convolutional neural network (CNN) that can learn the statistical information contained in the speckle intensity patterns captured on a set of diffusers having the same macroscopic parameter. We then show for the first time, to the best of our knowledge, that the trained CNN can generalize and make high-quality object predictions through an entirely different set of diffusers of the same class. Our work paves the way to a highly scalable DL approach for imaging through scattering media. However, in this work, our reconstruction image quality is slightly degraded compared with the “one-to-one expert” deep learning technique.
In this new paper, we demonstrate a novel adaptive learning framework, termed dynamic synthesis network (DSN), which dynamically adjusts the model weights and adapts to different scattering conditions. The proposed DSN can adaptively remove 3D scattering artifacts and achieve state-of-the-art performance for a wide range of scattering conditions. Different from the brute-force approach to train a “generalist” network using data from diverse scattering conditions, our new network adaptability is achieved by a novel “mixture of experts” architecture that enables dynamically synthesizing a network by blending multiple experts using a gating network.
As shown in Figure1, the DSN combines multiple DNNs to encode and decode a diverse set of multi-scale spatial features from the holographically backpropagated input volume for adaptively removing scattering artifacts in the input. The GTN provides the adapting mechanism by predicting the synthesis weights based on the matching hologram input. The DSN thus adapts to each input during the inference and computes the optimal weights and the resulting synthesized network “on-the-fly”, hence achieving “end-to-end” adaptive descattering.
Compared with the traditional network framework, the DSN works by continuously mixing the features maps extracted by different experts to synthesize an optimal feature representation of the input in high dimensional feature space (as illustrated by Figure 2). We show that this novel structure allows DSN to perform adaptive descattering in a continuum of scattering levels and achieve superior performance across a broad range of scattering conditions, as shown in Figure. 3.
Overall, our contribution is a novel adaptive DL framework that achieves generalizable performance across a broad range of scattering conditions using a single holistic DNN architecture. We demonstrated this framework on 3D particle imaging using inline holography with different particle densities, particle sizes and refractive index contrast. We expect that the same dynamic synthesis framework can be adapted to many other imaging applications, such as image denoising, imaging in dynamic scattering media, computational fluorescence microscopy, and imaging and light control in complex media, such as biological tissues. Broadly, our dynamic synthesis framework opens up a new paradigm for designing highly adaptive DL-based computational imaging techniques.
Journal
Light Science & Applications