Bio-inspired device captures images by mimicking human eye

UNIVERSITY PARK, Pa. – Drawing inspiration from nature, Penn State scientists have developed a new device that produces images by mimicking the red, green and blue photoreceptors and the neural network found in human eyes.

“We borrowed a design from nature — our retinas contain cone cells that are sensitive to red, green and blue light and a neural network that starts processing what we are seeing even before the information is transmitted to our brain,” said Kai Wang, assistant research professor in the Department of Materials Science and Engineering at Penn State. “This natural process creates the colorful world we can see.”

To achieve this in an artificial device, the scientists created a new sensor array from narrowband perovskite photodetectors, which mimic our cone cells, and connected it to a neuromorphic algorithm, which mimics our neural network, to process the information and produce high-fidelity images.

Photodetectors convert light energy into electrical signals and are essential for cameras and many other optical technologies. Narrowband photodetectors can focus on individual parts of the light spectrum, like the reds, greens and blues that make up visible light, the scientists said.

“In this work, we found a novel way to design perovskite material that is sensitive to only one wavelength of light,” Wang said. “We created three different perovskite materials, and they are designed in a way that they can only be sensitive to red, green or blue colors.”

The technology may represent a way around using filters found in modern cameras that lower resolution and increase cost and manufacturing complexity, the scientists said.

Silicon photodetectors in cameras absorb light but do not distinguish colors. An external filter separates the reds, greens and blues, and the filter only allows one color to reach each section of the light sensor, wasting two-thirds of the incoming light.

“When the light is filtered, there is some loss of information and that can be avoided using our design. So we propose this work may represent a future camera sensing technique that can help people to get a higher spatial resolution.”

And because the scientists used perovskite materials, the new devices generate power as they absorb light, potentially opening the door to battery-free camera technology, the scientists said.

“The device structure is similar to solar cells that use light to generate electricity,” said Luyao Zheng, a postdoctoral researcher at Penn State. “Once you shine a light on it, it will generate a current. So like our eyes, we don’t need to apply energy to capture this information from light.”

This research could also trigger further developments in artificial retina biotechnology. Devices based on this technology could someday replace dead or damaged cells in our eyes to restore vision, according to the scientists.

The findings, reported in the journal Science Advances, represent several fundamental breakthroughs in realizing perovskite narrowband photodetection devices — from materials synthesis to device design to systems innovation, the scientists wrote in the journal.

Perovskites are semiconductors, and when light hits these materials it creates electron-hole pairs. Sending these electrons and holes in opposite directions is what generates an electrical current.

In this study, the scientists created thin-film perovskites with heavily unbalanced electron-hole transport, meaning the holes are moving through the material faster than the electrons. By manipulating the architecture of the unbalanced perovskites, or how the layers are stacked, the scientists found they could harness properties than turn the materials into narrowband photodetectors.

They created a sensor array with these materials and used a projector to shine an image through the device. Information collected in the red, green and blue layers was fed into a three-sub-layer neuromorphic algorithm for signal processing and image reconstruction. Neuromorphic algorithms are a kind of computing technology that seeks to emulate the operation of the human brain.

“We tried different ways to process the data,” Wang said. “We tried directly merging the signals from the three color layers, but the picture was not very clear. But when we do this neuromorphic processing, the image is much closer to the original.”

Because the algorithm mimics the neural network in human retinas, the findings could provide new insight into the importance of these neural networks to our vision, the scientists said.  

“By joining our device and this algorithm together, we can demonstrate that the neural network functionality is really important in the vision processing in human eyes,” Wang said.

Also contributing to this research from Penn State were: Swaroop Ghosh, associate professor, and Junde Li, doctoral candidate, in the Department of Electrical Engineering and Computer Science; and Dong Yang, assistant research professor, Jungjin Yoon and Tao Ye, postdoctoral researchers, and Abbey Marie Knoepfel and Yuchen Hou, doctoral candidates, in the Department of Materials Science and Engineering. Shashank Priya, former associate vice president for research and director of strategic initiatives and professor of materials science and engineering, also contributed.

The Air Force Office of Scientific Research provided funding for this work. Researchers on the project were also supported by the U.S. Department of Energy, the National Institute of Food and Agriculture and the National Science Foundation.