Stereoscopic panoramas promise an inviting, immersive experience for viewers but, at high resolutions, distortions can develop that make viewing unpleasant or even intolerable. A team at Disney Research Zurich has found methods to correct these problems, yielding high-quality panoramas at megapixel resolutions.
The researchers will present findings related to their so-called Megastereo project at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 25-27, in Portland, Oregon.
Using software to digitally stitch overlapping images together into large panoramas has become popular and commonplace, with stitching tools available even in some consumer cameras. Methods for creating stereo panoramas that add the impression of depth to standard 2D panoramas also have been developed, but problems arise when applied to high resolution images.
"Images captured at higher resolution and quality often pose novel challenges," explained Alexander Sorkine-Hornung, research scientist at Disney Research Zurich. When such images are stitched into a stereo panorama, the results may be buildings that seem to sway wildly or expand and contract as perspective shifts. Likewise, people or objects that look three-dimensional from one perspective may appear flat as the line of sight changes.
The problems arise when left eye/right eye parallax views are introduced in a panorama to create the impression of depth. Such a panorama could be created using two cameras that capture images for each eye, but more commonly a 360-degree panorama is captured with a single, rotating camera. By mounting the camera on an offset arm, it's possible to capture images at varying centers of projection, so that a stereoscopic image can be produced by stitching together specific strips from the input views. But artifacts such as visible seams between photos that might be bearable in a regular panorama can lead to an intolerable experience when viewing stereo panoramas at high resolutions. For instance, the wide angle lenses typically used to create panoramas make objects near the edges of images appear larger than those in the center. When such images are stitched together in a horizontal stereo panorama, it can create vertical parallax that causes buildings or other objects at the top of the panorama to tilt or sway as the viewer's perspective changes.
The Disney Research team developed methods for correcting the optical distortion of the wide angle lenses, as well as correcting less-than-perfect camera orientations, such as when images are captured with a handheld camera. Other problems relate to visible seams between images that have been stitched together, such as lines or walls that should be straight but appear wavy or crooked. One reason for such problems is that cameras capture light rays from only a limited amount of pixels per view as well as a limited number of camera viewpoints. "Simply capturing more and higher resolution images usually isn't feasible and, more importantly, doesn't fully resolve the issues," Sorkine-Hornung said. To counteract the root problem, the team developed an optical flow-based upsampling method, which interpolates what light rays are missing in between.
The team found that their techniques for correcting problems in 360-degree stereo panoramas were robust and could apply to other sorts of panoramas, such as linear panoramas where, instead of rotating on a fixed base, a camera is mounted on a car or otherwise moves in a straight line.
In addition to Sorkine-Hornung, the team included Christian Richardt, Yael Pritch and Henning Zimmer, all of Disney Research Zurich; Zimmer also is a post-doctoral researcher at ETH Zurich and Richardt is a post-doctoral researcher in the REVES group at INRIA Sophia-Antipolis in France. More information about the study, including a video, is available at http://www.disneyresearch.com/project/megastereo/
About Disney Research
Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Disney Research is managed by an internal Disney Research Council co-chaired by Disney-Pixar's Ed Catmull and Walt Disney Imagineering's Bruce Vaughn, and including the directors of the individual labs. It has facilities in Los Angeles, San Francisco, Pittsburgh, Boston and Zürich. Research topics include computer graphics, video processing, computer vision, robotics, radio and antennas, wireless communications, human-computer interaction, displays, data mining, machine learning and behavioral sciences.