The ability to represent accurate and realistic 3D shapes with machine learning techniques has made a significant impact in the world of computer graphics and vision. More recently, a new technique called neural implicit representation has emerged as a novel way to more accurately represent 3D shapes, pushing the boundaries of innovation in automating complex 3D geometry, with ease and efficiency.
New work by computer scientists, Alec Jacobson, PhD, and Nicholas Sharp, PhD, at University of Toronto (U of T), addresses the next frontier: How to put these representations to work. The team have developed a novel technique to automate a wide range of geometric queries that enable detail-rich 3D representation in a number of applications.
“We wanted to develop tools to make it easier to apply these exciting representations to downstream applications in computer graphics like animation, physical simulation, computational fabrication, or robotics,” says Sharp, the paper’s first author and postdoctoral research fellow at U of T.
The researchers are slated to demonstrate their work titled, “Spelunking the Deep: Guaranteed Queries on General Neural Implicit Surfaces via Range Analysis”, at SIGGRAPH 2022 held Aug. 8-11 in Vancouver, BC, Canada. The annual conference, which will be in-person and virtual this year, spotlights the world’s leading professionals, academics, and creative minds at the forefront of computer graphics and interactive techniques.
“Ten years ago, we witnessed a dramatic change in computer vision research when deep learning methods took over. The current excitement around neural implicits could be a similar moment for computer graphics research,” says Jacobson, associate professor of computer science at U of T.
Neural implicit representations encode a shape by taking a point in 3D space as input; if the neural network outputs a positive value, that point is outside the shape, and a negative value means the point is inside the shape. But, if you have two shapes represented and want to determine if these two shapes intersect there is no clear, efficient way to do this.
For example in computer graphics, to make the character (shape one) interact with the scene or the environment, it is useful to know whether the character’s body intersects with objects (shape two) in the scene. How is this determined?
That’s where the researchers’ new approach comes in.
“Our new technique provides the low-level query that tests for this intersection, and many other queries as well,” says Sharp. “Our project helps build bridges between exciting new tools from machine learning, and these kinds of classic applications in computer graphics.”
The new technique shows how to analyze neural networks to evaluate such geometric queries by leveraging a decades-old approach called range analysis, which automatically determines whether certain regions are inside or outside the shape. This new method supports a number of geometric queries important in 3D modeling such as ray casting, constructing spatial hierarchies, fast mesh extraction, close-point queries, and more.
“While range analysis has been applied to geometric queries in the past, Nick’s remarkable insight is that this tool could be applied efficiently to deep learning-based representations,” says Jacobson. “This could substantially change the robustness and quality of learning for 3D data in the future.”
The researchers demonstrate how to perform queries on a broad class of existing networks in computer graphics, vision, and simulation. The queries can be efficiently evaluated on GPUs, and offer concrete accuracy guarantees even on randomly-initialized networks, enabling their use in training objectives and beyond. The aim is to make this new method immediately compatible with a wide range of past and future work on neural implicit representations.
Jacobson, who also is Senior Research Scientist at Adobe Research Toronto, and Sharp, now a Senior Research Scientist at Nvidia, are set to demonstrate their new method Aug. 10 at SIGGRAPH as part of the program, Roundtable Session: Convolutions and Neural Fields. The paper can be found here.
About ACM SIGGRAPH
ACM SIGGRAPH is an international community of researchers, artists, developers, filmmakers, scientists and business professionals with a shared interest in computer graphics and interactive techniques. A special interest group of the Association for Computing Machinery (ACM), the world’s first and largest computing society, our mission is to nurture, champion and connect like-minded researchers and practitioners to catalyze innovation in computer graphics and interactive techniques.