image: Dr. Sarah Murthi tests an augmented reality prototype that overlays ultrasound data directly on the patient. view more
Credit: Maryland Blended Reality Center
Ultrasound data displayed directly on a patient via augmented reality headsets. Immersive “grand rounds” for medical students and faculty even when they’re in different locations. Virtual reality landscapes matched with classical opera to transport people with painful injuries outside of themselves, reducing the need for potentially addictive opioids.
These medical examples of extended reality (XR)—the umbrella term used for technology based in virtual and augmented reality or other immersive media—are already being prototyped or tested in clinical trials. But its widespread use in hospitals and other health care settings is currently hampered by technical challenges and sparse regulatory guidelines.
Now, with $5 million from the U.S. National Science Foundation (NSF) and technology titans including Google, Microsoft and Meta (formerly known as Facebook), a trio of academic institutions are collaborating with industry and the federal government to develop, test and certify XR technologies in medicine and health care.
The new Center for Medical Innovations in Extended Reality, known as MIXR, joins University of Maryland computer scientists and engineers with physicians and clinicians at the University of Maryland School of Medicine in Baltimore and the University of Michigan to improve medical training, patient management and health care outcomes across all areas of clinical practice.
The award is part of NSF’s Industry-University Cooperative Research Centers (IUCRC) program, designed to jumpstart breakthrough research by enabling close and sustained engagement between industry innovators, world-class academic teams and government agencies.
Behrooz Shirazi, acting deputy division director of the NSF’s Division of Computer and Network Systems and a program director for IUCRC, called MIXR one of the first national centers at the intersection of medical and computing sciences. “We expect this vibrant collaboration to produce significant societal and health care impacts,” he said.
In addition to Google, Microsoft and Meta, other technology companies involved in MIXR are Sony, Magic Leap, Health2047, GigXR, Brainlab and apoQlar.
Another key partner in the MIXR initiative will be federal regulatory experts working at the U.S. Food and Drug Administration, ensuring that safe, effective and innovative clinical solutions make it to patients as soon as possible.
“We’ll work closely with our industry and government partners to answer any scientific questions regarding regulatory evaluations and decisions needed for the widescale clinical use of these devices,” said Amitabh Varshney, professor and dean of the College of Computer, Mathematical, and Natural Sciences at the University of Maryland.
Varshney is the lead site principal investigator on the project, and is joined by partner site PI’s Sarah Murthi, M.D., an associate professor of surgery at the University of Maryland School of Medicine, and Mark Cohen, M.D., a professor and vice chair of surgery at the University of Michigan Medical School with appointments in pharmacology and biomedical engineering. All three have extensive experience developing and using immersive technologies in a medical setting.
Varshney and Murthi co-direct the Maryland Blended Reality Center (MBRC), launched in 2017 as part of MPowering the State, the strategic partnership between the University of Maryland, College Park and the University of Maryland, Baltimore.
Early projects out of MBRC focused on prototyping new diagnostic tools to assist physicians at the renowned R Adams Cowley Shock Trauma Center in Baltimore, where Murthi is a critical care doctor and director of the critical care ultrasound program. This includes innovative AR medical displays that could improve how bedside procedures are done.
MBRC clinicians also teamed up with the Maryland Institute College of Art to test a virtual reality platform that can help patients deal with physical and emotional trauma through immersion in another world, with a focus on quadriplegic patients who are hospitalized with acute spinal cord injury.
In 2018, Murthi and Varshney co-authored an op-ed in the Harvard Business Review that detailed how augmented reality could improve patient care and lower costs in hospital settings.
“Immersive technologies have the potential to fundamentally change, improve and reduce the cost of medical training and of maintaining clinical skills across all aspects of health care,” Murthi said.
Cohen, who leads the Center for Surgical Innovation and trains new physicians at Michigan, said that using XR in medical rounds—adding in virtual reality-based illustrations or augmented reality data overlaid on a patient—makes for a richer experience for both teacher and student, both in-person or virtually observing from hundreds of miles away.
“We realized that having this ability to interact virtually with both the patients and residents—pulling up holographic windows and showing diagnostic imaging and labs—would greatly enhance the educational experience in these traditional grand rounds,” he said.
In an interview published last year, Cohen said he is also interested in combining XR technology with machine learning, hoping to leverage sophisticated immersive diagnostic imaging resources with artificial intelligence algorithms to “better predict when diseases will flare up, and how to improve the way we follow and treat chronic diseases like heart failure, cancer and diabetes.”
The MIXR initiative is heavily dependent on powerful computing resources. At Maryland, those resources will be handled by the University of Maryland Institute for Advanced Computer Studies. This includes building and maintaining a soon-to-be-unveiled “HoloCamera” studio, where more than 300 immersive cameras are fused together to bring unique visualization technology to bear on immersive medical environment captures.
The new camera system can be used to record cinematic-quality 3D demos of surgeons teaching intricate procedures like a lower extremity fasciotomy, a limb-saving technique of cutting the sheath of tissue encasing a muscle to treat for loss of circulation.
Joseph JaJa, professor and chair of the Department of Electrical and Computer Engineering at UMD, is also providing expertise. As the lead-site co-PI, he will support the integration of high-performance computing and machine learning into the XR technology being developed and use his extensive experience working with industry to foster stronger collaborative efforts.
Barbara Brawn, currently working with Murthi and Varshney as the associate director of MBRC, will act as the industry liaison contact between MIXR researchers and technology companies keen to see their latest hardware and software tools used to save lives and improve medical training.
“The synergy in MIXR will be contagious,” Varshney said. “Our industry partners will push forward new ideas and novel technologies. The scientists and physicians will help refine and test those ideas. And we both will work with the FDA to bring these technologies from the lab to the proper health care setting where they can have an exponential impact.”