Feature | Cardiac Imaging | March 17, 2019 | By Greg Freiherr

Sunday ACC 2019: Mixed Reality Offers Advantages of Virtual and Augmented Realities

Hybrid Mixes Digital Data With Natural Environment

Jennifer N. A. Silva, M.D., a pediatric cardiologist at Washington University School of Medicine in Saint Louis, Mo., describes “mixed reality” at ACC19 Future Hub.

Jennifer N. A. Silva, M.D., a pediatric cardiologist at Washington University School of Medicine in Saint Louis, Mo., describes “mixed reality” at ACC19 Future Hub.

Greg Freiherr

Greg Freiherr

Virtual reality (VR) and its less immersive kin, augmented reality (AR), are gaining traction in some medical applications. But a new entrant, “mixed reality,” may have the inside track in the interventional suite or operating room.

Mixed reality allows the user to interact with both the real world and digital data, according to Jennifer N. A. Silva, M.D., a pediatric cardiologist at Washington University School of Medicine in Saint Louis, Mo., who spoke during Future Hub at this year’s annual meeting (#ACC19) of the American College of Cardiology.

With ready access to both digital and physical reality, a surgeon or interventionalist can simulate a procedure on a digitally created model that is updated in real time – then perform the actual procedure on the patient.

“Mixed reality affords the ability to remain in your natural environment,” Silva told Imaging Technology News (ITN) and Diagnostic and Interventional Cardiology (DAIC). Yet the users of mixed reality can take advantage of a digital world otherwise found only in VR.

Mixed reality is somewhere in the middle of what Silva described as a spectrum of “extended realities.” At one end is virtual reality (VR), which completely immerses the user in a digital world. Because VR is a completely separate and alternative reality, it “does not allow you to have a meaningful interaction with your natural environment,” she said.

At the other end of the spectrum is reality, characterized by what we sense with our eyes.  In between these two ends are augmented reality (to which mixed reality is closely related). In AR, the user cannot interact with but rather only look at digital data imported into the natural environment, Silva said.

“Mixed reality allows you to have a meaningful interaction with digital data and with your natural environment,” she said. “So instead of just looking at a hologram, you can do something to it -- you can touch it; rotate it; turn it upside down.”

 

Fitting Round Pegs Into Round Holes

The technology – and the type of reality it harnesses – “has to fit what you are trying to do,” she told ITN and DAIC. “There has to be a match between the technology and the need.”

Different types of reality may be useful in one or more of the four major application areas: education, pre-procedural planning, rehabilitation and intraprocedural visualization. VR offers little value during procedures, because it immerses the user in a completely digital world and, therefore, blocks out the natural environment. For this same reason, however, VR may be useful in rehabilitation, as it may replace the user’s perception of the real world with a digital one.

VR may also be useful when practicing a surgical or interventional procedure. Augmented reality has proven useful in actual interventions, presenting data that a surgeon or interventionalist can use during the actual procedure. But mixed reality offers the ability to interact with digital data and with the real world in the same context and time frame.

“If I’m doing a procedure, I can’t fully remove myself from that environment (to interact in VR). I need something that will allow me to interact with my natural environment and with the digital data,” said Silva, who was one of the moderators during March 16 ACC Future Hub session titled “Are Clinical Virtual Reality and Augmented Reality Ready for Prime Time?”

“The ability to manipulate and control your data is perhaps the single greatest value add of the extended realities,” she told ITN and DAIC. Theoretically, mixed reality allows this manipulation and control.

“We are on the first step of what I think will be a long road,” said Silva, who is one of the founders of a mixed reality company, called SentiAR.

Silva and her colleagues right now are struggling to achieve connectivity among the different pieces of equipment found in an EP interventional lab. Examples of this equipment are fluoroscopy systems, vital signs monitors and cardiac mapping systems. “A lot of them are made by different vendors, so it lends itself to poor interoperability,” she said. The goal is to integrate the data from the equipment, as well as their workflows.

“I want to make the platform the glue that starts to allow me to interact with all my pieces of equipment,” she said. 

 

How Mixed Reality Might Impact EP

Mixed reality may prove particularly useful during electrophysiological procedures, when interventionalists map electrical activity inside the patient’s heart, then ablate tissues associated with electrical signals that cause arrhythmia. The device being developed by SentiAR creates a 3-D model of the patient’s heart that shows the map of electrical signals in real-time, said Silva who specializes in pediatric cardiology. The mixed reality platform being developed at SentiAR can be used on patients of any age, she noted.

The user can see 3-D images and other digital data, as well as the physical world, by looking through a head-mounted technology called HoloLens, which is made by Microsoft.  Silva described it as “a computer on your head” that allows the user to see digitally created structures superimposed on a view of the natural environment.  Simultaneously, sensors built into the head-mounted technology record head movements that allow the user to click through digital menus projected onto the user’s field of vision or to control different pieces of equipment in the interventional lab.

Although HoloLens has proven useful, the company is not inextricably tied to the Microsoft technology. Instead, the SentiAR is keeping its options open to take advantage of new technologies that may come along. “There are many mixed reality headsets out there, and there are many more in development,” she said. 

Silva said the product being developed by SentiAR is a long way from finalization. “As good technology does, it will continue to evolve,” she said, noting that future technologies may harness new types of reality that represent different points on the “extended realities” spectrum.

“This (spectrum) is where innovation is going to happen.”

 

Greg Freiherr is a contributing editor to Imaging Technology News (ITN). Over the past three decades, Freiherr has served as business and technology editor for publications in medical imaging, as well as consulted for vendors, professional organizations, academia, and financial institutions.

 

Additional Content:

Two Technologies That Offer a Paradigm Shift in Medicine at HIMSS 2017 

VIDEO: Augmented Reality for Surgical Planning

VIDEO: Users Can Touch This Virtual Reality Heart

A 40,000 Foot View of Trends in Cardiology

Applications for Artificial Intelligence in Cardiovascular Imaging

Machine Learning Approaches in Cardiovascular Imaging

Technology Report: Artificial Intelligence (Video report published January 2019) 

PODCAST: Shortcomings of CTA in Cardiology

PODCAST: How Technology Is Changing Cardiology

ACC.19 Future Hub Hosts “Shark Tank” of Emerging Technologies In Cardiology

Related Content

AI Metrics, LLC, a medical imaging startup focused on augmented intelligence to improve patient care, announced today that the U.S. Food and Drug Administration (FDA) has granted 510(k) clearance for the company’s flagship image analysis platform.
News | Artificial Intelligence | January 18, 2021
January 18, 2021 — AI Metrics, LLC, a medical imaging startup focused on augmented intelligence to improve patient ca
Coronary CT angiography images courtesy of Canon Medical Systems.

Coronary CT angiography images courtesy of Canon Medical Systems.

News | Computed Tomography (CT) | January 18, 2021
MRI of Nonferromagnetic Ballistics Suspended in Gelatin. 

MRI of Nonferromagnetic Ballistics Suspended in Gelatin. Scout (A), T1-weighted spin-echo (SE) (B), T2-weighted SE (C), T2-weighted gradient-recalled echo (GRE) (TR/TE, 500/10; D), and T2-weighted GRE (TR/TE, 700/30; E) MR images show jacket hollow point .45 automatic Colt pistol bullet (Corbon) (1), solid lead .45 Long Colt bullet (Winchester) (2), full metal jacket (FMJ) automatic Colt pistol bullet (Winchester) (3), 5.56-mm FMJ bullet (Federal Ammunition) (4), #7 lead shotgun pellet (Winchester) (5), and 5-mm lead air gun pellet (Sheridan) (6). On all sequences, metallic artifact is minimal. Although metallic artifact increases or blooms with increased TR/TE in GRE images (D and E), amount of surrounding distortion is still minimal.

News | Magnetic Resonance Imaging (MRI) | January 15, 2021
January 15, 2021 — 
Novel Coronavirus SARS-CoV-2 Transmission electron micrograph of SARS-CoV-2 virus particles, isolated from a patient. Image captured and color-enhanced at the NIAID Integrated Research Facility (IRF) in Fort Detrick, Maryland. Credit: National Institute of Allergy and Infectious Diseases, NIH

Novel Coronavirus SARS-CoV-2 Transmission electron micrograph of SARS-CoV-2 virus particles, isolated from a patient. Image captured and color-enhanced at the NIAID Integrated Research Facility (IRF) in Fort Detrick, Maryland. Image courtesy of  National Institute of Allergy and Infectious Diseases (NIH)

News | Coronavirus (COVID-19) | January 15, 2021
January 15, 2021 — In one of the first studies to examine the impact of the...
Multidisciplinary functionality supports high-capacity utilization and economic efficiency
News | Mobile C-Arms | January 14, 2021
January 14, 2021 — Siemens Healthineers has announced the Food and Drug Administration (FDA) clearance of the...
Use of telehealth jumped sharply during the first months of the coronavirus pandemic shutdown, with the approach being used more often for behavioral health services than for medical care, according to a new RAND Corporation study.

Getty Images

News | Teleradiology | January 13, 2021
January 13, 2021 — Use of telehealth jumped sha
This cutting-edge technology assists radiologists in the interpretation of screening mammograms and allows them to diagnose breast cancer earlier and more accurately
News | Mammography | January 12, 2021
January 12, 2021 — Therapixel announced that...
The U.S. Food and Drug Administration released the agency's first Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan. This action plan describes a multi-pronged approach to advance the Agency's oversight of AI/ML-based medical software.
News | Artificial Intelligence | January 12, 2021
January 12, 2021 — The U.S.
Myocarditis among recovering COVID-19 athletes less common than previously reported

Getty Images

News | Cardiac Imaging | January 11, 2021
January 11, 2021 — In a letter published in the December issue of the American Heart Association's...