99.999% Uptime is not Good Enough

“The industry standard of imaging system availability is 99.999 percent uptime, but that’s not good enough for Broward Health. Our vendor neutral archive (VNA) is a critical system in our enterprise imaging environment. It serves as the brains of our operation, supporting a variety of clinical customers within high acuity settings. Broward Health Medical Center is a Level I Trauma Center where every minute counts. We need zero downtime.” 

That’s what Boris Kalitenko, Sr., PACS administrator of Broward Health said to Brendan Devitt, professional services engineer at Mach7 Technologies. Devitt agreed and by leveraging Mach7 Enterprise Imaging Platform with its easy-to-use graphical interface, the team created a business continuity solution for Broward Health that has satisfied all stakeholders with essentially zero downtime and no headaches. In fact, during the last system upgrade, end users didn’t even know the enterprise image management system was taken out of production. 

Broward Health

Providing service for more than 75 years, Broward Health in South Florida is one of the 10 largest healthcare systems in the U.S. and currently operates more than 50 healthcare facilities. The system includes a comprehensive radiology department that averages more than 700 studies per day and a Level 1 Trauma Center, one of only seven in the state of Florida. Considering that nearly half of all standard emergency department visits include imaging services,¹ downtime is unacceptable. 

Downtime Happens

In our digital age, system downtime is inevitable. Whether it is an unscheduled surprise or scheduled downtime for software updates that include bug fixes, enhancements or operating system updates; downtime happens.  

Back in the early 2000s, when PACS solutions went down, the radiology department could print film and revert to a film-based workflow.² While a less than ideal scenario, work continued and critical cases went to the top of the worklist. 

Today, a film-based workflow is no longer an option. Not only is it difficult to find the equipment and materials, but finding the personnel with the expertise to print the film, and the radiologists to efficiently read it, would be a challenge.  

In a digital age, what’s a PACS administrator to do? Prior to implementing Mach7 Enterprise Imaging Platform and the business continuity solution (BCS) at Broward Health, scheduled downtime took weeks of preparation. Prior images for scheduled exams were pre-fetched days (if not weeks) before they were needed. Plans for manual routing of images were created, which included phone calls and training for every single site. The IT team scheduled additional resources to manage the increased manual intervention required to sustain network traffic. 

To minimize the amount of manual intervention, choosing a time where imaging volumes were reduced was critical. That meant the IT team would come in around 10 P.M. and if things went well, they would complete the task later the next morning. If things didn’t go well, it could take significantly longer. Stories abound on the Internet of 3-hour downtimes turning into 3-day catastrophes. 

Kalitenko noted, “Even relatively short upgrades were still done in the middle of the night, which had us all a little bleary-eyed, exhausted and stressed. This is not the condition you want anyone to be in when manually dealing with critical patient care data. The network would be hugely taxed before, during and after the downtime and when the system went back online, there was still a mountain of work to do to clean up and resolve the exceptions.”

A Better Solution 

Kalitenko searched for and found a solution. Working together with Devitt from Mach7, the team designed a solution to support Kalitenko’s dream. To start, the system is load-balanced where all modalities send to a virtual IP that sits above six production servers managing the over 700 radiology procedures that Broward Health completes each day. 

Broward Health added a seventh server, a copy of the production server acting as a BCS that replicates all production activities from archiving to routing and all specialty workflow. The seventh server (BCS) remains inactive until needed. During a scheduled maintenance, the six servers are stopped and the BCS server is activated. In the event of an unplanned outage, the BCS server is automatically activated (via artificial intelligence) and commences the support of the full production workload. The entire process takes less than 10 seconds with no impact to operations and workflow. Problem solved — end users experience zero downtime. 

“Zero downtime offers maximum benefit to our patients and our clinical team, and also benefits our IT team as we can now do software upgrades during the day,” Kalitenko elaborated. “Not only are we fully awake and alert during the process, but all the critical players are available, including our IT team, technologists, radiologists — everyone is available to help test and resolve any problems that may arise. Not that I would expect problems!”

A Partnership That Delivers 

Boris Kalitenko and the Broward Health image support team have been using the business continuity server for over a year without a single hiccup. Not only has this had a positive impact on patient care, but the IT team is happy to give up their sleepless, stressful upgrades.   

Kalitenko had this to say about the partnership: “Mach7 Technologies built a partnership with us that delivers great results. Working together, we proved that a core imaging component like our VNA can (and should) be 100 percent available — zero outages. A business continuity server with all functionalities of our production server is a life saver for a high acuity hospital system such as Broward Health.” 

References:

1. Wong et al, “Use of Imaging in the Emergency Department”, RSNA Publication, http://pubs.rsna.org/doi/full/10.1148/radiol.13130972. Accessed Sept. 14, 2016.

2. McBiles and Chacko, “Coping with PACS downtime in digital radiology” Journal of Digital Imaging: Aug. 2000, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3452972/. Accessed
Sept. 14, 2016.

Case study supplied by Mach7 Technologies.