May 18, 2021
In our most recent blog series, we’ve broken down what can go wrong when maintenance isn’t properly conducted for commercial HVAC systems, electrical systems, plumbing systems, and refrigeration equipment. In some instances, equipment or system failures resulting from maintenance neglect are manageable.
In other cases, though, those equipment failures can have harsh or even catastrophic consequences. Depending the function of a facility as well as the nature of the equipment failure itself, the results from basic maintenance neglect can quickly turn into a nightmare if facility managers or owners aren’t careful.
To paint a more vivid picture of this issue, we researched and uncovered three “horror stories” to show what can happen if planned maintenance is ignored or executed incorrectly
Horror Story 1: HVAC Burnout Shuts Down EHR System in California Hospital
For mission-critical facilities like data centers and hospitals, the effects of HVAC system failures are magnified. A northern California hospital had its EHR system shutdown due to an HVAC system burnout at its data center. As a result, medical professionals could not access vital patient records for over a week.
Reports showed that high temperatures, a direct result of the HVAC system failure, rose beyond the computer system’s tolerance levels. While some officials downplayed the impact of the HVAC breakdown and data center damages, there were undeniable consequences felt by hospital staff and their patients.
The ripple effects of the failure caused delays in patient care, including the postponement of important radiation treatments. In one specific instance, a woman visiting the hospital for a Lexiscan of her heart didn’t get the results of her test until two weeks later. This led to additional cardiac intervention for what doctors believed was a minor heart attack.
A lack of disaster recovery processes caused otherwise avoidable issues. Secondly, maintenance negligence led to an HVAC failure of not one, but two HVAC systems in the data center (the other HVAC system was overworked after the first system failed). These burnouts typically occur when an HVAC system must expend too much energy to function, which is often the result of overly delayed changes in air filters. It could also be the case that the unit itself had aged and deteriorated over time, which would have been a failure of capital planning for the mission-critical data center. In both of those cases, proper planned maintenance could have helped the data center identify risks (i.e. recommend an HVAC system replacement) and prevent total equipment failure.
Horror Story 2: Severe Water Leak Destroys Servers in Mission-Critical Data Center
Real-Time Data Center Solutions will, on occasion, publish nightmare scenarios for data centers around the country in a series titled “Confessions of a Data Center Manager.” One such story involves a data center’s servers going up in flames due to a short caused by an electrical fault, but another describes a situation in which the managers of a data center knew about issues with their plumbing system and procrastinated finding a solution until it was too late.
In the case of the plumbing snafu, the data center was housed in an older building with a documented history of minor water leaks. Management had already expressed their concerns that those minor leaks could be the sign of a catastrophic leak in the future. They knew the risk was significant, but instead of finding a planned maintenance solution for their plumbing system, they opted to install a monitoring system that could alert them in case a leak was detected.
Their project never made much progress due to indecision. One weekend, without anyone around to address the problem, a severe leak started that destroyed servers and resulted in lost customers. The project lead, who was tasked to come up with a maintenance solution before something like this happened, was let go.
This leak should have never occurred, especially given that management knew about the problem in advance. A simple pipe inspection with camera monitoring or a hydrostatic test would have shown the areas with active leaks or with a high propensity for leaks. For mission critical facilities, regular checks of HVAC, electrical (via infrared), and plumbing systems can be conducted without disrupting facility operations. Though there are costs associated with planned maintenance, they aren’t nearly as steep as the cost of servers and lost revenue due to unhappy clients.
Horror Story 3: Ground Fault in a High Voltage Cable Knocks Out 70% of a Hospital’s Power
More than most facilities (as noted in the first example), an electrical system failure at a hospital, especially when backup systems aren’t functional or properly used, are highly problematic because of the potential ramifications for patients and staff. This was demonstrated when a ground fault in a high-voltage cable between switchgear units knocked out more than two-thirds of a Swedish hospital’s electricity network.
Fortunately in this case, no serious harm was done to patients during the power outage, which lasted slightly less than 1.5 hours. Nevertheless, hospital functions took a considerably long time to recover even after the power was restored. During the outage, patients could not signal for assistance, medical cabinets with combination locks were inaccessible, food supplies were delayed, and surgeons could not connect cylinders of respiratory air. Likewise, hospital IT systems completely shut down.
In retrospect, the person manning the hospital’s control center did not have the knowledge nor the expertise to recognize faults or switch promptly to emergency power systems. When it was eventually determined the issue was a ground fault, the onsite technician called a maintenance company to solve the issue. The initial call was not effective as they didn’t have the expertise or equipment to locate the precise location of the fault. Finally, well after emergency power was restored, they located the fault and replaced the 300+ meter cable.
In the case a third-party facility maintenance staff was working in the control center, the staff should be trained to respond correctly in an emergency situation and recognize potential faults before they become a problem. If onsite staff calls a third-party for an emergency like this scenario, TD’s truck-based service group would have the equipment necessary to diagnose and properly locate a fault even if the replacement cable could not be installed in the same day.
Value of Planned Maintenance and Expert Emergency Staff
Planned maintenance can dramatically reduce the chances of horror stories from occurring in your own facility. Even if your facility wouldn’t be classified as “mission-critical,” there are still dangers that stem from an overreliance on reactive maintenance or under-trained staff.
Interested in identifying the main points of risk within your facility and how to evaluate them? We encourage you to download our helpful free e-book.
In the instance that you do have a maintenance emergency, you need experts that can quickly diagnose the issue and solve the problem right away. Having a reliable partner — one that values the on-going training of its staff — is critical in the worst-case scenarios.
Tag(s):
Maintenance