Home > The Monday Musings Column > We’ve got a backup for our backup – more things that went bump in the night

We’ve got a backup for our backup – more things that went bump in the night

Continuing in the run up to Halloween with tales of things that went wrong, this week we turn to a bit of a farce that we enjoyed along with our friends in Information Technology.

One site I inherited when I moved from Logistics to Facilities Management was a multi story office block that was almost wholly occupied by IT people and was one of two main centres for that trade. The building was also one of the main hubs for the company’s data network and, as such, was something of a sacrosanct site.

The FM work there had been part of the IT team, but we had inherited those people along with the site. They knew their job and they knew their building but, until we arrived, they had never had a ring fenced budget and, every year, something had been lopped off to fund IT project overspends.

As we dug deeper into the backlog of maintenance one thing that I had placed on the high priority list was the emergency backup generator system. This was a thing of legend at the site and beyond; “They have a backup generator for their backup generator” people would tell you around the company in terms of some awe.  The generator room in the basement had taken on qualities that might have been employed for a shrine, and the full time engineer that they had taken on to maintain the system played the role of high priest to the hilt.

Access to the room was something of a privilege, but my regional maintenance manager and I were reluctantly granted an entrance on the basis that we were now in charge. The room was pretty spotless and the two engines, one a Gardener and the other a Rolls Royce (no less) gleamed on the plinths.

The system was explained patiently to us. In the event of a power failure there was a battery backup that would allow a few minutes of power while the Gardener engine kicked in. If, for any reason that failed to fire up the Rolls Royce would deploy itself and, in the event of a long term power outage, the engines could be run alternately to keep the data flowing.

But it had never been tested. Yes, there was a switch that allowed a simulation power cut to see if these beauties would kick in and that was tried annually, but the overall system had never been tested. So I announced that we would, and requested a date when it would be convenient for us to do so.

The entire IT hierarchy were appalled and the ranks massed to oppose this folly, but in the end we got our way. We put in a bypass power source from the main switch so that the building would not actually lose power and threw the switch on the original circuit to make the generator room think that the mains had gone off.

The battery back didn’t work. It didn’t even have power enough to start the generator let alone support the building. But we had also found when we installed the by-pass that two thirds of the building, including a pair of new computer rooms, were already by-passing the backup system because corners had been cut in funding projects.

We found the money to put things right, but the backup myth died. These things have an importance at their own time, but times move on. We put a lot into that building to prepare it for the 21st century, but it has gone now, replaced by an apartment complex. Happy memories though!

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: