Archive

Posts Tagged ‘disaster recovery’

the lockdown log 20


Weather extremes are messing with my efforts to keep my various projects on track, the extreme heat not being conducive to labouring outside nor for some of the indoor jobs and now heavy rain has further complicated things; traipsing mud around is not popular and I have to wait for things that I could not paint because it was too hot to dry out before I can paint them.

These are the sort of buggerment factors that all project manage to face up to and I will manage somehow. The year is slipping by and it is now barely light when I get up at five in the morning and darkness is falling by around nine in the evening. There is still much to do if I am to meet my self-imposed plans, but if I look back there is a huge amount completed. I have written in my Monday Musings blogs about the need to occasionally stop and look back to see how far you have come and it is an important psychological boost when you start to feel that you are losing momentum.

Having written my last Monday Musing on crisis management I had to put my abilities into practice last week when one of the local foxes wandered into the house and then went berserk trying to get back out. Fortunately keeping calm and being patient Reynard was persuaded to leave, but in its blind panic it took a while for it to realise that it was rushing past an open door in its attempts to get out of a window. No damage was done in the house and the fox seems to have recovered as it has been several times since.

Such diversions are not always welcome, especially when, as happened here, I had just got everything ready to start a job when I was called shift our furry visitor. By the time that I had got rid of it, helped to restore order in the house and discussed with the Berkshire Belle (who had taken it all very calmly) how we could prevent further incursions I had largely gone off the idea of what I had gone out to do. Certainly it took me so long to get my head back into the game that I didn’t get the job finished.

With the Law of Sod in full swing it appears that my deck stain sill be delivered today or tomorrow. I will re-check the directions, but given that the weather has broken (after the thunder storm of last night it has already rained three times this morning in the two and a half hours that I have been up. Looking at the forecast my chances of getting the new decking fry enough to stain look bleak for the next ten days or so. Ho hum; Plan M I think I am up to now.

I cut my hair again this week, the fourth self-haircut of this Summer. My usual barber has apparently reopened, but I am a little twitchy about going there as we have a significant upsurge on the Covid-19 front here. So another sit in the back garden with the clippers and using my ‘phone camera as a substitute mirror has, at least, tidied me up again. In the process of using the ‘phone to check my work I inadvertently took a selfie; who is this old git I see before me? Small wonder that they want me to wear a mask when out; it must make me look less frightening. Age creeps up…

One of the biggest problems we face at the moment is that hope is being drained. I can only speak from a personal perspective, but the little things that we enjoy as a couple; going out to shop, to eat or to visit places is lost for the foreseeable future. An end to the pandemic is not in sight and we are trying to adapt, but many of the things that brought us joy are out of reach. Yes I know that there are millions worse off than we are and that we have many privileges that others crave, but that is where we have to try and get our mindset changed. We have become used to being free to do what we can afford to do and now we can’t much of that and, because we think that is is sensible, are choosing not to do other things.

This week we have learned that one of my nieces and her partner have bought Covid-19. Fortunately they seem to have been only mildly affected, but they are the first people that we know that have caught it and even though they live an hour’s drive away it somehow brings it closer.

At the end of the day we have each other and that matters enough to keep us fairly sane in these weird times. We hope that you are all staying safe too.

on crisis management


I have, over the years, had to manage a few some lasting a matter of hours, many for a couple of days and one or two that went into a second week. They come along with reasonable regularity and most you have planned for, for example there is a good chance that you will get a power failure at some point and you should have plans in place that you test. Occasionally you get some warning; floods for example, but most of the time a crisis will come out of the blue and you need to react.

Leaving aside the details of how you manage a crisis there is a common thread and that is that you need information in order to make decisions about what to do. The problem that always occurs is that the information you are getting is not static. Let’s take an equipment failure as a starting point. Something stops working so you do the obvious checks and then call for an engineer. They will be with you in two hours you are told and so you work on the basis that it will probably be three hours before they get to you (experience) and then at least another hour before they have diagnosed the fault.

So you are planning on at least four hours before you know what is wrong and then you should have some idea of how long it will take to fix. You make some decisions about what you can do to continue business and communicate an action plan. Half a day is potentially down the toilet and you will be trying to work out the implications of that. You should have some plans in place for this sort of problem and you will have kicked those off, but, as the military will tell you, no plan survives first contact with the enemy. In this case the enemy is life and it will conspire to screw you in all probability.

The fitter arrives five hours after the call and starts work. There is no point in asking how long at this point because they cannot tell you any more than that they need a certain amount of time to run their tests. Leave them to get on with the job and concentrate on the things that you can do something about, but, by now, you have lost today.

As you try to reschedule again and plan communicating the changes news reaches you that one group of people have, instead of doing something that you asked, decided that they had a better idea and have implemented it. It might actually be a better idea, but doing it in isolation has screwed up the overall plan (by now you are probably on Plan C).

Trying not to panic you accept that today is really now finished and look to work out what you will do tomorrow. There are still variables though and the engineer tells you that the part that has failed has been identified, has been ordered and is en-route by courier from Germany and that he will be back in the morning to accept it on delivery and fit it. You know not to ask when things will be running again, but you ask anyway. The response is a shrug. Plan D is worked out that evening along with plans E and F as contingency and you have all the communication ready to roll come the morning.

I won’t go through the story* any further because the moral has been set out: Things change all of the time around you and trying to juggle them all whilst new things keep getting thrown in as ones in the pattern are snatched away is all part of managing. Rarely does anyone tell you that you are doing well and, whilst you always have crisis plans, all they do is give you a rough framework and a few tools. Most of what happens you will be making ups as you go along based on information, predictions and experience of which the first two will be changing, often to the extent of complete U-turns.

All you can do is to keep your head and plough on. You know that you will, at times, look stupid or incompetent, but don’t let it phase you. The Monday Morning Quarterbacks will have a field day with the benefit of knowing the results and will glory in the bits that you didn’t do so well, but they were not there in the hot seat.

One of the problems is that there will always be some damage and anyone who try’s to say that there should n to have been is wrong (I have a lot of other descriptions for them, but I’ll keep this clean). What you want to do is to minimise that damage and as long as you do that you have succeeded.

Yes there needs to be a drains up review afterwards, but that has to be solely about learning; apportionment of blame can play no part because you want the truth not the smokescreen of a fighting defence.

  • The story is a real one. The new part arrived and failed that day. It took four more days to find the root of the problem and another two to implement an effective cure. It cost us dearly in terms both financial and to our reputation and all because a new piece of kit being operated by the company next door was causing spikes in the electricity supply. We did survive though and over the next twelve months we recovered, but one customer had left all of our daily briefings and delighted in trotting them out every time we got around to negotiating another contract with them. That’s life; suck it up and keep smiling.

on mob rule


We are living in a strange time and one where rampant mobs seem to be more tolerated than dealt with. Peaceful protest is one thing and I will always support the right of people to gather and march in support of a cause regardless of whether I agree with their point of view. But violent protest and damage to property are criminal acts.

For those involved in workplace management (or facilities management as we called it for a while) a plan of some sort is required to deal with the mob should one be encountered. It should be part of the organisation’s risk and crisis planning and taken seriously.

It is a while since I have had to worry about such things, but through the eighties and nineties into the noughties it was my problem and there were times when the biggest problem was not the possibility an unruly mob at the gates, but the unruly mob of senior managers clamouring for action.

There is a notion that the people in the upper echelons of an organisation have qualities above those of the people that they employ. This is basically true, but there are times when all sense of proportion is lost and stupidity takes over.

One example was a building that was under the flight path to an RAF base, our location, given the prevailing winds, normally being under the landing path. Within our disaster plan we had allowed for a normal major emergency evacuation of the type where we would hand over to the emergency services, but the personnel director insisted that we should have a specific plan for an aircraft crashing onto the site. I took the existing plan to the Fire Service who had no comment and then to the liaison officer at the airbase. Over coffee he solved my problem; “Why not just extract that bit of the plan and have it as an appendix with a title like “In the event of an Air Crash” he suggested. I did that and my problem went away.

Another piece of lunacy was the dreaded Millennium Bug. As the nineties ground to a close the threat of computers crashing and all sorts of problems occurring at midnight on the dreaded day were being bandied around. It was thought that there would be widespread civil unrest and that rioting and looting would ensue. This we considered with regard to the city centre properties in the portfolio that I was managing. We would have the normal small teams of security guards in each of these sites and my plan was that these people had a way of safely evacuating themselves in the event of trouble.

The idea was that we would keep the people safe and if a building got trashed then we had business continuity and recovery plans in place, but the likelihood of trouble seemed very remote. My own experience of computer programming and software design from the previous decade was that the century roll-over was covered.

Late one evening just before Christmas I got a call at home from one of the directors based in a City of London site. Effectively they wanted me to be at their building overnight on the 31st December in case of trouble. Quite what I was expected to do if faced by a rioting mob I was not sure, but they were insistent. I was equally firm about not going and I didn’t. Nor did they. Nor did the mob assemble.

Today though it is not a joking matter and there are real threats from mob behaviour that need to be addressed. How you deal with that is up to you if it is your responsibility, but my advice is to think first of the safety of your people. Then brush up on your business continuity and recovery plans so that the relevant people are aware and thinking about what they will need to do. Be realistic and think about what you will do if any part of the plan does not work. Contigency plans for your contingency plan? You had better believe it.

I hope that you have not problems, but good luck if you do. Just remember the golden rule; people before property.

Why am I so keen on planning and preparing for crisis management? I was born to it

September 5, 2011 1 comment

I’ve written here a few times about various aspects of incident management and, as one or two have remarked, maybe I’m a bit of an anorak about these things. They may have a point because, to some extent, incident management has been with me since I was in short trousers.

My childhood was spent living on country estates, more that usually with a farm attached. We didn’t own these places, my parents worked there so that explains my interest in customer service; I was, in fact, born into service. But the incident management side of things comes from that background too. In recent years risk assessments have become a fad in many ways, but they are just a formalisation of what I was taught to do in the late 1950s by people who understood such things intuitively.

So how does what I learned all those years ago down on the farm fit with the management of modern property? Well take one sort of incident management that a typical facilities management team should have down pat, that of fire. One of the things that we handled with considerable frequency was fire. Not just the risk of fire (and I have seen a barn fire at close quarters), but managing fires that we would start on purpose. We would have at least one managed conflagration a week as we burned refuse, burned off fields, bracken or whatever. And when I talk about burning refuse I mean bonfires that the average village would be proud of on November 5th; you can create a huge weekly pile from a 50+ acre estate.

These things are not done willy nilly, they are carefully arranged, taking into account the wind, time of day and nature of what you are burning. A compost heap large enough to keep  Time Team busy excavating it for a week will burn for days if it spontaneously combusts. Siting the bonfire, compost heap or whatever is carefully thought through. Precautions are taken and what you’ll do if things don’t go as planned are worked out. We were taught to understand consequences and about accepting responsibility.

On a farm or large country estate there is a lot of serious kit and danger lurks all around. As kids we were brought up to understand and respect things, so maybe it should be no surprise that it is so ingrained in me. That’s not to say that I don’t take risks; I do, but I think about it first. From what I learned as a child I got to think about things like always checking my escape route(s) when I stay somewhere away from home, like counting the seat rows between me and the nearest two exits when I fly, like my winter survival kit that I carry in the car from November through March.

I had the benefit of learning about these things to the degree that it became automatic. It was only later in life when I became responsible for large numbers of people that I began to think about it and to analyse what I was doing and why as part of developing drills and desktop run throughs. When you have a bloody great fire every week on purpose you are doing it for real, but when you are running big buildings, thankfully, you don’t, so you need to have dummy drills.

Practice does make perfect which is why I have ridden my teams hard on these things, and that is why we coped so well with some of the incidents that have described in these columns. It’s in my blood.