Saturday, September 13, 2008

Waiting for Hurricanes Ike, Josephine, Kyle, Laura ...

Hurricane Rita Traffic Jam
Houston, Texas residents were advised not to evacuate for Hurricane Ike this week, in an attempt to avoid a repeat of the chaos of Hurricane Rita in 2005, when ten times more people were killed during the evacuation than in the storm itself.

Meanwhile over 200,000 residents did not take the advice to evacuate the Galveston area, with authorities anticipating a major search and rescue operation after the hurricane has
passed.

Emergency services in Galveston have effectively shut down operations for the duration of the emergency.

It's becoming difficult to figure out when to follow government advice and when to ignore it. From a government perspective, it is important not to have failed to warn people to evacuate. From an individual perspective, each unnecessary evacuation carries a high personal cost.

With unpredictable but frequent high-consequence events such as these many alarms will turn out to be false, increasing the perception that the next alarm will also be false (people tend to overweight recent experience when estimating probabilities). Authorities know this and therefore have to overstate the facts to get people to act, people grow more cynical, and the cycle continues until a real disaster occurs.

Sometimes I wonder if the authorities should take a leaf out of the meterologists book and start using probabilities: (just what does a 10% chance of rain mean anyway?). "There is a 10% chance of you being killed by the next hurricane if you stay where you are" might be better advice to hear. But the evacuations would still have to be mandatory: figuring out a complex evacuation plan to keep traffic moving and provide safe havens where individuals can decide
whether and when to evacuate would be just about impossible.

Tuesday, September 9, 2008

When the internet fails...

Do Not Dig
You're ready for when your internet connection goes down, aren't you?

It's something you planned for. It's all in Appendix B of your Business Continuity Plan. The section entitled "How to Continue Operating When Someone Cuts Through Your T-1 With a Mechanical Digger."

Or possibly, as in our case, you Internet Service Provider has an unspecified software issue for 24 hours and takes 8 hours to admit it is their equipment, not yours, that is causing the problem.

So our Internet was down. I didn't think it was critical. Email to and from clients would be delayed for a while, but anything urgent could be transmitted by phone. We didn't host any services internally because we recognized that doing so would introduce too many unnecessary vulnerabilities.

Our VoIP link would be down, but we only use that for outgoing calls. We use the POTS (Plain Old Telephone Service) as where we are it is more reliable.

So none of our critical functions depended upon the internet. Or did they?
  • We couldn't organize an alternative dial-up connection quickly. The details we needed to set up an alternative dial-up connection were... on the internet. (Oops!)
  • We couldn't make travel arrangements for an imminent trip. (When did you last use a travel agent?)
  • We couldn't move money at the last minute to pay bills using online banking.
  • We couldn't research software support issues using the internet, and software development was hampered by a lack of online information.
  • Substantial effort was diverted from doing useful stuff to trying to convince our ISP's customer "support" staff that a problem existed, performing pointless tests on our equipment to placate them, and to devising alternative Internet access. (Sadly there is a very limited choice of broadband ISP in our area).
And of course I couldn't publish this blog or do research for the Risky Thinking newsletter.

Slowly, without realizing it, a reliance on the internet had crept into many of our business functions.

Has the same thing happened in your organization?

If the Internet went down would you still be able to:
  • Make travel arrangements?
  • Receive and process orders?
  • Ship goods?
  • Issue invoices?
  • Communicate with field staff?
  • Manage bank accounts?
If not, it may be time to improve and test your internet fallback arrangements.

It was not the end of the world when our internet connection failed, but it definitely caused us more problems than we were expecting.