Marc’s Mind Nothing to see here. Move along.

These are the times that try men’s souls…

Even though Thomas Paine was talking about the American Revolution when he wrote that famous passage, the phrase he used aptly describes the last week and what thousands of people in the Seattle and Pacific Northwest have been dealing with. Last Thursday, a major storm hit the region, bringing with it hurricane force winds and massive destruction causing damage and widespread power outages from Portland to Washington's Canadian border. The day before the wind storm, we had torrential rain that softened the ground and loosened up tree roots. Trees came crashing down everywhere leaving the area looking somewhat like we had just finished a war. It was surreal as I was driving into the office morning after the winds died down, everywhere I looked there were trees that had been snapped in half or uprooted completely. There were no streetlights, no lights in anyone's home and very few cars on the road. It was made even more disturbing by how grey everything was. Living in the area for the past 14 years, I've gotten used to grey skies, but this was a dirty grey that was more like a haze over everything, it was very unnatural. The governor even declared a state of emergency which wasn't surprising considering that three quarters of the population was without power and scrambling to find warm shelter, gasoline (since without power the pumps weren't running), fire wood, batteries, etc. since the temperature on Friday night was going to drop to below freezing. This storm came on the heels of a nasty bout the region faced a few weeks earlier with snow and ice that all but shut down the Puget Sound's transportation system. Things are mostly back to normal again now, but you can still hear chain saws clearing away the fallen trees that knocked down power lines, smashed cars and houses and blocked roadways.

I have to give my respect and compliments to the crews working at the power companies, specifically the Puget Sound Energy crews. From the news reports I've heard, they had more than three quarters of their substations down and had to repair major damage just to get them back online. They had to repair the substations and feeder lines before they could work on getting anyone's power back up. They did an amazing job and very quickly given how much damage was done and how cold the weather was here. Thanks again. I do have a major criticizm, though. I understand the motivation, but I don't approve of it. Friday Bellevue was dark, there weren't more than a few blocks that had power, including Overlake Hospital (who happens to be on the same substation our NOC is fed from -- part of why our NOC is where it is). The hostpital's generators performed and kept them with the power they needed until Sunday night. Saturday morning, though the Bellevue Square Mall was open for business. It really bothers me that the shopping mall got priority over other the hospital and other medical businesses that surround the hospital (and many of our neighbors). If it weren't Christmas shopping season, chances are pretty good that a hospital would have been a slightly higer priority than a shopping mall.

I've been debating for the last couple of days on whether or not I wanted to post anything at all here about how it impacted me, my employees and co-workers because I didn't want to seem petty or have any bad repercussions. The biggest problem is that the ISP business that I've run for more than 12 years had power reserves that failed. That caused our entire network to go down from Thursday at about 11:30pm until Sunday until about 5:30pm when we got power back -- 66 hours. DSL accounts came back online by about 6pm and web sites and email were back online and fully functional by about 7:30pm. During this outage we were cut off from the world. This impacted thousands of DSL subscribers and web sites. In short, this was one of my worst nightmares come true. Since then, we as a company, and me personally have been criticized by the vocal minority. Pretty harshly, too. Its hard not to take it personally when you've spent the last 12+ years building a business and a reputation for being reliable and then one day having that reputation shredded by a few people who claim that we did nothing to prevent it or recover from it. They're wrong. Of course, me posting that in any sort of public forum would just fuel the fires of the "troll's" and make me look bad. Thats the reason for this post. This is my venue, and I control what people say and don't say. I probably won't authorize any comments from this post because its puprose is not for me to wage a flame war with trolls, but for me to speak my peace in a place where those I know can read it and understand what happened from my viewpoint without the contradictory opinions from those who were not involved.

As I mentioned before, we were criticized for not having adequate disaster recovery plans in place. We did, but they failed. Disaster recovery plans are those documents and guidelines that tell people what to do in case of a disaster. Sounds pretty straightforward, and are necessary for any business to have. Until the disaster strikes, however, you never know how well the system will work. Ours didn't. Instead of pointing fingers at people and whining about all the things that wen't wrong, though, we're learning from it. I'm quite confident that after the changes we're laying out are implemented, we will be able to handle an extended outage like this with little to no impact to our customers. The people harping on us for this are, for the most part, your typical garden variety usenet/forum troll. They find something that they have an opinion on, and start picking apart whomever they are "debating" (using the term loosely) with. It doesn't matter what is said by the opposing debator, they will always find something wrong with what is said.

I've spoken on the phone with several customers and they are generally much more polite than they are online. I'm not sure why that is, though. To speculate, I'm guessing that the anonymity and the non-real time nature of a forum or usenet post gives them more time to think about how to craft their responses. Perhaps trying to make other people feel inferior in forums somehow makes them feel superior. I'm not a psychiatrist, nor do I claim to command some great grasp of the human psyche, but I can't come up with anything else. It does really bother me how people can be so rude, demadning and judgemental on a forum, but once you get them on the phone or face to face they are totally different.

Case in point. One of worst of this vocal minority has sent about half a dozen messages to our sales and support team in first half of this week. He's also posted in usenet and another forum. The first thing he did after he called and got me on the phone was give me a poorly veiled threat that he was going to just "have to take this to usenet". It seems he didn't want to actually talk to me about it, so I responded to him "You have me on the phone now, what exactly did you want to talk about?" Most of the conversation was one pretty sided with me talking and him being quiet. I don't think its "answers" or "information" (using the same terms he used with me) that he was after, rather something to debate on and criticize. Even if I were to provide him with all of the details and information that he says he "needs", what exactly would he do with it? There is absolutely nothing he could do with it. Despite having encountered this mindset countless times in the 15 years I've been on usenet, I still don't understand it.

Many of these people I talk to I want to ask them a few things. For example, what were they doing to help during this disaster (and it was a disaster, and not just for us)? The answer for many of them, I suspect, would be "trying to stay warm". For us the answer would have been a bit different. In addition to trying to stay warm, which was no easy feat for many of us, about a third of the staff was at the office (where our main NOC is) trying to figure out if there was anything else we could do to get power, which there wasn't. In an outage this size, we couldn't even find a house sized generator anywhere -- we looked. Two of our staff members were in the eastern part of Bellevue with their chain saws helping to clear away trees from the road and off of fallen houses. Two more of our staff were repairing the large hole in the roof of one of their houses that the wind created by ripping off the shingles and then part of the roof. Personally, I was at the office waiting and thinking, as well as making and recieving a few calls trying to look for other sources of power. I had to make calls from the office since it seems that Sprint's towers in Bothell were either down or also without power. I couldn't make or receive calls on my cell phone at all until power came back up there on Sunday night.

As I've said many times through this we've been given some harsh criticims. Our outage times have been exaggerated (by an extra 30 hours in some cases), some even made bold claims stating they thought we went out of business. One person also said, and I quote, "I've been in IT for 6 years and disaster recovery isn't that hard." My guess is that they've never faced a real disaster that they were in charge of recovering from. I've noticed that when people get frustrated they overreact, embelish and above all love to lay blame.

Its also frustrating that when we tell people what we are doing and what we have been working on for months they think we're shining them on. Saying that its "typical PR speak" or that we're only now attempting to put together plans. Its very trying. I have to thank our phone support staff, too. They've been getting the brunt of it when people call and have remained polite and professional through some very difficult calls.

I feel terrible for all of the businesses and individuals who were without service or power, our customers or not. Walking through the grocery stores who each had to throw out tens of thousands of dollars of spoiled food makes me shake my head and wonder about how and why some people can be so self righteous and blind at times. Eight people died in this storm, most of them because they made bad decisions about how keep warm, and granted one or two of them may be up for Darwin Awards, it doesn't change the fact that they died.

As usual, there is a lot more that I wanted to say, but the words escape me. I guess thats why I'm in IT and not an author...