Log in

No account? Create an account
01 June 2008 @ 11:30 am
This evening at 4:55 in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
Ryanequivoke on June 1st, 2008 06:57 pm (UTC)
Three walls? Intense. Good luck with the damage control, Ben.
Briangnomatron on June 2nd, 2008 12:47 am (UTC)
This is actually just a cut'n'paste from the data centre company where the R&M server lives - it's not like Ben's house now has only one wall left or anything...
Ryanequivoke on June 2nd, 2008 12:52 am (UTC)
Well yeah, but it is his/their/company datas and equipments.

Although I did picture that he, Sven, and the crew were toiling away in a the charred remains of a room with rock music playing because that's what all the forensics shows do to make things seem more interesting.
Briangnomatron on June 2nd, 2008 12:55 am (UTC)
Thankfully none of the actual servers were affected by the fires and such, so it's really more of a twidding-thumbs sort of thing. Much less exciting than jump-cutting shaky cam with bad rock music and gloves and evidence bags... "my god! look at this! this server was infected with an explosive virus! *dun-dun-duuuuuuuun*"
^..^meowpurrr on June 2nd, 2008 02:17 am (UTC)
no, we never have and never will even see the physical server, let alone touch it :>
(which is a pain in the tits when trying to resolve certain problems, compared with hands-on fiddling)
^..^meowpurrr on June 2nd, 2008 02:20 am (UTC)
it's not like Ben's house now has only one wall left
well, lean too hard on anything, or jump up and down, and it probably would :>

i especially like the forum posts from people complaining they'd lost $10k in sales in the first few hours, and demanding compensation. if they have that kind of turnover they can afford a second server (or 10) in another datacenter ffs.
Sicknessamelange on June 2nd, 2008 07:41 am (UTC)
Dude. I get kicked off my computer for 2 days, and look what happens...
I wait as patiently as possible.