“Good morning, just calling to alert to you the situtation we have at the IC Data Center, all client sites are down as of 3:30AM this morning. Clients will be unable to access the site, ETA is unknown. We will contact you with hourly updates.”
That’s the VM a lot of people woke up to yesterday morning. It was a hellish day. Sites were taken down at midnight for maint at the SBC data center in Irvine. Maint was complete at 3:00 AM, and we started the process of bringing all the sites back up. By 3:30 it was evident that sites weren’t going to be online in time for the East coast clients to begin work at 7AM Eastern time. So I had to start calling both east and west coast people to alert them to the outage.
The ones that answered thier phones weren’t too happy that I woke them at 3:30 am. Thankfully most of them got VMs instead of live calls. 🙂
Had to wake the other SAs at 4:00am because the one SA couldn’t handle all the problems on his own, also my boss had to be contacted at that time and alerted to the situation. All the SAs were all in the office by 4:30am, and the CEO of my company had to be alerted at that time to the outage which was now going to effect all US clients. He wasn’t too happy about that either.
Started Site down procedures at 4:30AM with e-mails and txt pages to all AM’s (about 20 people) and the appropriate people at Pfizer (about 10-15 other people).
Hourly updates with no ETA were given tell 8am when I left, and were continues after that tell noon when we finally got an ETA of 1.5 hours. All sites were back online by 3:30 PM pacific time, which means east coast didn’t get ANY work done yesterday, and central sites were down tell 5:30 pm, which means they probably didn’t get anything done either.
It was absolute hell, and I was on the phone from 3:30 tell 6:30 straight. Thankfully my boss took over the phones at 6:30, and after that I just got the occasional call when he was busy.
Hopefully this will put some more pressure on for an east coast datacenter, which would be good cause they’d need more SAs for that. 🙂 And possibly some east coast SAs, and since that’s where I want to move, that’d be good. 🙂 Cause hopefully I’d be in on that project.
Cisco is going good. I’m nearly complete with the first book for the CCNA exam. I think I’ll try and get the INTRO exam taken here soon.
Today, other then the site down issue at work. Was good. I slept, got my package from UPS and USPS, which my mother sent lots of candy in. Went back to bed about 6ish and slept tell 11:15 when I got up, showered and then came into work.
Anyways, that’s my life.
Price is Right tomorrow! Woot!
Oh, and fucking COX sent me a bill, even though my fucking cable hasn’t been hooked up yet. Fuckers. I’m going to call and complain sometime soon.
Lateers all.