Friday, June 6, 2008

Continued problems with our hosting provider; email and web move to our San Francisco datacenter under way

Regrettably, we're still running services from's web hosting facilities until we can migrate everything to 365 Main in San Francisco.

So this morning, I wake up to find that our mail server is unreachable again, and this series of messages on The Planet's service update site:

  • June 6 – 10:00am CDT - We have lost network connectivity to H1. We are confirming the extent of any power loss, and we will be updating shortly.
  • June 6 – 10:05am CDT - Transport for H1 temporarily fell offline and is restored. H1 Phase 2 did not lose power. H1 Phase 1 lost power. We will be updating again shortly.
  • June 6 – 10:10am CDT - The temporary generator powering Phase 1 failed. We switched over to the backup generators that were just brought in. The CRAC units have been powered on, and PDUs are having power restored right now. [THis is the second temporary generator that has failed in the last week at The Planet. Perhaps it is operator error? - Rusty]
  • June 6 – 10:15am CDT - We continue to power PDUs in Phase 1. We will update when all PDUs have been restored.
  • June 6 – 10:20am CDT - Power has been restored completely to Phase 1. Our DC Ops team will be walking through the aisles to confirm all racks are online.
  • Customer Support Overview (June 6, 11:30am CDT): -Technical Support Phone: No Hold Time

From our monitoring, the service went down at 7:40 AM pacific, or 9:40 AM CDT. They were a little slow to notice they lost communications with their data center!

Of course our mail server is still unreachable at 11:44 Pacific, or 1:44 PM CDT, 4 hours after they stated that power has returned.

What really annoys me is that they are stating on their site, "Technical Support Phone: No Hold Time". The reason for this is that they're sending all support calls to their sales people, who don't do much more than tell you they're going to escalate you to Level 2 support, but all those techs are busy and they'll need to call you back. I suspect they did that to reduce their 800 number call expenses, because they had hundreds of customers sitting on hold for 30-40 minutes all the time. They also get to make it look like their response time is much better than it really is.

After waiting 45 minutes for a callback that never came, I called in again. Finally I got them to connect me with a real tech support service, not just the person logging callbacks. I've been on hold with "real" technical support at The Planet for 10 minutes now, trying to get our mail server powered back on.

10 more minutes on hold, and the tech tells me, "Can you go online and submit a reboot request ticket, that will expedite things."

At this point I have no faith of when our mail will be back again.

I'll continue to move our services out of The Planet and to our own servers in San Francisco; our DNS is already moved (although we still don't have the redundant location DNS in place yet); the hardware for the new mail server is setup but the mail services aren't configured yet. There are also a few issues with some of the web services we run; the old systems at the Planet used a much older version of the Berkeley DB software package which isn't compatible with the current versions. So I have make a few changes to our "now playing" code as well as our stream server monitoring systems. The "now playing" database is the most important to our listeners, because that's got all the information on which album songs come from, as well as the info on where to buy the track or get more info on the artist.

The mail server is a bit harder to migrate, but I'm also working on that right now as well.

Hopefully, the good thing that will eventually come out of this is that we'll have redundant servers, in different geographic locations,

Labels: ,


Anonymous Anonymous said...

The planet told everyone to go to the forums, but when you ask questions you get no answers and if you ask what some feel are the wrong questions one of the planets managers calling customers trolls. When I posted telling this manager it was sad to see a company manager calling customers trolls as anyone that has spent any time on the Internet knows calling someone a troll is getting close to using the "N" word with someone.

After I posted that it was sad, he banned me from posting anymore comments. I have yet to receive any contact from his bosses (I sent copies of the comment to all of them) so I can only assume that they are aware or feel the same way about the customer.

Folks systems fail, it is just how it is. But to then have the company call it's customers trolls for asking about info related to the service that company is being paid to provide is just too much for me. I have moved my server and I know many who once they get the rest of their data from their servers will be gone as well. I feel the planet should know that for some of these folks the last straw was your manager calling the customers trolls.

In years to come when things are calmed down what will be remembered about this whole incident is how at a time of crisis the customer where called TROLLS by management

June 7, 2008 8:28 AM  
Blogger Shawn said...

Consider moving your mail services to something like Gmail. They will host mail services for a small company for cheap, and give you plenty of space.

June 26, 2008 7:42 AM  

Post a Comment

Links to this post:

Create a Link

<< Home