eagleharbor rebooting
Update / 4:30 AM, December 1st: Great news: nearly all clients’ sites have been restored and are back online. According to our records and cross-checking, about 98% of clients are back up and running.
- We have run into issues with a couple of sites which our technicians are working on.
- We still need to recompile Apache/PHP to include extended modules and libraries. When we do this, we will also install Zend, for those who need Zend.
- SSL certificates and nameserver zonefiles will be installed as soon as possible.
- If your site should be on a dedicated IP address and is not, please let us know by simply posting a comment on this blog post — our technicians will check into it right away for you.
Equally important, in my opinion, is that this mess that we went through yesterday (which I will be happy to talk more about, it’s no secret) thankfully has attracted the attention of data center management. Interestingly, it took one supervisor about a half-hour Thursday evening to fix the remaining problems present that the “Level 2 Tech” working on our server had left over the course of 10 hours… including locating and racking a missing drive which we had been told repeatedly was in the server!
My friends, if you or I pulled this level of incompetence and complete disregard for our customers like we were subjected to yesterday by the data center, we’d be burned to the ground within the day. Needless to say neither you nor I run our businesses this way, which is why I am so upset about it. I know how much better it could be.
So at least I have gotten management’s attention. We shall see if it makes any difference. I am not willing to have servers at a facility where they aren’t assured to be well cared-for. If it wasn’t for you, my clients, I frankly would pack my bags and get the heck outta dodge after what we were just put through. But I recognize that we all need stability, so I am willing to go face-to-face, offer constructive feedback and see if we can’t make a positive out of a negative for the good of our current stability as well as the good of their collective customer base. That’s a long sentence saying, if everybody wins, it’s worth it.
The other great news is that we have lined up a new provider for our future server acquisitions. At this time, we anticipate that any new servers will be deployed with them. As that relationship comes to fruition we will tell you all about it. :D The company is run by folks I have known a very, very long time, is a top-quality dedicated server provider, and has a very solid business model.
Hope this update from the gal at the “top” helps to put your mind at ease and answer your questions. I’ll also be e-mailing out an update/newsletter over the weekend with a LOT more infomation (some related, some not) so please keep an eye out for that as well.
Any questions or issues, please let us know! Please click the Comment link and we will be in touch ASAP.
Karin
Agile Hosting/DCN
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Update / 2:56 PM: I can’t believe we are still dealing with this. :(
The data center technicians have finally brought the server back online with a fresh drive, but incredibly, they appear to have done so without also loading the failed & backup drives in the server… so we have, essentially, a blank server sitting there with no data to restore to it! — despite no less than four explicit requests that the drives be mounted as slaves so we could recover data. Needless to say, I am just this side of *irate*. Data center management have already been notified.
We will begin site restores as soon as the data center loads our drives back in the chassis, as originally requested.
Thank you for bearing with us. This situation has us re-analyzing our choice of data centers. This is not acceptable to us.
Karin
Agile Hosting/Door County Networking
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Update / 8:50 AM: The primary drive has failed, requiring us to put a brand-new drive in the server. Once the new drive has been installed, we will copy everyone’s data to the new drive and bring sites up one-at-a-time. We do not have an ETA for this process yet; at this moment, Level 2 Techs at the data center are still working on mounting a fresh drive in the chassis. As soon as the server is live on the new drive we will start work on bringing sites up. Thank you for your patience!!!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Good morning,
The server ‘eagleharbor’ went down at about 4:15 A.M. Central. We attempted to reboot it immediately. Upon reboot the operating system went into a forced FSCK (”file system check”), which is like a “defrag” in Windows-land. This is a normal Linux activity which is designed to protect data on a drive.
The FSCK completed once, then upon restart it went back into a forced FSCK again. (Forced meaning, the operating system will not allow the drive to boot up without it — this usually means there is major filesystem fragmentation present. Again, this is a protective measure.)
The server will come back up once the operating system has finished the required FSCK. We’ll keep you up-to-date with its progress.
Thank you!!!
DCSN Team