Have a great day! ##

If you experience any issues or have any concerns, please contact us at your convenience at support@dcn911.com. Thank you!

Have a great day! :)

## DC Service Network Team

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Good morning!

We are upgrading our billing & support software at https://www.dcn911.com/support/ this morning.

If you have any support issues, please send an email to us at support@dcn911.com — we will still receive your request and respond to it ASAP. :)

The software upgrade should be done in about 1 hour. We’ll update you when it is complete. :) Thank you!!

Server: evensonfarm

• Recompiled kernel to latest version (reboot was necessary to boot onto the new kernel)
• Recompiled Apache to 2.2.12 and PHP to 5.2.10
• PHP configured with suPHP and Suhosin

Server: svensbluff

• Recompiled kernel to latest version (reboot was necessary to boot onto the new kernel)
• We will be recompiling Apache & PHP to the latest versions momentarily.

Please double-check all of your scripts and pages to make sure they are all working. PHP pages in particular (which are what is run by scripts like WordPress, Joomla, OSCommerce nad Magento) may be affected by either the suPHP or Suhosin. If you find any errors or other issues, please contact us immediately at support@dcn911.com and we will be happy to help you with it. We want your sites to be running smoothly!

We will make one last update here once the Apache & PHP recompile is completed on svensbluff. Once that is finished, all maintenance will have been completed for the evening.

Thank you very much!! ##

The server has been rebooted to the new kernel. Approximate downtime for the reboot was from 2:09-2:12 AM. We have confirmed all services are back up and running smoothly.##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

12:43 AM: We are recompiling the kernel, and upgrading Apache & PHP to the latest versions on evensonfarm.

A reboot will be necessary to boot the server onto the new kernel. We will be performing the reboot after all upgrades (including Apache & PHP) are complete.

The reboot should take a maximum of 5 minutes, unless an FSCK is required. In the event an FSCK is required, the reboot may take 20-30 minutes. This is necessary and normal maintenance, but not a cause for alarm.

If you have any questions, please do contact us. :)

DCSN Team

####

We have also upgraded to the current Linux kernel, and rebooted the server to make the new kernel take effect. The server was down from 2:22-2:25 PM Central time today (11/23/08) while the server was rebooting. We waited until today to do this work, knowing that weekend internet traffic is usually very low, and Sunday afternoons are often spent watching football rather than playing on the computer. :) We hope that this was accurate in your case as well, and resulted in no inconvenience to you!! The server is now back up and functioning properly.

If you encounter any issues, please do let us know. Thank you very much!! Have a wonderful weekend. ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE: Rescheduled for 9:00 PM Central 11/20/08 – Due to unforeseen scheduling conflicts early this morning, we are rescheduling these upgrades to run and complete between 9-10 PM Central time. We will post here once the work has been completed. Thank you! ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We will be performing routine maintenance on the server ‘svensbluff’ early Thursday morning, including a kernel upgrade and an Apache/PHP recompile. The purpose of these upgrades are to close potential security holes, as well as to implement several standards required for PCI DSS compliance.

A short period (10-15 minutes) of downtime is possible but not expected during the Apache rebuild. We will need to reboot the server to complete the kernel upgrade. Rebooting usually takes only a few minutes, so you may see 5-7 minutes of downtime as the server restarts all of its services.

Although our shared servers are not yet PCI-compliant, we are working to make them PCI compliant within the timelines required for Level 4 Merchants. We will post confirmation once PCI DSS compliance has been achieved on our shared servers. ##

All software has been upgraded to current versions as outlined below. cPanel has been re-upgraded to version 11.24.0, and this time it did not kill Apache. (This was the ultimate goal behind the software updates – that this new major version of cPanel would play more nicely with current versions of server software rather than the old versions we were running.)

We’ve verified everything is running correctly, except for the couple of sites which had old scripts or modules which are not PHP 5 compliant. (Such are very few & far between.) We will be working with those clients directly to get those issues resolved.

All systems are “Go” and the upgrade project has been completed! :) If you have any questions or need assistance with anything, please contact us anytime. We are always happy to hear from you! ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE – 11/03/08, 01:53 AM Central: We’re still at it! :)

A big “thank you” to cPanel, who graciously answered our MySQL ticket literally within 10 minutes, restored MySQL to version 4.1, escalated to Tier 2, and had the new MySQL 5 installed within the hour. They were friendly, helpful, and fast!

We then upgraded to Apache 2.2.10/PHP 5.2.6 with no further issues. All in all, the new Apache, MySQL and PHP are working beautifully.

Among other things, we were sure to compile in:

• cURL and cURL-SSL
• IMAP
• DAV
• GD
• json
• freetype
• gettext
• mhash
• mcrypt
• mbstring
• SOAP
• Zend
• and much more :)

We’ve gone through every site hosted on the server and verified they are all working. We surfed through various parts of sites trying to find errors. We found very few, and will be in touch with site owners regarding what we found.

We have one final thing to take care of re. Ioncube loaders, and then we will consider the server upgrade project complete.

We’ll be writing and sending out an email newsletter Monday to explain what was all done and what improvements this brings you.

As before, if you find any problems with your websites, please contact us immediately. We will do what we can to help you.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE – 11/02/08 11:23 PM Central: All data is safe and secure. We have full dumps of all the databases taken pre-upgrade, and we moved several heavy database-driven sites to a PHP 5 server to complete the upgrade on those on a per-site basis (and have those sites able to come right back online if there were any errors). Those transfers all went smoothly as well. Please rest assured there is NO data loss and no data has been affected. The issues lie strictly with operating system packages – how the server runs. This has nothing to do with data.

The upgrade from MySQL 4 to MySQL 5 would not complete. We’ve opened a ticket with cPanel and they are in the process of fixing and completing the upgrade to MySQL 5.

We will verify that MySQL 5 is running properly before moving forward; the next step is to recompile Apache to Apache 2 and PHP to PHP 5. ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We are in the process of upgrading shared server ‘svensbluff’ to use Apache version 2, MySQL 5 and PHP 5.

This is a big upgrade, but will bring major service enhancements to clients.

Additionally, PHP 4 has gone “EOL” (end of life) which means there are no further upgrades available for it, and we would be in a very awkward, reactive situation if (when) a new exploit is discovered in it. Rather than remain sitting ducks waiting for something to go wrong on old software, we are upgrading to current versions and packages. We can keep your sites and data more secure and better managed on the current versions and packages (as opposed to the old versions running now).

‘svensbluff’ is our last shared server to be upgraded. Our other servers are already running Apache 2, MySQL 5 and PHP 5 without any problems. Most web hosts have long-since upgraded their servers to use Apache 2, MySQL 5 and PHP 5. This is the standard software configuration in hosting now.

If you discover any problems with your website, please contact us immediately. We are here, on duty, waiting to hear from you and we want to help you!

We’ll post further updates as the upgrade proceeds. ##

In the meantime we have disabled this cPanel upgrade so it will not run again before we have this bug sorted out.

We are considering this case “closed” for the sake of this posting, however we will post here further as we continue to work on it later today.

Again we are very, very sorry for the trouble this morning!! If we had known cPanel’s handiwork was going to kill the server, we would not have allowed it to update.

Thank you for your patience!!

DCSN Support Team

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

9:14 AM – We’re still working on this with no ETA as of yet. We will post further updates as we know more. ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Good morning,

This morning our server ‘svensbluff’ auto-updated to cPanel 11.24.0, which appears to have killed Apache (the web server). Despite a thorough investigation, we’ve been unable to determine why exactly Apache will not run, other than it obviously appears related to the cPanel update.

We are recompiling Apache as we speak in an attempt to restore service.

We will post further updates here as they become available. Thank you for your patience!!

All servers and sites are back up as of this time. External monitoring showed routing returned after 7:36 PM. The issue is resolved and service has been restored to normal. Thank you for your patience! ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

7:25 PM Central

At 6:58 PM Central, our servers at the GNAX data center became unreachable. This is a network outage affecting all of our servers hosted at the GNAX data center; it is not limited to one or two servers.

It does appear to be an issue at the data center, as traceroutes are reaching the border routers.

We have notified the data center and will update you as soon as we know more.

Impact: This affects only some dedicated server customers who are hosted at GNAX. This does not affect dedicated servers nor our shared servers which are hosted in Denver.

##

If you notice any irregularities with your account, please let us know. We were not made aware of the issue until 3:30 AM Central time. We feel terribly that we weren’t aware of it sooner.

Unfortunately our monitoring system did not pick up on it because it monitors by IP address, and asks each service for a response at the IP. The services were all up, even DNS, and responding … but DNS wasn’t processing requests correctly. Monitoring just asks “are you responding?” — it doesn’t ask and analyze custom requests. That’s why our monitoring system didn’t pick up on this problem. :( I am very sorry for the trouble this caused you. We are going to look into our options for monitoring, in order to be able to better detect situations like this.

We always love to hear from you. We’d rather receive a false report than no report at all. :) So if you see anything weird, please let us know by opening a support ticket right away. We want your service to be top notch, A-#1 !!!

Thank you for all the feedback as we worked on this, it was very helpful to us and helped us to iron out what was/was not going on. ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Good morning,

Around 3:30 AM Central time, an as-yet unidentified DNS issue caused websites on the server ‘svensbluff’ to become inaccessible.

We have been trouble-shooting this issue since that time.

At this point we have been unable to find the exact cause.

We will restore DNS service to down websites as quickly as possible. We have a list of nameservers that we know are not working, and these are the nameservers that we will assure are answering authoritatively like they are supposed to be, as soon as possible.

Thank you for your patience! We will post updates here as soon as they are available.

DCSN Team