DCN911.COM Off-Network Support

UPDATE: Upgrade and Reboot complete 11/23/08 - Apache and PHP were upgraded Thursday night, 11/20/08 as rescheduled. Everything has compiled properly and is functioning as it should, however please do be aware that Apache 2 (which we have been running for a while) does handle some operations differently than Apache 1 did. Please double-check your sites and coding to ensure they are Apache 2 compliant.

We have also upgraded to the current Linux kernel, and rebooted the server to make the new kernel take effect. The server was down from 2:22-2:25 PM Central time today (11/23/08) while the server was rebooting. We waited until today to do this work, knowing that weekend internet traffic is usually very low, and Sunday afternoons are often spent watching football rather than playing on the computer. :) We hope that this was accurate in your case as well, and resulted in no inconvenience to you!! The server is now back up and functioning properly.

If you encounter any issues, please do let us know. Thank you very much!! Have a wonderful weekend. ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE: Rescheduled for 9:00 PM Central 11/20/08 - Due to unforeseen scheduling conflicts early this morning, we are rescheduling these upgrades to run and complete between 9-10 PM Central time. We will post here once the work has been completed. Thank you! ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We will be performing routine maintenance on the server ’svensbluff’ early Thursday morning, including a kernel upgrade and an Apache/PHP recompile. The purpose of these upgrades are to close potential security holes, as well as to implement several standards required for PCI DSS compliance.

A short period (10-15 minutes) of downtime is possible but not expected during the Apache rebuild. We will need to reboot the server to complete the kernel upgrade. Rebooting usually takes only a few minutes, so you may see 5-7 minutes of downtime as the server restarts all of its services.

Although our shared servers are not yet PCI-compliant, we are working to make them PCI compliant within the timelines required for Level 4 Merchants. We will post confirmation once PCI DSS compliance has been achieved on our shared servers. ##

UPDATE - 11/03/08, 3:42 AM Central: Project completed!

All software has been upgraded to current versions as outlined below. cPanel has been re-upgraded to version 11.24.0, and this time it did not kill Apache. (This was the ultimate goal behind the software updates - that this new major version of cPanel would play more nicely with current versions of server software rather than the old versions we were running.)

We’ve verified everything is running correctly, except for the couple of sites which had old scripts or modules which are not PHP 5 compliant. (Such are very few & far between.) We will be working with those clients directly to get those issues resolved.

All systems are “Go” and the upgrade project has been completed! :) If you have any questions or need assistance with anything, please contact us anytime. We are always happy to hear from you! ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(more…)

9:24 AM RESOLVED - We’ve rolled-back cPanel to 11.23.6 and Apache is magically running fine!!  :) We’re opening a ticket with cPanel to have them repair whatever is wrong with 11.24.0 that is breaking legacy Apache.

In the meantime we have disabled this cPanel upgrade so it will not run again before we have this bug sorted out.

We are considering this case “closed” for the sake of this posting, however we will post here further as we continue to work on it later today.

Again we are very, very sorry for the trouble this morning!! If we had known cPanel’s handiwork was going to kill the server, we would not have allowed it to update.

Thank you for your patience!!

DCSN Support Team

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

9:14 AM - We’re still working on this with no ETA as of yet. We will post further updates as we know more. ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Good morning,

This morning our server ’svensbluff’ auto-updated to cPanel 11.24.0, which appears to have killed Apache (the web server). Despite a thorough investigation, we’ve been unable to determine why exactly Apache will not run, other than it obviously appears related to the cPanel update.

We are recompiling Apache as we speak in an attempt to restore service.

We will post further updates here as they become available. Thank you for your patience!!