DCN911.COM Off-Network Support

Our shared servers ’svensbluff’ and ‘pilotisland’ were both rebooted around 2:00 AM Central time Monday, 12/31/07 to complete kernel upgrades. This was routine security work and caused no change in functionality on either server. Both servers were offline for about 2 minutes during the course of the reboots, and have been online and running at 100% since then.

If you have any questions please open a support ticket so we may assist you. Thank you very much!

DCSN Support Team ##

Hello,

We will be rebooting the shared server svensbluff tonight at approximately 10:30 PM to upgrade to a new kernel. The expected downtime is under 10 minutes. If you have any questions, please let us know.

Thank you!

UPDATE, RESOLVED - 11:28 AM: Apache has successfully recompiled, and we have verified that the SSL certificates are working again without error. We recompiled the same libraries into Apache as before. However, if your site is experiencing any problems due to missing libraries or modules, please contact us right away and we will be happy to assist you with it. Thank you and have a great weekend! ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We’ve learned that last night’s cPanel update has broken the SSL certificates on the svensbluff shared server.

Our technicians are currently working on repairing the problem. Included in the repair efforts are recompiling Apache (the web server), so if your site goes offline for a few minutes, that’s why. :) Please do not worry, we are aware of the issue and are working on repairing it as quickly as possible.

Please accept our apologies for the trouble! We are also working with cPanel to identify what caused this in the first place, and to try to get them to not break things this way again. :)

DCSN Team

This weekend, cPanel auto-updated all servers on the “Release” build to cPanel 11. The previous version of cPanel was 10.x, so that makes cPanel 11 a BIG update.

cPanel has been preparing your server for this update for some time, bringing many packages up-to-date over the past weeks and months, in the hopes of making the transition as smooth as possible. It is imperative for security reasons — not just cPanel 11 — that your server be running the latest packages of OS software. (Please understand we do not allow unpatched/out-of-date servers to run on our network. Running out-of-date software is prohibited by our TOS as it is a security risk to everyone, not just you and your server.)

The most common problem we have seen is servers running an old version of Perl. Perl needs to be version 5.8.8 for cPanel 11 and related perl modules to work properly. Once your server is upgraded to v. 5.8.8, then the dependent perl modules will be able to update. Here’s a tutorial on how to upgrade your Perl to version 5.8.8. :)

The second-most common problem we have been seeing are difficulties with the mbox to maildir conversion of the mail system. cPanel has been running maildir as the default mail system for over 18 months now, but many folks have been dragging their feet (out of fear — we do understand!!!) about converting. Recently, the mail delivery on many servers stopped working completely because clients were still using the out-of-date mbox system. mbox is deprecated, not supported by cPanel, and is not able to handle the extraordinary volume of mail that most ordinary e-mail accounts have these days. mbox also has functionality limitations that you will be able to break free from once your server has been converted to maildir. We will be posting a FAQ/How-to about mbox-to-maildir as well as dealing with some of the common issues after the conversion.

cPanel 11 is here to stay. It is NOT something to be afraid of. :) cPanel 11 brings a lot of fantastic feature enhancements and additional functionality that your clients are going to LOVE:

• Getting Started Wizard walks them through the initial set-up of their hosting account!
• New and better File Manager
• New and better HTML editor
• MySQL Database set-up wizard
• Install Perl Modules in client’s home directory
• Install PHP PEAR modules in client’s home directory
• Custom PHP Configuration on per-site basis
• User-level (per e-mail account) e-mail filtering
• WebDAV WebDisk functionality
• Dozens of new Video Tutorials embedded right inside the appropriate cPanel pages
• Much better organization of cPanel’s account features
• Clients can drag-and-drop cPanel sections to reorganize their cPanel so it best makes sense for them

When you login to WHM, you will see an improved GUI (much easier on the eyes) and WHM has also been somewhat reorganized to make things easier to find. Everything is still basically in the same place; you’ll just find more sections with shorter lists of operations in each.

If you are experiencing anything odd with your server, please open a support ticket with us right away. Our technicians will be happy to trouble-shoot and repair your cPanel 11-related issue promptly.

DCSN Team

We are pleased to be wrapping up our 3rd straight month of 100% network uptime and over 99.999% server uptime.

Our shared servers were rebooted once during that time as part of a normal kernel upgrade, which is an ordinary and necessary part of maintaining a web hosting server. Upgrading to the latest kernel helps assure your data remains as secure as possible.

Reboots about 5 minutes to complete, and are always done during the overnight hours, to minimize the number of visitors who might be impacted.

DCSN Support

UPDATE, 10:58 PM Central: Server svensbluff is back online. :)  The issue appears to have been a blown circuit breaker in the Cisco switch. The breaker has been replaced, svensbluff has been successfully rebooted, and all service is completely back to normal. ##
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE, 10:27 PM Central: The data center has 2 spare Cisco 3550’s on site in the event the bad switch needs complete replacement. As soon as diagnostics have been completed, the switch will be repaired or replaced. The data center is working with the goal of getting clients (you) back online as quickly as possible. As a new switch will still need to be configured once it is plugged in, they will do whatever is possible to route us temporarily through an existing live switch, complete the configuration work carefully (which means less chances for mistakes) and then re-route us through the new switch.

This would mean a couple of short blips later (in the overnight hours), but the net result is that everyone’s sites would come back online online much sooner.

We’ll post further updates as available.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The bad luck gremlins have struck again, fortunately this time it is on a Friday night while internet traffic is at one of its lowest levels throughout the week. The data center in Denver advises they are experiencing a switch failure within the data center, which is a very localized issue affecting only the servers behind that switch — including our shared server svensbluff.

The server itself is up and working fine. It is just that the internet cannot access it, and it cannot access the internet. It is similar to you or I standing in a room, and then shutting off the lights. We are alive, breathing, and perfectly fine… it’s just that it’s dark. :)

The data center is aware of the issue and has been working on it since it arose. We will post ETAs for repair as soon as they are available. At this time, the data center does not yet have an ETA, which is why we don’t have one to post yet.

If you have any questions please open a ticket at our helpdesk at https://www.dcn911.com/support-center/

We are sorry for any inconvenience this outage may be causing you or your visitors. It is our hope that since it’s a Friday night, impact to clients is low. Thank you very much!!

DCSN Support Team

Good afternoon! We are in the process of upgrading kernels on servers. Every time a kernel is updated, the server must be rebooted to make that kernel update “take.” A reboot takes only a few minutes, and service interruption is very short.
The server ’svensbluff’ is scheduled to be rebooted at 3 PM Central time. The server ‘cana’ is scheduled to be rebooted at 4 PM.

Dedicated server clients should schedule a reboot time that is best for them by opening a support ticket.

Thank you!! Have a great afternoon.
DCSN Support Team

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE, 3:04 PM: svensbluff has rebooted successfully, all work completed. ##

12:05 AM: We’ve received two reports of selective ISPs or backbones not being able to reach our shared server svensbluff. For the affected viewers, websites on the svensbluff server appear inaccessible by domain name. Although it appears that the sites are “down,” the server is indeed up and the domain names are working for the overwhelming majority of the internet.

Our upstream provider is turning up their new Time Warner connection, and turning off their line at the old data center (Data393). The BGP routers must re-calculate the routing for all connecting backbones. Most ISPs are completely unaffected by this BGP4 reconvergence, but there are a handful — very, very few — who are seeing blips/temporary “outages” for their websites. The reason this is occurring is because the viewer’s ISP does not know how to route to the data center on the new route yet. The BGP4 system is calculating the network and routing changes, and making those new routes “live” at the ISP level, as we speak. More information about BGP and how it works can be seen here.

We expect these interruptions to be resolved overnight.

There may be similar issues on Monday when the data center turns up the Level(3) circuit, and turns off the SAVVIS circuit. The majority of clients will be completely unaffected and see no outages or sluggishness whatsoever.

You can always reach your website via IP address, at http://72.18.145.36/~username where username is your hosting account’s username. The IP address continues to be routeable and live, because the server is still up.

If you have any questions or concerns about this please let us know right away via the Helpdesk. Thank you!!

## RESOLVED ## - 6:48 AM Central 02/19/07: The server is back online and running normally. The move is complete. We are up and running in the new data center and expect no further blips, outages, or burps. Thank you!!! Have a wonderful holiday~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE - 5:10 AM Central 02/19/07: The server has arrived at the new data center. There are no problems or issues with the move thus far. Servers are being re-racked and re-connected to the internet, and powered-up as soon as possible. As soon as the server is racked, re-connected and powered up, all websites and e-mail will be back online and functioning normally.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE - 10:56 PM Central: The server has been taken offline to be moved to the new data center. Everything is proceeding on schedule. We will update you as we hear more.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Handy Networks, the upstream provider for shared server svensbluff, is moving to a new data center in Downtown Denver. Our server svensbluff is scheduled to be moved between 10 PM Central Time Sunday, February 18 and 6 AM Monday, February 19, 2007.

The server may be offline for much or all of this time. The server’s being offline during this period is NOT cause for worry or alarm; it means everything is going according to plan.

Previous servers moved by Handy over the weekend have all been back online between 4 and 5 AM, so we are hopeful that our downtime will be shorter than the 8-hour window posted.

If you have any questions during this time, our helpdesk will be online throughout the move, and you are welcome to contact us at your convenience.

Thank you!!!

DCSN Team

02/11/07 10:16 PM Central: MAINTENANCE POSTPONED – due to cooling system issue at the new data center, the migration of the ’svensbluff’ server to the new data center has been postponed to Friday or Saturday, February 16-17, 2007. We have not been given an exact downtime window yet, but we will post it here, and e-mail all customers, once the new window has been confirmed by our upstream.

Click below for more details about the move and the new facility.

————————————————————— (more…)