DCN911.COM Off-Network Support

7:56 PM Central – UPDATE – RESOLVED

All servers and sites are back up as of this time. External monitoring showed routing returned after 7:36 PM. The issue is resolved and service has been restored to normal. Thank you for your patience! ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

7:25 PM Central

At 6:58 PM Central, our servers at the GNAX data center became unreachable. This is a network outage affecting all of our servers hosted at the GNAX data center; it is not limited to one or two servers.

It does appear to be an issue at the data center, as traceroutes are reaching the border routers.

We have notified the data center and will update you as soon as we know more.

Impact: This affects only some dedicated server customers who are hosted at GNAX. This does not affect dedicated servers nor our shared servers which are hosted in Denver.

##

UPDATE: 7:35 AM: Great news! :) The DNS issue on svensbluff has been resolved. No accounts had to be moved. Rebooting the server appeared to resolve the problem, which would suggest there was a tmp file corruption that needed to be flushed out of the system via a power cycle.

If you notice any irregularities with your account, please let us know. We were not made aware of the issue until 3:30 AM Central time. We feel terribly that we weren’t aware of it sooner.

Unfortunately our monitoring system did not pick up on it because it monitors by IP address, and asks each service for a response at the IP. The services were all up, even DNS, and responding … but DNS wasn’t processing requests correctly. Monitoring just asks “are you responding?” — it doesn’t ask and analyze custom requests. That’s why our monitoring system didn’t pick up on this problem. :( I am very sorry for the trouble this caused you. We are going to look into our options for monitoring, in order to be able to better detect situations like this.

We always love to hear from you. We’d rather receive a false report than no report at all. :) So if you see anything weird, please let us know by opening a support ticket right away. We want your service to be top notch, A-#1 !!!

Thank you for all the feedback as we worked on this, it was very helpful to us and helped us to iron out what was/was not going on. ##

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Good morning,

Around 3:30 AM Central time, an as-yet unidentified DNS issue caused websites on the server ‘svensbluff’ to become inaccessible.

We have been trouble-shooting this issue since that time.

At this point we have been unable to find the exact cause.

We will restore DNS service to down websites as quickly as possible. We have a list of nameservers that we know are not working, and these are the nameservers that we will assure are answering authoritatively like they are supposed to be, as soon as possible.

Thank you for your patience! We will post updates here as soon as they are available.

DCSN Team

Uptime

• evensonfarm
• svensbluff