DCN911.COM Off-Network Support

1:25 PM: UPDATE
————————————————————————
The server has been rebooted, and we are recompiling (rebuilding) Apache and PHP. It will take several builds and re-builds to fix it, so please be patient as we fix it. :D

We have left the mail server online so that clients can still send & receive e-mail without interruption.

Thank you very much!

DCSN Team
————————————————————————

Must be a Monday! :(

We upgraded Apache and PHP this morning to close a security hole with PHP 4.4.1 and phpList. Unfortunately the upgrade resulted in a misconfiguration which has caused PHP to run improperly, and the server is running a very high load.

We have technicians rebooting the server currently, and we will repair the PHP configuration as soon as it is back up. Thank you for your patience. :)

DCSN Team

Hello,

Our shared server “eagleharbor” is being attacked by spammers. The high volume of e-mail going through this server is driving the load up. We are working hard to identify and remove the relays/sending scripts (there are multiple) and shut out the spammers. We are also trying very hard to do this without interrupting clients’ regular, valid e-mail running through the server.

Thank you for your patience as we work on this for you.

DCSN Support

—————————————————————————————–

Update, 5:26 PM Central Time

GOT ‘EM! The spammer has been caught and blocked from the server. The scripts which were being used to send spam, have been removed. And, the spammer has been reported to his data center (LeaseWeb — spammer hosts).

The spammer was able to access a handful of hosting accounts by guessing the passwords and logging into them via FTP. We have verified, through server logs, that this person did not access any cPanels nor e-mail accounts… all he wanted was web space to upload & run his spam script from.

We have changed the passwords on the handful of accounts which were compromised by the spammer. Clients can find their new FTP/cPanel passwords by logging in to our secure Billing System at https://www.dcnsafe.com/modernbill/. We are also e-mailing the new passwords, and a full explanation of this event, to our affected customers. If you cannot access your website via FTP, please watch for your new password at the e-mail address which is your registered contact address with us.

If you have any questions, please do contact us directly. We are always happy to hear from you and help you!

Thank you very much!

DCSN Support

Our network provider, SAVVIS, is scheduling the following maintenance on their network. Please note dates and locations to identify whether you may be affected.

Maintenance Scheduled for CH3 IDC:
Maintenance Time: 0001- 0500 CST Sat Mar 4
Maintenance Action: Backbone router hardware maintenance
IDC Affected: Chicago-3

In this window, Savvis engineers will be replacing defective hardware on
a local backbone router. Customers attached to this device may
experience service outages, latency, or other connectivity issues as the
work is performed.

This action has been scheduled to occur between 0001 and 0500 CST on Sat
Mar 4.

6:48 AM Central: The server is back online and functioning normally. We are investigating the reason for the downtime and will report our findings as soon as we have identified the issue.  Thank you very much! :)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The server “eagleharbor” has gone down and is being rebooted as we speak. Please stay tuned for further details as we find out more.

Thank you!

DCSN Team

UPDATE 02/11/06, 01: 56 AM Central: The server is coming back online now. Loads are still a bit high as the file system finishes rebuilding, and the various programs which run on the server re-create their caches and run their standard start-up procedures. We anticipate the load will drop and optimal service will resume as things finish running and settle down.

Thank you very much!

DCSN Team
- serving Door County Networking & Agile Hosting

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

UPDATE 02/11/06, 01: 41 AM Central: The server is indeed running an FSCK. The data center reports the FSCK is proceeding slowly. (Although irritating, this is normal)  They are watching the server to make sure the FSCK completes properly. Again, the server will be up just as soon as the filesystem check finishes.

Thank you very much for your patience! If you have any questions please contact us directly at supportteam@dcn911.com

DCSN Team
- serving Door County Networking & Agile Hosting

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We are experiencing a routine filesystem burp with the “eagleharbor” server and the server is being rebooted.

An FSCK (File System ChecK), which is similar to a Windows disk defrag, will likely be necessary. This usually takes 10-15 minutes, but might take longer. Please be patient; the server will be back on online just as soon as the Linux disk system has completed its maintenance. This is the system that actually protects our data, by demanding a rebuild when it’s necessary.

Thank you very much for your patience!

DCSN Team
- serving Door County Networking & Agile Hosting

We are in the process of migrating accounts from the refuge.dcnsafe.com server, to the eagleharbor.dcnsafe.com server. We are about 1/3 of the way through and so far everything is proceeding as planned.

All parts, files, subdomains, add-on domains, databases, cron jobs, SSL certificates, and e-mail is being copied over intact, so you will not lose any data or functionality with your site.

Direct clients do not need to make any changes to their nameservers. We are carefully controlling the DNS on websites and moving your viewers over to the new server as soon as your website has been copied. We will take care of the nameservers on our end.

Resellers, however, will need to make changes to their nameserver IPs. We will notify each reseller directly of the changes which need to be made to complete the transition.

The SSL certificate for refuge clients will remain https://www.dcnsafe.com/~username/

If you have any questions or observe any issues with your website, please open a support ticket at the support desk, at http://www.dcnsafe.com/tt/ and our technicians will look into your issue right away.

Thank you very much!

DCSN Tech Support Team

9:30 AM: The problem which was preventing the drive from booting (the server was going into constant repeated FSCKs) was identified and fixed. The server has been rebooted on the original drive and all data appears intact!

E-mail sent to you over the past six hours, if queued in the sending mail server’s queue will continue to attempt to deliver to you. We are seeing a tremendous amount of e-mail coming through the server even as we post this now. If you were expecting  a message from someone, please contact them and ask them to re-send it. It will deliver now.

We will be working with our technical staff to identify the best, smoothest way possible to move accounts over to our new system (which has brand-new drives that are exhibiting no errors at this time) over the next 24-48 hours. We will notify each client individually to share their scheduled migration time as well as their new IP address.

Please continue to use your hosting account like normal today. Now that the server is up, everything is working.

If you have any questions please feel free to open a support ticket at our helpdesk, or simply send us an e-mail at supportteam@dcn911.com.
———————————————————————————-

8:25 AM: The techs are having significant issues bringing the drive back online so it will boot. They are continuing to work on it. We are also looking into getting the backup drive online so we can get client accounts online promptly via backups if that will be faster and have better-integrity data. We will update here as we know more. Thank you very, very much for your patience. /DCSN Tech Support
———————————————————————————-

6:48 AM: Issues exist with the drive that require its replacement, hence the lengthy analysis that we have experienced. We have conferred with our technicians and are working to bring the server up as quickly as possible, considering that it is the beginning of the business day. We will be moving clients off this server within the next 24-48 hours.

Clients will see no downtime from the server move. We will be moving accounts smoothly with no interruption in service expected from the actual migration. The only instance where service interruption may occur, is if massive drive failure occurs before all accounts are moved over. In that case we will simply restore the remaining accounts from our backup files, which are on a good drive and are unaffected by this issue.

Thank you for your patience as we are working to get the server back online as quickly as possible. /DCSN Tech Support
———————————————————————————-

5:28 AM:The techs report that the drive analysis continues to run. It is taking longer than expected, but thus far there is no evidence of lost data. We expect the analysis to finish at any time, at which time the server will be brought online immediately. Thank you very much for your continued patience! /DCSN Tech Support
———————————————————————————-

4:39 AM: The drive analysis continues on schedule and without any reported issues. We are still anticipating that the server will return to service promptly. If there is any change to this timeline we will let you know here, immediately. /DCSN Tech Support
———————————————————————————-

As of 2:45 AM: Our shared hosting server refuge.dcnsafe.com has been pulled offline for emergency maintenance.

Moments ago, the hard drive sent a message to our technicians indicating it may be failing. Rather than taking a chance and gambling with possible massive data loss and many hours of unexpected downtime, senior level techs have pulled the system down to run a drive integrity analysis. This will tell us whether the drive is indeed failing, or if this was simply a “fluke.”

We estimate the process may take as long as 2 hours, depending on the condition of the drive and the data. The server went offline at approximately 2:35 AM Central time and we anticipate that it will be back in service no later than 5 AM Central time.

We apologize for the inconvenience that this work may have caused. It is our hope that taking a shorter time now to assess the condition of the drive now, will save us all from a larger mess (and possibly lost data) in the future.

If you have any questions, please contact our support team here at supportteam@dcn911.com.

Door County Networking
Agile Hosting

We are rebooting resolute.dcnsafe.com (the former refuge.dcnsafe.com) to complete some security work that we have done. Please bear with us, sites will be back online in just a couple of minutes. :)

Support Team
DCN & Agile Hosting

After careful consideration we have determined it is time for a hardware upgrade on our shared server refuge.dcnsafe.com.

If you are not sure whether this upgrade will affect you, please login to your account’s control panel at http://www.yourdomain.com/cpanel/ (using your FTP username & password) and scroll to the bottom of the main page. On the right, you will see the header “General server Information”. The first item is Server Name.

If your cPanel reads refuge as in the example provided above, then you are on the refuge.dcnsafe.com server and will have your server upgraded.

Why are we upgrading? Our reasons for the upgrade are numerous and include the following:

• We have exhausted the available physical upgrade options for the current server. We are changing our systems over to AMD models which can be upgraded to 4 GB of RAM. Currently, clients are running on Pentium 4s with a maximum of 2 GB of RAM. We have already maxed out the P4 systems and would like to offer our clients faster, larger CPUs, faster L2 cache, as well as additional RAM.

• We experienced a network-wide power loss at The Planet’s data center last month which lasted “only” 81 minutes for us, but was several hours in length for others at the same data center.  Although our server remained powered-up, the network at the data center went offline during this time. The two redundant UPS systems in place (battery backup and diesel-powered generator) failed to fire up and keep the connections live. Unfortunately we have no way of verifying that this has been remedied, as obviously routine testing, if done, did not pick up on such a state of failure being present. We simply cannot risk another such outage for our clients.

• Hands-on support at the data center — the people who are our “eyes and hands” at the facility — has declined in quality and speed over the past year. We have given what we feel is ample time to work through “growing pains” and bringing new staff online to accomodate a larger customer base, but unfortunately the response times to our work requests (including simple reboots) has gotten longer and longer. The last reboot took nearly an hour to be tended to. This is another example of service not up to our high standards, and our experience has shown that there is no indication these trends will reverse.

• Spam blocks on the server’s IP space — which do not result from any spam sent from our server — remain in place after 6 months’ time. Simply put, this is extraordinary. A data center should be able to get a spam block cleaned out within 2 weeks. The history provided at OpenRBL shows that the Network and Abuse Teams do not respond promptly nor reliably to complaints and hence spam blocks are not removed. Although spam blacklists are an irresponsible to filter mail for spam (because as in our case, nobody on our server has sent spam), the fact is some ISPs erroneously insist on subscribing to them. This results in mail that might be refused and bounced back to you. Although it is the ISPs choice to use the blacklist, we simply don’t want our clients to have their mail bounced, nor do we want our reputation as a quality provider tarnished by others’ misbehavior.

The New Data Center
We are pleased to announce we are setting our shared clients up (as each server is upgraded) at the SAVVIS data center in Dallas, Texas. We will provide detailed network information shortly, but in the meantime, can offer you the following details about the change in service:

• Most of our clients find that SAVVIS is faster, with less latency and a shorter trip (fewer “hops”), than servers at The Planet’s data centers.  The SAVVIS network has tested to be superior in uptime and reliability than The Planet’s infrastructure and peers, over the past year.

• Your server will be upgraded to a system with specs much more powerful than your current system. We are now configuring our shared servers on a minimum of AMD Opterons with 1 MB L2 cache and 2 GB physical RAM (upgradeable to 4 GB). This means more power and speed for your website!

• Your websites will see no changes in functionality. Your pages and databases will work exactly the same. Your e-mail accounts and passwords will also stay the same. No work or changes are required of you whatsoever. We will continue to use the cPanel control panel system, the same operating system, the same OS patching system, the same upgrade schedule.  Everything will feel and look the same.

• Your IP address will change. Unfortunately this is unavoidable. However, very few people actually reference or access their site by IP address anyway, so we do not anticipate this to be any sort of issue for most clients. For those who do use their IP address, in most instances you will only need to change your IP address one time in your FTP program or e-mail program. After that, it is saved and remembered automatically by your PC.

•  There will be no downtime as a result of the move. Period.  We do site moves between servers everyday, and have the “no downtime move” down to a science.

• There will be no lost e-mail as a result of the move. Period. See #5. :)

• The move will be completed in the late-night/overnight hours so that your website traffic will be at its very lowest. This means that any risk of worst-case scenarios and errors are limited to the least-traffic times.

• We will provide you with plenty of notice prior to the actual move. We have not yet set a date/time for the move but we will provide you with 5-7 days’ notice so you are fully aware of it and can plan accordingly.  This is a planned equipment upgrade, the sort of which we perform routinely. We believe in keeping our customers “in the loop” so you don’t get hit with any unfortunate surprises.

This is the first of a series of planned upgrades. As our remaining shared servers at The Planet are scheduled for upgrade, we will contact clients on those servers directly and point them to this post for further information (as the basics remain the same, only the server name will be different).

If you have any questions about the upgrade, please contact our Support Team directly by opening a helpdesk ticket at https://www.dcnsafe.com/supportdesk/

Thank you!
DCN & Agile Hosting Support Team

Shared server refuge.dcnsafe.com at The Planet/Dallas is currently rebooting. The server went down on a high load as maintenance was being run to fix the quota system. It is possible this issue is related to yesterday morning’s power outage & spike at the data center. We will keep you updated …