In true Nubby Admin form, I just learn the hard way why separating systems from unnecessary dependencies on eachother is a Good Thing. Specifically, separating your DNS providor from your web host. I knew it theoretically, but had not yet lerned it experientially. Think about this for a moment: If two things are conjoined in some way, the likelihood that the failure of one will affect the other is high.

At one of the places I do work for, the webhost not only handles their website, but also the entire domain’s DNS records. MX Records, SRV record for Outlook Anywhere, the Phone System’s record, everything. The other morning, I logged into Outlook only to receive a strange certificate error:

There is a problem with the proxy server's security certificate. The name on the security certificate is invalid or does not match the name of the target site [my domain]. Outlook is unable to connect to the proxy server. (Error Code 10)

Pinging remote.[mydomain].com came back with a different IP address than it should have. My pulse quickened, expecting the worst. Did someone get into our web hosts’s control panel? Have they hijacked our site? Is it redirecting to some V!@gra campaign? Can I at least get a cheap R0l3x watch out of the deal?

I logged into our Plesk control panel… only to be told our account was suspended. I figured that our hosting company had objected to our hacked website erupting spam all over the intertubes. I called the webhost’s support number and was greeted by a staffer that embodied all of the enthusiasm of a three-toed sloth attempting to fight his way out of a medically induced coma.

After some discussion, and him apparently IM’ing an admin, it turns out we were over our limit for storage on the server. Sweet. Due to that, our entire hosting account was disabled. Including DNS. Double-sweet.

After getting our account reactivated long enough to jettison some files, all was well. However, I learned a valuable lesson and methinks I’ll be switching DNS authority over to our registrar, Network Solutions to separate an unnecessary dependency. Yes, we used Network Solutions as a registrar but that was before I was involved with this organization so there is no blood on my hands.

Have you ever had a system fail and bring down another system that you weren’t anticipating? How did you dig out of that hole and what did you do to break that dependency?