The Failure of a SysAdmin – A Tale of Laziness, Good Fortune and Self Loathing

I sat staring at a blinking insertion point in an open KeePass window. My mind was having trouble processing what was being seen. Moments earlier I had the need to log into a client’s Microsoft SBS 2008 machine that was humming along happily in an office about 600 miles away from me. That SBS box’s domain policy requires the password be changed every three months and I had changed the password about a month or so earlier. Since I hadn’t done much work for them recently I couldn’t remember the new password.

Not a problem,” I smugly thought to myself, “for I am the password master!“. I keep a rather meticulous KeePass file on their network which I manually update with all account passwords for that office. I opened the file and found the domain admin account’s entry. “That’s strange…” I thought while glaring suspiciously at the “password” field. It contained the previous quarter’s password. I keep a running history of each quarter’s password for a “just in case” scenario, so I looked back to see if I had mistakenly labeled the entry for the new password as an old password. I noticed that two of the old password entries had the same password. I stared as the underweight hamster in my mind turned the flywheel of salient thought. My breathing was surprisingly even considering what possibilities were only just beginning to be seen on the horizon.

It appears that when I last changed the password, I typed it into KeePass like a good little button masher, but then when moving the old password into a historic entry, I must have been careless and pasted the old password over the new password and saved the file. Either that or I closed the KeePass file without saving changes. I’ve certainly done that before (once just today on a VPS’s root account; Yay for keypair SSH tunnels!). Certainly something in my password documentation workflow could be improved (not the least of which is my concentration). Regardless of what I could do in the future to prevent this from happening, nothing was changing the fact that I didn’t have the new password documented in the proper place. Oh, and did I mention that my domain admin passwords are nearly incomprehensible strings of 16 or more characters? That means I had no chance of remembering it.

Hark! There was another possibility. I occasionally sync the remote office’s KeePass files to my local network for convenience. Perhaps I entered the new domain admin password into the local copy of the files. If by chance in my carelessness I edited a local copy and not their copy I might be saved! “Oh please, Oh please, Oh please be in there…”

It was not to be so. My local copies were identical to the office’s copies. Apparently I studiously synced their KeePass file to my local network after mangling the entry for the domain admin. The noose was tightening and I was getting angrier at myself by the second. Had I just been more careful and double checked my work…

But Wait! I have a simple Word document that I keep updated with that office’s most important information (safely stored, of course – but still it makes me uncomfortable. Alas). It’s an old-fashioned hard copy of the most important passwords and disaster recovery information for the office’s computer systems. I check it for freshness at the beginning of every month and print two copies out. One copy gets kept at the accountant’s home (he also swaps out the backup hard drives every week and keeps them in his home office as an offsite backup) and the other copy goes to the Assistant to the Board of Directors. Both fellows are former bank managers, both are sufficiently concerned with security for the needs of that organization. I trust them to keep good care of those hard copies. In theory I could simply look at the hard copy information and get the domain admin password.

I didn’t even need to check that document, however. I knew I hadn’t updated it in months. The last update I performed was well before the last domain admin password change. The time it took for hope to spring up inside of me over the idea of the hard copies only to be let down by my dereliction of duty took about half a second. I winced at how I had shirked those responsibilities.

Breach of contract!” “Lying down on the job!” “Filthy moocher taking their money!” All things many people would be justified in shouting at me. However, this office is a non-profit, run by family friends for which I do all of my work on a volunteer basis. They know and are okay with me giving them whatever time I can spare to them between my other responsibilities. Does that mean that I can glibly botch domain admin passwords with nary a care in the world? Well, no. If I’m going to take on responsibilities, then I need to either carry them out to the utmost or divest myself of them entirely. However, at least I didn’t have the added guilt of taking money for a job not done.

I slumped in my office chair. Visions of taking a 600 mile road trip to reset the domain admin password SRVANY / INSTSRV style passed before my eyes. Of course, the SBS 2008 machine has an onboard ILO chip that would have helped me handle the situation nicely from the relative comfort of my desk chair, but as fate would have it, the firmware has some issues preventing me from logging into it and I needed to schedule my yearly trip there to physically handle that procedure (I don’t trust anyone locally to do it with me coaching them over the phone). The mishaps just keep stacking up.

What a monumental convergence of carelessness and stupidity…”

Wait… what?” A sparkle in my memory glinted like a distant Tinkerbell swerving between stacks of mental cruft. “I seem to recall… yes… I think I did…” If anyone has been reading this blog with regularity, they’ll know that I’ve been migrating to Fedora 14 from Windows 7. I’ve been using Fedora about 85% of the time; only booting into the Windows partition for apps I haven’t migrated or virtualized yet such as QuickBooks and Steam. My Windows-based remote connection manager of choice is mRemote. That SBS 2008 machine is one of my saved connections, and I made the habit of saving the password in the connection as well. It was possible that I connected to the SBS 2008 machine at least once right after changing the password and saved the new password! I wouldn’t be able to see the password’s characters (of course it’s hashed in the connection file), but I’d at least be able to log into the machine and reset the password to something else.

I nervously rebooted my machine and dropped into my Windows partition. I launched mRemote with great trepidation. mRemote itself doesn’t have a great track record for keeping saved connections safe. If memory serves correct, I’ve had two XML files that had my entire saved connection library suddenly go corrupt with no ability to repair it. Wouldn’t it be awesome if the connection file was corrupted when I most need it? Fortunately it wasn’t.

I found the saved connection to the SBS 2008 server within mRemote’s interface. The password field was filled out with obfuscating black circles so I now knew that I had saved at least some password. Was it the correct password or was this old information as well? I could only hope as I opened the connection. With my breath held captive, the RDP connection began to open. Would I see the remote screen whine “Incorrect Password”? Part of the RDP protocol’s henavior can be something of a tease. You will still get to see the remote login screen if the password is wrong.

Never in my life have I been so happy to see the words “Loading User Profile” across the Windows login screen. I could breathe a bit easier now. After resetting the password and making sure to document it properly, I did a bit of a post mortem on my actions.

Post Mortem

To my knowledge, I exhibited two kinds of laziness here, and neither were of the admirable “Lazy Admin” variety. Most greviously, I did not take care to document my actions. I carelessly handled the single most important secret in that organization’s technology infrastructure.

The second kind of laziness that I showed is a bit borderline. I feel that I was lazy in saving a password in a connection. I’m sure most of us save passwords in a similar manner, and perhaps that’s okay if you know the security level of the encryption used to store the saved password. You also need to secure against your PC being attacked and connections being made using those saved connections. I’m a bit iffy on this part of the story, but maybe I can get a pass. I don’t work for the DoD or need a Top Secret security clearance. Saved passwords do make things easier, especially since the passwords I choose are so complex. Even if the practice itself gets a pass, I don’t believe it’s even close to the kind of laziness in an admin that’s a virtue. Having to fall back on my kind of laziness is frighteningly arbitrary. An especially bad reflection on my character is that several times before this incident happened I wondered as I opened one of the saved connections “What password is this connection even using and have I written it down?” but I did nothing about it.

When it comes down to it, I have all the head knowledge and all of the resources I need to make proper documentation so that things like this never have to happen. I just screwed up. Okay, fine – I’m busy, but who isn’t? Starting a business, transitioning away from a lot of volunteer work, attempting to get some new contracts, working on an existing contract, doing many other different projects and that’s not even mentioning non-professional items (not that there are very many at this moment in my life). But regardless of all that, nothing can excuse performing a job poorly like that. There’s nothing wrong with the tools (unless you count me). That non profit could very well have been in some serious trouble if I had lost the domain admin password. Trouble in the form of lost productivity or even broken compliance and fines.

The Result:

I woke up and smelled the coffee. I’m trying to make it a long term change. It made me stop and take stock of if I’m just too busy to have all the responsibilities that I currently do. It’s made me taste what I already know: You can’t take things for granted. The problem is that if the responsibilities that I have right now are taking up so many hours of my time when I’m really only performing a 60% job on them, then to do a 100% job on them I’ll need to drop some things. Time to make some decisions.

Have you ever had a painful moment that made you take stock of your attitude toward work? Ever had some unknown or unacknowledged irresponsible actions shown to you in a painful way? You can comment below, and I promise I’ll never judge you or reveal your true identity. =)


  1. Fred Woodbridge

    March 21, 2011 at 8:37 am

    These same problems are prevalent in aviation. Pilots have to perform incredibly mundane actions thousands of times in a career, but also the right ones or people die. This is not to speak of emergencies, of course.
    They use checklists for this purpose. Check out The Checklist Manifesto by Atul Gawande for more information.


    • Wesley David

      March 21, 2011 at 2:38 pm

      Thanks for the book recommendation! I do think I need to make a better habit of checklists. The tyranny of the urgent has encouraged me to develop some pretty bad habits in that regard.


  2. Zoredache

    March 24, 2011 at 12:41 pm

    Consider keeping your databases on a SVN/GIT/VCS server. This will help since you can easily check them out where you need access, and won’t be manually copying them around or worrying about over-writing copies you modified.. You also will be less tempted to keep older passwords since you will be able to retrieve any version of your database from the SVN history.


    • Wesley David

      March 24, 2011 at 12:48 pm

      You know, I’ve been really wanting to make my own SVN / etc. server. I’ve never used one and am unfamiliar with their workings, but the general concepts that I see look awesome. Thanks!


Leave a Reply

Follow TheNubbyAdmin!

follow us in feedly

Raw RSS Feed:

Contact Me!

Want to hire me as a consultant? Have a job you think I might be interested in? Drop me a line:

Contact Me!

Subscribe via Email

Your email address is handled by Google FeedBurner and never spammed!

The Nubby Archives

Circle Me on Google+!

Photos from Flickr

Me on StackExchange

The IT Crowd Strava Group

%d bloggers like this: