Author Archive

SolarWinds has announced their “Tales from the Trenches” System Administrator Appreciation Day contest. It has been launched in honor of the 13th annual SysAdmin Appreciation Day started by Ted Kekatos. System Administrator Appreciation Day is July 27th this year, for those not in-the-know.

This latest SolarWinds contest is pretty simple. Share a “trench-born tale” about your job as a systems administrator and a hand selected panel of Systems Administrators will cluck over it like mother hens with codependency issues. You will have until July 27th 2012 to submit your tale. Once the judges have determined a winner, the prize for the best tale will be an iPad, a ThinkGeek gift card or an original piece of artwork depicting the winner’s story!

But just who are these SysAdmin judges who will be scrutinizing the hopefully-not-too-tall tales? They are:

• Ted Kekatos – Mr. SysAdmin Day himself!
• Denny LeCompte - VP of product management at SolarWinds
• Trevor Pott – SysAdmin writer for The Register
• Matt Simmons - The Standalone SysAdmin of much repute
• Me – Yes, I managed to charm my way onto the judging panel somehow.

Oh but there’s more! The SolarWinds Thwack community will vote on the best tales as well. The top four community-voted tales will each receive a $50 gift card to ThinkGeek.

Some Rules of Note

There are some rules that readers of my blog should note. In fact, make sure to read the entire official rules page just to be safe. One of the rules which might be a disappointment to many of you (since my audience is rather international) reads thusly:

The SolarWinds ‘Tales from the Trenches’ System Administrator Appreciation Day Contest (the “Contest”) is open only to legal residents of United States who are eighteen (18) years of age or older.

So USA tale-tellers only. Sad International Panda.

Let’s move on to some of the other rules that seem to be of special importance (note that this is not a full list of the rules. Makes sure to check out the official rules page or don’t say I didn’t warn you!). Of course, be punctual, like all good SysAdmins are!

The Contest will commence on 6:30 a.m. CST on Wednesday, June 27, 2012, and an Entrant must enter by posting a comment in the appropriate thread on thwack® community forum prior to 11:59 p.m. CST on Friday, July 27, 2012 (“Entry Period”).  Entrants submitted before or after the Entry Period will not be eligible for the Contest.

Also, you’ll need to sign up on Thwack:

Entrants must have a thwack account and must submit their best original story about their experiences “in the trenches” as a SysAdmin on thwack during the Entry Period. Entries should be 500 words or less and should include a subject line..  By submitting the completed survey, you will receive one (1) entry into the Contest.

Tactfully Tell Your Tale of Torturously Tortuous Technology Trials

Crack open your text editor of choice, start putting your terror-tinged tales into words and submit your story here. At worst you’ll experience the catharsis of sharing your woes, at best you’ll be richer some cool prizes. Share the news with some of the social media icons on my blog and let’s get together for some group session therapy. It’s okay to cry. No one will judge.

We’ve all been there. Googling around for an answer to a problem we’re having. We stumble upon a forum post that mirrors our dilemma exactly. Various remedies are suggested, and we furrow or brows in consternation because we’ve already tried what is being suggested.

And then we see it. At the very end of the thread. And it makes us cry.

“Thanks every1. It’s fixed.”

Are you kidding me? It’s not like they just forgot about the forum thread. They remembered enough to come back, log in, and submit a post. Would it kill people to simply add one sentence that gives even the slightest hint about what solved their problem? Just a simple “applied a udev patch” or “syntax error in script” would be better than “It’s fixed.”

So to all Good Guy (or Girl) SysAdmins out there, thanks for posting back with the full solution. Even more heroic are the people with the exact same problem as the original poster, and who revive the dead thread to post their solution. You are truly wonderful people. Three cheers for active community members who either participate on forums, Q/A sites or write blogs. You’re Good Guys (and Girls).

(For those not versed in the world of memes, this is based on Good Guy Greg.)

TechMentor is a yearly conference designed for information technology professionals that largely work with Microsoft technologies. Last year I mentioned the TechMentor event that took place in Las Vegas. This year, TechMentor 2012 will be taking place at Microsoft HQ in Redmond Washington! The date is August 20-24.

Some of the big names showing up to the event will be Don Jones, Mark Minasi, Bruce Rougeau and Greg Shields. Some of the listed topics that will be dsicussed are Virtualization (assuming a heavy focus on Hyper-V), Application Delivery, MCITP Certification (and hopefully some demystifying of the new MCSE/MCSA certifications), as well as the ubiquitous PowerShell. Actually, if you didn’t have an idea that PowerShell would be a topic just from seeing the name “Don Jones” then you’re not familiar enough with PowerShell. =)

I’ve been given the opportunity to offer readers of this blog a $300 discount off of the 5 Day Best Value Package rate (discount applies to the Standard rate and new registrations only). FYI, this is at no benefit to me; I get no kickbacks or affiliate-anything.

To claim that $300 discount, simply use this link (or paste this URL into a new browser tab: http://bit.ly/TMRDReg) and use the code TMRTU when signing up.

Let me know if you plan on making it to the conference! I’d love to have you guest blog about your experience.

A Phoenix, Arizona IT professional who I am familiar with has launched an indie funded project that any techie who comes in contact with physical infrastructure will appreciate. It a Keychain Punchdown Tool! Take 0:36 of your time to see what you think.

There is an Indie GoGo project for the keychain punchdown tool. It will work on both 66 and 110 blocks, fit in your pockey and make you the life of every party! (Two of the three things in the preceding sentence were true). The funding goal for the project is $2,000 and there are four funding tiers: $1USD, $10, $50 and $75.

If you’re an IT pro that’s on the go and you use punchdown tools frequently, consider giving a few dollars to this indie project. Please share this with anyone who might be interested either through this post and the social sharing options to the side and below the post or through the Indie GoGo page itself.

I saw this infographic recently concerning Microsoft’s stagnation and Apple’s phenomenal growth in the last ten years. I regard myself as rather agnostic concerning brand wars and really don’t care what technologies a person or company uses, just so long as it works well. However, it might be interesting to note the lackluster decade that Microsoft has had. The Zune, Windows Mobile and Bing are all products that Microsoft as invested heavily in (especially Bing) and yet nothing much has come of any of them.

Clicking the infographic takes you to the site “MBAOnline” for a look at the larger graphic.

Created by: MBAOnline.com

I do see some good tools and products that Microsoft has made in the last ten years, but those are mostly systems tools. Aside from the Xbox, I can’t find a major consumer product that has taken off and set Microsoft apart in any space. Am I missing something? Anyone have a contrary opinion they’d like to share? Let me know below.

My Problem

I have an HP ProLiant N40L MicroServer that needed its BIOS upgraded. I downloaded the proper firmware update package from HP’s support site which includes ROMPaq and the update flat files. I created a bootable USB thumbdrive using the ROMPaq utility, however attempting to update the server’s BIOS receives this error:

RomPaq may take a few minutes to get started, please be patient...
Command or filename not recognized

My Solution

Open the SWSetup folder that the ROMPaq installer creates and copy all files from the “Flat Files” folder over to the USB drive that the ROMPaq utility modifies to be bootable.

The Long Story

After the “Command or filename not recognized” error was received, I popped the USB drive into a different machine and inspected the contents. Looking at the AUTOEXEC.BAT file that is on the root showed this:

@echo off
echo ROMPAQ may take a few minutes to get started. Please be patient...
rompaq.exe /l:us /!

However, there was no rompaq.exe file anywhere on the drive! Furthermore, there wasn’t even a ROM file to be seen. Clearly the utility did not create a drive that was capable of flashing my server. Part of the unpacked files included a folder called “Flat Files” that included things which made more sense. For example, in that flat files folder was an AUTOEXEC.BAT file that looked like this:

@echo off
 
if errorlevel == 1 goto ENDIT
if errorlevel == 0 goto other
:other
echo Next Please!!!
flash.bat
 
:ENDIT

As well as a flash.bat file that looked like this:

kbd /"flash O41072~1.rom -r 100000,10000 -r 1e0000,10000;q"

And finally an actual ROM file that matched filenames with the line in the flash.bat file. I merely copied all the files in the “Flat Files” folder over to the USB key and chose to replace any existing files.

Apparently HP’s QA group let this get past them. The tool and documentation explicitly state that it will, once run, leave the USB media in a state that can be used to directory update the ProLiant firmware. No additional steps should be needed, according to the documentation. That is clearly not the case in this scenario.

Ever had a similar experience with ROMPaq or is this an oddball case? Let me know in the comments.

Plesk has some baked in magic concerning MySQL and the admin account. If you try to use mysql or mysqldump using your root user name or password, it will not be allowed. You’ll see something like this:

[root@server] mysql -u admin -p
ERROR 1045 (28000): Access denied for user 'admin'@'localhost' (using password: YES)

If you want to access MySQL as root, you simply use the command my. What exactly does my do?

[root@server] type my
my is aliased to `mysql -A -u admin -p`cat /etc/psa/.psa.shadow`'

So, in order to use mysqldump, one has to use the following command syntax:

mysqldump -u admin -p`cat /etc/psa/.psa.shadow` [database] > [outfile-name]

Strange but true. Of course, this is for a vanilla installation of Plesk. I’m sure you can customize your MySQL permissions and groups to behave differently, however I would not advise that. Plesk likes to have things its own way, and it will either break if you change things or set things back to how it likes them on the next update.

This won’t be my usual “Problem, Solution, Long Story” style troubleshooting post. There are a few complexities involved that don’t allow it to fit into that template so easily.

I have a client-facing server running CentOS 5.7 and Plesk 10.3. When clients need web space, I put them on my Plesk server so they have shiny buttons to click when managing their own web space. Recently I had a series of unfortunate events cause an outage on one client.

It starts with my craving to have things standardized. All client account domain directories are in lower case. All, that is, except for one: AmazingClient. Their main domain’s vhost directory is /var/www/vhosts/AmazingClient which, in Plesk-land means that any reference to that client’s domain is always in that case. It bugs me. More than it should. When I created the client account several months ago, for some inexplicable reason, I used CamelCase in their name. One recent evening I decided to change the capitalization for their account’s main domain. Simple, right?

I did say that I’m using Plesk, did I not?

Before I go any further, I know what you might be thinking. “Domains aren’t case sensitive! What nonsense are you on about?!” They’re not case sensitive when approaching domains from a DNS perspective. However, I’m looking at this from a filesystem and Plesk user account perspective.

To change something as simple as the case of a domain’s vhost directory, one cannot merely rename it. There are many configuration files to consider as well as Plesk-specific tasks that rely on the domain’s directory not being glibly swapped out from underneath it. To change a domain’s name in Plesk, one has to go into the client’s control panel, and click on the Websites & Domains tab.

From there you will find the domain that you want to change the case of (remember, this isn’t about “domain” in the DNS sense, but rather the representation of that domain within Plesk and on the filesystem) and click on its link. From there you will come to the Host Settings page for that domain. Once on the Host Settings page, you’ll have the option to change the domain name. Here comes the trouble: you can’t change the name merely based on case. Even though Plesk sees the client domain differently in the backend based on case, in this Host Settings interface case is not taken into account. Plesk will complain that the domain already exists. You need to change the domain name to something different, then change it back to the original domain name, minus the capitalization. (Plesk FAIL #1)

In my case, I wanted to swing it from AwesomeClient.com, to awesomeclienttemp.com, and then back to awesomeclient.com (sans the capital “A” and “C”).

Tipping Over the Edge of Doom

When trying to move from AwesomeClient.com to awesomeclienttemp.com I received this error:

Internal error: [domain path] is out of webspace
Message is out of webspace
File Webspace.php
Line 334
Type PleskFatalException

After that error, the Websites & Domains tab is no longer accessible to that client account. Trying to use it receives the same “Internal Error: [domain path] is out of webspace” error.

You see, it appears that Plesk, upon requesting a domain rename, copies the domain’s existing files and then deletes the old ones. It does not perform a mere rename action (Plesk FAIL #2). This client uses quite a bit of space and it apparently maxed out their quota. I say “apparently” because, by a strict accounting for free space and quotas on the server, it should have been allowed – but just barely. Perhaps there’s more space that Plesk needs than a simple doubling of existing files. (Plesk FAIL #3?) Plesk certainly didn’t perform any kind of filesystem or account limitation checking prior to attempting the move. (Plesk FAIL #4)

The client site was still responsive; there didn’t appear to be any negative effects. I needed to investigate further, but as the night wore on I decided to postpone a thorough examination until another day.

Ask Not For Whom Your Cell Phone Tolls

Bright and early the next morning, I got a call. It was from the client.

“Our website seems to be down, so… uhh… if you could look into that…”

Super.

Nothing was being served up in response to any page requests for this domain. Apache’s error logs were showing requests for this client’s files as hitting in the default vhost root, not their own. Then, it hit me.

Plesk does not use the standard Apache configuration files. I mean, it does, but not really. It auto-generates Apache configuration files based on the information that is stored in its own customer database within MySQL. That’s why the domain was just fine the evening before, but didn’t fail until the wee hours of the morning. The configuration files had been latently generated based on the failed attempt at changing the domain account name.

Silly me… I expected there to be rollback statements in any of the SQL DML statements made to the database. I expected that a fatal error would be caught and changes rolled back. They weren’t. (Plesk FAIL #5) Silly, silly me.

Of course, I wasn’t going to be able to change the domain information because the Websites & Domains tab bombed out permanently with an internal error. I couldn’t access the officially sanctioned means of modifying the domain account. This called for some database mangling.

Let Pry Through the Portage of the Database

I logged into mysql and dumped the psa database. From there, I used grep to scour the .sql file for any mention of awesomeclienttemp. Sure enough, the bad change was recorded in the database. There were dozens of records in several tables that pointed to the bad domain. That was causing Apache configuration files to be written with bad data, among other applications. There was also mention of the original, unsullied domain. I guess not all of the SQL statements that are part and parcel of a domain change were able to be executed before the error condition was achieved. (Side note: ROLLBACK!! ROLLBACK!! ROLLBACK!!)

Solving the problem was a simple as searching for and replacing the string awesomeclienttemp with AwesomeClient. I used mysql to perform that, but it could have been done on the dump file and then imported. For those interested, I used the replace() function and performed a select statement first just to make sure that I was changing the data that I expected to. Once satisfied with the results I performed an update statement also using the replace() function. Here’s an example of changing some values in the dns_recs table of Plesk’s psa database:

mysql&gt> SELECT REPLACE(displayVal,'clienttemp', 'Client') FROM dns_recs WHERE displayVal LIKE '%clienttemp%';
+--------------------------------------------+
| REPLACE(displayVal,'clienttemp', 'Client') |
+--------------------------------------------+
| mail.AwesomeClient.com.                    |
| AwesomeClient.com.                         |
| AwesomeClient.com.                         |
| AwesomeClient.com.                         |
+--------------------------------------------+
mysql> UPDATE dns_recs SET displayVal=REPLACE(displayVal,'clienttemp', 'Client') WHERE displayVal LIKE '%clienttemp%';

With the database in a better state, there is still one more thing left to do. Plesk doesn’t dynamically look to the database for configuration information. It looks to regular files that have been dynamically generated from the database’s information. That generation happens on a schedule, but can be expedited using the httpdmng command. Specifically, I used:

/usr/local/psa/admin/bin/httpdmng --reconfigure-domain AwesomeClient.com

You could also use the –reconfigure-all option to perform a regeneration of all domain configuration files. After running httpdmng the domain was up and running.

Apache Test Page or Blank Page Problems

I glossed over some of the troubleshooting techniques I used while tracing the problem to its root. If you’re having trouble with seeing the Apache test page, then search through your httpd.conf file and make sure that your DirectoryIndex directive is set to look for all of the variants of an index.html page that you use. For example, index.html, index.htm. index.php, etc.

Furthermore, just to reiterate, check all of your vhost conf files, such as yourdomain/conf/vhost.conf (or any conf files that reside in that directory) for the DocumentRoot directive and make sure that it’s pointed to what you want it to be pointed at. Do not edit the files that are named similar to 13279881860.14852200_httpd.include. Those are auto-generated by Plesk and at worst you could cause destruction of files in your domain; at best you will have to re-edit those files every time a new one is generated.

Of course, do a dummy check to make sure that the domain you are trying to access is really resolving to the IP address of your web server. Just… do it. It takes 5 seconds and you have the outside chance of being pleasantly surprised.

The Takeaway

Plesk is rickety. If anyone has used a better control panel for client-facing servers, let me know. I’ve worked with cPanel and Plesk, but never with any of the others that I’ve listed in this giant list of web based server control panels. Most people will shout “Just don’t use a control panel!” but that’s not a terribly client friendly option. I’m not categorically against control panels when used in the correct situations. I am, however, against misbehaving control panels.

Let me know your experiences in the comments below.

Previously I explored how to limit a user’s ability to runs commands with sudo. As a tangential topic, I needed to restrict the commands that a user account had access to when they connected to the server via SSH. Specifically, I needed just a few commands to be strung together and executed every time this account connected.

The mechanism that I used to do this was with the authorized_keys file. For a thorough explanation of that file, take a peak at the man page for sshd. To explain it very simply, the authorized_keys file holds the public keys of other users/systems that are allowed to connect to that machine. For example, I place my main user account’s public RSA key into the authorized_keys file on the Linux servers that I manage. When I connect to the remote servers using SSH, it checks to see if I’m who I say I am by challenging me with the public key that it has stored. The user account on my laptop uses the private key to validate itself (yes, the private key is password protected) and I am then allowed to haxor on the servers to my heart’s content.

Here’s an example of a public key:

ssh-rsa AAAB3NzaC1yc2EAAAADAQABAAABAQDclBxY7lOaolHGaogdcc9GaTQLWMcn2PK4hnQfWlJgeeGqgS66jL4XJyiR9HcgaebBW88Z2sevUxd7g25WhuuRAazfOcElEaE+h6MMPZ94gHY+x+iVAdlNKxLT/bTvCUXLEft/yZFpnknnv7jX4ChfSiII9OiAiCzuSdyHt1/1LgEHgvDIwKMzvTgImm5X/3IhtOitjJY3Q6yhKQ6LdenQtG/v+ANqKe6opDuUKc3k9hRmj7aHROxL52paQTEgEMoVLbIoZY4/yGUzmrZQU45jNqMrbXdAxG4XexZxb7bpTLu91s0DJQGx43JNXwhJVinPgxHLmfyoCSqR1WPqn8E3 testuser@testserver

The public key, when placed in a system’s authorized_keys file, can have some extra tidbits added to it that sshd honors. An SSH protocol 2 public key follows this format:

options, keytype, base64-encoded key, comment

In the above public key, you see the keytype as ‘ssh-rsa’ followed by a space, then the key itself followed by a space and finally a comment, which in this case is a username and hostname combination. That’s a helpful hint to know who this key supposedly belongs to. Notice that there are no options included in the above key, which would come before the keytype.

Some of the options that are available to be parsed by sshd include:

• environment= Changes an environmental variable for the user that is on the receiving end of the connection.
• from= Only allows connections that use this public key to be initiated from certain hosts. Helpful for the extremely paranoid or the very security conscious (the only difference between the two being pay grade).
• no-X11-forwarding Because we don’t need users installing xorg and then browsing the web on a remote instance of Chrome.

There are plenty of other options, however the final one that I’ll mention is the most crucial to this topic: command="command"

With the command= option, you can cause a command to be run immediately upon a successful connection to a remote host. Once the command is run, the connection is closed. Notice how that works. The command is immediately run and then once the command finishes, the connection is closed. This is not something that you’d want to do to a key that is intended to be used interactively by a human.

What could this be good for? In my specific scenario, I am using a backup tool that moves all of the data to stdout which is then piped to ssh for a secure transfer to remote storage. The remote connection would normally look like this: ssh remoteuser@remoteserver ” cat > backupfile.zip” However, if I edit the authorized keys file, I can restrict the incoming ssh connection to only be allowed to use that specific command.

It’s just another layer of security to keep people from doing things that they shouldn’t be doing. Have different ways of achieving a similar goal? Any caveats you know about? Let me know in the comments.

In a recent post named The Making of a Meta Server or “Why I Bought a Mac Mini as a NMS” I explained why I had chosen a brand new, 2012 Mac Mini as my NMS hardware. After two weeks of mind numbing work, I have officially declared the Mac-Mini-as-a-NMS project a failure.

The main problem surrounded Apple’s custom EFI. Apple hardware does not use a BIOS, but instead uses EFI (note: not, specifically speaking, UEFI). Or rather, it uses an ancient, bastardized version of EFI 1.1. There is a BIOS compatibility layer that allows OSs that can only communicate with a BIOS to operate on the hardware. Most notably Windows. Apple’s OS also runs on a hard disk that has been partitioned using the GPT partitioning scheme, which isn’t itself a huge deal, but you might be surprised at the anemic support for GPT boot disks in even modern operating systems.

To use the Mac Mini to boot an OS that needs BIOS compatibility and a MBR disk should be relatively easy. Right? Right!

Unless Apple is involved.

There are several things that Apple has mutated away from the EFI standard, one of them being not using the EFI system partition for anything except firmware updates. Their custom EFI implementation has the boot process (as well as some extra filesystem drivers) baked in. The whole EFI experience just never worked like I expected it to. The other trouble is that Boot Camp has been changed in OS X Lion. If you wanted to be hand held through the partitioning process and the creation of a hybrid GPT/MBR disk, you’re invited to use Boot Camp. However the latest alterations only allow media with Windows images to be accepted. You can no longer (from my ability to understand) use Boot Camp to install non-Windows OSs. Of course, it was always unsupported, but at least it was doable.

During the whole process, I used the EFI boot manager rEFIt which apparently only recently works with OS X Lion. I read more about the GPT partitioning scheme than I ever have previously. I learned more about EFI than I ever wanted to know (although all of that information will come in very handy in the near future). I hand-rolled bootable USB thumbdrives. I tweaked partition tables. I did very nearly everything I could think of except rolling my own EFI boot partition. After the hours had steadily ticked away I decided it was no longer worth it.

After countless errors concerning boot media, partition problems, and blinking cursors, I concede that the latest Mac Mini has defeated me. It has been shipped back to Amazon and I can go back to my Apple-less existence. Speaking of Amazon, I believe that they deserve some praise in this.

Amazon made the returns process easier than any return I have ever made. Anywhere. I stated that the reason I returned it was because software I had intended to use with it was not compatible. As a result of the return not being their fault, I had to pay return shipping. Within just a few clicks, Amazon created a return label. I printed it out, boxed the mini up, taped the label to the box and handed it over to the man behind the UPS Store counter. Within 15 seconds I was walking out of the store. I have the fortune of living just a few hundred miles from an Amazon return center located in the Las Vegas area so the return was processed and money credited back within two days. Thank you, Amazon. You were the only bright spot in this debacle.

I am now investigating other pieces of hardware for this project based on the recommendations of several colleagues. If you have a recommendation, share it with me and the rest of my readers in the comments below. I’ll certainly write about my second attempt at this project as it happens.

In the end, I’m not mad. The Apple wasn’t designed to do what I was asking it to do. It was my fault. My only lingering frustration is that the Mac seems to take any standard technology that it uses and twists it in new and different ways so that your familiairty with a standard becomes more of a liability than an asset. Sound like another familiar company that SysAdmins like to pick on? Then again, Apple isn’t intended to be in the business market. Let us pause and mourn the passing of the Xserve (I handed my G5 Xserve over to Best Buy for free recycling last year. So, so sad…).

Any similar experiences with an Apple product? Have you managed to wedge an alternate OS on 2012 Apple hardware? Let me know in the comments below.