The Nubby Admin

Last year, Arial Jolo put together the 2011 Root Election in honor of SysAdmin Day. This year, he’s cooked up something else for SysAdmin day! It’s called AdminFest and it’s a SysAdmin day meetup at Breoghan Brew Bar  Bolivar 860, San Telmo (Buenos Aires) on Friday July 27th, 2012. (The site may present you with the Spanish language version, so I provided the “forced English” link above. Remove the fiddly bits and just leave the domain name if you want to see the Spanish version.)

According to Arial:

This year we want SysAdmin to celebrate their day in peace, to sit and relax and have somebody else doing their job. That’s why we tell them: Put somebody else on call.

Furthermore you can check them out on Twitter at @PasaLaGuardia, chat them up on Facebook, or email them at [email protected].

Oh but not so fast, my brew swilling South American SysAdmin! You’ve got to pass the gauntlet first. The gauntlet is a SysAdmin test that you’ll have to take in order to prove your belonging in the group. Don’t worry. I have every bit of confidence in you.

Two promotional videos have been made for the 2012 Admin Fest. Have a look and don’t forget to turn on the English subtitles in case your Spanish is a little out of use. Oh, and who hasn’t had to cradle a PC back to sleep?

Are you going?

If you manage to make it past the gauntlet and into Breoghan Brew Bar on Friday, July 27th, make sure to take some pictures. I’ll post the best ones here after the festivities are over. Just remember to take care of those flapping Nagios alerts before you head out of the office so you’re not nagged incessantly while you’re trying to socialize with your fellow SysAdmins.

I’m going to assume that anyone reading this post with at least a few weeks of professional information technology work to their credit has at one time asked a question on a forum in pursuit of a technical solution. I have my own favorite forums and communities that I keep on the short list for when I need to ask questions. I’m also fairly active in answering people’s questions when I have the time.

Over the years of participating on forums, it’s fairly common to see people post logs or diagnostic information as they and the forum’s population try to troubleshoot the issue. Most of the time people remove any sensitive information, but once in a while you’ll see someone post the entire config file for a router that includes username and password (unencrypted, of course).

Recently I’ve noticed that, sometimes, the inhabitants of a forum or community will get a little cranky if not enough somewhat-private information is posted. We’re not talking about usernames and passwords, of course. Usually it revolves around domain names and DNS entries. That is, if a person is having issues surrounding DNS, especially public DNS records, a forum member will sometimes complain that the original author hasn’t provided enough information to help solve the problem. What follows is usually a request for the author to provide the public DNS name of the domain that is having problems.

I take immediate issue with wanting to know the public domain information of a post’s author, and have finally put my finger on the controversy. This is why I do not think people should post DNS information in public forum posts:

My Position, Part 1

If you post information on a public website concerning your workplace or a client, one should exclude as much information as is possible. Any amount of information concerning a client can be used to bootstrap a potential intruder into your infrastructure. I recommend avoiding the mention of any real names that could trace your post back to a real company. That includes even public DNS entries.

Counter Argument: The above idea is commonly countered by something along the lines of “If attackers can gain entry into your systems simply by knowing a few relatively innocuous bits of information, then you have larger problems.” The rebuttal is that, while that might be a certain shade of true, narrowing down the scope of an attacker’s knowledge is undeniably helpful. If a potential attacker knows that a specific company uses SuperMicro, it’s just a little bit easier to either spearfish or attempt to attack external services using known exploits for that vendor and the stock hardware / software / firmware that often comes with it.

(Side note: That’s why I’m on my guard when working for American non-profits. Each non profit has to file what is known as a “Form 990″ which includes major purchases. I’ve perused through a few 990s for organizations that I’ve worked for or supported and learned a lot about equipment purchases and brand preferences. Knowing major vendor relationships can bolster an attacker’s ability to do evil deeds.)

(Second side note: I don’t live this out perfectly because I do enjoy and see good business sense in being open with how an infrastructure is built. When building my own hosted services, I prefer to swing the door open and show build and status information that violates this position. However, in certain scenarios, it should still be carefully considered before information is shared about an infrastructure.)

My Position, Part 2

Furthermore, if you volunteer your time on a forum or in a community of some kind, don’t ask for specific information concerning a poster’s problem that could identify a company. Certainly don’t complain about people not posting those specifics. Don’t ask for real domain names, even if it’s a DNS question concerning public records.

First Counter Argument: “But wait! DNS names are public information!” you might howl. That’s true, however if a poster reveals the domain name of the company he is working with, then suddenly the relationship between the author and a company has been established. Now that poster’s history on a forum or group of forums (people often use the same nickname on multiple forums) has a greater liklihood of being pertinent to that specific company. Did a person ask a series of questions months ago concerning how to run SMTP services on IIS 7? Oh but they were cautious and didn’t mention any company affiliations! Months later, tracking down a public DNS issue, if at a forum volunteer’s request they post the results of nslookup including full domain information, suddenly their posting history could be correlated to a specific company.

Second Counter Argument: “If they’re asking for help for a public DNS issue, then how can I help the person without being able to perform my own dig and nslookup queries?” Teach a man to fish, don’t hand him a plate of fish and chips.

Oh sure, perhaps they no longer work for the same place that used IIS 7 or maybe they’re a consultant like myself and have multiple clients come and go so relationships would be very hard to correlate. However, the likelihood is that there’s a solid relationship between that author, his posting history and a specific company. Certainly a quick look on LinkedIn or some other social networks could easily reveal if the poster is a consultant or has changed jobs in a certain time period. Nevertheless, all of this shows that a person’s history and relationships can be relatively easy to track down and will help in the event that an intruder starts an information gathering mission as a precursor to an attack.

My Position, Part 3

Finally, there’s this thing called Google. Or Yahoo, or Bing, or Baidu, or DuckDuckGo, or Wolfram Alpha, or Cuil (har har), or whatever search engine you like to use. They index anything and everything and, unless you’re asking on a private site that disallows indexing, your question will be indexed within mere minutes of being posted. That can be bad for two reasons:

1. If someone searches the company / domain name that you’ve posted, your post may come back. If you’re working for a SMB, your post could be on the first page of results. Now, anyone who searches for that company will see a page that says in essence “Something ain’t working right over here!” From a customer perspective, that could mean the difference between a relationship or them looking elsewhere. From the company’s perspective, well, it’s not uncommon for a company executive to have a Google search alert for their company’s name. Don’t be surprised if they see your post as a result. Depending on the company culture and your relationship to the company (internal IT or external consultant) that might not endear you to them.
2. If someone searches for a name associated with you, such as a real name, your consultancy’s name if you’re independent, or your forum name, then they will see who you’re working for and what problems and weak spots might exist. The trouble associated with that has largely already been dealt with above.

Counter Argument: “That’s just silly.” No it isn’t. Nya nya nya you’re stupid and ugly. Or… something.

The Overarching Counter Argument to All of This:

“But still, you’ve got larger problems if information like that can lead to a break in!”

I disagree. Everyone has problems. Every grouping of infrastructure can be compromised, broken into and owned given enough time and effort. Every last thing. Every speck of code, every network appliance, everything that has a network presence and anything that plugs into an electrical socket can be compromised and turned against an organization.

The goal is to reduce not only the possible targets of attack, but to reduce the known information about the existing targets. “Security through obscurity” has gotten a bad reputation that isn’t universally deserved.

Don’t leave behind a public trail of problems that you have been working on solving that have a direct connection to a specific company. I realize it’s already easy enough to check someone’s LinkedIn profile for work history and then correlate the person to forum accounts. However, hanging it out directly in a forum post is just more low hanging fruit that is unnecessary.

This isn’t always easy to practice, and certainly public facing services like SaaS, Iaas, and Whatever-aaS do get a customer-relationship benefit from being open and transparent with what they’re building and the products that are being used to build it, however those scenarios seem to be edge cases. In most cases, privacy is preferred.

Disclaimer

I am paranoid.

What Say Ye?

The TL;DR summary of this post would be: “Don’t post info that can reveal who you’re working for. Don’t ask for info that could reveal who a person is working for. Don’t believe the fallacy that because public DNS records are public that it’s okay for an author to include it in their forum post.”

Do you agree? Am I unreasonably paranoid? Support or slay me in the comments below.

Have you heard of the term “Brogrammer?” There’s a lot of controversy surrounding the term and the supposed culture. I say “supposed” because I’m still not sure that the culture truly exists in the form that it is caricatured as.

A quick Urban Dictionary definition of a “Brogrammer” is thus:

A programmer who breaks the usual expectations of quiet nerdiness and opts instead for the usual trappings of a frat-boy: popped collars, bad beer, and calling everybody “bro”. Despised by everyone, especially other programmers.

For now, suffice it to say that the term brogrammer is a facetious term for a probably-nonexistent type of person that is supposedly a cross between Stiffler and Ken Thomspon. There’s even a Brogrammer Quiz! Yes, that’s right, you can find out if you’re a brogrammer or not. In fact, let’s take the quiz together!

The Results – (not) Pretty?

So, apparently, I’m a brogrammer. What’s scarier is that I was not once asked if I wear pink polo shirts (I do).  I’m also seriously considering buying Aeroshot, getting designer sunglasses and I’m currently working on a six pack. I suspect that one year from now, I will become a caricature of everything wrong with IT. (Anyone willing to put a calendar item down, hit me up a year from today and see if I need to be batted in the kneecaps for becoming a BroAdmin.)

Now I’m curious what other people get as a score. Whether you’re a SysAdmin or developer it doesn’t matter. The questions are vague enough to where anyone with a basic background in professional information technology can do fine. Let me know your score in the comments below. Please, please, please tell me I’m not a freak with a score of 70 and a potential score of over 100. =(

Addendum

For those interested in some further research:

• Rob “Chad” Spectre’s definitive Brogrammer Primer.
• Quora’s thread “Brogramming: How does a programmer become a Brogrammer?”
• Bloomberg article from March 2012 titled “The Rise of the ‘Brogrammer’“
• Oh, but Gizmodo says “There’s No Such Thing as a Brogrammer“

SolarWinds has announced their “Tales from the Trenches” System Administrator Appreciation Day contest. It has been launched in honor of the 13th annual SysAdmin Appreciation Day started by Ted Kekatos. System Administrator Appreciation Day is July 27th this year, for those not in-the-know.

This latest SolarWinds contest is pretty simple. Share a “trench-born tale” about your job as a systems administrator and a hand selected panel of Systems Administrators will cluck over it like mother hens with codependency issues. You will have until July 27th 2012 to submit your tale. Once the judges have determined a winner, the prize for the best tale will be an iPad, a ThinkGeek gift card or an original piece of artwork depicting the winner’s story!

But just who are these SysAdmin judges who will be scrutinizing the hopefully-not-too-tall tales? They are:

• Ted Kekatos – Mr. SysAdmin Day himself!
• Denny LeCompte - VP of product management at SolarWinds
• Trevor Pott – SysAdmin writer for The Register
• Matt Simmons - The Standalone SysAdmin of much repute
• Me – Yes, I managed to charm my way onto the judging panel somehow.

Oh but there’s more! The SolarWinds Thwack community will vote on the best tales as well. The top four community-voted tales will each receive a $50 gift card to ThinkGeek.

Some Rules of Note

There are some rules that readers of my blog should note. In fact, make sure to read the entire official rules page just to be safe. One of the rules which might be a disappointment to many of you (since my audience is rather international) reads thusly:

The SolarWinds ‘Tales from the Trenches’ System Administrator Appreciation Day Contest (the “Contest”) is open only to legal residents of United States who are eighteen (18) years of age or older.

So USA tale-tellers only. Sad International Panda.

Let’s move on to some of the other rules that seem to be of special importance (note that this is not a full list of the rules. Makes sure to check out the official rules page or don’t say I didn’t warn you!). Of course, be punctual, like all good SysAdmins are!

The Contest will commence on 6:30 a.m. CST on Wednesday, June 27, 2012, and an Entrant must enter by posting a comment in the appropriate thread on thwack® community forum prior to 11:59 p.m. CST on Friday, July 27, 2012 (“Entry Period”).  Entrants submitted before or after the Entry Period will not be eligible for the Contest.

Also, you’ll need to sign up on Thwack:

Entrants must have a thwack account and must submit their best original story about their experiences “in the trenches” as a SysAdmin on thwack during the Entry Period. Entries should be 500 words or less and should include a subject line..  By submitting the completed survey, you will receive one (1) entry into the Contest.

Tactfully Tell Your Tale of Torturously Tortuous Technology Trials

Crack open your text editor of choice, start putting your terror-tinged tales into words and submit your story here. At worst you’ll experience the catharsis of sharing your woes, at best you’ll be richer some cool prizes. Share the news with some of the social media icons on my blog and let’s get together for some group session therapy. It’s okay to cry. No one will judge.

We’ve all been there. Googling around for an answer to a problem we’re having. We stumble upon a forum post that mirrors our dilemma exactly. Various remedies are suggested, and we furrow or brows in consternation because we’ve already tried what is being suggested.

And then we see it. At the very end of the thread. And it makes us cry.

“Thanks every1. It’s fixed.”

Are you kidding me? It’s not like they just forgot about the forum thread. They remembered enough to come back, log in, and submit a post. Would it kill people to simply add one sentence that gives even the slightest hint about what solved their problem? Just a simple “applied a udev patch” or “syntax error in script” would be better than “It’s fixed.”

So to all Good Guy (or Girl) SysAdmins out there, thanks for posting back with the full solution. Even more heroic are the people with the exact same problem as the original poster, and who revive the dead thread to post their solution. You are truly wonderful people. Three cheers for active community members who either participate on forums, Q/A sites or write blogs. You’re Good Guys (and Girls).

(For those not versed in the world of memes, this is based on Good Guy Greg.)

TechMentor is a yearly conference designed for information technology professionals that largely work with Microsoft technologies. Last year I mentioned the TechMentor event that took place in Las Vegas. This year, TechMentor 2012 will be taking place at Microsoft HQ in Redmond Washington! The date is August 20-24.

Some of the big names showing up to the event will be Don Jones, Mark Minasi, Bruce Rougeau and Greg Shields. Some of the listed topics that will be dsicussed are Virtualization (assuming a heavy focus on Hyper-V), Application Delivery, MCITP Certification (and hopefully some demystifying of the new MCSE/MCSA certifications), as well as the ubiquitous PowerShell. Actually, if you didn’t have an idea that PowerShell would be a topic just from seeing the name “Don Jones” then you’re not familiar enough with PowerShell. =)

I’ve been given the opportunity to offer readers of this blog a $300 discount off of the 5 Day Best Value Package rate (discount applies to the Standard rate and new registrations only). FYI, this is at no benefit to me; I get no kickbacks or affiliate-anything.

To claim that $300 discount, simply use this link (or paste this URL into a new browser tab: http://bit.ly/TMRDReg) and use the code TMRTU when signing up.

Let me know if you plan on making it to the conference! I’d love to have you guest blog about your experience.

A Phoenix, Arizona IT professional who I am familiar with has launched an indie funded project that any techie who comes in contact with physical infrastructure will appreciate. It a Keychain Punchdown Tool! Take 0:36 of your time to see what you think.

There is an Indie GoGo project for the keychain punchdown tool. It will work on both 66 and 110 blocks, fit in your pockey and make you the life of every party! (Two of the three things in the preceding sentence were true). The funding goal for the project is $2,000 and there are four funding tiers: $1USD, $10, $50 and $75.

If you’re an IT pro that’s on the go and you use punchdown tools frequently, consider giving a few dollars to this indie project. Please share this with anyone who might be interested either through this post and the social sharing options to the side and below the post or through the Indie GoGo page itself.

I saw this infographic recently concerning Microsoft’s stagnation and Apple’s phenomenal growth in the last ten years. I regard myself as rather agnostic concerning brand wars and really don’t care what technologies a person or company uses, just so long as it works well. However, it might be interesting to note the lackluster decade that Microsoft has had. The Zune, Windows Mobile and Bing are all products that Microsoft as invested heavily in (especially Bing) and yet nothing much has come of any of them.

Clicking the infographic takes you to the site “MBAOnline” for a look at the larger graphic.

Created by: MBAOnline.com

I do see some good tools and products that Microsoft has made in the last ten years, but those are mostly systems tools. Aside from the Xbox, I can’t find a major consumer product that has taken off and set Microsoft apart in any space. Am I missing something? Anyone have a contrary opinion they’d like to share? Let me know below.

My Problem

I have an HP ProLiant N40L MicroServer that needed its BIOS upgraded. I downloaded the proper firmware update package from HP’s support site which includes ROMPaq and the update flat files. I created a bootable USB thumbdrive using the ROMPaq utility, however attempting to update the server’s BIOS receives this error:

RomPaq may take a few minutes to get started, please be patient...
Command or filename not recognized

My Solution

Open the SWSetup folder that the ROMPaq installer creates and copy all files from the “Flat Files” folder over to the USB drive that the ROMPaq utility modifies to be bootable.

The Long Story

After the “Command or filename not recognized” error was received, I popped the USB drive into a different machine and inspected the contents. Looking at the AUTOEXEC.BAT file that is on the root showed this:

@echo off
echo ROMPAQ may take a few minutes to get started. Please be patient...
rompaq.exe /l:us /!

However, there was no rompaq.exe file anywhere on the drive! Furthermore, there wasn’t even a ROM file to be seen. Clearly the utility did not create a drive that was capable of flashing my server. Part of the unpacked files included a folder called “Flat Files” that included things which made more sense. For example, in that flat files folder was an AUTOEXEC.BAT file that looked like this:

@echo off
 
if errorlevel == 1 goto ENDIT
if errorlevel == 0 goto other
:other
echo Next Please!!!
flash.bat
 
:ENDIT

As well as a flash.bat file that looked like this:

kbd /"flash O41072~1.rom -r 100000,10000 -r 1e0000,10000;q"

And finally an actual ROM file that matched filenames with the line in the flash.bat file. I merely copied all the files in the “Flat Files” folder over to the USB key and chose to replace any existing files.

Apparently HP’s QA group let this get past them. The tool and documentation explicitly state that it will, once run, leave the USB media in a state that can be used to directory update the ProLiant firmware. No additional steps should be needed, according to the documentation. That is clearly not the case in this scenario.

Ever had a similar experience with ROMPaq or is this an oddball case? Let me know in the comments.

Plesk has some baked in magic concerning MySQL and the admin account. If you try to use mysql or mysqldump using your root user name or password, it will not be allowed. You’ll see something like this:

[root@server] mysql -u admin -p
ERROR 1045 (28000): Access denied for user 'admin'@'localhost' (using password: YES)

If you want to access MySQL as root, you simply use the command my. What exactly does my do?

[root@server] type my
my is aliased to `mysql -A -u admin -p`cat /etc/psa/.psa.shadow`'

So, in order to use mysqldump, one has to use the following command syntax:

mysqldump -u admin -p`cat /etc/psa/.psa.shadow` [database] > [outfile-name]

Strange but true. Of course, this is for a vanilla installation of Plesk. I’m sure you can customize your MySQL permissions and groups to behave differently, however I would not advise that. Plesk likes to have things its own way, and it will either break if you change things or set things back to how it likes them on the next update.