Archive for 'SysAdmin'

In a recent post named The Making of a Meta Server or “Why I Bought a Mac Mini as a NMS” I explained why I had chosen a brand new, 2012 Mac Mini as my NMS hardware. After two weeks of mind numbing work, I have officially declared the Mac-Mini-as-a-NMS project a failure.

The main problem surrounded Apple’s custom EFI. Apple hardware does not use a BIOS, but instead uses EFI (note: not, specifically speaking, UEFI). Or rather, it uses an ancient, bastardized version of EFI 1.1. There is a BIOS compatibility layer that allows OSs that can only communicate with a BIOS to operate on the hardware. Most notably Windows. Apple’s OS also runs on a hard disk that has been partitioned using the GPT partitioning scheme, which isn’t itself a huge deal, but you might be surprised at the anemic support for GPT boot disks in even modern operating systems.

To use the Mac Mini to boot an OS that needs BIOS compatibility and a MBR disk should be relatively easy. Right? Right!

Unless Apple is involved.

There are several things that Apple has mutated away from the EFI standard, one of them being not using the EFI system partition for anything except firmware updates. Their custom EFI implementation has the boot process (as well as some extra filesystem drivers) baked in. The whole EFI experience just never worked like I expected it to. The other trouble is that Boot Camp has been changed in OS X Lion. If you wanted to be hand held through the partitioning process and the creation of a hybrid GPT/MBR disk, you’re invited to use Boot Camp. However the latest alterations only allow media with Windows images to be accepted. You can no longer (from my ability to understand) use Boot Camp to install non-Windows OSs. Of course, it was always unsupported, but at least it was doable.

During the whole process, I used the EFI boot manager rEFIt which apparently only recently works with OS X Lion. I read more about the GPT partitioning scheme than I ever have previously. I learned more about EFI than I ever wanted to know (although all of that information will come in very handy in the near future). I hand-rolled bootable USB thumbdrives. I tweaked partition tables. I did very nearly everything I could think of except rolling my own EFI boot partition. After the hours had steadily ticked away I decided it was no longer worth it.

After countless errors concerning boot media, partition problems, and blinking cursors, I concede that the latest Mac Mini has defeated me. It has been shipped back to Amazon and I can go back to my Apple-less existence. Speaking of Amazon, I believe that they deserve some praise in this.

Amazon made the returns process easier than any return I have ever made. Anywhere. I stated that the reason I returned it was because software I had intended to use with it was not compatible. As a result of the return not being their fault, I had to pay return shipping. Within just a few clicks, Amazon created a return label. I printed it out, boxed the mini up, taped the label to the box and handed it over to the man behind the UPS Store counter. Within 15 seconds I was walking out of the store. I have the fortune of living just a few hundred miles from an Amazon return center located in the Las Vegas area so the return was processed and money credited back within two days. Thank you, Amazon. You were the only bright spot in this debacle.

I am now investigating other pieces of hardware for this project based on the recommendations of several colleagues. If you have a recommendation, share it with me and the rest of my readers in the comments below. I’ll certainly write about my second attempt at this project as it happens.

In the end, I’m not mad. The Apple wasn’t designed to do what I was asking it to do. It was my fault. My only lingering frustration is that the Mac seems to take any standard technology that it uses and twists it in new and different ways so that your familiairty with a standard becomes more of a liability than an asset. Sound like another familiar company that SysAdmins like to pick on? Then again, Apple isn’t intended to be in the business market. Let us pause and mourn the passing of the Xserve (I handed my G5 Xserve over to Best Buy for free recycling last year. So, so sad…).

Any similar experiences with an Apple product? Have you managed to wedge an alternate OS on 2012 Apple hardware? Let me know in the comments below.

I have had to repeatedly jog my faulty memory concerning how to set up a SOCKS proxy. Thus, I am using this blog as my public sticky note to remind myself. Specifically, I forget what options I like to pass through the SOCKS aware SSH client when connecting to the SOCKS aware SSH server.

A SOCKS proxy is easy to set up. It’s nothing more than an SSH server and an SSH client that speak the SOCKS protocol. In my case, I use OpenSSH. If you use a different SSH server or some other form of making a SOCKS proxy, this little post will be of little use to you. However, stick around because there’s a note down below concerning the false sense of security many people have when using a SOCKS proxy.

There’s a handy little option in the OpenSSH client that allows for the creation of a a local port binding that immediately forwards traffic to that port to another machine: -D. After that part of the command, simply include the username and host for the OpenSSH server that you want all local traffic bound for that local port to be relayed through. It makes it all the sweeter if you have RSA keypairs set up between hosts.

In my case, I usually use this set of options:

ssh -fCND localhost:8080 user@myserver.thenubbyadmin.com

Let me peel back those other three options that I use:

• -f sends ssh to the background just before the command is executed.
• -N refuses to execute remote commands. This way I know nothing is going to be run via the SSH connection on the remote machine. I’m paranoid.
• -C compresses the TCP traffic. This might not be ideal if you have a good connection as it is stated in the man pages for the OpenSSH client that -C slows down your throughput on fast connections.

Application Support

The applications that you want to use with the SOCKS proxy need to have explicit options to support it. It’s not something that can be done underneath the application without its knowledge. For example, most web browsers have an option to use a SOCKS proxy within their advanced options section.

You will want to go to the options page of your application and search for SOCKS support. From there, tell the application to use localhost:port# as the proxy. In my case, I made port 8080 to be the local port that listens for traffic and then forwards it to my remote server.

If you need a secure connection that can be put in place without an application’s knowledge, you’ll need to implement a VPN.

You’re Not as Anonymous as You Think You Are

If you’re using the SOCKS proxy for the purposes of secure browsing, know that your DNS requests are an entirely different application layer traffic. Unless your DNS client is also set up to use the SOCKS proxy, your DNS requests will be plainly visible on the network that you are trying to remain anonymous / protected on. This can cause problems if you’re on an untrusted network. Owning the DNS servers that a machine is using is one of the most sure ways of wreaking havoc.

Have any other SOCKS tips? Do you use a different client or server? Let me know in the comments.

When making a bootable USB drive to install CentOS 6 from, you will need two primary partitions, one of them marked with the boot flag. One partition will be the boot partition and one will be a data partition that has the ISO file on it. As of this blog post, to make a CentOS 6 bootable USB drive, you’ll need a USB drive that has a little more than twice the space that your ISO file itself takes up. There is a bug that requires the ISO’s contents to be on the boot partition and the .iso file itself to be on the data partition. In essence you’re duplicating the ISO file and you still need some space left over for bootloader information. In my case, I’m using the minimal CentOS image, so space requirements are under 1GB.

At this point, go out and grab the CentOS ISO that interests you. Have it on your filesystem because we’ll be mounting it and copying some files from it. Once you’ve got the ISO you can move on to partitioning the drive.

Partitioning

First, you’ll want to partition the USB drive. We’ll be using plain ol’ MBR style partition tables and two primary partitions. I’m not going to hand-hold you through this part of the process. Use whatever partitioning tool you want and follow the guidelines below. GParted is fine if you use Gnome, parted is great if you want to use a shell, and fdisk works on both Windows and *NIX environments.

The partition layout will be thus:

1. A primary partition that uses the FAT16 filesystem and is at least as big as your ISO plus about 50MB. You need to give it the boot flag.
2. A primary partition that uses ext2 and is at least as big as your ISO. Preferably you’ll just use up the rest of your USB drive’s free space for this partition.

Once your partitions are set up, we’ve got some file moving to do.

Setting the Filesystems Up

You’ll want to mount your two partitions so that you can access them. In my case, the first partition (the FAT16 boot partition) is /dev/sdc1 and the data partition (the one formatted in ext2) is /dev/sdc2. I’ve mounted sdc1 as /mnt/usbboot and sdc2 as /mnt/usbdata. I will be using that nomenclature throughout the rest of this post.

You’ll also want to mount your CentOS ISO as a filesystem because we need to copy some files off of it. In my case, I ran mount -o loop /path/to/iso/file.iso /mnt/centosiso and will be using /mnt/centosiso in my examples below. Now that we’ve got all of our filesystems mounted, we’ll start the procedures.

First, go to the mounted CentOS iso and copy the /isolinux directory to the boot partition of the USB drive.

cp -r /mnt/centosiso/isolinux /mnt/usbboot

Rename the isolinux folder on the USB drive to syslinux

mv /mnt/usbboot/isolinux /mnt/usbboot/syslinux

Rename the isolinux.cfg file to syslinux.cfg

mv /mnt/usbboot/syslinux/isolinux.cfg /mnt/usbboot/syslinux/syslinux.cfg

Now we need to copy the contents of the /mnt/centosiso/images folder to the USB boot partition. Notice that I emphasis that this is a copy of the contents within the ISO’s images folder. A little later on we’ll be copying over the entire ISO as a file.

cp -r /mnt/centosiso/images /mnt/usbboot

Finally, we copy the .iso file itself to the data partition (not the boot partition that we were just working with!):

cp /path/to/iso/file.iso /mnt/usbdata

Once all that is done, we have to install a bootloader. I’ll use the simple syslinux loader. We want to use our smaller volume (the one that we set the boot flag on up in the partitioning section) as the target for the syslinux command.

syslinux -i /dev/sdc1

Now, we dismount our USB drive and test it out by booting from it on another system!

 

---

Finoto!

You should now have a bootable CentOS 6 USB drive. CentOS 6 is somewhat unique as a result of the bug that requires the images directory to be included on the boot partition, but other than that it’s relatively straight forward.

To find out what version (and often build number as well) of OS X you’re running using the Terminal, you’ve got several options. The first and likely the most obvious (and simple) is using sw_version.

macmeta:~ user$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.7.3
BuildVersion: 11D50d

You can retrieve the specific information that you want using the following options: -productName | -productVersion | -buildVersion

macmeta:~ aoi$ sw_vers -productVersion
10.7.3

You can cat out /System/Library/CoreServices/SystemVersion.plist and eyeball the XML that comes back. In my case:

macmeta:~ user$ cat /System/Library/CoreServices/SystemVersion.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>ProductBuildVersion</key>
	<string>11D50d</string>
	<key>ProductCopyright</key>
	<string>1983-2012 Apple Inc.</string>
	<key>ProductName</key>
	<string>Mac OS X</string>
	<key>ProductUserVisibleVersion</key>
	<string>10.7.3</string>
	<key>ProductVersion</key>
	<string>10.7.3</string>
</dict>
</plist>

Or you could merely add grep -C 2 ProductVersion.

macmeta:~ user$ cat /System/Library/CoreServices/SystemVersion.plist | grep -C 2 ProductVersion
	<key>ProductUserVisibleVersion</key>
	<string>10.7.3</string>
	<key>ProductVersion</key>
	<string>10.7.3</string>

Using system_profiler SPSoftwareDataType gives you lots of information including kernel version:

macmeta:~ user$ system_profiler SPSoftwareDataType
Software:
 
    System Software Overview:
 
      System Version: Mac OS X 10.7.3 (11D50d)
      Kernel Version: Darwin 11.3.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: macmeta
      User Name: USER (user)
      Secure Virtual Memory: Enabled
      64-bit Kernel and Extensions: Yes
      Time since boot: 56 minutes

Those accustomed to using ye olden *NIX uname -a will find that it only gives the Darwin kernel information.

macmeta:~ user$ uname -a
Darwin macmeta.tc.ph.cox.net 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64

And I saved the ugliest for last! osascript is a tool that can be used to run AppleScript. with the -e option, it will run a single line script that you enter. Let’s try osascript -e 'system info':

macmeta:~ user$ osascript -e 'system info'
AppleScript version:2.2.1, AppleScript Studio version:1.5.2, system version:10.7.3, short user name:aoi, long user name:AOI, user ID:501, user locale:en_US, home directory:alias Macintosh HD:Users:aoi:, boot volume:Macintosh HD, computer name:macmeta, host name:macmeta.tc.ph.cox.net, IPv4 address:192.168.11.119, primary Ethernet address:40:6c:8f:0d:48:a6, CPU type:Intel 80486, CPU speed:2300, physical memory:2048

As you can see, it brings back a ton of info, including the system version.

Know of any other ways to find the system version and build numbers of OS X? Let me know in the comments.

There comes a time when it is expedient to know some version information about the EFI firmware running on Apple OS X. In my case, I just wanted to make sure that I had EFI64. I knew I did, but… seeing is believing.

As of OS X 10.7.3, to find out information concerning what version of Apple’s EFI firmware you are running, perform the following command at a terminal:

ioreg -p IODeviceTree -b -n efi | grep efi -C 4

In my case, the output is as follows:

+-o Root  <class IORegistryEntry, id 0x100000100, retain 10>
  +-o /  <class IOPlatformExpertDevice, id 0x100000110, registered, matched, active, busy 0 (15221 ms), retain 37>
    +-o chosen  <class IOService, id 0x100000101, !registered, !matched, active, busy 0, retain 5>
    | +-o memory-map  <class IOService, id 0x100000102, !registered, !matched, active, busy 0, retain 6>
    +-o efi  <class IOService, id 0x100000103, !registered, !matched, active, busy 0, retain 8>
    | | {
    | |   "firmware-revision" = <0a000100>
    | |   "device-properties" = <fe04000001000000040000000e0200000500000002010c00d041030a000000000101060000027fff04001e00000073006100760065006400$
    | |   "firmware-abi" = <"EFI64">
    | |   "name" = <"efi">
    | |   "firmware-vendor" = <4100700070006c0065000000>
    | | }
    | | 
    | +-o kernel-compatibility  <class IOService, id 0x100000104, !registered, !matched, active, busy 0, retain 4>

And the text that I’m most interested in is “firmware-abi” = <”EFI64″>

To explain the above, ioreg is used to query the Apple I/O registry. We then need to select the Device Tree plane and do so with -p IODeviceTree. -b is just a nicety to put the object name in bold letters. -n scopes the query down to only those things that have a certain name in them. In our case, we want the name of “efi”.

We then pipe the whole mess to grep where we search for the word “efi” and then, with -C, give four lines of context above and below each mention of the word “efi”.

Simple!

My Problem

I have a new Mac Mini running OS X Lion that I need remote access to. I’ve enabled “Screen Sharing” in Sharing Preferences, created a VNC password and ensured that Screen Sharing is allowed through the firewall.

Using various VNC clients, I receive different forms of bizarre errors and refused connections. In Remmina on Fedora 14, I cannot connect to the Apple VNC server. Either a connection is made and I see a very brief flicker of a remote connection before the window is closed or the connection perpetually hangs at the “Connecting…” stage.

Using UltraVNC on Windows 7 I receive the error:

Server closed connection

- the server running as application

My Solution

Edit your bit-depth settings in your VNC client (not on the receiving Apple computer). Change the bit depth to something other than 256 colors or 32-bit. For example, I can connect to the Mac’s VNC server using 15, 16 or 24-bit color depths. As soon as I choose 256 colors or 32-bit I am unable to make a connection.

I suppose Apple wants their GUI to be enjoyed in its full glory and 256 colors is just too shabby. 32-bit? Well, that’s not minimalistic enough.

There’s a small office that I’ve done consistent work with for the last five years. I’m their contracted IT director on a part-time schedule. Anything that could even loosely be called “technology” is up to me to understand, explain, and make work. More than just a technology worker, I have to understand the business’s goals, mission and workflows first and foremost to then be able to profitably apply technology. I’ve submitted many board reports over the years and had to counsel the director and president on more technology related topics than I can ever hope to remember.

However, I’ve been a bad IT person. I haven’t proactively monitored their IT assets. I can make any number of excuses for myself, but none really mollifies me (nor would it satisfy anyone else with even just a hint of a desire to do a job right). An interesting fact is that some of my very first attempts at blogging (back in 2007 or 2008) were as a result of my attempts at making a monitoring system for this organization. That blog, a shared Drupal CMS between me and a friend, is long gone. However the nagging need for a thorough “Meta Server” has haunted me ever since.

The organization is suddenly expanding into a new field for them. They have high hopes and are tackling this new project head on. Their exact growth potential is unknown at the moment, but I want to put the long needed information reporting infrastructure in place now before much more moves forward. A few new websites will be made for the organization’s endeavors and both will be a large part of the success (or failure) of this new phase of growth. If I don’t know about their website / server / PC infrastructure health before I get panic-stricken phone calls, then I can consider myself a complete failure as a SysAdmin.

“What is a Meta Server?!”

A “Meta Server” is what I like to call any node whose sole purpose is to collect and display information that is largely of no interest to a standard user. You’re unlikely to see the term anywhere else because I just made it up (or if it is in use elsewhere, please, whoever came up with it, don’t sue me).

It’s the kind of thing that is where notes are stored, wikis are stood up, NMSs sweep, trends are graphed and bitknobs are virtually twiddled. In larger places, you might have a “Meta Rack” but I’ve never worked in an environment so large as to need stacks of meta servers. Oh if only…

My intention for this server is not to be just an NMS, in spite of the title of this blog post. It’s much more than that. Let’s take a look at what my plans are for it.

The Software

My plans for the meta server are many. I want monitoring, alerting, trending, help desk, asset/inventory management, log collection, imaging, perhaps a wiki… lots of stuff. Since it’s a small office with little need for a multitude of servers I can’t separate these roles out onto difference pieces of hardware, nor do I really need to unless there’s some glaring incompatibility between packages. Even if there was some kind of package incompatibility between tools, I’d prefer to just use Linux-VServer or something similar to stand up a virtual instance.

To expand on the list of topics just above, I’m very interested in the health of the network itself as well as the nodes on the network. I want to scan the network for devices and get alerts when new things show up. I want to poll each device, PC, laptop, printer, server, WAP, modem, switch… you name it, for vital statistics. That can be through SNMP, netflow, sflow, or an agent installed onto the operating system (in the case of a PC with a full OS on it). I want speed and latency statistics for our ISP connection too. This bundle of requirements necessitates probably three, maybe four separate tools.

I haven’t settled 100% on the applications that I’ll be using, but I have a pretty good idea that either OpenNMS or Pandora FMS will be the main monitoring and alerting system. For prettier graphs to look at, Observium is high on my list. I might use ntop for netflow analysis, rTPL for scheduled throughput tests and smokeping for latency monitoring. Munin may play a part as well; I haven’t decided yet.

For log collection, I’m interested in Splunk‘s community edition, but graylog2 is appealing as well. I’d probably use the Snare Agent for Windows to collect logs from my Windows hosts and send it all to graylog2 – if that’s the direction I go. However, Alien Vault’s OSSIM is also in the running.

For a help desk, I’m almost 100% sold on RT. I’m currently using SpiceWorks‘s help desk on a Windows server, but that’s a bit heavy for my needs. I don’t use most of its other management tools. It’s asset management and network monitoring is… okay. It’s a bit rigid for my tastes, however.

I’m also interested in having a simple PXE boot imaging tool on the network. I have long been a fan of the FOG project. This goal of mine isn’t to start creating an extensive image library. Instead, I just want to take the occasional quick image of a PC before a major change and also to be able to boot a PC over the network onto an anti-virus image to perform an offline virus scan. I’ll need some decent storage space to keep a few images around depending on which user’s PC may need to be quickly backed up. A few hundred GBs would be nice.

I’m considering the use of Monit for some automated response, but since I don’t have many *nix devices to contend with, that might be wasted effort. Then again, automating things is never wasted effort!

There might be a documentation wiki thrown in there for good measure. Currently I use the hosted wiki service of Zoho, but I am considering moving it in-house.

It has occurred to me that such a small device may be a prime target for theft so I’m considering volume encryption to protect the data. If someone wants the hardware bad enough, they can have it. However, I don’t want them to have any valuable data to play with. I’m sure it’s very unlikely that a smash-and-grab thief would have the interest or skills to do much with the data, but… still.

The Hardware

First, let’s clear something up. As often as I refer to this needed device as a “server” it is not, in fact, a “server” by any enterprise understanding of that word. It is just a device that serves, but is not intended to be made up of components that are traditionally thought of as “server grade.”

Since it’s a small office, it doesn’t have a proper server rack. It has more of a closet than anything. The closet is better than the “Just set that server down next to my desk; it’ll be fine, my door locks!” situation I found myself in years ago. The device needs to be small.

I’ve been collecting a large list of small form factor PCs for quite some time now. I like small things, especially when the alternative is cramming a 6 year old workstation next to the building’s demarc point to act as a caching proxy (I bet you’ve done that). I’ve got to narrow the market of small PCs down to a manageable pool using some base requirements.

My goals for the hardware are the following:

1. I don’t want a development board. Things like the RaspberryPi or the Hawkboard won’t cut it. I want a production piece of equipment that is manufactured in decent quantities and is intended for a broader consumer base than hardware hackers and developers. The BeagleBoard is on the edge of that grouping because of its wide acceptance, but it’s still iffy. I’m sure some will say “Oh but the Hawkboard…” or “Hey, the BeagleBoard does…” and that’s fine. It’s just that in my research, they don’t seem to be dead ringers for well supported, hard-working devices.
2. At least 250GB of internal storage. I don’t want to deal with CF cards like many micro-ITX boxes use. They don’t have enough storage and regardless of how advanced the wear leveling algorithms are, I don’t want to worry about block wear on a device that already has precious little space. NAND flash is also prone to soft read errors that makes you reliant on the ECC of the card itself. That’s too many variables for my comfort at this moment. Also, I want the storage to be internal as a matter of preference. I’m trying to get away from a “Just attach another USB hard drive!” mentality. That and I just don’t like USB buses as a rule.
3. At least 2GB of RAM. I’m going to have a lot of daemons running and I’d prefer to take a RAM ceiling out of the equation.
4. Price: I want to keep it around the $500 mark. If it can happen for less that’s great.

Over the years of keeping this project on the backburner and also considering building other similar meta servers, I’ve looked at NetGate cases, the Fit-PC, BeagleBoard-based systems, SlimPRO, Pearl D series, and various models of thin clients and plug PCs. I do have a soft spot for plug PCs, but none of them meet the above criterion.

The Final Choice

After searching far and wide, the one PC that kept coming to the top of the pack was the Fit-PC. I’ve had my eye on it for years and watched as its design have iterated past version 1, through version 2 and now on to version 3. It’s a handy little thing (form-factor pun intended) that has some decent resources. The latest version can include a 250 GB platter hard drive with 2GB of RAM and a 1GHz APU G-T40N processor. The price is a tad steep at $480 plus VAT and shipping.

I mulled the option. After shipping and tax, it would be over $500. Could I really justify that much money for what I was getting? Certainly the value of what I was going to do with it was worth it. I just… I wasn’t sure.

Then it hit me. A Mac Mini. The cheapest brand-new Mac Mini one can find (legitimately) is about $569 USD. And what does one of those shiny suckers have in it? 4GB of RAM, a 500GB hard drive and an Intel i5 processor. Furthermore, I can get it on Amazon.com straight from Apple’s store with no shipping fees and no sales tax. It has twice the RAM (2GB would probably suffice, but more is always better!), twice the hard drive space (very much useful considering the imaging server portion of the project) and an Intel i5 (could come in handy for encryption and report generating). Furthermore, love or hate Apple, their hardware is rock solid (iPhone recpetion issues notwithstanding).

For just a handful of dollars more, I get double the resources (more than double if you consider the CPU). I had been approved for a $500 purchase, but with just one five minute phone call later a $569 Mac Mini had been approved.

As for the OS, I am highly unlikely to be using OS X (highly). I’ll almost certainly put CentOS 6 on it and be on my merry way.

Retrospective

In talking with colleagues, I’ve taken a tiny bit of flak for making such an expensive NMS. Certainly, I think I could perhaps build a similar box for slightly cheaper, but without the i5. I’d likely need to use an Atom processor to keep the price down. However, the time for me to build the thing still costs my client money. Perhaps other solutions exist off-the-shelf with similar specs – but I wasn’t able to find them. Once again, research time costs.

In the end, I’m sitting here, a brand new Mac Mini still in its box next to me. I don’t have any regrets… yet. I’m eager to get this project going and hope to blog more about its progress, starting with the installation of CentOS on Apple hardware.

So what do you think? Did I blow it? Did I have other compelling options that I missed? Would you be happy with a Mac Mini “Meta Server?” Let me know in the comment below.

My Problem:

Attempting to install an RPM using the rpm -i command causes a series of errors:

error: open of <!DOCTYPE failed: No such file or directory
error: open of HTML failed: No such file or directory
error: open of PUBLIC failed: No such file or directory

The shell may hang and not return control to you.

It looks like it’s trying to parse an HTML document as a series of commands. Let’s think about that for a moment, shall we?

The Solution:

You are not attempting to install an rpm file, you are attempting to install a web page. Most likely either a redirect or 404 error page. Don’t believe me? Use cat to view the rpm file. In my case, trying to install rpmforge as a repo, I used curl to get what I thought was the proper rpm, however in reality I was retrieving the following:

$ cat rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://rpmforge.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm">here</a>.</p>
<hr>
<address>Apache/2.2.3 (Red Hat) Server at pkgs.repoforge.org Port 80</address>
</body></html>

Once you find the proper RPM file, I’m sure the installation will proceed without a hitch.

This post will be short and sweet. There is a list of network monitoring and management tools that I’ve gone to for years now whenever I feel the need to kill hours and hours of my life. It’s run by the Stanford Linear Accelerator Center (SLAC). Every year new tools are added. It’s really just a big text dump with some HTML tables that throw me back to the mid-nineties sipping coffee at 2AM while reading the Koala Book (this was well before XHTML was in the book’s title).

If you have ever had to work on a network, of any kind and for any reason, then you are probably going to want to look at this list (if you don’t already know about it). I disclaim any negative effects on your family life once you click through to it.

I give you: SLAC’s behemoth list of network monitoring tools

Enjoy and may you live to see the sun once more.

One of the perennial problems I see no matter where I work or who I contract for is mysteriously slow network speeds to or from a Windows machine. I’ve amassed quite a list of tips and tricks for addressing this issue, and now I’m listing them all here.

I won’t go into a thorough treatment of exactly what’s going on with each command and feature that is being enabled or disabled. I’ll leave that as an exercise for the reader. This is a quick-n-dirty jumping off point for deeper problem resolution procedures. Also, these troubleshooting steps are not offered in any particular order, with the exception of the first four which try and scope the problem down to hardware versus software.

With no further rambling, here is my list of tricks when trying to solve a slow network connection on a Windows host:

Start With the Physical Layer

It’s almost always the easiest thing to check and is more often the cause of problems than most people would suspect. Thanks to Pauska in the comment below for reminding me of this. Switch cables out, switch NICs if possible (I like to keep a USB NIC around for this), try different switch ports, wall jacks — everything. It’s quick work and can reap a quick reward. Plus, with the physical layer out of the way, you can trust the observations that you make in the software layer.

Boot From a Live CD

Remove the OS from the equation and see if you can isolate the issue to hardware. Grab a Live CD that has an OS on it with support for your hardware. Once you boot from it, perform some tests on the bandwidth to see if the problem still exists. If so, then you may be safer in assuming that the problem exists somewhere other than the operating system (unless the same configuration that’s causing the problem exists in both operating systems).

Search for Network Related Errors

Perhaps there’s a lot of collisions on the network or the network card is having a large amount of CRC errors. A quick way to see current TCP/IP statistics is to run netstat -s. Look for any interesting numbers that speak to receive errors or re-transmissions.

Use Performance Monitor counters to analyze error data live. If errors and re-transmissions seem unusually high, you have a jumping off point for further exploration.

Inspect Traffic with Network Monitor

Launch Microsoft Network Monitor or Wireshark (or whatever packet sniffer you prefer) and inspect the packet stream. There will almost certainly be a trail of information that can lead you to the ultimate problem. The trouble is: can you persevere to the end? It’s no easy thing to digest TCP conversations en mass.

In reality, this is where the root cause analysis will begin and often where it will end. However, if you want to flail at some network related options to try and narrow down the culprit, read on.

Disable Windows Network Task Offloading

Add a DWORD registry key titled ‘DisableTaskOffload’ with value of 1 to the registry hive  HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\.

Check that it currently exists and what the value is with the following PowerShell cmdlet:

get-itemproperty -path HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ -name "DisableTaskOffload"

Check the whole parent hive if you want:

get-itemproperty -path HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Create the new registry entry:

New-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\" -Name "DisableTaskOffload" -Value 1 -PropertyType "DWord"

Disable TCP chimney offloading

You will need to disable TCP offloading in the Windows OS as well as the hardware’s drivers, however we’ll talk more about disabling hardware offloading in the next point. By the way, TCP offloading only works if it is enabled both in Windows and in the hardware’s driver.

First, let’s check to see if any connections are currently offloaded to hardware using netstat -t

InHost means that the TCP connection is being handled… well… in the host. If there are connections being offloaded to the hardware, know that disabling this will wreak some havoc with them.

To determine the state of offloading within the OS, run the following at a command prompt:

netsh int tcp show global

Look at the state of the “Chimney Offload State” setting. If it’s enabled, disable it with the following command:

netsh int tcp set global chimney=disabled

Disable All Hardware Network Offloading

Now you need to inspect your network card’s capabilities. Go to Device Manager, open up the properties of the NIC and select the Advanced tab. Search for any options that reference offloading. TCP, UDP, checksum, whatever. Disable it. “But! But! Offloading roxors!!” I know, this is just for troubleshooting purposes. Once you figure out where the bottleneck is, you can start determining the root cause. That’s for later though.

Each card has different features and terminology, so I can’t be more specific. For now, just disable anything to do with offloading.

Disable Receive Side Scaling

Check to see if it’s enabled with the following command:

netsh int tcp show global

Disable receive side scaling with:

netsh int tcp set global rss=disabled

Disable NetDMA

Once again, check to see if it’s enabled with the following command:

netsh int tcp show global.

See if the registry key for the setting exists using PowerShell:

get-itemproperty -path HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters -name EnableTCPA

To disable it, create its registry key and give it the proper value. Using PowerShell:

New-Item -Path HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableTCPA
New-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\" -Name "EnableTCPA" -Value 0 -PropertyType "DWord"

Disable Autotuning

Check to see if autotuning is enabled with:

netsh interface tcp show global

Disable it with:

netsh int tcp set global autotuning=disabled

Uninstall Remote Differential Compression

Go to Add/Remove Programs or Programs and Features (run >> appwiz.cpl). Choose the option to turn Windows features on or off. Uninstall Remote Differential Compression.

More information about RDC can be found at the Wikipedia page on Remote Differential Compression.

Alter NIC and Switch Port Speed and Duplex Settings

First, document your NICs current link speed and duplex settings. Then document the switch port’s settings.

In Windows, go into Device Manager, open the NIC in question and go to the advanced tab. The exact naming of the property for the card’s speed and duplex settings will vary, but you’ll know it when you see it.

Auto-negotation can be a pain. Set your NIC to 100 or 1000 Mbps Full Duplex if possible. Continue to frob with the possibilities. Personally, I wouldn’t bother with half-duplex settings, but – as they say - any port speed in a storm!

Update your NIC Drivers

Sounds simple. Sounds stupid. It works. Do it.

Not only should you use the latest drivers, but also look for discussions concerning your network card and its performance relative to the driver version. Perhaps it’s an older driver that you need. See if you can track down older versions and try those.

Check for Third Party Security Tools

If an antivirus utility is set to scan live traffic for malicious payloads, that can negatively impact throughput. Check to see what security tools are installed on the node that is having throughput problems and temporarily disable any features that affect live traffic.

Reset the TCP/IP Stack

You know that you’re flailing when you start resetting the TCP/IP stack. Read more about the procedure in Microsoft KB299357. At an elevated command prompt, run the following command:

netsh int ip reset resetlog.txt

Reset Winsock2

To read more about the practice of repairing winsock2 corruption read Microsoft KB811259. To reset winsock, use the following command:

netsh winsock reset

Reset only the catalog with the following command

netsh winsock reset catalog

Note that if you are using Windows XP SP1 or earlier, you will have to manually reset winsock using the instructions in Microsoft KB811259.

Do you have any tips or tricks for a slow Windows network connection? Let me know in the comments below and I’ll include them here!