(Today’s post is written by guest blogger Scott Pack!)
I have a bottom of the line box sitting on my desk that I use for some testing. It is a hardware clone of the oldest Snort sensors I have deployed, and by old I mean corporate desktop grade vintage 2003. I have it running and set up so that I can test configuration changes, new rules, software updates, etc. This all makes the system pretty mundane and cookie cutter. I’ll often get a hankering to start from a blank slate, tell this thing to go koan itself, and come back to a clean install. This is all a rather long and drawn out way of saying that I don’t really care about the health of this system and don’t pay very much attention to it.
This morning, in the email containing the output from the auto-update script for this system, I saw gobs of errors of the form:
Error unpacking rpm package 4:perl-5.8.8-38.el5_8.i386
error: unpacking of archive failed on file /usr/bin/a2p;509097e4: cpio: open failed - Read-only file system
If there’s one way to catch my interest, it is to tell me that system partitions are read-only. Some quick research showed me that yes, every single file-system was in fact read only. SMART also was showing errors out the wazoo. Over the years I have learned that SMART’s false-negative rate was astronomical, but its false-positive rate was approaching zero. That is, a “healthy” report from SMART was meaningless, but a “failed” report is completely trustworthy. This was really no big deal, since the system wasn’t used for anything time sensitive, I could just pull a hard drive off the shelf and reprovision.
Since one thing I had used this system for was to do some performance profiling of snort, I had mocked up a couple of analysis and test scripts on it. The scripts themselves were easily rebuildable, but to save me the effort I used the old tar+ssh trick to archive the home directories for myself and root for later extraction:
tar -zc /home/packs /root | ssh [email protected] ‘cat - > snort-test_homes.tar.gz’
This is where things got hairy. Since the last time I had used ssh to go to node1 it had also been rebuilt, resulting in a host key change. With StrictHostKeyChecking enabled ssh refuses to connect if there is a host key mismatch. Ordinarily, I would just delete the key from the known_hosts file and move on. With the file-system being read only….
I worked around this by changing my known_hosts file on the command line. Since all the file-systems were read-only I couldn’t actually write any files, nor did I need to save the information. This left me with the perfect choice of
/dev/null. Adding in this option made my final command look something like this:
tar -zc /home/packs /root | ssh [email protected] -o UserKnownHostsFile=/dev/null 'cat - > snort-test_homes.tar.gz'
I liked this because it worked and was easy. I don’t like it because it felt skeezy.