Archive for 'Linux'

EDIT: Yes, I mistyped the original post and put “March” as the month. It’s not March, it’s May. Thanks Iain. =)

The second CentOS Dojo is happening May 10th, 2013 in Scottsdale Arizona at the GoDaddy offices in the Scottsdale Airpark. Quoting from the CentOS website:

The CentOS Dojo’s are a one day event, organised around the world, that bring together people from the CentOS Communities to talk about systems administration, best practises in linux centric activities and emerging technologies of note. The emphasis is to find local speakers and tutors to come together and talk about things that they care about most, and to share stories from their experiences working with CentOS in various scenarios.

The location is at GoDaddy.com Studios 14455 N Hayden Rd #219 Scottsdale, AZ 85260:

View Larger Map

I will probably live blog the event as it happens. Talks are still being registered, so if you’ve got a talk that you want to give and are going to be in the Phoenix area, get in contact with CentOS project leader Karanbir Singh.

Some of the talks that I’m most interested in include:

• Metrics at Pinterest by Jeremy Carroll
• Leveraging CentOS and Xen for the Go Daddy private cloud by Mike Dorman
• Logging Love: Build a log analysis system with Logstash, Elasticsearch and Kibana by Rashid Khan

Check back at the official CentOS Dojo Phoenix page to see how the talks start lining up. Is anyone out there going? Let me know in the comments below.

(Today’s post is brought to you by the letters N, F, and S. Oh, and Scott Pack too.)

Let’s face it. NFS is a magical thing. It allows you to centralize your storage, share volumes across systems, and all while maintaining sane permissions and ownership. Unfortunately, it can also be a bit of a fickle beast. Let’s say you just had your volume configured and you set up the mounts. You go and run this command:

mount -t nfs 10.10.10.1:/vol1/fs1 /data

Works like a champ, you now have your data partition mounted over NFS. So you add this line to your /etc/fstab and make it mount automagically.

10.10.10.1:/vol1/fs1 /data nfs defaults 0 0

A few weeks go by and you apply a kernel update. No big deal, you apply the updates and during your next maintenance window reboot to apply the new kernel. Then you start to see applications failing and notice the volume isn’t actually mounted. This is an unfortunate result of the automounter subsystem.

It’s like this. At boot time the root partition gets mounted, automounter reads the /etc/fstab file, and boots any filesystem that doesn’t have noauto as a mount option. We’re still very early in the boot process so the network isn’t up yet, so naturally any network filesystems fail. The real problem here is that at no point does automounter go back and attempt to remount those systems. So your NFS mount points fail because there is no network, and done is done.

The developers were nice enough to provide a fix for this. There exists a mount option called _netdev. If we quote directly from the man page (sourced from RHEL 6.4):

_netdev
The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).

This is awesome, and exactly what we want. So you modify your entry in fstab to look like this:

10.10.10.1:/vol1/fs1 /data nfs defaults,_netdev 0 0

You’ve been bitten by NFS mounting in the past so you throw this in your test environment and reboot immediately. After the system comes up you notice a problem. Your NFS volumes are still unmounted. You see, there’s a bit of a hitch. Automounter followed the same procedure that it did before, except this time it didn’t even attempt to mount /data. The _netdev option doesn’t tell the system to mount the filesystem when network comes up, it says don’t attempt to mount it at all if the network isn’t up. There is still a missing piece to the puzzle. If you look at your init scripts there is a service called netfs. If you read the script you can see in the chkconfig header this description:

# description: Mounts and unmounts all Network File System (NFS), \
# CIFS (Lan Manager/Windows), and NCP (NetWare) mount points.

This is exactly what you need. It is a service whose sole purpose is to read your /etc/fstab and mount network filesystems. All you have to do is enable it

chkconfig netfs on

and watch the magic happen. Now your mount boot process should look something like this:

1. Automounter reads /etc/fstab
2. Ignores /data since it has _netdev option set
3. Mounts all other filesystems
4. Finishes mount jobs and allows system to continue booting
5. Network comes up
6. Service netfs started
7. netfs reads /etc/fstab and finds an nfs filesystem
8. netfs mounts /data

What’s funny is that while I was researching this problem I never stumbled across netfs as a service. I had even gone so far as to start planning out my own custom init script that would do exactly this, except specifically for my mount points instead of generalizing. It’s nice to see that I was on the right track, but even better that the tools already existed.

When using a *nix variant of one kind or another, you can press a certain key combination and be transported to another land. A land of magic. Or at least a fresh shell. If you’re using a GUI DE then pressing this certain key combination will take you from the land of pretty colors to a land of monochrome… unless you’ve gone all crazy with your bashrc file, but that’s for another post.

What is this strange key combination? It depends on your distribution, but it’s going to most likely be Alt-F1 (through F7) or Control-Alt-F1 (through F7). Before you go pounding that key combination, know that your desktop environment is likely on the F1 or F7 terminal. If you get whisked away from your sparkly world of windows and menus, but you want to come back, just press Control-Alt-F1/F7.

The concept is known as “Virtual Terminals” or “Virtual Consoles” and I suggest that if you’re not familiar with it, that you read up on it. Especially if you’re considering getting your RHCSA like some friends and I are. Oh, and make sure to check out the Stack Exchange thread of epic proportions titled “What is the exact difference between a ‘terminal’, a ‘shell’, a ‘tty’ and a ‘console’?” as well as the nearly-as-edifying Q/A “Why is a virtual terminal “virtual”, and what/why/where is the “real” terminal?“

Anyway, I use virtual terminals to have multiple commands running simultaneously but not in the background of a single shell (e.g. running top on virtual console 2 while I go back to frobbing on console 1). I find it easier to check the progress of a running program when it’s in a virtual console rather than as a background job in a single shell.

In the course of using virtual consoles, sometimes I want to know exactly where I am. Am I on vterminal 4? 6? How would one find that information out?

The answer is with the command tty. In fact, if you read the two Q/A session I linked to above, you’ll know exactly why the command is called `tty`. When you run that command, you’ll be told which terminal you’re on.

[user@computer ~]$ tty
/dev/tty3

Now we know exactly where we are! You can also do some stupid tricks with virtual terminals since they are represented by device files in the Linux filesystem. For example, you can send output from one tty to another. echo woot > /dev/tty3 would send the text “woot” onto the console located at tty3 (to be accessed with Alt-F3). I’m sure there’s a useful application for that ability, but I haven’t found it yet.

It should also be noted that if you want to see how many virtual terminals currently have someone logged in, the who command can help you.

[user@computer ~]$ who
wesley     tty3     2012-10-13 12:13
snipes     tty2     2012-10-13 12:13
crusher    tty4     2012-10-13 12:14
borland    tty1     2012-10-13 12:10 (:0)

Have any handy bits of wisdom for the use of virtual consoles? Want to share? Share with the class in the comments section.

Because I’m tired of forgetting how to set this option, I’m putting it here.

In Vim, and I presume vi as well, if you want to search your document with case insensitivity, simply use the following command:

:set ignorecase

From then on you can search a document with no regard to case. /AllowUsers is the same search as /allowusers.

Previously I explored how to limit a user’s ability to runs commands with sudo. As a tangential topic, I needed to restrict the commands that a user account had access to when they connected to the server via SSH. Specifically, I needed just a few commands to be strung together and executed every time this account connected.

The mechanism that I used to do this was with the authorized_keys file. For a thorough explanation of that file, take a peak at the man page for sshd. To explain it very simply, the authorized_keys file holds the public keys of other users/systems that are allowed to connect to that machine. For example, I place my main user account’s public RSA key into the authorized_keys file on the Linux servers that I manage. When I connect to the remote servers using SSH, it checks to see if I’m who I say I am by challenging me with the public key that it has stored. The user account on my laptop uses the private key to validate itself (yes, the private key is password protected) and I am then allowed to haxor on the servers to my heart’s content.

Here’s an example of a public key:

ssh-rsa AAAB3NzaC1yc2EAAAADAQABAAABAQDclBxY7lOaolHGaogdcc9GaTQLWMcn2PK4hnQfWlJgeeGqgS66jL4XJyiR9HcgaebBW88Z2sevUxd7g25WhuuRAazfOcElEaE+h6MMPZ94gHY+x+iVAdlNKxLT/bTvCUXLEft/yZFpnknnv7jX4ChfSiII9OiAiCzuSdyHt1/1LgEHgvDIwKMzvTgImm5X/3IhtOitjJY3Q6yhKQ6LdenQtG/v+ANqKe6opDuUKc3k9hRmj7aHROxL52paQTEgEMoVLbIoZY4/yGUzmrZQU45jNqMrbXdAxG4XexZxb7bpTLu91s0DJQGx43JNXwhJVinPgxHLmfyoCSqR1WPqn8E3 testuser@testserver

The public key, when placed in a system’s authorized_keys file, can have some extra tidbits added to it that sshd honors. An SSH protocol 2 public key follows this format:

options, keytype, base64-encoded key, comment

In the above public key, you see the keytype as ‘ssh-rsa’ followed by a space, then the key itself followed by a space and finally a comment, which in this case is a username and hostname combination. That’s a helpful hint to know who this key supposedly belongs to. Notice that there are no options included in the above key, which would come before the keytype.

Some of the options that are available to be parsed by sshd include:

• environment= Changes an environmental variable for the user that is on the receiving end of the connection.
• from= Only allows connections that use this public key to be initiated from certain hosts. Helpful for the extremely paranoid or the very security conscious (the only difference between the two being pay grade).
• no-X11-forwarding Because we don’t need users installing xorg and then browsing the web on a remote instance of Chrome.

There are plenty of other options, however the final one that I’ll mention is the most crucial to this topic: command="command"

With the command= option, you can cause a command to be run immediately upon a successful connection to a remote host. Once the command is run, the connection is closed. Notice how that works. The command is immediately run and then once the command finishes, the connection is closed. This is not something that you’d want to do to a key that is intended to be used interactively by a human.

What could this be good for? In my specific scenario, I am using a backup tool that moves all of the data to stdout which is then piped to ssh for a secure transfer to remote storage. The remote connection would normally look like this: ssh remoteuser@remoteserver ” cat > backupfile.zip” However, if I edit the authorized keys file, I can restrict the incoming ssh connection to only be allowed to use that specific command.

It’s just another layer of security to keep people from doing things that they shouldn’t be doing. Have different ways of achieving a similar goal? Any caveats you know about? Let me know in the comments.

When making a bootable USB drive to install CentOS 6 from, you will need two primary partitions, one of them marked with the boot flag. One partition will be the boot partition and one will be a data partition that has the ISO file on it. As of this blog post, to make a CentOS 6 bootable USB drive, you’ll need a USB drive that has a little more than twice the space that your ISO file itself takes up. There is a bug that requires the ISO’s contents to be on the boot partition and the .iso file itself to be on the data partition. In essence you’re duplicating the ISO file and you still need some space left over for bootloader information. In my case, I’m using the minimal CentOS image, so space requirements are under 1GB.

At this point, go out and grab the CentOS ISO that interests you. Have it on your filesystem because we’ll be mounting it and copying some files from it. Once you’ve got the ISO you can move on to partitioning the drive.

Partitioning

First, you’ll want to partition the USB drive. We’ll be using plain ol’ MBR style partition tables and two primary partitions. I’m not going to hand-hold you through this part of the process. Use whatever partitioning tool you want and follow the guidelines below. GParted is fine if you use Gnome, parted is great if you want to use a shell, and fdisk works on both Windows and *NIX environments.

The partition layout will be thus:

1. A primary partition that uses the FAT16 filesystem and is at least as big as your ISO plus about 50MB. You need to give it the boot flag.
2. A primary partition that uses ext2 and is at least as big as your ISO. Preferably you’ll just use up the rest of your USB drive’s free space for this partition.

Once your partitions are set up, we’ve got some file moving to do.

Setting the Filesystems Up

You’ll want to mount your two partitions so that you can access them. In my case, the first partition (the FAT16 boot partition) is /dev/sdc1 and the data partition (the one formatted in ext2) is /dev/sdc2. I’ve mounted sdc1 as /mnt/usbboot and sdc2 as /mnt/usbdata. I will be using that nomenclature throughout the rest of this post.

You’ll also want to mount your CentOS ISO as a filesystem because we need to copy some files off of it. In my case, I ran mount -o loop /path/to/iso/file.iso /mnt/centosiso and will be using /mnt/centosiso in my examples below. Now that we’ve got all of our filesystems mounted, we’ll start the procedures.

First, go to the mounted CentOS iso and copy the /isolinux directory to the boot partition of the USB drive.

cp -r /mnt/centosiso/isolinux /mnt/usbboot

Rename the isolinux folder on the USB drive to syslinux

mv /mnt/usbboot/isolinux /mnt/usbboot/syslinux

Rename the isolinux.cfg file to syslinux.cfg

mv /mnt/usbboot/syslinux/isolinux.cfg /mnt/usbboot/syslinux/syslinux.cfg

Now we need to copy the contents of the /mnt/centosiso/images folder to the USB boot partition. Notice that I emphasis that this is a copy of the contents within the ISO’s images folder. A little later on we’ll be copying over the entire ISO as a file.

cp -r /mnt/centosiso/images /mnt/usbboot

Finally, we copy the .iso file itself to the data partition (not the boot partition that we were just working with!):

cp /path/to/iso/file.iso /mnt/usbdata

Once all that is done, we have to install a bootloader. I’ll use the simple syslinux loader. We want to use our smaller volume (the one that we set the boot flag on up in the partitioning section) as the target for the syslinux command.

syslinux -i /dev/sdc1

Now, we dismount our USB drive and test it out by booting from it on another system!

 

---

Finoto!

You should now have a bootable CentOS 6 USB drive. CentOS 6 is somewhat unique as a result of the bug that requires the images directory to be included on the boot partition, but other than that it’s relatively straight forward.

My Problem:

Attempting to install an RPM using the rpm -i command causes a series of errors:

error: open of <!DOCTYPE failed: No such file or directory
error: open of HTML failed: No such file or directory
error: open of PUBLIC failed: No such file or directory

The shell may hang and not return control to you.

It looks like it’s trying to parse an HTML document as a series of commands. Let’s think about that for a moment, shall we?

The Solution:

You are not attempting to install an rpm file, you are attempting to install a web page. Most likely either a redirect or 404 error page. Don’t believe me? Use cat to view the rpm file. In my case, trying to install rpmforge as a repo, I used curl to get what I thought was the proper rpm, however in reality I was retrieving the following:

$ cat rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://rpmforge.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm">here</a>.</p>
<hr>
<address>Apache/2.2.3 (Red Hat) Server at pkgs.repoforge.org Port 80</address>
</body></html>

Once you find the proper RPM file, I’m sure the installation will proceed without a hitch.

The Task

I have a situation on a CentOS server where I need to grant one low privileged user account the ability to run a single command as root. Here’s how I did it:

Enter visudo and the sudoers File

This probably deserves its own post, but for now let it suffice to know that if you are editing the sudoers file, you need to use visudo. It checks your syntax before saving the file which will prevent you from swearing like a drunken stevedore in between hysterical crying fits.

Run visudo as root and scroll down to the section that assigned rights to user accounts. You’ll almost certainly see see a line that says

root ALL=(ALL) ALL

That’s the beginning of the section that we’re interested in. But, what does that even mean? Let’s talk about that before we edit anything.

The syntax for the user lines in the sudoers file follows this syntax:

who host=(accounts) commands

Broken down, that means:

• who: the account that is having its ability to use sudo privileges modified
• host: the system that the account is able to run these sudo commands on (the sudoers file can be shared across multiple computers, so that’s when this would come into play)
• accounts: what other accounts on the machine the user running sudo can act as
• commands: the commands that the account represented by who can run as sudo

That means root ALL=(ALL) ALL is broken down thusly: The root account can use sudo on all computers that have this sudoers file and assume the identity of any of the accounts on those machines to perform any command that is available on them.

There are a few other additional options that can be placed on the line to further define each user’s sudo privileges. I won’t go into detail about those options (mostly because I just learned about them the other day and I’m still clueless), but you can read much more about the whole thing using its man pages.

The specific option that I’m interested in is the NOPASSWD option. You see, I need to call sudo to run a specific command as root within a script and not be prompted for a password. In that case, I place the NOPASSWD option just before the commands that I want use as root without a password. It would look something like this:

backupuser ALL=(ALL) NOPASSWD: /usr/bin/backuptool

And that is how I restricted an account’s ability to use one single command as root using sudo without a password. Any thoughts? Caveats? The sudoers file is a behemoth invention that can do quite a few different things. Let me know your ideas below.

(Today’s post is by guest author Erik Skålerud!)

Following up on Wesley’s post “What Version of CentOS / RedHat am I running?” and also to answer a twitter question from Barry Morrison – here’s how you quickly check what version of Ubuntu you’re running.

Both of the following methods should give you identical results.

Method #1

lsb_release -a

The result should be similar to this depending on your version/release:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 10.04.4 LTS
Release:        10.04
Codename:       lucid

Method #2

cat /etc/lsb-release

Result (again, depending on version/release):

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"

(Today’s post is by information security analyst Scott Pack!)

One of the worst problems I have with tabbed consoles is knowing exactly which console I’m working in. Sure, I can simply look at either the shell’s title, or the prompt, but I still inevitably will type a command into the wrong console. Normally this will result in something more akin to “file not found” because you catted the wrong file, rather than “Now running koan on all servers” because you did something in mcollective prod instead of test. While chatting up a guy at work about this problem we threw around changing the background color depending on which system you’re on.

This is one of those things that’s a little trickier than one might think at first blush. For instance, my first thought was to overload the ssh operator and make the changes client side. This works for me since I use cygwin as my ssh client and proxy through an aggregator (see this ServerFault answer about proxying</shill>). My coworker, however, uses puttycm which pretty much excludes any kind of local overloading. In the end, I opted to have the colorization occur on login. This made the process more portable, but it also meant that I have to make changes to my ~/.bashrc on every system.

In the end, I wanted this setup to work for both of our environments. So for my uses I wanted the background to get reset whenever I logged off a remote system, so that if I *did* manually ssh from one system to another, the background color should always be correct no matter what else I’ve been doing. So the colors needed to be set on login, which is easy, and also whenever ssh exits (I suppose I should also account for telnet/rsh but I really don’t want to). To that end, I decided to make the colorization a separate function so it could easily be reused.

It’s also worth noting that this was designed for a system running bash-3.x. If you’re in an entirely bash-4.x environment we can use an associative array, which works like a perl hash. Since I still deal with a lot of RHEL5 systems, which still uses bash-3.x, I had to code to it. To account for this I had to do some odd stuff with the array that basically amounts to magic.

Without further ado, below is everything you’ll need to add to your ~/.bashrc file to make this garbage work. With any luck, the comments are sufficient to explain.

function setcolor {
# Set up the colormap using hex codes for the colors
  colormap=( "node1:#330000"
             "node2:#003300"
             "node3:#330033"
             "node4:#333300"
             "node5:#FF0000"
             "node6:#0000FF" )
# Generate my own short hostname, i.e. turn node1.example.com into node1
  short=`echo ${HOSTNAME} | sed "s/\..*$//"`
color="#000000" # Set default color to black
# Iterate through the colormap looking for the hostname. Also, some bash magic.
  for host in ${colormap[@]}; do
    if [[ ${host%%:*} == ${short} ]]; then
      color=${host##*:}
    fi
  done
# Wrap the color in the xterm escape sequences to set the background color
  echo -ne "33]11;${color}07" # Set the background color
}
# Only run the setcolor function if we are using xterm or xterm-color as our termtype
if [ $TERM = "xterm" ]; then
  export TERM="xterm-color"
  setcolor
elif [ $TERM = "xterm-color" ]; then
  setcolor
fi
# Replaces the shell title with the name of the host we are sshing to
function ssh {
echo -ne "33]0;${1}07"     # Set the terminal title to the host we are sshing to
  /usr/bin/ssh $1 $2 $3
  setcolor  # Once ssh exits, reset the color back to what it should be
}