How to View All .bash_history Files on Your Server

I recently had a moment involving a CentOS server that caused me to circle the wagons and ask “Who just did what in their shell?!”

After quickly checking to see who was currently logged in (as well as those that had just recently been logged in), I wanted to see the command history for each user on the server.

Before I go any further, let me say a few important things:

There are more shells than bash

Each shell has its own history options and files. Don’t assume that because you found all the .bash_history files on a machine that you have all shell histories.

And all the zsh proselytes said “Amen.”

.bash_history is a suggestion not a rule

Bash’s history file (that’s the $HISTFILE variable) can be changed. Just because you found all the .bash_history files on a machine doesn’t mean you have all of bash’s history.

Bash history is a convenience not a reporting tool

Bash history can easily be altered for both good and bad purposes. It is not to be relied on as a a way of seriously auditing what has been done on a server. For that kind of thing, look at auditd.

Scan all .bash_history files

The above notwithstanding, if you want to quickly scan your machine’s .bash_history files consider the following options.

The first is dead simple, and I thank @etrever, @evilchili and Gilles over at for this method (I’m still getting my *nix chops).

grep -e "stuff goes here" /home/*/.bash_history

Yep, simple as that. This is of course assuming that 1) All user folders are standardized, and 2) your history files all share a common name. If the previous two things are true, this is a great, quick way to see things like… oh… say… Who just went all chmod -R 777 on the httpdocs folder?!

However, if you want a slightly more robust way of searching through all bash history on a machine that takes the home folder ambiguity out of the equation, Gilles from the Unix & Linux Stack Exchange had an awesome solution.

getent passwd |
cut -d : -f 6 |
sed 's:$:/.bash_history:' |
xargs -d 'n' grep -H -e "$pattern"

I had never seen the ‘getent’ tool before which gets entries from the following administrative databases: ahosts, ahostsv4, ahostsv6, aliases, ethers, group, gshadow, hosts, netgroup, networks, passwd, protocols, rpc, services, and shadow. ‘Cut’ segments the input by a colon and then selects the sixth field which is each user’s home directory. Sed works its magic to take the input and append it with the probable location of the .bash_history file. Finally grep is fed each path and searches for our pattern.

Certainly, if there is a question about the existence of other shells or if you want to be certain that your history file really is called .bash_history, you’ll need to add some extra logic in. However, for my scenario, this was enough to get me going.

Unfortunately, I was made painfully aware of how bash history is a mere user level convenience and not an auditing tool. Nothing malicious was done to the server and nothing terribly bad was done, however as I looked deeper into what could have happened, I realized that a much more thorough auditing trail might be needed in the future.

How do you handle shell history? Do you implement any special tricks to make it more reliable or do you use an entirely different system to keep track of commands that have been run?


  1. Jason Wellband

    February 26, 2012 at 3:54 pm

    Typically, if you’re into auditing, you either force everyone to use a shell that audits what they do into a non-user configurable/mutable area (e.g. syslog; check out lshell for an example) -or- you have them use sudo, which logs to syslog out of the box. Keep in mind that you don’t have to run things as root with sudo. It can run commands as any user available as long as you configure sudoers to do so.

    Rule 1 of auditing is do not permit root logins via remote means and secure the system to where only trusted people can touch it or access its lights-out functionality. That way, even if your root account is not properly audited, you could at least tell who had logged in as root during that time.

    Just a word of warning – your getent command would also query any sort of centralized account repository (e.g. LDAP). Especially with LDAP, said service may be configured to only send back so many results on a query (since most queries should only send back one or a few results at most). In that case, you’d not get every user. You might want to integrate the last command to only look for certain users, e.g.

    [[email protected] ~]# last | grep -ve ‘^reboot’ | tail -7 | head -5 | awk ‘{ print $1 }’ | sort -u | xargs -n1 getent passwd | cut -d: -f6 | sed ‘s|$|/.*history|g’ | xargs -i sh -c “grep -nH chmod {}”
    /home/jwell/.bash_history:86:chmod 644 file
    /home/jwell/.bash_history:87:sudo chmod 644 file
    /root/.bash_history:144:chmod 755 My
    /root/.bash_history:145:chmod 755 My Documents/
    /root/.bash_history:893:chmod 775 /export/desktop/
    /root/.bash_history:980:chmod 777 .
    /root/.bash_history:988:chmod 1777 .

    Adjust the head and tail lines to match the number of user logins you want to check. The head argument should be the number of logins you want to go back and the tail should be that number plus two. This will gobble up duplicate logins (no use in looking at the same file twice), so you may want to run last by hand to decide how many logins back you want to look.

    Adjust your inner grep to taste. The -n gives you the line # of the shell history so you can easily find what you’re looking for. You could add a tail in there to only look at the last N lines of the history (no use in looking at old history…), but since the history is rarely dated, then there’s no magic bullet for that.

    I have the latter xargs spawn off a shell by hand so that the wildcard is parsed, thereby giving you all files that match .*history, which I believe will take care of most of the common shells.

    Hit me up on G+ if I’m not clear :)


    • Wesley David

      February 27, 2012 at 1:28 pm

      Thanks for the tip about lshell. That’s an interesting possibility – I had never considered just switching to a security-specific shell.

      I’m not terribly familiar with getent nor do I work with Linux machines that are connected to a directory service, so that caveat is good to be aware of before I find its faults out too late. =)

      Oh, and I’m glad we agree that Coke > Pepsi. =)


  2. Phil Hollenback

    February 27, 2012 at 12:05 am

    Couple notes:

    1. getent is not available on all platforms. For example, it doesn’t exist on the mac. You can use straight perl like I did here:!/philiph/statuses/162921853185310721

    perl -e ‘print +(getpwnam($ARGV[0]))[6].”n”‘ username

    which would need to be tweaked a little bit to find the .bash_history file as your above example does.

    2. I really like the idea of stuffing this in to some sort of central logging facility like maybe logstash. There are certainly privacy concerns there, but it might appropriate if you just do this on production boxes where people don’t do any personal work.

    3. Is it possible to make all users on a box send history to the same file? I’ve never played with that but maybe you can do something along those lines. Or, maybe you can at least configure everyone to put their bash history files in a central directory to make things simpler.

    I think that the shell history is a rich source of information, and I really like the idea of collecting and mining that data.


    • Wesley David

      February 27, 2012 at 1:33 pm

      1. I knew you’d go all PERL up in this hizouse. =)

      2. I’m thinking for production servers and not for anything personal. I’d really like to look into auditd. I was talking with @packscott and he mentioned how Cisco ACS works. If you connect with an ACS user, every command intended for the Cisco device will first be sent to the ACS server to discern if that user can perform that action on that device. If something like that could be done on POSIX systems, combined with role based user separation as well as zones of hardware and software… oh my. Me gusta.

      3. I dunno. I am, after all, just a nublet.


    • Damian

      May 23, 2012 at 6:31 am

      Note to self:X (capital x) means “execute/search only if the file is a directory or aleardy has execute permission for some user”u=blah or u+blah has always been my preferred way of setting permissions. Since it avoids needing to work in octals (have enough trouble with decimal).


Leave a Reply

Follow TheNubbyAdmin!

follow us in feedly

Raw RSS Feed:

Contact Me!

Want to hire me as a consultant? Have a job you think I might be interested in? Drop me a line:

Contact Me!

Subscribe via Email

Your email address is handled by Google FeedBurner and never spammed!

The Nubby Archives

Circle Me on Google+!

Photos from Flickr

Me on StackExchange

The IT Crowd Strava Group

%d bloggers like this: