I recently had a moment involving a CentOS server that caused me to circle the wagons and ask “Who just did what in their shell?!”
After quickly checking to see who was currently logged in (as well as those that had just recently been logged in), I wanted to see the command history for each user on the server.
Before I go any further, let me say a few important things:
There are more shells than bash
Each shell has its own history options and files. Don’t assume that because you found all the .bash_history files on a machine that you have all shell histories.
And all the zsh proselytes said “Amen.”
.bash_history is a suggestion not a rule
Bash’s history file (that’s the $HISTFILE variable) can be changed. Just because you found all the .bash_history files on a machine doesn’t mean you have all of bash’s history.
Bash history is a convenience not a reporting tool
Bash history can easily be altered for both good and bad purposes. It is not to be relied on as a a way of seriously auditing what has been done on a server. For that kind of thing, look at auditd.
Scan all .bash_history files
The above notwithstanding, if you want to quickly scan your machine’s .bash_history files consider the following options.
grep -e "stuff goes here" /home/*/.bash_history
Yep, simple as that. This is of course assuming that 1) All user folders are standardized, and 2) your history files all share a common name. If the previous two things are true, this is a great, quick way to see things like… oh… say… “Who just went all chmod -R 777 on the httpdocs folder?!“
However, if you want a slightly more robust way of searching through all bash history on a machine that takes the home folder ambiguity out of the equation, Gilles from the Unix & Linux Stack Exchange had an awesome solution.
getent passwd | cut -d : -f 6 | sed 's:$:/.bash_history:' | xargs -d 'n' grep -H -e "$pattern"
I had never seen the ‘getent’ tool before which gets entries from the following administrative databases: ahosts, ahostsv4, ahostsv6, aliases, ethers, group, gshadow, hosts, netgroup, networks, passwd, protocols, rpc, services, and shadow. ‘Cut’ segments the input by a colon and then selects the sixth field which is each user’s home directory. Sed works its magic to take the input and append it with the probable location of the .bash_history file. Finally grep is fed each path and searches for our pattern.
Certainly, if there is a question about the existence of other shells or if you want to be certain that your history file really is called .bash_history, you’ll need to add some extra logic in. However, for my scenario, this was enough to get me going.
Unfortunately, I was made painfully aware of how bash history is a mere user level convenience and not an auditing tool. Nothing malicious was done to the server and nothing terribly bad was done, however as I looked deeper into what could have happened, I realized that a much more thorough auditing trail might be needed in the future.
How do you handle shell history? Do you implement any special tricks to make it more reliable or do you use an entirely different system to keep track of commands that have been run?