Using Linux utmpdump for Forensics and Detecting Log File Tampering
In this post we’re going to show you how to use utmpdump for investigating Linux audit logs for signs of compromise. Seemingly unknown by many, the utmpdump command is a great tool for Linux forensics and detecting log file tampering.
To start, utmpdump is a utility to dump the system audit logs called utmp, wtmp, and btmp. These logs contain the following data:
/var/run/utmp – Contains currently logged in users. /var/log/wtmp – Contains all current and past logins and additional information about system reboots, etc. /var/log/btmp – Contains all bad login attempts.
Because the utmp, wtmp and btmp files contain login information about all users, they are prime targets by intruders and malware on Linux to either destroy or alter.
Many pieces of Linux malware will simply erase the files and replace them with ones that are zero bytes long. This is a very ham-fisted way to cover your tracks and is easily spotted. However, more sophisticated attackers will make an attempt to clean the logs. Aside from simply obliterating the contents, there are two basic ways to alter these files with the log cleaners commonly found:
Overwrite the suspicious entries with nulls.
Remove the suspicious entries and splice the log back together cleanly.
In this post we’ll cover the first technique and will talk about the second in the future. The basic thing to know is that when an attacker overwrites an entry with nulls (zeroes), the entry will not show anything about what was there. It is effectively wiped and recovery of the entry is not possible.
In this post, we’ll be doing our investigation on a Raspberry Pi device running a version of the Raspbian OS which is a Debian variant.
Utmpdump for Forensics
The utmp, wtmp and btmp files are a binary format. In order to read them you will need a utility like utmpdump. In the most basic form, utmpdump allows us to quickly dump the logs and save them for later review as text. The utmpdump command is this:
utmpdump /var/run/utmp
utmpdump /var/log/wtmp
utmpdump /var/log/btmpYou can also redirect the output to a text file for archiving:
utmpdump /var/run/utmp > utmp.log
utmpdump /var/log/wtmp > wtmp.log
utmpdump /var/log/btmp > btmp.logDoing the above will give you an output like below:
sandflysecurity# utmpdump /var/log/wtmp
Utmp dump of /var/log/wtmp
[8] [00658] [ ] [ ] [pts/0 ] [ ] [0.0.0.0 ] [2019-07-08T00:38:05,868666+0000]
[2] [00000] [~~ ] [reboot ] [~ ] [4.9.80-v7+ ] [0.0.0.0 ] [1970-01-01T00:00:02,375926+0000]
[7] [00507] [:0 ] [pi ] [tty7 ] [:0 ] [0.0.0.0 ] [2019-07-10T01:55:32,589988+0000]
[5] [00439] [tty1] [ ] [tty1 ] [ ] [0.0.0.0 ] [2019-07-10T01:55:34,780689+0000]
[6] [00439] [tty1] [LOGIN ] [tty1 ] [ ] [0.0.0.0 ] [2019-07-10T01:55:34,780689+0000]
[7] [00439] [ ] [pi ] [tty1 ] [ ] [0.0.0.0 ] [2019-07-10T01:55:34,928778+0000]
[1] [00053] [~~ ] [runlevel] [~ ] [4.9.80-v7+ ] [0.0.0.0 ] [2019-07-10T01:55:35,888562+0000]
[7] [00998] [ts/0] [pi ] [pts/0 ] [192.168.1.109 ] [192.168.1.109 ] [2019-07-24T11:23:01,119990+0000]
[8] [00998] [ ] [ ] [pts/0 ] [ ] [0.0.0.0 ] [2019-07-24T11:23:20,384719+0000]
[7] [01048] [ts/0] [pi ] [pts/0 ] [192.168.1.109 ] [192.168.1.109 ] [2019-07-24T11:24:24,148680+0000]
[8] [01048] [ ] [ ] [pts/0 ] [ ] [0.0.0.0 ] [2019-07-24T11:24:24,178681+0000]
[7] [01048] [ts/0] [pi ] [pts/0 ] [192.168.1.109 ] [192.168.1.109 ] [2019-07-24T11:24:24,589624+0000]
[8] [01048] [ ] [ ] [pts/0 ] [ ] [0.0.0.0 ] [2019-07-24T11:24:24,720678+0000]
[7] [01048] [ts/0] [pi ] [pts/0 ] [192.168.1.109 ] [192.168.1.109 ] [2019-07-24T11:24:25,019377+0000]
[8] [01048] [ ] [ ] [pts/0 ] [ ] [0.0.0.0 ] [2019-07-24T11:24:25,047173+0000]
...The above shows things like the terminal used to login (e.g. pts/0, tty1), the username used (“pi”), the IP address they used and the timestamp of the login and logout. You can also see system events like reboots and so on. The utmpdump command has identical formats for the utmp, wtmp and btmp logs.
Log Cleaner Impact on Forensics
But what if an attacker runs a log cleaner? If they run it against the utmp log then it deletes their user ID if they are logged in. We’ll demonstrate with a user called “www” that has logged in to the Raspberry Pi we are using for our investigation. Commands like who or w will not show them logged in as you can see below.
After a log cleaner is run the user vanishes:
The system wtmp file shows all users that have logged in over time. Below we’ll use utmpdump to see the results before and after our log cleaner has ran.
But again, we’ll run our log cleaner and the user entry is overwritten.
Tampered Log Files are Great
Now, we’re going to let you in on a secret about tampered logs:
Although log tampering looks bad, the reality is it’s a good thing when intruders tamper with logs. This is because it shows 100% of the time there has been a compromise without any guesswork when you do spot it.
The above seems counterintuitive, but in reality most log tampering is done poorly and even if done well often it leaves inconsistencies on the target system that can be spotted. Once you spot these problems, then there is clear intent that someone, or something, is on that system and is trying to hide. We call this:
Think of it this way: Detecting log tampering is the equivalent to coming home and seeing your door kicked in due to a burglary and knowing something bad has just happened vs. looking for a family heirloom six months later and only then figuring out it’s been stolen some time back. Finding a compromise early is a very good thing, and a tampered log file is a gigantic flashing neon sign of trouble. The trick though is knowing how to spot tampering and being vigilant about looking for the problem.
Detecting Tampered Linux Audit Logs with Utmpdump
Now we can use the utmpdump command to quickly check our logs to see if log cleaning tactics have been used. Although it won’t tell you immediately what user was erased, it will give you near 100% proof that someone is operating on the host maliciously. Knowing a system is compromised is always the first step in figuring out who or what did it.
As it turns out, if you look carefully you can see how these log entries look a little off. They are all zeroes across the board and this is not normal. Even the date shows a year of 1970-01-01 which in Unix-speak means it is also all zeroes (epoch time). Let’s look at the log again with these details pointed out:
To determine quickly if our logs were hit with a null overwrite style log cleaner, we’re going to use these simple grep commands:
utmpdump /var/run/utmp | grep "\[0\].*1970-01-01"
utmpdump /var/log/wtmp | grep "\[0\].*1970-01-01"
utmpdump /var/log/btmp | grep "\[0\].*1970-01-01"What these commands do is search the logs for entries that appear to be all zeroes (nulls). This is not normal behavior on Linux and sticks out like a sore thumb when you look for it. The following screenshots demonstrate detecting log file tampering in the utmp, wtmp and btmp files.
You can even spot bad login attempts that were erased after entry was gained by searching the system /var/log/btmp file.
Bracketing Timestamps to Determine Intrusion Time
Even though the entry was deleted, you can use the timestamps in the entries above and below the erased values to bracket when the intrusion happened. Here we see the btmp log file has some good time windows to start our bracketed search. Btmp is a really good log file for this on Internet connected hosts because bad logins happen all the time so there is almost always one coming and going to bracket the deleted entry.
Sandfly Hunts for Tampered Logs Even on Embedded Devices
Tampering with log files is like moth to a flame for intruders. They almost always try to cover their tracks and Sandfly looks for a variety of ways logs can be tampered with and that includes erasing entries with nulls. Below we see two detections from Sandfly during a normal scan with our agentless security bot. The first is the evidence of a log entry overwritten, the second is a more specific alert where we are able to actually determine which user was deleted and what time it happened.
Notice we were doing this automatically and agentlessly on a Raspberry Pi device used in the above examples. This is because Sandfly’s agentless approach can work across all versions of Linux, even embedded and Internet of Things devices. We don’t need to run agents on endpoints so we can work effectively, even on devices with limited resources that can’t run software agents.
Pay Attention to Your Logs
Log files contain heaps of information to tell you what happened in the event of a compromise. Even if the log file has been tampered with that tells you something as well (100% proof of compromise). Watching system logs with tools like utmpdump is a quick way to check a host against common log cleaning tactics. Using something like Sandfly to do it for you automatically 24 hours a day is an even better idea. Give it a try for free today.