Basic Linux Malware Process Forensics for Incident Responders
Let’s go step-by-step and do some basic live process forensics for Linux. Today’s attack is a bind shell backdoor waiting for a connection on Ubuntu.
Simulated Bind Shell Attack
If you want to simulate the attack in this post, you can use the netcat command which opens a TCP port on 31337, but sends all data to /dev/null instead of a real shell. The commands below delete the binary as well after it starts so you can experiment with recovering the deleted process binary.
cp /bin/nc /tmp/x7
./x7 -vv -k -w 1 -l 31337 > /dev/null &
Suspicious Network Port Spotted
In our example we saw something odd when we ran:
netstat shows a process named “x7” PID with a listening port that we don’t recognize.
Obtain /proc Listing for Suspicious Process ID
First thing we’ll do is list out Process ID (PID) under /proc/<PID> to see what is going on. Our PID of interest is 5805:
ls -al /proc/5805
Below we see a couple odd things.
The current working directory (cwd) is /tmp.
The binary was in /tmp, but was deleted.
A lot of exploits work out of /tmp and /dev/shm on Linux. These are both world writable directories on most all Linux systems and many malware and exploits will drop their payloads there to run. A process that is making its home in /tmp or /dev/shm is suspicious.
Recover Deleted Linux Malware Binary
Before we do anything else, we’ll recover the deleted binary. As long as the process is still running, it is very easy to recover a deleted process binary on Linux:
cp /proc/<PID>/exe /tmp/recovered_bin
Obtain Deleted Linux Malware Hashes
Now that we’ve saved the Linux binary somewhere off the system, we can recover the hashes easily.
If you are using netcat to simulate the attack, you can recover the deleted binary and run a hash on the system netcat command and the recovered binary and see they match.
<identical hash here>
Explore Linux Malware Command Line
The command line is stored under /proc/<PID>/cmdline and the command name is shown under /proc/<PID>/comm.
Some malware will cloak this data to masquerade as another process. You may see different names for the program in this case, or even names that are trying to hide as something else like apache or sshd.
If you see multiple different names, then it is very likely the program is malicious.
Explore Linux Malware Process Environment
Now let’s take a look at the environment our malware inherited when it started. This can often reveal information about who or what started the process. Here we see the process was started with sudo by another user:
Investigate Linux Malware Open File Descriptors
We’ll now investigate the file descriptors the malware has open. This can often show you hidden files and directories that the malware is using to stash things along with open sockets:
ls -al /proc/<PID>/fd
Investigate Linux Malware Process Maps
Another area to look into is the Linux process maps. This shows libraries the malware is using and again can show links to malicious files it is using as well.
Investigate Linux Malware Process Stack
The /proc/<PID>/stack area can sometimes reveal more details. We’ll look at that like this:
In this case we see some network accept() calls indicating this is a network server waiting for a connection. Sometimes there won’t be anything obvious here, but sometimes there is. It just depends what the process is doing so it’s best to look.
Get Linux Malware Status
Finally, let’s look at /proc/<PID>/status for overall process details. This can reveal parent PIDs, etc.
Get the Linux Command Line Forensics Cheatsheet
Those are some basics of Linux live process analysis. The big thing is this:
Don’t kill a suspicious process until you have investigated what it is doing.
If you kill a suspicious process out of panic, then you can lose and destroy a lot of useful information. We have created a Linux command line cheatsheet to help you look for these and other artifacts here:
Sandfly’s agentless security platform for Linux hunts for suspicious bind shells and many other process attacks on Linux 24 hours a day. We find the above attacks plus many more without loading agents on your endpoints. You can learn more here.