Basic Linux Malware Process Forensics for Incident Responders

Let’s go step-by-step and do some basic live process forensics for Linux. Today’s attack is a bind shell backdoor waiting for a connection on Ubuntu.

Simulated Bind Shell Attack

If you want to simulate the attack in this post, you can use the netcat command which opens a TCP port on 31337, but sends all data to /dev/null instead of a real shell. The commands below delete the binary as well after it starts so you can experiment with recovering the deleted process binary.

cd /tmp
cp /bin/nc /tmp/x7
./x7 -vv -k -w 1 -l 31337 > /dev/null &
rm x7

Suspicious Network Port Spotted

In our example we saw something odd when we ran:

netstat -nalp

netstat shows a process named “x7” PID with a listening port that we don’t recognize.

Obtain /proc Listing for Suspicious Process ID

First thing we’ll do is list out Process ID (PID) under /proc/<PID> to see what is going on. Our PID of interest is 5805:

ls -al /proc/5805

Below we see a couple odd things.

1. The current working directory (cwd) is /tmp.

2. The binary was in /tmp, but was deleted.

A lot of exploits work out of /tmp and /dev/shm on Linux. These are both world writable directories on most all Linux systems and many malware and exploits will drop their payloads there to run. A process that is making its home in /tmp or /dev/shm is suspicious.

Recover Deleted Linux Malware Binary

Before we do anything else, we’ll recover the deleted binary. As long as the process is still running, it is very easy to recover a deleted process binary on Linux:

cp /proc/<PID>/exe /tmp/recovered_bin

Obtain Deleted Linux Malware Hashes

Now that we’ve saved the Linux binary somewhere off the system, we can recover the hashes easily.

If you are using netcat to simulate the attack, you can recover the deleted binary and run a hash on the system netcat command and the recovered binary and see they match.

sha1sum /bin/nc
<hash here>
sha1sum /tmp/recovered_bin
<identical hash here>

Explore Linux Malware Command Line

The command line is stored under /proc/<PID>/cmdline and the command name is shown under /proc/<PID>/comm.

Some malware will cloak this data to masquerade as another process. You may see different names for the program in this case, or even names that are trying to hide as something else like apache or sshd.

If you see multiple different names, then it is very likely the program is malicious.

cat /proc/<PID>/comm 

cat /proc/<PID>/cmdline

Explore Linux Malware Process Environment

Now let’s take a look at the environment our malware inherited when it started. This can often reveal information about who or what started the process. Here we see the process was started with sudo by another user:

strings /proc/<PID>/environ

Investigate Linux Malware Open File Descriptors

We’ll now investigate the file descriptors the malware has open. This can often show you hidden files and directories that the malware is using to stash things along with open sockets:

ls -al /proc/<PID>/fd

Investigate Linux Malware Process Maps

Another area to look into is the Linux process maps. This shows libraries the malware is using and again can show links to malicious files it is using as well.

cat /proc/<PID>/maps

Investigate Linux Malware Process Stack

The /proc/<PID>/stack area can sometimes reveal more details. We’ll look at that like this:

cat /proc/<PID>/stack

In this case we see some network accept() calls indicating this is a network server waiting for a connection. Sometimes there won’t be anything obvious here, but sometimes there is. It just depends what the process is doing so it’s best to look.

Get Linux Malware Status

Finally, let’s look at /proc/<PID>/status for overall process details. This can reveal parent PIDs, etc.

cat /proc/<PID>/status

Get the Linux Command Line Forensics Cheatsheet

Those are some basics of Linux live process analysis. The big thing is this:

Don’t kill a suspicious process until you have investigated what it is doing.

If you kill a suspicious process out of panic, then you can lose and destroy a lot of useful information. We have created a Linux command line cheatsheet to help you look for these and other artifacts here:

Linux Command Line Forensics Cheat Sheet

Try Sandfly

Sandfly’s agentless security platform for Linux hunts for suspicious bind shells and many other process attacks on Linux 24 hours a day. We find the above attacks plus many more without loading agents on your endpoints. You can learn more here.