Using Linux Process Environment Variables for Live Forensics

In this post and video we’re going to explore the Linux /proc directory further by digging into a live process and the environment variables it presents. The basic format for the command we’ll use is this:

strings /proc/<PID>/environ

In the above, the PID represents the Process ID that you want to investigate. The strings command is used to present the data nicely, but you can use the cat or more command if that’s all you have loaded.

Linux Process Environment Variables Background

When a process is started on Linux, many of the environment variables of that process remain for the duration it is running. What does this mean? It means many of the shell variables active when it was started still are visible even after the original shell that started the process has long since gone away.

Now this may not sound that great at first blush, but for tracking down a malicious process it actually becomes extremely helpful. Why? Because an attacker that launches a malicious binary will leave the traces of that activity on the process even after they have left.

What can you find in the process environment? Here are some examples:

• Evidence of history anti-forensics.

• Command line and paths used for the binary.

• Users and sudo users that started the command.

• Custom variables used by the binary on startup which could contain keys and other

• SSH connection information of who or what started the process.

• Etc.

That bullet on SSH connection information is extremely useful. Many Linux versions will staple the address of the SSH client that started the process right onto the item each time. Even if an attacker cleans their IP address from the logs, chances are good that if they came in over SSH that the IP address they used is just sitting there on the process they started waiting to be grabbed.

Instead of trying to go through a tangle of logs to find out what user or IP started something, why not just look at the process directly which is telling you?

Below are some screen shots of a mock malicious netcat process to show these ideas.

Mock Linux Network Server Attack

First, we’ll setup our mock malicious netcat process. We’ll rename it to “x” and run it on TCP port 4444 to simulate a listening network server. We’ll include some history anti-forensics as well to show that idea on top of it all.

Next, we’ll take a look at this process from the view of the admin. We’ll run the basic netstat -nalp command to dump all the listening daemons and the PIDs and process names. We see our nefarious program “x” running on a weird TCP port.

Now that we see our suspicious process, we’ll use the PID to dig into the environment it has when running.

Looking at the above we have a lot of useful information for an investigator:

• Evidence of anti-forensics

• The path the process is using as the working directory (/root)

• The home directory of the process (/root)

• Confirmation of the user that started it (if they used sudo that would show up here as well).

• SSH connections of the hosts that did the deed.

NOTE: The IP addresses used in the above demo were cloud VMs that long since been destroyed. Please do not bother those systems as they are not connected with our company.

Last Piece of Advice: Don’t Panic

In our Linux command line forensics cheat sheet, the first thing we put in there is “Don’t panic.” We encourage admins to never do something rash like kill a suspicious process before they investigate what it is doing. By killing a process before you are done figuring out what is going on, you instantly destroy a lot of useful information like process environment variables. So slow down, isolate the system if you want to stop it communicating on the network, and look at the process closely to see what you can find before you do anything drastic.

Experiment on Your Own

That in a nutshell is just some of the data present by looking at /proc/<PID>/environ and we encourage you to explore it on your own Linux systems. It may help you during your next investigation.

Sandfly Hunts for This Activity for You

Of course, Sandfly investigates process environment variables hunting for suspicious activity for you 24 hours a day. If a suspicious process is found on your hosts, we grab the environment variables as part of the automated investigation along with other