Detecting Linux Kernel Process Masquerading with Command Line Forensics

Linux kernel process masquerading is sometimes used by malware to hide when it is running. Let’s go over how you can unmask a piece of Linux malware using this tactic.

What is Linux Kernel Process Masquerading?

On Linux the kernel has many threads created to help with system tasks. These threads can be for scheduling, disk I/O, etc. When you use a standard process listing command like ps, these threads will show up as having [brackets] around them to denote that they are threads of some kind. Ordinary processes will not normally show up with [brackets] around them in the ps listing. The brackets denote that the process has no command line arguments which usually means it was spawned as a thread.

For example, the below listing shows kernel threads vs. normal processes:

ps -auxww

What Linux Kernel Thread Masquerading Looks Like

Linux malware uses a variety of techniques to hide from detection. One method they will use is to try to impersonate a kernel thread by making the process show [brackets] around its name in the ps listing. Administrators can easily overlook a malicious process this way.

If you look at the listing below, we have started a process to hide itself by trying to look like a kernel thread. Can you see it?

How to Impersonate a Linux Kernel Thread

Now that you know what Linux kernel thread masquerading looks like, let’s setup a test so you can play with how to find it using command line forensics. We’ll use the sleep command for our simulation as you can do it on any system without fear of causing trouble.

export PATH=.:$PATH
cp /bin/sleep /tmp/[kworkerd]
cd /tmp
"[kworkerd]" 3600 &

The export path sets things so we can execute the file in the local directory without needing to put a “./” in front of it. This makes it look more legit. We next copy the sleep command over to /tmp and then run it under the bogus name [kworkerd]. We put on a value of 3600 seconds to the sleep command so it will quietly exit after an hour once testing is over.

Let’s look at our handiwork and we should see [kworkerd] running when we do our ps command.

ps -auxww

De-cloaking Linux Kernel Thread Masquerading with Process Maps

The first method we’ll use to de-cloak a masquerading process is to see if it has any contents under /proc/<PID>/maps. This location is normally where processes show libraries they are linking to and where they mapped to in memory. For real kernel threads it should be empty. If you look at this location for a process that is named in [brackets] but it shows any data, then it is not a real kernel thread.

The basic command we’ll use is cat /proc/<PID>/maps where <PID> is the process ID we are investigating. In the above example we think that [kworkerd] looks suspicious with PID 2121 so we’ll check it out:

cat /proc/2121/maps

If you see anything listed under this area and the process has [brackets] around it, then it’s likely malicious and trying to hide.

If you want you can run this command to quickly go over all the system PIDs and see which ones are named with brackets but have maps files. Normally you should see nothing here. Anything that shows data should be investigated further.

ps auxww | grep \\[ | awk '{print $2}' | xargs -I % sh -c 'echo PID: %; cat /proc/%/maps' 2> /dev/null

This command outputs the image below if it finds something.

In the /proc/<PID>/maps listing you’ll see some paths to investigate where the binary has links to itself or libraries it is using. In the above we see the path /tmp/[kworkerd] which would be a high priority location to investigate. You may also see libraries that are suspicious, references to hidden directories, etc. Take a close look at it and be sure you don’t miss anything!

De-cloaking Linux Kernel Thread Masquerading with Cryptographic Hashing

Another way to de-cloak a masquerading Linux kernel thread is to see if it shows a binary attached to the running process. Basically, you just use the technique we discussed on this Twitter thread.

A process binary on Linux can be quickly copied if you simply look at /proc/<PID>/exe. You can copy this file to a new location and have an instant snapshot of the binary that started the process. You can also use this link to get an instant hash to check against databases of known malware. Real kernel threads won’t have this data available, only imposters will.

In our case, we’ll use this knowledge to investigate our suspicious PID 2121 like this:

sha1sum /proc/2121/exe

Now we see the hash, let’s recover the binary and copy it somewhere so it can be analyzed offline. Using the command below we’ll make a copy to /tmp/suspicious_bin. Now, we have our own copy in case the malware tries to delete itself in self-defense:

cp /proc/2121/exe /tmp/suspicious_bin

If you want to automatically crawl through the PIDs and get SHA1 hashes of imposters, you can run this command:

ps auxww | grep \\[ | awk '{print $2}' | xargs -I % sh -c 'echo PID: %; sha1sum /proc/%/exe' 2> /dev/null

The above command will try to get a SHA1 hash of all processes with [brackets] around them. Any that return a hash are likely imposters:

Kernel Thread Masquerading Detection the Easy Way

There you have two solid ways to use the Linux command line to investigate suspicious processes trying to masquerade as a kernel threads. If you want to do it even easier, you can use Sandfly which is our agentless compromise and intrusion detection system for Linux.

Sandfly has many more methods for finding this kind of attack plus hundreds of others. We monitor and investigate for signs of compromise (such as kernel thread masquerading) 24 hours a day without loading anything on your Linux endpoints. Get a free trial instantly.