Detecting Linux Stealth Rootkits with Directory Link Errors

Detecting stealth rootkits on Linux can be done from the command line. The secret is to ask the same question multiple ways to make sure all answers agree. If they don't all agree, something is likely hiding from you.

Let's demonstrate this idea by finding a directory from the Reptile stealth rootkit on Linux. We are going to ask the same question multiple ways to see if they all agree. The question is simple:

"How many directories are here?"

It is surprisingly hard for a stealth rootkit to provide consistent answers to simple questions like this. You can use this method in other ways to find things hiding on Linux.

h/t Hal Pomeranz for the link check method he talked about years ago in this post.

When you run a rootkit like Reptile, they often come with the ability to hide a directory if named a certain way. Any directory with the name "reptile" in it will be hidden with this rootkit as you see below under /lib/udev. We'll look at this directory with some simple command line tools to spot rootkit activity.

This is what the directory looks like when you issue a secret command to make it visible.

Let's use the tactic of asking the same question multiple ways to find something hiding. Here we'll list only directories then pipe it through a command to get a count. Only two directories are shown. The commands are:

ls -d */ 

ls -d */ | wc -l

Now we ask the same question, but this time we'll use the "stat" command and check directory link counts. The stat command says more directories are here with the link count:

stat . 

NOTE: This trick may not work on file systems such as btrfs.

It's one thing to get the kernel to lie. It's another entirely to get the file system to agree. Here the kernel rootkit cannot get the stat command to agree that the file system only has two directories.

Here is another view of the stat command to get the same information that does not match what ls is showing. This inconsistency error is common with stealth malware.

ls -al
 
stat .

Doing manual investigation works, but it's tedious. You should automate it with Sandfly Security. We spot directory link errors like this, but we look for many more problems as well that reveal stealth rootkit activity on Linux.

Command line tactics let you know something is wrong. However, you may have to isolate and take the machine into single user mode to dig deeper or do a memory dump to try to find the culprit. But, at least you know an investigation is warranted.

Hiding on Linux is difficult. Getting everything to agree with the hiding lie is expensive and prone to error. Even if you get a stealth rootkit to work on an isolated box, it likely will break in the wild.

There have been increasing reports of stealth rootkits being deployed on some high-profile attacks so we don't want to dismiss them. However, while stealth rootkits seem like a great idea, in our experience they are often more bark than bite.