Detecting Linux memfd_create() Fileless Malware with Command Line Forensics

A developing threat to Linux over the last several years has been the idea of fileless malware. Fileless Linux malware is difficult to detect. It's designed to inject itself into a running Linux system and leave no traces on the disk. There are different ways to accomplish this, but here are some of the better-known tactics:

1. Executing the binary and the deleting itself from the disk.

2. Injecting code into a running server without writing to the disk (e.g. making a PHP server run your PHP code through a vulnerable input).

3. Use of debugging calls such as ptrace() to attach to a running process and inserting code into the memory space to execute.

4. Using calls such as memfd_create() to create an anonymous file in RAM that can be run.

In this post we’re going to talk about how to detect the last attack vector as it has gained more popularity recently. It has been seen in commonly available malware encryption tools such as Ezuri to make Linux malware harder to detect of late.

What is a memfd_create() Fileless Attack?

Let’s quickly discuss what the memfd_create() call does. Basically, it allows you to create a part of RAM on Linux that is a memory resident file. It’s like saving a file to RAM instead of to the file system itself. Once you do this, then you can reference that memory-resident file and get it to run just as you can with any file located in a standard disk directory. The man page describes it the same:

memfd_create() creates an anonymous file and returns a file descriptor that refers to it. The file behaves like a regular file, and so can be modified, truncated, memory-mapped, and so on. However, unlike a regular file, it lives in RAM and has a volatile backing storage.

Think of it as instead of calling /bin/ls on a Linux host, you instead loaded ls into RAM and simply referred to it as <memory_resident_file_location>/ls instead. That’s all it is. Now, like many system calls, this can have legitimate use for performance reasons, etc. However it can also be abused by malware that doesn’t wish to be seen. Also, if the system reboots or is shut down the malware is erased instantly which can be valuable to intruders wishing to protect themselves from discovery.

Running the Attack

We’ll launch an attack using one of any number of memfd_create() style exploits. They all have similar patterns of detection so it doesn’t matter which you use, but today we’ll use one from this article: In Memory Only ELF Execution.

Here is the attack from the intruder’s side. We are sending the binary over SSH directly so there is nothing written to disk. There is also no interactive shell which lowers the intruder’s footprint on the box.

This attack was setup to send a bindshell backdoor onto the target over SSH and execute it. Once run, the code goes resident and opens up a listening port waiting for connections. This will simulate a standard attack vector, but it can be anything an intruder may want to do on the remote host.

Rapid Detection for Fileless Linux Attacks

The class of attack we are dealing with today can be found with a simple ls and grep command:

ls -alR /proc/*/exe 2> /dev/null | grep memfd:.*\(deleted\)

This command goes over the /proc directory for all running processes and checks to see if they have an executable path that is of the form: memfd: (deleted). This entry is extremely suspicious and common with this kind of fileless attack.

Running this command should show no results. If you see any results come back as shown below, then the process should be immediately investigated. There is always of course a risk of a false positive, but for this type of attack it would be very uncommon to see (contact us if you have an example).

This is showing us one hit for the directory entry /proc/14667/exe. In the path the Process ID (PID) is 14667 and this is what we will want to focus on going forward.

What About Stealth Binaries?

If you are dealing with a binary that has deployed stealth hiding techniques, then all bets are off for this tactic. You’ll need to use other advanced methods which we are not going to cover in this article. However for common attacks using this method it is fast and effective.

Investigating the Fileless Linux Attack with Command Line Forensics

If the attack is present and visible with standard tools like ps and netstat you are in luck. In this case the bindshell launched and immediately bound to TCP port 31337. But it could have easily done a reverse connection outbound, started sending stolen data, etc. The first order of business though is to see what it may be doing. Using netstat and ps is a good place to start.

The netstat command shows us the first sign of trouble with an unknown port open with a weird process name:

netstat -nalp

We see the PID is 14667 as above. Now let’s look in our ps listing to see what else is there:

ps -auxw

The process is now more visible. We see it is calling itself [kworker/0:sandfly] in our example. This will help it hide among legitimate processes with similar names. You can see in the listing above how hard it would be to casually spot this if named without our sandfly identifier.

Investigating the Suspicious Linux Process

Now that we have confirmed our suspect, we can use some very simple commands to figure out what it may be. Most of the action will take place in the /proc directory. Specifically we will go to /proc/<PID> where PID is our suspect process ID. You can go to that directory like this:

cd /proc/14667

Process Directory Listing

In the process directory we can list what’s there. The Linux kernel will build the directory in real-time and show us many interesting things without doing anything else:

ls -al

This simple ls command shows:

1. The date stamps when the process was started.

2. The current working directory where the user started the process under /root most likely.

3. The exe link points to a now non-existent binary location.

The last point is the most important as it means the Linux kernel believes the binary that started the process is now missing on the file system.

Using Comm and Cmdline Parameters to Find Hiding Linux Processes

Now that we know this process is a problem child, we’ll go further. The files comm and cmdline under the /proc directory above will show what the system thinks the command and command line were when it started. This is a great place to look as often malicious programs try to hide their command names or command lines and this creates inconsistencies we can spot.

We’ll use the cat command to look at the comm file. We’ll use the strings command to look at the cmdline file. This is because cmdline has null characters in it that can affect formatting with cat. But if you only have cat installed you can just use that.

cat comm
strings cmdline

The comm and cmdline files show the following:

The thing that really stands out here is that in most cases on Linux these files should show a binary name that largely matches each other. For instance if you ran this against a web server like nginx you’ll see the nginx name somewhere in both of these files. But here we see something pretty odd. The comm file is just the single number “3” which is an artifact in how it was run (referencing file descriptor 3 where it was inserted/executed). The cmdline is odd too for reasons discussed in our article on kernel thread masquerading on Linux. But for now, let’s just note that these files do not reflect what the process name is in both. We would expect each file to have some relation to each other with the name, but we don’t see that here.

Using Process Maps to Validate Binary Name

The file /proc/<PID>/maps will contain process mappings for a binary. This will show the binary name, plus other library files it is using when it runs. Normally the first part of this file contains a reference to the actual binary that is running (e.g. /usr/bin/vi). But when we look here we see the strange reference to /memfd: (deleted) again.

cat maps

Investigating Linux Process Environments

Many people are not aware that on Linux when you start a process it will often have an extensive list of environment variables attached to it. Certainly any process started by a user will have it here in most cases. In this situation of our hiding process we’ll look at /proc/<PID>/environ to see what is there.

Again for formatting reasons we’ll use the strings command which makes it easier to read.

strings environ

Our malware process was inserted in over SSH and here you can see what IP did it in the environment. That’s very handy[Watch the video here]Even if an attacker has managed to scrub the logs of their IP addresses, often the processes they started will leave this information behind in various places.

Malware Evidence Conclusion

What have we seen so far:

1. Process file system grep flagged suspicious executable path running.

2. Process with weird network ports open.

3. Process with a name trying to look like a kernel thread.

4. Process comm and cmdline files do not have same command name referenced.

5. Process comm is just one character long.

6. Process current working directory is under /root.

7. Process exe points to deleted /memfd: location and not a legitimate binary path again.

8. Process maps file shows same deleted location and not a legitimate binary path.

9. Process environment shows SSH connection started this strange deleted process.

The above are all indicators of something trying to hide. Linux does not try to hide anything. When you see a process going to great lengths to not be noticed, it’s not friendly. We are clearly in Ninja on Rooftops territory here. Or as we say here at Sandfly: Trying to hide is an intrusion signature.

Grabbing the Binary

Going on the above, this process is absolutely malicious. Let’s grab it. Even though the binary is “fileless” you can actually recover the binary file that was injected very easily on Linux. We cover it fully in our article on recovering deleted binaries, but here it is in short.

cp /proc/<PID>/exe /destination_dir/filename

Assuming we are still in the correct directory from our demo above:

cp exe /tmp/malware.recovered

The file will now be under /tmp/malware.recovered. You can run a cryptographic hash for use with malware look-ups and the binary can be analyzed offline like normal. Just run the cryptographic hash of your choice. For example, use SHA1 to generate a hash on the live process binary and then the recovered one:

sha1sum exe
sha1sum /tmp/malware.recovered

Let’s Cheat

Of course doing the above on a one-off basis is fine (assuming you even know to look for a problem). However if you want to automatically check your systems on a constant basis for these issues, you should use our agentless security platform Sandfly.

We make this attack and many others no longer safe to use for intruders. This particular attack in fact is obvious to Sandfly as seen below with 10 alerts alone.

Best of all, we can do these investigations for you 24 hours a day without loading any software on your Linux endpoints. Sandfly works on all modern and legacy Linux distributions without any system impacts. Drop us a line if you want to see how our agentless security platform for Linux can automate detecting Linux malware.