Leaked North Korean Linux Stealth Rootkit Analysis

Phrack Magazine issue #72 recently released a data dump from a suspected North Korean hacking group that contained a large trove of exploit tactics, compromised system information, and a stealth rootkit targeting Linux. We have reviewed the rootkit and are providing additional detection and operation details for incident responders.

We highly encourage teams to read the initial Phrack article which lays out the data obtained. Specifically they note:

1. Chinese APTs are hacking South Korea and Taiwan together with the Kimsuky North Korean APT with tool sharing and assistance.
2. Attackers have been able to backdoor legitimate software for at least 20 years.
3. Dump suggests that the attackers accessed internal South Korean government networks and had access to sensitive certificates.
4. Screenshot of the attacker’s desktop shows active backdoor development.

Phrack has made the entire data dump available below along with the PDF article of the leaked information. Please be aware that this archive contains live malware binaries for multiple platforms and should be handled carefully:

Phrack North Korean Data Dump

Rootkit Details

As the Phrack article states, the malware is a Loadable Kernel Module (LKM) style rootkit with ability to hide from detection and operate on any network port. The rootkit has the following features:

• Based on the khook library which is commonly used by malware to hook and intercept Linux kernel system calls to evade detection.
• Hides its kernel module from listing with tools like lsmod.
• Hides processes and backdoor network activity.
• Hides persistence files dropped under /etc/init.d and /etc/rc*.d directories.
• Receives magic packet on any port to activate backdoor.
• The backdoor can execute shells, upload and download files, activate proxies, or chain hosts together for lateral movement.
• Commands have anti-forensic features for the backdoor binary and shells spawned.
• All traffic is encrypted.

Rootkit Detection

The rootkit can be found by Sandfly out of the box without any updates for customers. We have had detections for this style of rootkit for some time. We strongly urge customers use automated tools to find this rootkit so detection can be done accurately and at scale. However, we will provide manual commands you may choose to use on individual hosts you are investigating for compromise in this article. Below are the default alerts Sandfly will generate on a host with this kernel rootkit and backdoor activated.

Rootkit Basic Operation

As with almost all LKM style rootkits, they tend to be fragile and kernel version specific. This rootkit is no different. To be deployed successfully, the attacker would need to have a version built for the specific victim kernel version. This is tedious and prone to risk, but once working the module goes live and sits quietly. However, it also means that system updates that involve a new kernel may very well cause the rootkit to fail entirely and not load or work correctly.

Kernel Module Location

The malicious module is stored under /usr/lib64/tracker-fs in the disclosed version. This is easily changed by attackers however and should not be relied upon for certain detection. However, for initial incident response it is not a bad idea to look for it using the following commands:

These command should not show any files. If you get data back then the module is present and this means it is active, or was once active, on the host. Here are a couple examples of what you is seen if the module is active.

Unsigned Module Kernel Taint Indicator

The next sign of trouble is that the kernel module is not signed. Unsigned kernel modules on Linux can be used by legitimate vendors (e.g. video cards), but if your systems do not use any unsigned modules it can mean trouble. We cover the detection of tainted kernel modules in this post, but we can also find it with some basic commands.

For this rootkit, the default malicious kernel module name is vmwfxs. This can obviously be changed by attackers easily so we don't recommend searching for this string directly, but you can use the following commands to see what tainted modules may be in use on a system and this can expose new variants.

These three commands will show taint status, or the deliberate loading of the module vmwfxs which is the name used by this rootkit.

The output from an affected system will look like the below when searching for taint status in the dmesg command. Evaluate any output to see if a module name looks suspicious or unexpected.

Since dmesg is a ring buffer, it's possible that messages could have rolled out over time. Another area to check that may have more persistence is /var/log/kern.log found on some systems. Below we see multiple tainted messages when the system rebooted and loaded the kernel module on each startup. The dmesg command won't show this continuity like the on-disk /var/log/kern.log will if available.

In Sandfly we also will identify tainted kernels, but go a step further to see if there is kernel taint but no module listed as having caused it. This often means a module loaded and then removed itself from listing to hide. Unfortunately while this is a great check, there is no direct command line tool to run it broadly so we recommend using our detection if available.

Additionally, for customers using drift detection we will alert if taint status has shifted from the expected profile. This is another great option if available to you.

Hidden Files and Directories

When the rootkit installs, it creates files to hide the kernel module, backdoor program, and persistence mechanisms to make sure it loads on boot. The files are:

Hidden Malicious Module

The malicious kernel module is located under /usr/lib64/tracker-fs. This file cannot be seen when the module is active, but can be read with normal system utilities if you reference it directly like below:

The two commands above both show file information if the kernel module is present, even if you can't see it in the directory listing with standard ls commands.

In this example we did an ls of the /usr/lib64 directory and do not see the module. But, by running the stat and file commands on the filename directly we can see that they do in fact exist but are simply invisible.

Hidden Backdoor Binary

Like the kernel module, there is a hidden binary that is executed when the backdoor is activated. It is hidden under /usr/include/tracker-fs/tracker-efs. Below we see a simple ls command in /usr/include that does not show the hidden tracker-fs directory. However you can go into this directory and the ls command will show the backdoor binary like below:

Persistence Mechanisms

The backdoor drops multiple redundant persistence mechanisms on the host. The files in this variant are located in the following locations and are hidden by the rootkit when active:

The files contain the following simple script to insert the malicious kernel module on boot:

Security teams can simply cat or list the files and see the data. The rootkit is hiding the files from ls but does not prevent them being read like so:

While the above works on the current version, obviously it will not work if these filenames are changed. As administrators cannot see the file with a file list command, if the names change it will make it harder to know what the files are unless you de-cloak them with other means.

Systemd Persistence

The scripts above can also be revealed by listing the systemd services like below.

If you are running Sandfly, you can see the new systemd service under the Host -> Services view. Below we see the tracker-fs.service. The red icon means it also was flagged as a drift detection which was unexpected on this host.

Automating Hidden File Detection

If you think doing the file de-cloaking manually like above is tedious and prone to failure, you'd be right. We again strongly urge customers to use Sandfly to do the hunting for them. Sandfly makes it extremely easy to find these problems as we will instantly de-cloak any file being hidden by this and other rootkits. Sandfly's detection will work regardless of what they rename the files. Below we see all hidden files with full file de-cloaking. We are designed to do this type of work agentlessly, quickly, and safely.

Backdoor Binary Strings Analysis

A simple strings command on the backdoor binary will show malicious data confirming the situation. Below we see very suspicious content in this file such as anti-forensics strings and more.

Communications Socket

The kernel module and backdoor communicate over a socket located in the /proc directory. On the current version it is located at /proc/acpi/pcicard and can be found by running:

The Phrack article details the operation of the backdoor communicating over this socket. For our analysis we did not investigate it closely beyond initiating the backdoor on the system as we will discuss shortly.

Backdoor Operation

The main purpose of the rootkit is to create a persistent and stealthy backdoor. The backdoor is activated when a special magic packet is received combined with a correct password to initiate an SSL connection. The backdoor can be activated on any port. This is important to understand because a firewall alone may not protect the target system. If the magic packet is able to hit the victim, then the backdoor may be activated. Because it can also run on active ports, the backdoor traffic can blend in with encrypted services like a web server or SSH. Or, it can go to a completely unused port and potentially avoid other defenses.

Backdoor Password

The backdoor requires a password to operate. While it's set to a single value in the variant that was made public, it is trivially changed and teams cannot rely on searching for it alone. The password however will prevent activation by unauthorized users.

Backdoor Features

The backdoor itself has some basic but powerful features as we can see in the help menu below:

The basic commands are:

• Start a shell on target system.
• Upload/Download files.
• Move to another machine using backdoor for lateral movement.
• Open a SOCKS5 proxy.
• Show connection information.

Hidden Backdoor Process

The backdoor binary is located under /usr/include/tracker-fs/tracker-efs. When active, it will be a hidden process and cannot be seen by system commands like ps. It also hides network port activity from tools like ss. Below we see the output of the ss command to list all processes with active ports. One of the entries is missing the process data entirely which is the active backdoor.

Even though the process is hidden from system commands, Sandfly does in fact de-cloak it entirely.

Backdoor Anti-Forensics

The shell will open with basic anti-forensics capabilities to prevent logging of commands and timing out. It does this by setting various history environment variables to /dev/null. It also tries to set the shell timeout to 0. Unfortunately since the process is hidden it is difficult to inspect. However, you can use our free sandfly-processdecloak tool to de-cloak hidden processes like below:

sandfly-processdecloak GitHub Repo

Once you get the cloaked PID, you can see the process environment with this command:

The output for our cloaked process (PID 2533) is below showing the suspicious environment variables setting the history files to /dev/null.

Below we see the anti-forensics of the active backdoor as reported by Sandfly. Shells spawned by the backdoor process inherit the anti-forensic features as well.

Backdoor Multi-Hop, SOCKS5 Proxy and File Utilities

The backdoor allows the operator to chain together multiple rootkits to move between systems. It also has the ability to upload and download files and start a SOCKS5 proxy.

Below we see the help of the trans command. There are many interesting parameters the attacker can use to move between systems including changing port knock and protocol types. Using this mechanism they can use the stealth rootkit to move between systems without leaving traces on the affected hosts.

Likewise, the SOCKS5 option also has commands for the attacker to leverage:

Backdoor Shell

There is of course an option to launch a shell on the host to directly attack the impacted system.

When the shell launches, it too will be hidden as part of the backdoor parent process. You will need special tooling to de-cloak it again like Sandfly below does or our free process de-cloaking tool referenced above.

Letting Sandfly's AI Analysis Help

The above is a lot to take in, however we recently introduced AI LLM integration into Sandfly 5.5. Our agentless Linux forensics data is excellent and this results in very good analysis from the leading LLM providers and on-prem solutions. Below is our AI analysis of the rootkit alerts with a very on-point summary of what is going on. We encourage customers to use this feature to help with their triage of alerts.

Cleaning Up Affected Systems

As we always get this question, we'll just state very clearly: You should not attempt to clean up any Linux host impacted by a rootkit. The system should be isolated, have root cause analysis performed to figure out how the attacker got onto the box, and then destroyed. Once root-level access is obtained it is simply not possible to know what was done to the system and it should be re-built.

Final Words

We try to not make these write-ups too much of an advertisement, but the reality is that a stealth rootkit like this needs to have automated hunting done and manual review only after systems impacted have been identified. In the case of this rootkit, it uses a fairly well-known base (khook library). However, it adds on a powerful backdoor allowing silent operation through multiple hosts to achieve lateral movement. The Phrack article lays out the case that it's part of a nation state toolkit and this is likely true based on our review along with the other leaked data.

We encourage customers to keep an eye on all their Linux systems. We offer a free trial and incident response license to use our agentless Linux security solution. Please contact us if you would like to take advantage of this.

Acknowledgements

We'd like to thank the authors at Phrack for the data dump and initial analysis. Please visit their website for more details:

Phrack

The raw dump of the threat actor's hard drive complete with Phrack write-up is below:

Phrack North Korean Data Dump

Indicators of Compromise

We include the following list of Indicators of Compromise below, with the warning that these values are easily changed by the rootkit authors and we'd expect them to be stale in short order. We urge generic attack tactics hunting vs. searching for filenames and other fragile indicators on Linux. We also encourage security teams to not use hashes to find Linux malware as they are extremely unreliable.