Sandfly 2.6.0 – Elasticsearch Replication, Linux Docker Container Security Scanning, Hidden Process De-Cloaking and More

Sandfly 2.6.0 has been released and now has the ability to use external Elasticsearch databases. This new feature allows you to use Elasticsearch’s Kibana and other tools to analyze and display Sandfly data.

We also added Docker container scanning for file and directory Linux attacks. This is on top of our process level Docker container attacks we already searched for in prior versions.

Further, we added in new forensic engines to detect and de-cloak processes being actively hidden by Linux Loadable Kernel Module (LKM) stealth rootkits. Plus, new modules to hunt for other signs of compromise and intrusion such as incorrect permissions on critical files and strace artifacts.

Finally, we’ve improved internal caching so security sweeps are faster yet again, taking just seconds each time we investigate a host.

External Elasticsearch Database

We now give users the ability to deploy Sandfly using an external Elasticsearch database. This means customers can now use their own Elasticsearch cluster they may already have, or use a cloud-based Elasticsearch hosting service to hold Sandfly data.

Sandfly still uses an internal Elasticsearch database for customers with smaller deployments that do not want the trouble of maintaining their own cluster. This works well, but it means that customers wishing to use tools such as Kibana to view and search the data were not able to easily do so. Now, customers can host their own Elasticsearch database and use tools like Kibana as they see fit.

During install, customers will have the chance to enter in the URL for the remote Elasticsearch database they wish to use. After that, all results will be available using standard Elasticsearch features including Kibana, machine learning modules, redundancy and so on.

Elasticsearch Results Replication

We also added in a new feature where results from Sandfly can be replicated to a secondary Elasticsearch database. Customers can deploy Sandfly servers inside different network segments but have all results go to a central location for consolidated viewing and analysis.

Sandfly will only send results data to the replication database. Encrypted credentials, host, scheduling and related data are not sent to this database. This feature allows customers to have SOC personnel view results but they will have no ability to change system operations. Customers can however leverage Sandfly’s REST API to control scanning with security orchestration tools or custom scripts. Or, they can use the built-in user interface as normal.

During install you can setup an optional URL for results replication. Existing installations can simply add the replication database URL into their setup files and it will work immediately. Customers that wish to use this feature can contact us for details on how to set this up quickly.

Elasticsearch Kibana Dashboards

With either external database option above, you can use Kibana to create dashboards and search the extensive Linux system and forensic data Sandfly delivers to you agentlessly. Not only can you quickly find security events, but you can also generate tables around operating system versions, hardware, users, and anything else you can probably think about running on your Linux hosts.

We have sample Kibana dashboards available to customers. Here we’re using Sandfly to monitor Linux hosts locally, at Amazon AWS and Digital Ocean at the same time. We’re doing this on Linux kernels back to 2.x all the way to modern versions. Plus, we’re watching AMD and Arm based processors because Sandfly works across Intel, AMD, Arm and MIPS CPUs. Sandfly allows you to monitor all your Linux hosts with one platform and without needing to load agents everywhere.

Here is another example where Sandfly gives you visibility to spot processes running with network connections on your hosts. Hot spots show the most common process and ports, while lighter colors show less used processes with open ports.

You can also obtain user login data without persistent agents on your hosts. Sandfly can easily pull over user login data and save it over time to tell you who is and is not logging into your systems. You can easily report failed logins across all your Linux hosts regardless of version, kernel patches, distribution, or even CPU type. You can also search across hosts for any usernames that did login and at what times. We pull user login data from all of your Linux systems agentlessly.

Sandfly with Kibana’s tag cloud can quickly and easily show you suspicious processes. Below we’ve flagged processes running from /tmp or /dev/shm directories and also a couple processes running with suspicious extensions. You can customize your threat hunting and query against hundreds of system and forensic artifact keywords we retrieve from your Linux hosts.

There is so much more to show about how to leverage Sandfly’s agentless visibility into Linux with Kibana and other tools like Splunk. Stay tuned for more updates.

Docker Container Scanning

Sandfly has been able to detect Linux process attacks inside Docker containers since version 1.x of the product. Today we are introducing container aware scanning for our extensive list of file and directory attack signatures as well.

Customers running Docker containers do not need to do anything to enable this feature. Sandfly is container aware and will see if containers are running on a host and automatically check them with our suite of signatures, completely agentlessly of course. This process is fast and has no risks to the running containers as we never write anything to the images or attach to the instances with invasive kernel probes. The containers have no idea we are there (and neither do any attackers which may be operating inside them).

Alerts detected by Sandfly will show the container ID, and container layers that are affected to allow investigation. Alerts show up as normal and no changes need to be made by customers to have this new capability. Additionally, a new process flag has been introduced to mark any process that is running inside a container (process.flags.containerized). This flag is easily searchable inside tools like Kibana or Splunk to quickly isolate containerized processes vs. host operating system processes.

Linux Stealth Rootkit Process De-Cloaking

We have added in new forensic checks that will completely de-cloak and identify processes being hidden by Linux Loadable Kernel Module (LKM) stealth rootkits. These checks sweep a system and quickly identify any process that is trying to hide. We will flag hidden processes with a new flag (process.flags.hidden) to allow it to be easily searched inside your SIEM tools and allow new rules to be built around it as well.

Default Command Excessive Time Running

Sandfly now checks for default commands that are taking too long to execute. Malware is known to try to hide long-running processes as common names to avoid detection. For instance, you might want to know if commands like date, ls and swapoff have been running for five hours. At least, we would want to know that. Now, you do.

Risky Permissions on /etc Files

We are now flagging risky permissions on files such as /etc/sudoers, /etc/group and more. We have seen tactics where these files have been set with open permissions to allow attackers to add in accounts and gain privileges. It is easy to overlook open permissions on these files and not know someone is using them as a backdoor to elevate access on a host. Sandfly will keep an eye on them for you.

Enhanced Malicious Packet Sniffer Detection

We have improved rules to spot malicious use of network sniffers on a host. This detection will look for invocations of sniffers being used to grab legacy unencrypted credentials. Protocols like FTP, POP3, and more are still in use and offer no protection to login credentials theft if grabbed over the network. If we see a process running with flags indicating it is interested in these protocols and is sniffing traffic, we’ll alert.

Suspicious User Home Directories: /dev/shm or /tmp

We’ll now flag any users with home directories such as /tmp or /dev/shm. These directories are a crowd favorite for malware and automated backdoor accounts. At best, a legitimate user with these directories set as their home is a security risk and should be changed. In either case, we’ll let you know if we see it.

strace Artifacts from Keyloggers or Other strace Activity

We have been looking for suspicious strace use in process tables for some time, but now we’re now hunting for artifacts of strace left behind in common areas. Strace can be used for legitimate purposes, but can also be done for unfriendly reasons such as grabbing SSH login credentials. We will flag not only strace files left behind that look like part of SSH key logging activity, but also any kind of strace file that was left behind in a publicly readable areas such as /tmp or /dev/shm. Even if an strace log is legitimate, they can often contain sensitive information about a program and should be in a private directory.

Expanded Reconnaissance Checks for Files

We have added in a variety of reconnaissance checks to pull file attributes (such as cryptographic hashes) from common system areas such as /etc and system binary directories such as /bin, /sbin, /usr/bin and so on. These reconnaissance modules can pull the data back to store in your Elasticsearch backend for change tracking and detection of modified files. These modules can generate a lot of data and are disabled by default. However if you have the space, they can be very powerful for helping to search for malicious binary hashes back in time or other file attributes for incident response.

Higher Performance Caching

We have added in better caching for higher performance on hosts. Sandfly can do a complete Linux host assessment for compromise typically in under 30 seconds. That’s running almost every security sweep we have which is not the default configuration. For normal scheduled security threat hunting we can be on and off a host in about 15 seconds without leaving any traces on your endpoints. Incident response checks have also been sped up as a result and you can have answers to investigations faster than ever (typically again in under 15 seconds). Agentless security is fast and accurate.

785 Sandfly Checks and Growing

Sandfly 2.6.0 has now brought the number of compromise and incident response checks we do on Linux up to 785. We can spot a tremendous amount of Linux malware, rootkits and intruder activity without loading any agents on your endpoints and without disruptive updates. Thank you for using our product.