Getting an Attacker IP Address from a Malicious Linux At Job

Attackers on Linux often try to establish persistence so they can maintain access to a host. The most common way to do this is with a cron job that is scheduled to run and re-install malware if it has been removed. However, an often overlooked persistence method is putting scheduled jobs in the Linux at job scheduler.

The at command allows users to schedule commands to run just like with cron, but they tend to be commands to run one time in the future and not commands to run repeatedly like cron. But consider if an attacker inserts a command to run six months from the date of intrusion. You may think you got rid of them only to find they return at a future date when their command runs and rolls out the welcome mat again for them.

This is why it’s always a good idea to check your Linux hosts for at commands you don’t recognize with a simple atq command line call when on a host. But also, it’s not a bad idea to manually review at jobs to make sure they are not running things they shouldn’t. For incident responders, checking for malicious at jobs is mandatory. This is because if an attacker has inserted an at command you are in luck because by default many Linux systems put the IP address of the system into the entry.

Checking for Malicious Linux At Jobs

There are two ways to view at jobs on a host. You can use the atq command to dump all at jobs and then the at command to view them.

The above we see four jobs for the root user. The first column is the at job number which we can list individually with the at -c <jobnum> command:

This dumps each job number and you can see some interesting things. The most important bits are all those environment variables and the information they contain.

If you want to see all the at jobs at once quickly, you can simply use the cat command to dump them like this:

The above will dump out all the entries so you can scroll through them looking for anything odd. In a typical job entry you'll find things like:

• User and binary names to investigate.
• Shell settings that could indicate the type of system that inserted the attack.
• SSH connection data with IP addresses of the local and remote systems.
• Other IP addresses and even proxy information setup in the environment.
• Random bits of leaked data set automatically by the shell during login that the attacker would rather you not find.

At jobs are a goldmine of forensic data and shouldn't be overlooked. See the image below for some real-world examples of what an Ubuntu system at job would leave in the entry.

Always Check At Jobs During Incident Response

In general, at jobs are much less frequently used than cron on Linux hosts. If you see an at job and you didn’t put it there, it’s a good idea to find out who did and why. If you’re investigating an incident, checking for roque at jobs is a good idea and can give you valuable clues about an attacker even if they deleted their connection information from other logs on the target host.

Have Sandfly Check for Malicious At Jobs

Even better than waiting for an incident before checking your at jobs, you can have our agentless security platform do it for you. We check for persistence attacks on Linux 24 hours a day. Why not give Sandfly a free trial?