How to check your server for suspicious activity in Linux
If you have even a small hunch that your server has been compromized, it bears further investigation. In fact, it's good practice to periodically review your server even if you suspect nothing. But what should you be looking for? How can you tell what files have been modified, whether you have malicious software on your server, and which processes are problematic? This guide is designed to answer all of those questions.
Checking your recently modified files
Looking at the recently modified files list in Linux can be an excellent way of spotting suspicious changes to your server – provided you know what you're looking for. You can find a list of files using the find command, which we have an extensive guide on here. For this use case, though, the most useful syntax would be something like this:
find -type f -mtime -15 -ls
This will find all files that were modified in the last 15 days. You can naturally change this number to reflect a period that is more realistic for your use case. You can also use +15 for all files that were modified 15 days ago (for example, on March 5 if you run the command on March 20). This is useful if you suspect a compromize on a specific day.
Checking a specific folder is often useful too – for example, your log folder, to see if any have been modified or updated:
find /var/log/ -type f -mtime -15 -ls
To investigate further, you can then check the timestamp of a specific file using stat. For example:
stat /var/log/malware.sh
File: malware.sh
Size: 6 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 1871 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-03-10 08:13:10.903053851 +0000
Modify: 2023-03-10 08:13:10.903053851 +0000
Change: 2023-03-10 08:13:10.903053851 +0000
Birth: 2023-02-10 08:13:10.903053851 +0000
You should be looking for anything that does not make sense – unusual permissions, dates of files that don't add up, and files (such as logs) that should be there but aren't.
Checking your log files
Most actions on your Linux server will be logged, which makes checking your log files one of the best ways to spot suspicious activity. However, you must be aware that a clever attacker will modify your log files to remove their activity. You can use the stat command to tell when a file was last modified, but be aware that this can be falsified too. Checking log files is not a foolproof way of spotting malicious activity, then, but it will help to catch low-hanging fruit – malware or actors who are not proficient enough to hide their activity.
Some of the logs you should check include:
- /var/log/auth.log: Shows all authentication logs. This allows you to investigate failed login attempts password changes, etc. On RedHat or CentOS, this will instead be
var/log/secure. - /var/log/faillog: A list of failed logins, useful for investigating brute force/login credential hacks.
- /var/log/wtmp: A record of every login/out. Check whether someone was accessing the server at a time they shouldn't have been.
- /var/log/apt: Logs when an application is installed using
apton Ubuntu machines. Useful for checking if any unsanctioned applications have been installed. On CenOS this will be/log/yum. - /var/log/syslog: Shows a log of activity data throughout the system, providing an overview of the general goings on. On CentOS/RHEL this will be
/var/log/messages.
Provided the attacker has not covered their tracks, combing these logs should give you a good idea of whether something suspicious has occurred.
Checking for suspicious processes
Most servers are hacked for their resources, not their information. If your server is compromized, it's most commonly so that an attacker can use it in a botnet or for brute force attacks on other servers. To do this, the attacker (most likely via an automated tool) will establish several outbound connections and create an associated process.
By analysing our processes, we may be able to figure out if anything suspicious is going on. But, be aware – it's not uncommon for hackers to replace the tools you'll be using to investigate with hacked versions that won't give you accurate information. Remain skeptical if other signals are pointing to a server compromize.
First things first, check your active connections using netstat -an. This will show you all connections and their port numbers in a list.
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 193.149.189.216:38038 91.189.91.38:80 TIME_WAIT
tcp 0 0 193.149.189.216:22 81.229.38.48:60860 ESTABLISHED
tcp 0 1112 193.149.189.216:22 83.121.2.254:50537 ESTABLISHED
tcp 0 0 193.149.189.216:22 83.123.9.169:60272 ESTABLISHED
tcp 0 0 193.149.189.216:22 2.187.234.37:22363 ESTABLISHED
tcp 0 33 193.149.189.216:22 83.121.2.254:49256 LAST_ACK
tcp 0 33 193.149.189.216:22 5.216.59.94:51280 LAST_ACK
The entries above are quite normal looking. TCP traffic on ports commonly used by genuine processes. However, let's say we want to investigate what processes are making connections on a particular port. We'll go with port 50537, as that's on our list:
lsof -i tcp:50537 -P -R
The lsof command (list running files) should return a list of which processes that are using this port, as well as their parent process.
COMMAND PID PPID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 49728 3453 root 4u IPv4 334132 0t0 TCP 64117a5178d79d0001fa80a7:22->81-229-38-48-no2744.tbcn.telia.com:50537 (ESTABLISHED)
We can find out which files the process is using by filtering by its PID:
lsof -p 49728
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 49728 root cwd DIR 8,1 4096 2 /
sshd 49728 root rtd DIR 8,1 4096 2 /
sshd 49728 root txt REG 8,1 917192 2757 /usr/sbin/sshd
sshd 49728 root mem REG 8,1 309600 23258 /usr/lib/x86_64-linux-gnu/libnss_systemd.so.2
sshd 49728 root mem REG 8,1 18424 3109 /usr/lib/x86_64-linux-gnu/security/pam_env.so
sshd 49728 root mem REG 8,1 26696 3120 /usr/lib/x86_64-linux-gnu/security/pam_limits.so
sshd 49728 root mem REG 8,1 18424 3124 /usr/lib/x86_64-linux-
This will be a long list, but it will tell you where the program is located. We can see here that ours started in /usr/sbin/sshd .
Dealing with the malicious program
If you believe it is malicious, you can kill it with kill 49728 (replace this with the right PID). You can then remove the file's execution rights with chmod -x /usr/sbin/httpd.
That should stop it running for now, but it's worth making sure that it, or another start-up command that will recreate it, isn't set to run whenever you boot your server. For this, we can use the usual crontab -l and remove entries coming from your suspect folder by using crontab -e and deleting its entry.
That (hopefully) deals with this particular malicious program. But there could easily be more, which is why it's usually safest to just rebuild your server entirely. Check common folders such as /etc and /tmp/ and try to root out other suspicious processes using the steps above.
Once you're done, update your software and server to ensure they're patched against the latest vulnerabilities.
Scanning for malicious software with RKHunter
You should never 100% rely on automated tools for security, but RKHunter or other anti-virus software can be a good supplement to manual checks. RKHunter helps you to identify whether your server has a rootkit by checking for common variants in common locations. It will also check for malware by checking running processes for suspicious files, identifying login backdoors, searching for suspicious directories and apache behavior, checking for suspicious network port usage and system start-up files, etc.
It's important to understand that passing the test doesn't necessarily mean that you don't have a rootkit or malware just that RKHunter couldn't find any. There is a possibility that a malicious process is interfering with RKHunter's output or that it simply does not have the capability to detect the malware you have. Still, it is a useful time-saving tool for identifying common security issues and can be used to run regular precautionary scans.
First, install it:
sudo apt update
sudo apt install rkhunter -y
Now let's perform some configuration. Open the config file using nano:
sudo nano /etc/rkhunter.conf
Look for WEB_CMD="/bin/false" manually or by using the find function via Ctrl + W. Comment out the line by adding a # in front of it.
#WEB_CMD="/bin/false"
Find UPDATE_MIRRORS and set it to option 1:
UPDATE_MIRRORS=1
This will make RKHunter check the mirrors file for updates when we use the --update option later.
Also set MIRRORS_MODE to 0. This will allow RKHunter to use any mirror when looking for updates, rather than just local ones.
MIRRORS_MODE=0
Press Ctrl + O followed by Ctrl + X to save and exit. Double-check that you haven't made any typos by running sudo rkhunter -C. No output means your config file has passed the test.
Now you can update RKHunter's data files and run a check:
sudo rkhunter --update
sudo rkhunter --check
RKHunter will provide a helpful list of what it has checked for and whether it has found any issues. Once it has finished, you can view the full log by typing:
sudo nano /var/log/rkhunter.log
Should RKHunter find any suspicious files, use the process outlined above to remove its execution rights and perform other necessary actions, such as using rm to delete it.
Closing words
That about covers the basics of checking for malicious server activity in Linux, but there is obviously a lot more to it. If you are in doubt and using your VPS for important or sensitive tasks, pay a Linux security professional.
For additional help, BitLaunch customers can reach out to our support via live chat. Our support agents cannot diagnose the security issues of every customer's server, but they may be able to point you in the right direction.