Recovering from a VPS server compromise on Linux

BitLaunch -

If you have analyzed your Linux VPS for suspicious activity and discovered unauthorized activity, you are unfortunately in a very difficult position. Though it is tempting to ignore it and hope for the best, you must assume your server is compromized and take the necessary steps to protect yourself and your users.

It is very difficult to determine how bad a Linux system compromize is and whether you have spotted all instances of tampering. If you are using the server in a production environment, it is simply not enough to remove revert the tampering you notice or run a virus scan.  How you proceed will likely depend on what your server is being used for and the type of compromize.

For example, If you are quite sure your attacker has root access then you'll need to take the nuclear option. If it was a DDoS attack or your domain was forwarded to another via your registrar account, you may not need to take action on your server at all.

If you cannot determine the source of the compromize or suspect your server has been hacked, you have a few options:

  • Create a fresh new server (safest)
  • Rebuild your server completely, erasing all of its data (safe)
  • Examine your data and copy it to another VPS or an offsite VM (risky)
  • Restore your server from a backup (risky)

We'll go through these in more detail in a bit, but first, you need to make sure that your VPS account is secure.

Securing your VPS account

If your VPS account has been compromized, any of the changes you make are pointless. You can rebuild or re-create servers all your like, but the attacker will likely just get back in and make the same changes. As a result, we recommend that BitLaunch users do the following after noticing a compromize:

  • Check their account history for unauthorized use ( Settings > Security)
  • Change their account password
  • Enable two-factor authentication
  • Create a new SSH key to use with your server(s) and reset the server root passwords (if applicable)
  • Check your DNS records for any tampering and remedy them

You should ideally follow these steps in the above order.

Creating a new server from scratch

Starting from scratch with a fresh server that has a new SSH key/password, IP address, etc. ensures that the attacker's ability to influence your server has been expunged (provided you have secured your account). BitLaunch users can achieve this by pressing the green plus icon in their control panel.

When creating your server, we recommend using a new password or preferably and SSH key for the best security. Point your DNS records to the new server if required and destroy the hacked one once you have finished recreating it. Make sure you are careful about how you set up your new server so that you do not introduce the same vulnerabilities. For example, if you are using WordPress, be careful about the plugins you install.

Rebuilding your server

A server rebuild keeps your login authentication and IP address but builds your server based on a chosen system image, destroying all data currently on the server. This is potentially less secure than creating a server from scratch, as if the attacker has compromized your login credentials they will remain the same (at least for a time). However, it's worth noting that you can quickly change your root password after the server has been rebuilt in your BitLaunch control panel. You can also manually replace the SSH key on your server after it has been rebuilt.

BitLaunch users can rebuild their server by pressing the three dots next to it and selecting "Rebuild".

You will be asked to specify an OS image and can then press "Rebuild as x".

Once your server has finished rebuilding, click on it to open its settings and change the root password in the "Access" section.

You can then generate a new SSH key using PuTTy and transfer the public key to ~/.ssh/authorized_keys. Check that the file has the correct permissions and that login is working and then remove your old key.

Copying your files to a new VPS

It may be tempting to create a new server and copy across the files from your website/app. This can work if the compromize comes in the form of an unsanctioned program installed on your server or a crypto miner, etc. However, unless you examine every single file, you cannot be sure that they have not been tampered with also. Even if you do check every file, you may miss something. As a result, this is hard to recommend in a production environment unless you have the ability to hire a security expert to audit your files.

If you decide to go this route, you can create a disk image with:

ssh root@your.ip.address "dd if=/dev/sda " | dd of=/home/folder/backup.img status=progress

Replace your.ip.address with the IP of your new server and /folder/backup.img with the path where you want it saved and the name of your disk image.  It will take a while to copy, after which you can use the disk image to restore your files.

Restore your server from a backup

You may think this is the perfect time to restore your server from one of your recent backups, but that could be a mistake. This is because it's unlikely you'll be able to determine exactly when your server is compromized. A malicious program or code could have been sitting on your server for years waiting for the go ahead from a command and control center. As a result, restoring from a backup is only recommended if you can determine when the first intrusion happened and have one that is sufficiently far away from that activity.

If you are a BitLaunch customer, you will already know that first-party servers do not support backups. We encourage you to make your own backups for the best data control and privacy. However, those using BitLaunch to pay for their Digital Ocean, Vultr, or Linode server can enable the backup option for weekly backups. You can restore backups via the "Backups" section in your server settings.

After restoring a backup, you should of course keep a close eye on your server and check it for malicious activity and audit it regularly.

Final tips

It's important that you do not introduce the same vulnerability that got your server compromised in the first place. Follow our guidance to secure your VPS, search the web for open vulnerabilities in your OS and applications, and be careful who you give access to.

If you are a BitLaunch customer and are unsure of how to proceed which your compromized server, reach out to our support team via live chat. They may be able to assist you further.