Introduction

Security is paramount importance for system admins. There are so many tools for security such as firewalls and network sniffers and so on. However sometimes we need to expose a service to the public and still we need to make sure all our users are legitimate.

A simple example of this is an FTP server, we don't know where the client will connect to the server -so we can not use a simple IP firewall- however we still need to protect from password guessing bots. There are so many applications similar to this. Fail2ban shines as a security tool in these situations with its flexible configuration and default filters. Fail2ban works with the daemons/services log files and matches the unwanted entries with regex. Once unwanted requests made it follows and -depending on the configuration- bans the IP that made the request.

Prerequisites

Any recent Ubuntu Server would work with all the commands provided.

We are using an Ubuntu 20.04 LTS in this article.

Preparing Our Server

Let's prepare our server for our setup, we will update the local package index and upgrade any packages that are outdated.

$ apt-get update
Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists... Done                        


$ apt-get -y upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
....
....

Installing Fail2ban

Ubuntu default repositories includes Fail2ban so we do not need to add any external repositories.

Let's install Fail2ban package using apt package manager.

$ apt install -y fail2ban
Reading package lists... Done
Building dependency tree       
Reading state information... Done
....
....

Enable Fail2ban to persist boot

$ systemctl enable fail2ban
Synchronizing state of fail2ban.service with SysV service script with /lib/systemd/systemd-sysv-install.

Start the Fail2ban service

$ systemctl start fail2ban 

Let's check the Fail2ban service's status

$ systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
     Loaded: loaded (/lib/systemd/system/fail2ban.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2020-06-03 12:43:39 UTC; 3s ago
       Docs: man:fail2ban(1)
    Process: 1395 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
   Main PID: 1396 (f2b/server)
      Tasks: 7 (limit: 614)
     Memory: 12.2M
     CGroup: /system.slice/fail2ban.service
             └─1396 /usr/bin/python3 /usr/bin/fail2ban-server -xf start

Jun 03 12:43:39 5eca7132aa8cc30001a9eb8b systemd[1]: Starting Fail2Ban Service...
Jun 03 12:43:39 5eca7132aa8cc30001a9eb8b systemd[1]: Started Fail2Ban Service.
Jun 03 12:43:40 5eca7132aa8cc30001a9eb8b fail2ban-server[1396]: Server ready

Fail2ban Configuration Files

There are several configuration files Fail2ban uses, here is the directory structure at the time of writing.

$ /etc/fail2ban$ ll
total 72
drwxr-xr-x   6 root root  4096 Jun  3 10:19 ./
drwxr-xr-x 112 root root  4096 Jun  3 12:20 ../
drwxr-xr-x   2 root root  4096 Jun  3 10:19 action.d/
-rw-r--r--   1 root root  2817 Jan 11 10:01 fail2ban.conf
drwxr-xr-x   2 root root  4096 Mar  2 13:20 fail2ban.d/
drwxr-xr-x   3 root root  4096 Jun  3 10:19 filter.d/
-rw-r--r--   1 root root 25740 Jan 11 10:01 jail.conf
drwxr-xr-x   2 root root  4096 Jun  3 10:19 jail.d/
-rw-r--r--   1 root root   645 Jan 11 10:01 paths-arch.conf
-rw-r--r--   1 root root  2827 Jan 11 10:01 paths-common.conf
-rw-r--r--   1 root root   573 Jan 11 10:01 paths-debian.conf
-rw-r--r--   1 root root   738 Jan 11 10:01 paths-opensuse.conf

Operation

Let's check our default enabled ssd jail

$ fail2ban-client status
Status
|- Number of jail:      1
`- Jail list:   sshd

Let's view our jail's status

$ fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     40
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 1
   |- Total banned:     8
   `- Banned IP list:   z.z.z.z

We can see that sshd jail is using filter monitoring /var/log./auth.log file. We can see that fail2ban already got on with it and banned an IP trying to guess our password.

To see the process better let's try to trigger ssh filter via trying the wrong password multiple times, you will see that at the end "connection refused" message, following commands are entered from a different server/client

$ ssh x.x.x.x
[email protected]'s password: 

Permission denied, please try again.
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 
[email protected]: Permission denied (publickey,password).
$ ssh x.x.x.x
[email protected]'s password: 

Permission denied, please try again.
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 
[email protected]: Permission denied (publickey,password).
$ 
$ ssh x.x.x.x
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 
[email protected]: Permission denied (publickey,password).
$ ssh x.x.x.x
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 
[email protected]: Permission denied (publickey,password).
$ ssh x.x.x.x
[email protected]'s password: 
Permission denied, please try again.
[email protected]'s password: 


^C
$ ssh x.x.x.x
ssh: connect to host x.x.x.x port 22: Connection refused
$ 

You can see that we are blocked.

Let's have a look to our hosts' iptables, as we can see y.y.y.y (our ssh client machine) is banned and will be refused to connect.

$ iptables -L -v -n
.....
Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   24  2028 REJECT     all  --  *      *       z.z.z.z              0.0.0.0/0            reject-with icmp-port-unreachable
   26  2028 REJECT     all  --  *      *       y.y.y.y              0.0.0.0/0            reject-with icmp-port-unreachable
 3352  268K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 .....

As you can see fail2ban creates chains in iptables to manage the reject actions, we are seeing the one for ssh here.

Whitelisting

This is a good place to talk about whitelisting, sometimes we can try multiple times legitimately and can block ourself from the machine, to avoid this we can whitelist our own IP addresses in the jail.conf as follows:

....
[DEFAULT]

ignoreip = 127.0.0.0/8 10.0.0.0/8 y.y.y.y
....

We have whitelisted following:

  • 127.0.0.0/8 is an internal block
  • 10.0.0.0/8 is our own VPN block
  • y.y.y.y is our external management server

Also you can remove an existing ban with the unbanip command

$ fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 7
|  |- Total failed:     581
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 1
   |- Total banned:     29
   `- Banned IP list:   z.z.z.z


$ fail2ban-client set sshd unbanip z.z.z.z
1

$ fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 7
|  |- Total failed:     582
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     29
   `- Banned IP list:

Demonstration with Nginx

Here we apply the required configurations from scratch to ban clients that request non-existing pages repeatedly.

First we install Nginx

$ apt install nginx
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  nginx
0 upgraded, 1 newly installed, 0 to remove and 10 not upgraded.
Need to get 3616 B of archives.
After this operation, 45.1 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 nginx all 1.17.10-0ubuntu1 [3616 B]
Fetched 3616 B in 0s (161 kB/s)
Selecting previously unselected package nginx.
(Reading database ... 78873 files and directories currently installed.)
Preparing to unpack .../nginx_1.17.10-0ubuntu1_all.deb ...
Unpacking nginx (1.17.10-0ubuntu1) ...
Setting up nginx (1.17.10-0ubuntu1) ...

Create a file under /etc/fail2ban/filter.d/ called nginx-404.conf , copy the contents of the below and save.

[Definition]

# x.x.x.x - - [03/Jun/2020:13:34:35 +0000] "GET /admin HTTP/1.1" 404 163 "-" "curl/7.68.0"
failregex = ^<HOST> -.*"(GET|POST|HEAD).*" 404 .*$

ignoreregex =

Here we tell Fail2ban what to look for, this is a regex expression for logs lines to match

Now create a file under /etc/fail2ban/jail.d/ called nginx-404.conf, and copy the contents of the below and save

[nginx-404]

enabled = true
filter  = nginx-404
logpath = /var/log/nginx/access.log
port    = http,https
maxretry= 5
findtime= 60

Reload fail2ban so it reads the new configuration files

$ systemctl reload fail2ban

Let's check our new jail is there

$ fail2ban-client status
Status
|- Number of jail:      2
`- Jail list:   nginx-404, sshd

On an another machine run the following 6 times

$ curl http://x.x.x.x/admin
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.17.10 (Ubuntu)</center>
</body>
</html>

At the end you will be banned

$ curl http://x.x.x.x/admin
curl: (7) Failed to connect to x.x.x.x port 80: Connection refused
$ fail2ban-client status nginx-404
Status for the jail: nginx-404
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     5
|  `- File list:        /var/log/nginx/access.log
`- Actions
   |- Currently banned: 1
   |- Total banned:     1
   `- Banned IP list:   y.y.y.y

In this demonstration we have configured maxretry a small number, for the real use cases this number must be increased.

You can again remove the ban with the unbanip command

Conclusion

Fail2ban is a great security tool which is highly configurable and already includes many default configuration you can use. Please explore filter.d directory to see what filters can be applied. Big thanks to authors of Fail2ban.