New Fail2Ban: Nice config, nice reporting insights :)

(UPDATE: I changed 15 to 14, having discovered the implementation is off-by-one from what I thought! 14 gives 229 days…)

Debian 11 and other OS releases have new versions of fail2ban supporting some nice features:

  • Incremental auto-increases of blocking, so that repeat attackers don’t overwhelm your server
  • Revised underlying database (this also may work in Debian 10 ie f2b 0.10) allowing smart queries and reports so you can better see what’s going on, and see the impact of your configuration

Below, I share how I’ve set up our fail2ban, along with one example of a nice shell-script report. With this as a start, you can hopefully see how to do what you might want… and perhaps some of this may make its way into the Webmin GUI in the future :cowboy_hat_face:

I’ll first provide simple instructions to replicate my setup, then discuss why I did what I did.

SETUP

  1. In /etc/fail2ban/jail.local (Network->Fail2ban->Edit Config Files, and scroll to the bottom of the list), edit or add these uncommented lines:
[DEFAULT]
ignoreip = <provide space-separated list of local ipv4 or ipv6 CIDR addresses, eg 127.0.0.1 192.168.0.0/16 ...>
bantime = 1h
maxretry = 3
findtime = 1d

bantime.increment = true
# 20 minutes
bantime.factor = 1200

#plh start at bantime, ADD shifted count times factor. Max of 14 is 229 days.
bantime.formula = ban.Time + (1<<(ban.Count if ban.Count<14 else 14)) * banFactor

# find bad IP in ANY jail, not just the current one. If they're bad, they're bad.
bantime.overalljails = true
  1. Now set specific jails to a longer default bantime.
  • You can do this in the GUI by clicking on specific jails in Filter Action Jails, or do it right here in jail.local while editing. Just go to the section for a specific jail and add a line like:
bantime = 2d

DISCUSSION

Why set it up this way:

  • The default findtime and maxretry allows someone 3 tries per day to get in by default, without complaint.

    • For some jails, I’ve reduced that further:
      • postfix and postfix-sasl: 2 tries
      • dovecot, anvil and dkim (I created new jails for the latter 2): only one try. People making invalid login attempts are immediately blocked from our server! It’s amazing how many bots are doing that.
  • I played with the parameters of my auto-increment setup in a spreadsheet until I had a result I like. This is not the same as any of the examples provided in the fail2ban templates!

    • I want a reasonably long minimum ban. That’s “bantime” – from an hour to two days. Most jails have a two day minimum for me.
    • I want the ban to grow reasonably quickly to…
      • A long enough time that an attacker isn’t harming my server performance
      • Not so long that a banned IP may well be owned by someone else when unblocked. (One of my early formulas had a max bantime of over 200 years :smiley:
    • With my formula, at 15 tries it’s 229 days, almost 8 months. Ten tries, a little over a week.

A SAMPLE REPORT (Not available in any existing app)

Fail2ban uses a sqlite3 database. You can use the sqlite3 CLI utility to extract data quickly and with relative ease. My script below does much of the hard work for you (took a bit to discover how to extract that final field :wink: )

NOTE: you may not have the sqlite3 CLI package installed, even though the database is in use. Just use Webmin tools or the shell (eg apt install sqlite3) to get it.

Example: list the 25 IP’s with with longest future ban time, including why they are banned:

sqlite3 -header -column 'file:/var/lib/fail2ban/fail2ban.sqlite3?mode=ro' "select datetime(timeofban,'unixepoch','localtime') as startofban, datetime(timeofban+bantime,'unixepoch','localtime') as endofban, ip, jail, bantime, bancount, substr(json_extract(data,'$.matches[0][2]'),1,100) as Detail from bips where endofban > datetime('now','localtime') order by endofban desc limit 10"

This creates a table with:

  • startofban - timestamp when the ban began
  • endofban - timestamp when the ban will end
  • ip - banned address
  • jail - which jail it’s in
  • bantime - seconds represented by (endofban - startofban)
  • bancount - how many times this IP has been banned
  • detail - the specific reason for the ban. Usually includes things like what address and PW was used for a failed login or email attempt, etc. Limited to first 100 characters of the details.

Example: to see the shortest bans, remove " desc" from the ‘order by’ query
Example: want to see this for each jail? change ‘order by’ to “jail, endofban”

WHY THIS HELPS
It’s easy to imagine that system admins shouldn’t be concerned about details like this. In general, I agree! For example, Webmin provides a nice level of information in the Jails Status and Actions tab, which lets me know I have several thousand IP’s banned in my jails, and none in a few (hmmm… need to check the configuration of those jails!)

However, sometimes a quick look at a detail report like above/below can be incredibly helpful. Some questions I find important…

  • Are my bans working correctly?
  • Am I banning IP’s too long?
  • Is there a significant pattern to the bad apples going after my servers?

Here’s a real report extract… the most recent ten bans on my own server:

startofban           endofban             ip               jail          bantime  bancount  Detail
-------------------  -------------------  ---------------  ------------  -------  --------  ----------------------------------------------------------------------------------------------------
2023-03-08 16:42:22  2023-03-10 16:42:22  182.70.242.21    dovecot       172800   1          aster dovecot: auth: passwd-file(cseab@ds.org,182.70.242.21): unknown user
2023-03-08 16:42:35  2023-03-10 16:42:35  218.22.187.66    dovecot       172800   1          aster dovecot: auth: passwd-file(cseab,218.22.187.66): unknown user
2023-03-08 16:42:45  2023-03-10 16:42:45  46.148.40.131    postfix-sasl  172800   1          aster postfix/smtpd[7302]: warning: unknown[46.148.40.131]: SASL LOGIN authentication failed: UGFzc
2023-03-08 16:42:57  2023-03-10 16:42:57  115.93.160.115   dovecot       172800   1          aster dovecot: auth: passwd-file(azcqwqz@ds.org,115.93.160.115): unknown user
2023-03-08 16:43:00  2023-03-10 16:43:00  221.225.147.85   dovecot       172800   1          aster dovecot: auth: passwd-file(postmaster,221.225.147.85): unknown user
2023-03-08 16:43:03  2023-03-10 16:43:03  187.103.74.193   dovecot       172800   1          aster dovecot: auth: passwd-file(wuxiwkhtj@ds.org,187.103.74.193): unknown user
2023-03-08 16:43:09  2023-03-10 16:43:09  122.176.115.248  dovecot       172800   1          aster dovecot: auth: passwd-file(elyn@ds.org,122.176.115.248): unknown user
2023-03-08 16:43:10  2023-03-10 16:43:10  203.124.60.115   dovecot       172800   1          aster dovecot: auth: passwd-file(azcqwqz,203.124.60.115): unknown user
2023-03-08 16:43:27  2023-03-10 16:43:27  122.187.230.134  dovecot       172800   1          aster dovecot: auth: passwd-file(px@ds.org,122.187.230.134): unknown user
2023-03-08 16:43:28  2023-03-10 16:43:28  122.187.234.38   dovecot       172800   1          aster dovecot: auth: passwd-file(zbnigye@ds.org,122.187.234.38): unknown user

What this immediately tells me:

  • I’m getting about ten attempts a minute on my POP/IMAP ports, just from new IP’s
  • It does not look like a dictionary attack. Truly random character strings.
  • My fail2ban config is functioning

I’ll stop there. Hopefully this little info-tutorial has been helpful!

SYSTEM INFORMATION
Operating system Debian Linux 11
Webmin version 2.013
Usermin version 1.861
Virtualmin version 7.5
5 Likes

been using this for quite a while now works fine … the only bug bear is not being able to see the results in a structured way through webmin … the firewalled module just dumps all IP’s in one page and the fail2ban module only allows you to see the first 25 entries ( I have 1k+ banned via fail2ban now).
other niggle with this is it does take a time for fail2ban to restore the banned IP’s on a reboot (machine resource dependant) so you will see some ‘already banned’ lines in your fail2ban logs

(Made a small but significant edit: 14 as formula max gives 229 days.)

Excellent summary on fail2ban. I have similar setup but I also add reporting to Abuseipdb.com which is a free service for reporting bad actors. For my CMS where I have failed logins, I send the IP to abuseipdb and if it is 100% abuser I tell fail2ban to ban the IP.

Similarly, when one of the fail2ban jails bans an IP I also have it report that ban and details to abuseipdb for the next person to learn from. The process was not trivial but the script is easy in php. My intrusions are now a few percent from what they were before I started. Maybe a dozen or so daily versus hundreds or thousands daily.

In this script, my CMS detects bad logins, checks the reputation and bans the 100% automatically and I review the remainder visually and correct any mistakes. The reply is from a curl script to abuseipdb.com to report the IP. I also automatically ban certain countries (China, Vietnam, African nations) that would not be part of my CMS website. I allow latitude for some other listed countries and I give more latitude for north america IPs. I seems to work well.

The other thing I do is limit the max ban to 40-59 days and my sqlite db cleanses itself after 60 days. There was comment from author about keeping the ban limit less than the sqlite cleaning limit.

Here is the recent report to abuseipdb, that’s me as “anonymous” and it is clear the IP is a ban candidate.

Interesting. So this is NOT fail2ban seeing the login fail? If I understand correctly:

  • Your CMS (running in v’min context) sees a login fail
  • It queries abuseipdb
  • If abuseipdb says it’s a bad IP, you then pass that info to fail2ban for long term banning

Correct?

Correct. I have two sources of banned IPs. One is the CMS log for bad logins and the other is your standard set of jails that work across all the logs for the the virtualmin servers I have. sshd, wordpress-hard, spamassassin jails etc.

The CMS was getting hit from all over the world but there is typically no easy way to stop access. I wrote a log checking script that after a bunch of failed attempts it checks with abuseipdb. I have a jail for that and the php script calls fail2ban-client set abuseipdb badip xxx.xxx.xxx.xxx. The first one fail bans for 1 day and the second is 30 days and third is max bantime.

There was some learning to do because fail2ban works as root:root but my script is not run as root.

Regarding the CMS, I can say that once I processed a few thousand bad IPs the total number of intrusions basically dropped almost to zero. This is seen as the blue lines which are requests from CMS to abuseipdb starting in early March. As they hit 500 daily the ips were banned and the number of intrusions then dropped to a very small manageable number. It was a fun experiment. The green line shows that fail2ban is reporting about 500 daily bans it catches in the normal jails.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.