Faced recently storage shortage on a server again, so made my homework, its a Debian GNU/Linux 7.11 (wheezy)
An sqlite3 dump into a gzipped textfile shows an amazing size difference, that means without a reqular maintenance of the fail2ban database, the system may go low on free diskspace.
# du -hs fail2ban.sqlite3
2.5G fail2ban.sqlite3
# sqlite3 ./fail2ban.sqlite3 .dump | gzip -c - > fail2ban.sqlite3.sql.gz
# du -hs fail2ban.sqlite3.sql.gz
188M fail2ban.sqlite3.sql.gz
# sqlite3 ./fail2ban.sqlite3 .dump | wc -l
4389727
# zcat fail2ban.sqlite3.sql.gz | wc -l
4389727
It is known, that before v.0.11, fail2ban had a bug, so purge was never invoked.
On this system there is only fail2ban v0.9.7, so i looked around for a solution.
Known problems, that accompanies a self made vacuum solutions are widely known:
- changes, where a lot of rows are involved, are creating a journal approx. same size than the db itself
- without the purge there will be lot of rows to delete starting 2016.Jul.17 02:23:27, if i want to keep at most 1 week
My estimations of time required for the whole operation caused me to consider on-the-fly cleanup, and copy the db and do the cleanup, with the risk, that by the time i finsh there will be too much differences to merge. I decided to stop fail2ban, and take the risk of being without it for the time of the cleanup.
Now move on with the numbers.
- The grand total of the dump: 4389727 rows
- Counting a sum of the items grouped by jails that are more than 1 week old: 4388242 rows
- So there were 4388242 rows to delete from fail2ban.sqlite3
# sqlite3 ./fail2ban.sqlite3 .dump | wc -l
4389727
# sqlite3 ./fail2ban.sqlite3 "select jail, count(*) as bancount from bans WHERE timeofban < (CAST(strftime('%s') as INT) - 604800) group by jail;"
recidive|31826
sshd|4247455
wprecidive|108961
# echo $(( 31826 + 4247455 + 108961))
4388242
Lets start the delete
# sqlite3 ./fail2ban.sqlite3 "delete from bans WHERE timeofban < (CAST(strftime('%s') as INT) - 604800);"
After the delete process (which took long hours, and the journal grew to the size of the db) the size of the fail2ban.sqlite3 has not changed. It was time to do the cleanup, so i ran VACUUM on the db Luckily, the process did not last long, and as expected, the journal grew to almost the size of the db, but both times the journal vanished after finishing the operation.
# sqlite3 ./fail2ban.sqlite3 "VACUUM;"
The grand total of the resulting db's dump: 1485 rows Size of the final database less than 1 MB!!!
# sqlite3 ./fail2ban.sqlite3 .dump | wc -l
1485
# du -hs fail2ban.sqlite3
904K fail2ban.sqlite3
After replacing the fail2ban.sqlite3 and starting fail2ban-server i saw there were no issues with the result of my work.
The next step will be to create a shell script, that does a daily delete and vacuum, which i expect to finish quickly so it can be done on-the-fly.
But that is the future.