How the FSF sysadmins block botnets with reaction

I have written about the FSF facing DDoS attacks several times, including on doing our part to clean up the internet and on Uptime Kuma, as well as "Defending Savannah from DDoS attacks". But I realized recently that I left out an important tool that we use extensively. Today I will talk about reaction. With the rise of aggressive scrapers building datasets to train LLMs, we needed to invest a lot of resources into thwarting the attackers. During this journey, we had several growing pains.

We noticed patterns in the scrapers that were abnormal, which gave us material for writing regular expressions. Searching for the regular expression then gave us a large lists of IP addresses. Looking up the origin of those IP addresses revealed that some of the crawlers were using botnets of residential IP addresses to scrape faster and avoid detection.

We looked for what kinds of botnets might be generating the kind of traffic that we were seeing, and one that we suspected was called the "Vo1d" botnet, comprised of smart TVs running some sort of compromised app. Recently, more information has come out about the Vo1d/Popa botnet. Security researchers at Qurium, a digital forensics investigation organization, intentionally ran a node and published the data. We got confirmation that at least some of the botnet traffic hitting GNU Savannah was originating through the Vo1d/Popa botnet.

We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules. fail2ban is a program that can read the stream of log files and perform actions when certain conditions are met. The actions are typically banning an address with a firewall like UFW, which blocks or allows connections to and from a server. We learned about ipset and configured fail2ban to add IP addresses that it found to IP sets. Using ipset, we kept building larger IP sets and did not find instability with as large as five million rules.

We used fail2ban with ipset for a while but ran into limits of its architecture with Python and SQLite. We rolled our own solutions as a quick patch several times with BASH, awk, Perl, etc. to find the address matching the pattern and add the rule to ipset without an additional intermediary database. The custom scripts worked for a moment, but it became difficult to manage when we had to run several little scripts as more patterns appeared. Not all of the patterns should be banned on first appearance either so we needed a solution like fail2ban. We looked for fail2ban alternatives written recently. We found several, but many of them were written as one-off showcases rather than well-maintained solutions that would last. We eventually found a promising project on Framasoft's forge Framagit called reaction written by ppom.

It took us a bit of time to understand how to configure reaction. Unlike fail2ban, reaction does not come with a working configuration and you must build your own configuration with what is relevant to your needs from example documentation. There was not an example for using ipset so we had to build one. Overall, this approach is very good because we now have a configuration that has everything we need and very little that we do not need.

After we ran into scaling issues with our initial implementation, we developed a much faster implementation where the reaction shutdown process would export the IP sets to disk and the reaction startup process would restore the IP sets. This allowed us to have nearly instantaneous restarts of the service to apply new rules. We published both of our configurations upstream to reaction's wiki so that everyone can benefit from it. reaction's getting started documentation now leads to the method that we proposed.

This work is not possible without your support. Because of free software supporters just like you, we can continue to find new patterns, block crawlers and mitigate DDoS attacks, and share our improvements with everyone. We know not everyone is in a position to give, but if you can, support our efforts by joining the FSF as an associate member. An associate membership is a great show of support we can rely on. The FSF does not compromise when it comes to defending your freedom. Without the continued commitment of people like you, the progress we have made is in danger, and software freedom could be reduced to a mere wish instead of today's reality. Please help us continue to protect software freedom and increase its global support. Every membership this summer will help us towards our goal of 175 new members. Associate members will also be able to enjoy all the member benefits which include merchandise discounts, a 16GB bootable membership card, and use of our associate member videoconferencing server.

Many sysadmins know about fail2ban, but not enough people know about reaction. I am very grateful to ppom for the help they have provided and for the tremendous project they have released to the world with reaction. We have implemented other defenses as well, but reaction is doing the majority of the automated work keeping our sites online.