./www/anubis, Proof-of-work check to stop AI bots

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 1.24.0nb1, Package name: anubis-1.24.0nb1, Maintainer: bsiegert

Anubis weighs the soul of your connection using a sha256 proof-of-work
challenge in order to protect upstream resources from scraper bots.

Installing and using this will likely result in your website not being
indexed by some search engines. This is considered a feature of Anubis,
not a bug.

This is a bit of a nuclear response, but AI scraper bots scraping so
aggressively have forced my hand. I hate that I have to do this, but
this is what we get for the modern Internet because bots don't conform
to standards like robots.txt, even when they claim to.

In most cases, you should not need this and can probably get by using
Cloudflare to protect a given origin. However, for circumstances where
you can't or won't use Cloudflare, Anubis is there for you.


Master sites:

Filesize: 16144.47 KB

Version history: (Expand)


CVS history: (Expand)


   2026-01-15 20:55:00 by Benny Siegert | Files touched by this commit (202) | Package updated
Log message:
Revbump all Go packages after go125 update
   2026-01-10 18:38:34 by Benny Siegert | Files touched by this commit (3) | Package updated
Log message:
anubis: update to 1.24.0

Anubis is back and better than ever! Lots of minor fixes with some big
ones interspersed.

- Fix panic when validating challenges after privacy-mode browsers strip
  headers and the follow-up request matches an ALLOW threshold.
- Expose WEIGHT rule matches as Prometheus metrics.
- Allow more OCI registry clients based on feedback.
- Expose services directory in the embedded (data) filesystem.
- Add Ukrainian locale (#1044).
- Allow Renovate as an OCI registry client.
- Properly handle 4in6 addresses so that IP matching works with those
  addresses.
- Add support to simple Valkey/Redis cluster mode
- Open Graph passthrough now reuses the configured target Host/SNI/TLS
  settings, so metadata fetches succeed when the upstream certificate
  differs from the public domain. (1283)
- Stabilize the CVE-2025-24369 regression test by always submitting an
  invalid proof instead of relying on random POW failures.
- Refine the check that ensures the presence of the Accept header to
  avoid breaking docker clients.
- Removed rules intended to reward actual browsers due to abuse in the wild.
   2025-12-02 20:25:26 by Benny Siegert | Files touched by this commit (202) | Package updated
Log message:
Revbump all Go packages after go125 update
   2025-11-28 18:38:31 by Ryo ONODERA | Files touched by this commit (3)
Log message:
www/anubis: Update to 1.23.1

Changelog:
## v1.23.1: Lyse Hext - Echo 1

- Fix `SERVE_ROBOTS_TXT` setting after the double slash fix broke it.

### Potentially breaking changes

#### Remove default Tencent Cloud block rule

v1.23.0 added a default rule to block Tencent Cloud. After an email from their \ 
abuse team where they promised to take action to clean up their reputation, I \ 
have removed the default block rule. If this network causes you problems, please \ 
contact [abuse@tencent.com](mailto:abuse@tencent.com) and supply the following \ 
information:

- Time of abusive requests.
- IP address, User-Agent header, or other unique identifiers that can help the \ 
abuse team educate the customer about their misbehaving infrastructure.
- Does the abusive IP address request robots.txt? If not, be sure to include \ 
that information.
- A brief description of the impact to your system such as high system load, \ 
pages not rendering, or database system crashes. This helps the provider \ 
establish the fact that their customer is causing you measurable harm.
- Context as to what your service is, what it does, and why they should care.

Mention that you are using Anubis or BotStopper to protect your services. If \ 
they do not respond to you, please [contact me](https://xeiaso.net/contact) as \ 
soon as possible.

#### Docker / OCI registry clients

Anubis v1.23.0 accidentally blocked Docker / OCI registry clients. In order to \ 
explicitly allow them, add an import for `(data)/clients/docker-client.yaml`:

```yaml
bots:
  - import: (data)/meta/default-config.yaml
  - import: (data)/clients/docker-client.yaml
```

This is technically a regression as these clients used to work in Anubis \ 
v1.22.0, however it is allowable to make this opt-in as most websites do not \ 
expect to be serving Docker / OCI registry client traffic.

## v1.23.0: Lyse Hext

- Add default tencent cloud DENY rule.
- Added `(data)/meta/default-config.yaml` for importing the entire default \ 
configuration at once.
- Add `-custom-real-ip-header` flag to get the original request IP from a \ 
different header than `x-real-ip`.
- Add `contentLength` variable to bot expressions.
- Add `COOKIE_SAME_SITE_MODE` to force anubis cookies SameSite value, and \ 
downgrade automatically from `None` to `Lax` if cookie is insecure.
- Fix lock convoy problem in decaymap \ 
([#1103](https://github.com/TecharoHQ/anubis/issues/1103)).
- Fix lock convoy problem in bbolt by implementing the actor pattern \ 
([#1103](https://github.com/TecharoHQ/anubis/issues/1103)).
- Remove bbolt actorify implementation due to causing production issues.
- Document missing environment variables in installation guide: `SLOG_LEVEL`, \ 
`COOKIE_PREFIX`, `FORCED_LANGUAGE`, and `TARGET_DISABLE_KEEPALIVE` \ 
([#1086](https://github.com/TecharoHQ/anubis/pull/1086)).
- Add validation warning when persistent storage is used without setting signing \ 
keys.
- Fixed `robots2policy` to properly group consecutive user agents into `any:` \ 
instead of only processing the last one \ 
([#925](https://github.com/TecharoHQ/anubis/pull/925)).
- Make the `fast` algorithm prefer purejs when running in an insecure context.
- Add the [`s3api` storage backend](./admin/policies.mdx#s3api) to allow Anubis \ 
to use S3 API compatible object storage as its storage backend.
- Fix a "stutter" in the cookie name prefix so the auth cookie is \ 
named `techaro.lol-anubis-auth` instead of `techaro.lol-anubis-auth-auth`.
- Make `cmd/containerbuild` support commas for separating elements of the \ 
`--docker-tags` argument as well as newlines.
- Add the `DIFFICULTY_IN_JWT` option, which allows one to add the `difficulty` \ 
field in the JWT claims which indicates the difficulty of the token \ 
([#1063](https://github.com/TecharoHQ/anubis/pull/1063)).
- Ported the client-side JS to TypeScript to avoid egregious errors in the future.
- Fixes concurrency problems with very old browsers \ 
([#1082](https://github.com/TecharoHQ/anubis/issues/1082)).
- Randomly use the Refresh header instead of the meta refresh tag in the \ 
metarefresh challenge.
- Update OpenRC service to truncate the runtime directory before starting Anubis.
- Make the git client profile more strictly match how the git client behaves.
- Make the default configuration reward users using normal browsers.
- Allow multiple consecutive slashes in a row in application paths \ 
([#754](https://github.com/TecharoHQ/anubis/issues/754)).
- Add option to set `targetSNI` to special keyword 'auto' to indicate that it \ 
should be automatically set to the request Host name \ 
([424](https://github.com/TecharoHQ/anubis/issues/424)).
- The Preact challenge has been removed from the default configuration. It will \ 
be deprecated in the future.
- An open redirect when in subrequest mode has been fixed.

### Potentially breaking changes

#### Multiple checks at once has and-like semantics instead of or-like semantics

Anubis lets you stack multiple checks at once with blocks like this:

```yaml
name: allow-prometheus
action: ALLOW
user_agent_regex: ^prometheus-probe$
remote_addresses:
  - 192.168.2.0/24
```

Previously, this only returned ALLOW if _any one_ of the conditions matched. \ 
This behaviour has changed to only return ALLOW if _all_ of the conditions \ 
match. I expect this to have some issues with user configs, however this fix is \ 
grave enough that it's worth the risk of breaking configs. If this bites you, \ 
please let me know so we can make an escape hatch.

### Better error messages

In order to make it easier for legitimate clients to debug issues with their \ 
browser configuration and Anubis, Anubis will emit internal error detail in base \ 
64 so that administrators can chase down issues. Future versions of this may \ 
also include a variant that encrypts the error detail messages.

### Bug Fixes

Sometimes the enhanced temporal assurance in \ 
[#1038](https://github.com/TecharoHQ/anubis/pull/1038) and \ 
[#1068](https://github.com/TecharoHQ/anubis/pull/1068) could backfire because \ 
Chromium and its ilk randomize the amount of time they wait in order to avoid a \ 
timing side channel attack. This has been fixed by both increasing the amount of \ 
time a client has to wait for the metarefresh and preact challenges as well as \ 
making the server side logic more permissive.
   2025-10-16 20:00:06 by Benny Siegert | Files touched by this commit (200) | Package updated
Log message:
Revbump all Go packages after go125 update
   2025-10-08 08:54:42 by Benny Siegert | Files touched by this commit (200) | Package updated
Log message:
Revbump all Go packages after go125 update
   2025-09-11 23:56:24 by Benny Siegert | Files touched by this commit (4) | Package updated
Log message:
anubis: update to 1.22.0

In this release, we finally fix the odd number of CPU cores bug, pave the way
for lighter weight challenges, make Anubis more adaptable, and more.

Big ticket items

Proof of React challenge

A new "proof of React" has been added. It runs a simple app in React \ 
that has
several chained hooks. It is much more lightweight than the proof of work
check.

Smaller features

- The segments function was added for splitting a path into its
  slash-separated segments.
- Added possibility to disable HTTP keep-alive to support backends not
  properly handling it.
- When issuing a challenge, Anubis stores information about that challenge
  into the store. That stored information is later used to validate challenge
  responses. This works around nondeterminism in bot rules.
- One of the biggest sources of lag in Firefox has been eliminated: the use of
  WebCrypto. Now whenever Anubis detects the client is using Firefox (or Pale
  Moon), it will swap over to a pure-JS implementation of SHA-256 for speed.
- Proof of work solving has had a complete overhaul and rethink based on
  feedback from browser engine developers, frontend experts, and overall
  performance profiling.
- Optimize the performance of the pure-JS Anubis solver.
- Web Workers are stored as dedicated JavaScript files in
  static/js/workers/*.mjs.
- Pave the way for non-SHA256 solver methods and eventually one that uses
  WebAssembly (or WebAssembly code compiled to JS for those that disable
  WebAssembly).
- Legacy JavaScript code has been eliminated.
- When parsing Open Graph tags, add any URLs found in the responses to a
  temporary "allow cache" so that social preview images work.
- The hard dependency on WebCrypto has been removed, allowing a proof of work
  challenge to work over plain (unencrypted) HTTP.
- The Anubis version number is put in the footer of every page.
- Add a default block rule for Huawei Cloud.
- Add a default block rule for Alibaba Cloud.
- Added support to use Traefik forwardAuth middleware.
- Add X-Request-URI support so that Subrequest Authentication has path
  support.

Fixes

Odd numbers of CPU cores are properly supported

Some phones have an odd number of CPU cores. This caused interesting issues.
This was fixed by using Math.trunc to convert the number of CPU cores back
into an integer.

Smaller fixes

- A standard library HTTP server log message about HTTP pipelining not working
  has been filtered out of Anubis' logs. There is no action that can be taken
  about it.
- Added a missing link to the Caddy installation environment in the
  installation documentation.
- Downstream consumers can change the default log/slog#Logger instance that
  Anubis uses by setting opts.Logger to your slog instance of choice (#864).
- The Thoth client is now public in the repo instead of being an internal
  package.
- Custom-AsyncHttpClient's default User-Agent has an increased weight by
  default.
- Add option for replacing the default explanation text with a custom one.
- The contact email in the LibreJS header has been changed.
- Firefox for Android support has been fixed by embedding the challenge ID
  into the pass-challenge route. This also fixes some inconsistent issues with
  other mobile browsers.
- The default favicon pattern in data/common/keep-internet-working.yaml has
  been updated to permit requests for png/gif/jpg/svg files as well as ico.
- The --cookie-prefix flag has been fixed so that it is fully respected.
- The default patterns in data/common/keep-internet-working.yaml have been
  updated to appropriately escape the '.' character in the regular expression
  patterns.
- Add optional restrictions for JWT based on the value of a header
- The word "hack" has been removed from the translation strings for \ 
Anubis due
  to incidents involving people misunderstanding that word and sending
  particularly horrible things to the project lead over email.
- Bump AI-robots.txt to version 1.39
- Inject adversarial input to break AI coding assistants.
- Add better logging when using Subrequest Authentication.

Security-relevant changes

Add a server-side check for the meta-refresh challenge that makes sure clients
have waited for at least 95% of the time that they should.

Fix potential double-spend for challenges

Anubis operates by issuing a challenge and having the client present a
solution for that challenge. Challenges are identified by a unique UUID, which
is stored in the database.

The problem is that a challenge could potentially be used twice by a dedicated
attacker making a targeted attack against Anubis. Challenge records did not
have a "spent" or "used" field. In total, a dedicated \ 
attacker could solve a
challenge once and reuse that solution across multiple sessions in order to
mint additional tokens.

This was fixed by adding a "spent" field to challenges in the data \ 
store. When
a challenge is solved, that "spent" field gets set to true. If a future
attempt to solve this challenge is observed, it gets rejected.

With the advent of store based challenge issuance, this means that these
challenge IDs are only good for 30 minutes. Websites using the most recent
version of Anubis have limited exposure to this problem.

Websites using older versions of Anubis have a much more increased exposure to
this problem and are encouraged to keep this software updated as often and as
frequently as possible.

Breaking changes

- The "slow" frontend solver has been removed in order to reduce \ 
maintenance
  burden. Any existing uses of it will still work, but issue a warning upon
  startup asking administrators to upgrade to the "fast" frontend solver.
- The legacy JSON based policy file example has been removed and all
  documentation for how to write a policy file in JSON has been deleted. JSON
  based policy files will still work, but YAML is the superior option for
  Anubis configuration.
   2025-09-06 15:17:15 by Benny Siegert | Files touched by this commit (195) | Package updated
Log message:
Revbump all Go packages after go125 security update