Skip to content

[Bots] Web Bot Auth docs#23099

Merged
Oxyjun merged 14 commits intoproductionfrom
jun/bots/wba
Jun 26, 2025
Merged

[Bots] Web Bot Auth docs#23099
Oxyjun merged 14 commits intoproductionfrom
jun/bots/wba

Conversation

@Oxyjun
Copy link
Contributor

@Oxyjun Oxyjun commented Jun 18, 2025

Summary

We're now introducing "Web Bot Auth" (WBA), which is a more secure authentication method for verifying bots. Previously, verifying bots was only possible through two flavours of IP validation, public IP list, and reverse DNS.

Web Bot Auth may become the new IETF standard in the near future, and paves the way for better bot detection across the Internet.

This PR restructures and adds information for WBA. Specifically:

  1. Creates a new chapter Verified bots requirements, which explains what's involved for a bot to be verified.
  2. Cuts down the existing Verified bots policy chapter to only talk about the policy
  3. Creates a new chapter Verification methods, which :
    • Guides users through verifying their bot via WBA
    • Guides users through verifying their bot via IP validation (text taken from existing Policy chapter)
    • Talks about generic user agents (text taken from existing Policy chapter)
    • Points users to other useful resources under "Additional resources" header.

Things that we need to improve:

  1. Link to blog
  2. Possible restructure such that WBA is its own page (and how this affects rest of IA)
  3. Consider absorbing Categories chapter into Verification methods (or somewhere else) Decided against after discussion.
  4. Consider defining the three main required headers for WBA into its own subheadings (such that they can be hyperlinked). This relates to point (2).
  5. Work out where the FAQs should go. Possibly a new Reference > FAQs > "WBA FAQs"?

Screenshots (optional)

Documentation checklist

  • The documentation style guide has been adhered to.
  • If a larger change - such as adding a new page- an issue has been opened in relation to any incorrect or out of date information that this PR fixes.
  • Files which have changed name or location have been allocated redirects.

@Oxyjun Oxyjun requested review from a team and patriciasantaana as code owners June 18, 2025 16:39
@github-actions github-actions bot added the product:bots Related to Bots product label Jun 18, 2025
@hyperlint-ai-deprecated
Copy link
Contributor

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.


PR Change Summary

Introduced Web Bot Auth (WBA) documentation, enhancing bot verification methods.

  • Created a new chapter on Verified Bots Requirements detailing verification criteria.
  • Restructured the Verified Bots Policy chapter to focus solely on policy aspects.
  • Added a new chapter on Verification Methods, outlining WBA and IP validation processes.

Modified Files

  • src/content/docs/bots/concepts/bot/verified-bots/categories.mdx
  • src/content/docs/bots/concepts/bot/verified-bots/policy.mdx

Added Files

  • src/content/docs/bots/concepts/bot/verified-bots/requirements.mdx
  • src/content/docs/bots/concepts/bot/verified-bots/verification.mdx

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 18, 2025

This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:

Pattern Owners
/public/__redirects @GregBrimble, @KianNH, @pedrosousa, @WalshyDev, @cloudflare/pcx-technical-writing
/src/content/docs/bots/ @patriciasantaana, @cloudflare/pcx-technical-writing
* @cloudflare/pcx-technical-writing

@github-actions
Copy link
Contributor

github-actions bot commented Jun 18, 2025

Oxyjun and others added 2 commits June 19, 2025 09:15
Co-authored-by: Patricia Santa Ana <103445940+patriciasantaana@users.noreply.github.com>
@github-actions
Copy link
Contributor

github-actions bot commented Jun 19, 2025

This PR changes current filenames or deletes current files. Make sure you have redirects set up to cover the following paths:

  • /bots/troubleshooting/frequently-asked-questions/

@github-actions github-actions bot added size/xl product:1.1.1.1 Related to 1.1.1.1 product product:agents Build and deploy AI-powered Agents on Cloudflare that can act autonomously. product:ai-gateway AI Gateway: https://developers.cloudflare.com/ai-gateway/ product:analytics Related to Analytics product product:api-shield product:AutoRAG product:billing product:cloudflare-one product:cloudflare-for-platforms product:d1 D1: https://developers.cloudflare.com/d1/ product:data-localization product:durable-objects Durable Objects: https://developers.cloudflare.com/workers/learning/using-durable-objects/ product:fundamentals product:learning-paths product:log-explorer product:logs Related to Logs product:magic-transit and removed size/m labels Jun 23, 2025
Copy link
Contributor

@AkshatM AkshatM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One or two final comments, but marking as appriove to unblock you

@JohnDaWalka
Copy link

JohnDaWalka commented Jun 23, 2025 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

product:bots Related to Bots product size/m

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants