Skip to content

Feat/llm txt improvements#3479

Merged
aditya520 merged 7 commits intomainfrom
feat/llm-txt-improvements
Feb 17, 2026
Merged

Feat/llm txt improvements#3479
aditya520 merged 7 commits intomainfrom
feat/llm-txt-improvements

Conversation

@aditya520
Copy link
Member

@aditya520 aditya520 commented Feb 16, 2026

Summary

Restructures the LLM-consumable documentation system from a monolithic llms full.txt approach to a Tiered + Protocol Hybrid architecture. AI agents can now progressively load context -- starting with a lightweight routing index (~500 tokens), drilling into curated product guides (~2k tokens each), and optionally fetching individual .mdx pages for full detail. This replaces the old architecture where a single monolithic file dumped all documentation at once, wasting token budget on irrelevant content. A new machine-readable manifest, dynamic sitemap, token counting toolchain, and hardened GitHub Action CI round out the system.

Rationale

How has this been tested?

  • Current tests cover my changes
  • Added new tests
  • Manually tested the code

@vercel
Copy link

vercel bot commented Feb 16, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
api-reference Ready Ready Preview, Comment Feb 17, 2026 4:42pm
component-library Ready Ready Preview, Comment Feb 17, 2026 4:42pm
developer-hub Ready Ready Preview, Comment Feb 17, 2026 4:42pm
entropy-explorer Ready Ready Preview, Comment Feb 17, 2026 4:42pm
insights Ready Ready Preview, Comment Feb 17, 2026 4:42pm
proposals Ready Ready Preview, Comment Feb 17, 2026 4:42pm
staking Ready Ready Preview, Comment Feb 17, 2026 4:42pm

Request Review

aditya520 and others added 3 commits February 16, 2026 11:38
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These are local-only planning documents, now covered by .gitignore.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Handle fumadocs index slug stripping in mdx route, remove dead
getLLMTextByPaths, fix SKILL.md Content-Type to text/markdown,
and standardize CONTENT constant naming.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Pin all GitHub Action third-party refs to full commit SHAs
- Move ${{ }} expressions to env: vars to prevent shell injection
- Create shared LLM_FILES config (single source of truth for manifest,
  sitemap, and token counter)
- Re-run token counter after Action content updates
- Return 410 Gone from deprecated /llms-full.txt endpoint
- Add sitemap reference to robots.txt
- Standardize all routes to module-level const CONTENT
- Fix SKILL.md sed pattern in Action after CONTENT rename

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-script

- Fix Solana/Sui progressive disclosure URLs (add pull-integration/ segment)
- Remove per-request Generated on timestamp from all route handlers
  (contradicts revalidate=false, latent CDN caching bug)
- Move sitemap lastModified to module-level BUILD_DATE constant
- Move remaining ${{ }} expressions in github-script to env/process.env

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aditya520
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f10cacae55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


- name: Read static headers from affected product files
- name: Read content from affected product files
if: steps.changed-files.outputs.significant == 'true' && steps.changed-files.outputs.affected_products != ''

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Run SKILL review even when no product route is mapped

This gate makes the entire update pipeline skip whenever affected_products is empty, even if include_skill was set to true. A PR that only changes key docs outside the hardcoded product mapping (for example content/docs/express-relay/index.mdx or content/docs/metrics/index.mdx) will be marked significant but never evaluate/update SKILL.md, which defeats the new include_skill logic and leaves the playbook stale for those docs changes.

Useful? React with 👍 / 👎.

if [ "$INCLUDE_SKILL" = "true" ]; then
SKILL_FILE="$ROUTE_DIR/SKILL.md/route.ts"
if [ -f "$SKILL_FILE" ]; then
sed -n '/const CONTENT = `/,/^`;/{/const CONTENT = `/d;/^`;/d;p}' "$SKILL_FILE" > "/tmp/header-SKILL.md.txt"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve SKILL frontmatter when extracting CONTENT

This sed extraction deletes the entire const CONTENT = ... line, but SKILL.md starts its markdown payload on that same line (---). As a result, the opening frontmatter delimiter is dropped before sending content to the model, and any rewritten SKILL.md can lose valid YAML frontmatter structure after update.

Useful? React with 👍 / 👎.

Fix two bugs in update-llms-txt workflow:

1. sed extraction was deleting the entire `const CONTENT` marker line,
   silently dropping the first line of content (frontmatter `---` in
   SKILL.md, `# Heading` in product files). Now strips only the prefix.

2. Pipeline steps were gated on `affected_products != ''`, skipping
   SKILL.md review when doc changes don't map to a product route.
   Now also runs when `include_skill == 'true'`.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aditya520 aditya520 merged commit d18b599 into main Feb 17, 2026
10 checks passed
@aditya520 aditya520 deleted the feat/llm-txt-improvements branch February 17, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants