feat: Support sitemap.xml by cristiklein · Pull Request #2071 · lycheeverse/lychee

cristiklein · 2026-03-04T14:43:37Z

Fixes #2062

This PR adds support for extracting links from <loc> tags from sitemap.xml. The implementation is kept minimal, relying on a regex, similar to how links are extracted from CSS.

In future, the XML extractor could be extended with support for links in SVGs and perhaps a proper XML parser. However, those are out-of-scope for this PR.

mre

You're on a roll! Great work! 🚀

I've added a few comments. 😊

thomas-zahner · 2026-03-11T14:24:04Z

@cristiklein Regarding the clippy warnings/errors, by rebasing or merging master you should be able to resolve the issues. (edit: I did a rebase)

thomas-zahner

Thank you for the PR!

Tested with https://validator.w3.org/feed/check.cgi Got "[Valid Atom 1.0] This is a valid Atom 1.0 feed."

cristiklein · 2026-03-11T20:10:45Z

@mre @thomas-zahner Thanks for the feedback on this PR. I believe I addressed all your comments. Otherwise, please let me know. 😄

thomas-zahner · 2026-03-13T10:57:00Z

@cristiklein Thank you for this cool addition to lychee! 🚀

cristiklein · 2026-03-13T15:07:38Z

@cristiklein Thank you for this cool addition to lychee! 🚀

And thank you for the kind words and for shepherding this PR. ❤️

The first Docs workflow run on main after #173 (commit f3f3e5e) failed in deploy → "Check deployed docs links" with "No files found for this input source". Root cause: the previous step ran lychee --dump on the deployed sitemap URL, but lychee 0.23.0 (the lycheeverse/lychee-action v2 default) only extracts <a href> from HTML and matching patterns from markdown — it does not parse <loc> entries from XML sitemaps. The dump produced an empty list and the follow-up --files-from step had nothing to read. Upstream already fixed this. lycheeverse/lychee#2071 (merged 2026-03-13, tagged in v0.24.0 on 2026-04-24) adds <loc> extraction from sitemap.xml, closing lycheeverse/lychee#2062 and #1819. Verified locally on 0.24.0: $ lychee --version lychee 0.24.0 $ lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 Pin LYCHEE_VERSION at the workflow env level and reference it from every lychee-action call so future bumps are one-line. v0.24.0's breaking changes are in lychee-lib (the Rust API consumers); the CLI surface we use is unchanged. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com>

Replace the previous v0.24.0 bump with the only combination that actually works against the current lychee release pipeline: - lycheeverse/lychee-action SHA 8646ba3 (tagged v2.8.0) → faea714 (post-v2.8.0 master). Adds subfolder-aware install needed for any lychee 0.24.x tarball. - LYCHEE_VERSION 'v0.24.0' → 'v0.24.1'. Why both moves: * lychee 0.24.0 added <loc> extraction from XML sitemaps (lycheeverse/lychee#2071), which is what the deploy and check-deployed jobs need to feed --files-from. lychee 0.23.0 dumps zero links from a sitemap, which is what produced the "No files found for this input source" failure on f3f3e5e. * lychee 0.24.0's release tarball was repackaged with a top-level subfolder AND the asset filename was renamed to lychee-lychee-v0.24.0-{arch}-... — both incompatible with lychee-action v2.8.0's hardcoded download URL and flat-extract logic. * lychee 0.24.1 (released the same day) reverted to the original asset filename but kept the subfolder layout AND kept the sitemap fix. * lychee-action faea714 (unreleased; current HEAD of master) bumps the default to 0.24.1 and adds subfolder-aware install. Pinning the SHA is the same security model we already use for v2.8.0. The combination 8646ba3 + 'latest' or 8646ba3 + 'v0.24.x' both fail. The combination faea714 + 'v0.24.1' works. Verified locally: $ lychee-v0.24.1/lychee --version lychee 0.24.1 $ lychee-v0.24.1/lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com>

) * ci(docs): bump lychee to v0.24.0 to fix sitemap URL extraction The first Docs workflow run on main after #173 (commit f3f3e5e) failed in deploy → "Check deployed docs links" with "No files found for this input source". Root cause: the previous step ran lychee --dump on the deployed sitemap URL, but lychee 0.23.0 (the lycheeverse/lychee-action v2 default) only extracts <a href> from HTML and matching patterns from markdown — it does not parse <loc> entries from XML sitemaps. The dump produced an empty list and the follow-up --files-from step had nothing to read. Upstream already fixed this. lycheeverse/lychee#2071 (merged 2026-03-13, tagged in v0.24.0 on 2026-04-24) adds <loc> extraction from sitemap.xml, closing lycheeverse/lychee#2062 and #1819. Verified locally on 0.24.0: $ lychee --version lychee 0.24.0 $ lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 Pin LYCHEE_VERSION at the workflow env level and reference it from every lychee-action call so future bumps are one-line. v0.24.0's breaking changes are in lychee-lib (the Rust API consumers); the CLI surface we use is unchanged. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com> * ci(docs): bump lychee-action and lychee for sitemap URL extraction Replace the previous v0.24.0 bump with the only combination that actually works against the current lychee release pipeline: - lycheeverse/lychee-action SHA 8646ba3 (tagged v2.8.0) → faea714 (post-v2.8.0 master). Adds subfolder-aware install needed for any lychee 0.24.x tarball. - LYCHEE_VERSION 'v0.24.0' → 'v0.24.1'. Why both moves: * lychee 0.24.0 added <loc> extraction from XML sitemaps (lycheeverse/lychee#2071), which is what the deploy and check-deployed jobs need to feed --files-from. lychee 0.23.0 dumps zero links from a sitemap, which is what produced the "No files found for this input source" failure on f3f3e5e. * lychee 0.24.0's release tarball was repackaged with a top-level subfolder AND the asset filename was renamed to lychee-lychee-v0.24.0-{arch}-... — both incompatible with lychee-action v2.8.0's hardcoded download URL and flat-extract logic. * lychee 0.24.1 (released the same day) reverted to the original asset filename but kept the subfolder layout AND kept the sitemap fix. * lychee-action faea714 (unreleased; current HEAD of master) bumps the default to 0.24.1 and adds subfolder-aware install. Pinning the SHA is the same security model we already use for v2.8.0. The combination 8646ba3 + 'latest' or 8646ba3 + 'v0.24.x' both fail. The combination faea714 + 'v0.24.1' works. Verified locally: $ lychee-v0.24.1/lychee --version lychee 0.24.1 $ lychee-v0.24.1/lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com> * ci(docs): add TODO(lychee-action-sha-pin) marker Companion to #179, which establishes the convention. Mark the spot where the SHA pin needs to be reverted once lycheeverse/lychee-action cuts a tagged release at or after faea714, with a back-link to the tracked entry in TODO.md so a single grep finds both ends. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com> --------- Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com> Co-authored-by: Claude <noreply@anthropic.com>

) * ci(docs): bump lychee to v0.24.0 to fix sitemap URL extraction The first Docs workflow run on main after #173 (commit f3f3e5e) failed in deploy → "Check deployed docs links" with "No files found for this input source". Root cause: the previous step ran lychee --dump on the deployed sitemap URL, but lychee 0.23.0 (the lycheeverse/lychee-action v2 default) only extracts <a href> from HTML and matching patterns from markdown — it does not parse <loc> entries from XML sitemaps. The dump produced an empty list and the follow-up --files-from step had nothing to read. Upstream already fixed this. lycheeverse/lychee#2071 (merged 2026-03-13, tagged in v0.24.0 on 2026-04-24) adds <loc> extraction from sitemap.xml, closing lycheeverse/lychee#2062 and #1819. Verified locally on 0.24.0: $ lychee --version lychee 0.24.0 $ lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 Pin LYCHEE_VERSION at the workflow env level and reference it from every lychee-action call so future bumps are one-line. v0.24.0's breaking changes are in lychee-lib (the Rust API consumers); the CLI surface we use is unchanged. Co-authored-by: Claude <noreply@anthropic.com> * ci(docs): bump lychee-action and lychee for sitemap URL extraction Replace the previous v0.24.0 bump with the only combination that actually works against the current lychee release pipeline: - lycheeverse/lychee-action SHA 8646ba3 (tagged v2.8.0) → faea714 (post-v2.8.0 master). Adds subfolder-aware install needed for any lychee 0.24.x tarball. - LYCHEE_VERSION 'v0.24.0' → 'v0.24.1'. Why both moves: * lychee 0.24.0 added <loc> extraction from XML sitemaps (lycheeverse/lychee#2071), which is what the deploy and check-deployed jobs need to feed --files-from. lychee 0.23.0 dumps zero links from a sitemap, which is what produced the "No files found for this input source" failure on f3f3e5e. * lychee 0.24.0's release tarball was repackaged with a top-level subfolder AND the asset filename was renamed to lychee-lychee-v0.24.0-{arch}-... — both incompatible with lychee-action v2.8.0's hardcoded download URL and flat-extract logic. * lychee 0.24.1 (released the same day) reverted to the original asset filename but kept the subfolder layout AND kept the sitemap fix. * lychee-action faea714 (unreleased; current HEAD of master) bumps the default to 0.24.1 and adds subfolder-aware install. Pinning the SHA is the same security model we already use for v2.8.0. The combination 8646ba3 + 'latest' or 8646ba3 + 'v0.24.x' both fail. The combination faea714 + 'v0.24.1' works. Verified locally: $ lychee-v0.24.1/lychee --version lychee 0.24.1 $ lychee-v0.24.1/lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 Co-authored-by: Claude <noreply@anthropic.com> * ci(docs): add TODO(lychee-action-sha-pin) marker Companion to #179, which establishes the convention. Mark the spot where the SHA pin needs to be reverted once lycheeverse/lychee-action cuts a tagged release at or after faea714, with a back-link to the tracked entry in TODO.md so a single grep finds both ends. Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com> Co-authored-by: Codex <codex@openai.com>

) * ci(docs): bump lychee to v0.24.0 to fix sitemap URL extraction The first Docs workflow run on main after #173 (commit f3f3e5e) failed in deploy → "Check deployed docs links" with "No files found for this input source". Root cause: the previous step ran lychee --dump on the deployed sitemap URL, but lychee 0.23.0 (the lycheeverse/lychee-action v2 default) only extracts <a href> from HTML and matching patterns from markdown — it does not parse <loc> entries from XML sitemaps. The dump produced an empty list and the follow-up --files-from step had nothing to read. Upstream already fixed this. lycheeverse/lychee#2071 (merged 2026-03-13, tagged in v0.24.0 on 2026-04-24) adds <loc> extraction from sitemap.xml, closing lycheeverse/lychee#2062 and #1819. Verified locally on 0.24.0: $ lychee --version lychee 0.24.0 $ lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 Pin LYCHEE_VERSION at the workflow env level and reference it from every lychee-action call so future bumps are one-line. v0.24.0's breaking changes are in lychee-lib (the Rust API consumers); the CLI surface we use is unchanged. * ci(docs): bump lychee-action and lychee for sitemap URL extraction Replace the previous v0.24.0 bump with the only combination that actually works against the current lychee release pipeline: - lycheeverse/lychee-action SHA 8646ba3 (tagged v2.8.0) → faea714 (post-v2.8.0 master). Adds subfolder-aware install needed for any lychee 0.24.x tarball. - LYCHEE_VERSION 'v0.24.0' → 'v0.24.1'. Why both moves: * lychee 0.24.0 added <loc> extraction from XML sitemaps (lycheeverse/lychee#2071), which is what the deploy and check-deployed jobs need to feed --files-from. lychee 0.23.0 dumps zero links from a sitemap, which is what produced the "No files found for this input source" failure on f3f3e5e. * lychee 0.24.0's release tarball was repackaged with a top-level subfolder AND the asset filename was renamed to lychee-lychee-v0.24.0-{arch}-... — both incompatible with lychee-action v2.8.0's hardcoded download URL and flat-extract logic. * lychee 0.24.1 (released the same day) reverted to the original asset filename but kept the subfolder layout AND kept the sitemap fix. * lychee-action faea714 (unreleased; current HEAD of master) bumps the default to 0.24.1 and adds subfolder-aware install. Pinning the SHA is the same security model we already use for v2.8.0. The combination 8646ba3 + 'latest' or 8646ba3 + 'v0.24.x' both fail. The combination faea714 + 'v0.24.1' works. Verified locally: $ lychee-v0.24.1/lychee --version lychee 0.24.1 $ lychee-v0.24.1/lychee --dump https://jackin.tailrocks.com/sitemap-0.xml | wc -l 45 * ci(docs): add TODO(lychee-action-sha-pin) marker Companion to #179, which establishes the convention. Mark the spot where the SHA pin needs to be reverted once lycheeverse/lychee-action cuts a tagged release at or after faea714, with a back-link to the tracked entry in TODO.md so a single grep finds both ends. --------- Signed-off-by: Alexey Zhokhov <alexey@zhokhov.com> Co-authored-by: Claude <noreply@anthropic.com>

mre reviewed Mar 4, 2026

View reviewed changes

Comment thread lychee-lib/src/extract/xml.rs

Comment thread lychee-lib/src/types/file.rs

Comment thread lychee-lib/src/types/file.rs

mre reviewed Mar 5, 2026

View reviewed changes

Comment thread lychee-lib/src/extract/xml.rs

cristiklein added 9 commits March 11, 2026 15:37

Add test for sitemap.xml

57cab8c

Wire XML extractor

f717469

Add test for extract_xml

909c8f9

Implement XML (sitemap) extractor

8fa7739

Final polishing

c8347a1

Make it clear that only sitemap XMLs are supported

7cd3eb3

Make linter happy

61c55c5

Add support for RSS and Atom feeds

1fe28e8

Fix test

a4ec801

thomas-zahner force-pushed the support-sitemap-xml branch from c8e9129 to a4ec801 Compare March 11, 2026 14:37

Extract variable

d7435cd

thomas-zahner requested changes Mar 11, 2026

View reviewed changes

Comment thread lychee-lib/src/extract/xml.rs

Comment thread lychee-lib/src/extract/xml.rs Outdated

Comment thread lychee-lib/src/extract/xml.rs Outdated

cristiklein added 6 commits March 11, 2026 18:54

Ensure Atom test validates

167b2f9

Tested with https://validator.w3.org/feed/check.cgi Got "[Valid Atom 1.0] This is a valid Atom 1.0 feed."

Clarify span calculation

f27682a

Test all attributes in extract_xml

b945661

Make cargo check happy

f6f5fdd

Make typos happy

5955fe1

Make cargo fmt happy

d7617e7

cristiklein requested review from mre and thomas-zahner March 11, 2026 20:10

thomas-zahner approved these changes Mar 13, 2026

View reviewed changes

thomas-zahner merged commit 556d424 into lycheeverse:master Mar 13, 2026
7 checks passed

mre mentioned this pull request Mar 13, 2026

chore: release v0.24.0 #2048

Merged

thomas-zahner mentioned this pull request Mar 13, 2026

Parsing sitemaps not working #1819

Closed

donbeave mentioned this pull request Apr 25, 2026

ci(docs): bump lychee-action and lychee for sitemap URL extraction jackin-project/jackin#176

Merged

3 tasks

donbeave mentioned this pull request May 1, 2026

chore(deps): bump lychee to v0.24.2 jackin-project/jackin#206

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Support sitemap.xml#2071

feat: Support sitemap.xml#2071
thomas-zahner merged 16 commits into
lycheeverse:masterfrom
cristiklein:support-sitemap-xml

cristiklein commented Mar 4, 2026

Uh oh!

mre left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomas-zahner commented Mar 11, 2026 •

edited

Loading

Uh oh!

thomas-zahner left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cristiklein commented Mar 11, 2026

Uh oh!

Uh oh!

thomas-zahner commented Mar 13, 2026 •

edited

Loading

Uh oh!

cristiklein commented Mar 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

cristiklein commented Mar 4, 2026

Uh oh!

mre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomas-zahner commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomas-zahner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cristiklein commented Mar 11, 2026

Uh oh!

Uh oh!

thomas-zahner commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cristiklein commented Mar 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thomas-zahner commented Mar 11, 2026 •

edited

Loading

thomas-zahner commented Mar 13, 2026 •

edited

Loading