fix(ci): Correctly update and restore cache for `lint:check-cpp-lint-static-full`'s generated files (fixes #1419): by kirkrodrigues · Pull Request #1430 · y-scope/clp

kirkrodrigues · 2025-10-16T21:33:24Z

Save cache entries using unique key per entry.
Restore latest entries using key prefix.
Avoid using outputs from optionally-run restore task.

Description

#815 added caching for the input and output of lint:check-cpp-lint-static-full, but two issues were preventing the cache from being updated correctly:

I misunderstood the docs for the cache action, so the cache was not being updated on every push to main. Specifically, cache entries are immutable for a given key, so every new entry needs to have a new key. I mistakenly set the cache key to be the same for every entry, so no new entries could be written.
In scheduled runs, the cache was not being saved since the key was empty. This is because the key was derived from the restore step, but scheduled runs don't run the restore step (intentionally).

This PR resolves the issues as follows:

When saving a cache entry, we now append a hash (of the entry's contents) to the key. When restoring the cache, we use a prefix of that key, and actions/cache/restore will automatically select the newest entry.

Note

Although the action/cache docs imply that cache entries are stored in different namespaces determined by the OS used, running
gh api -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" /repos/y-scope/clp/actions/caches shows that cache entries for different OSes have the same version. So we still must use a different cache key prefix per OS and build configuration.

For action/cache/save, we specify the cache key manually rather than trying to rely on the value from action/cache/restore.

This fix should reduce clp-core-build-macos workflow runtimes from 2-3 hours to ~20 minutes.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Merged the change to main in my fork.
Validated that the new cache entries were created successfully.
Re-ran the clp-build-artifact and clp-macos-build workflows and validated that they utilized the cache and ran much more quickly.
Changed a C++ file on this branch and validated that clang-tidy was only run on that file, while all others were cached.

Note

clang-tidy violations are not being detected correctly, so we can't test if violations will be detected by the workflow. This will be resolved in another PR.

Summary by CodeRabbit

Chores
- Improved CI workflow caching: adjusted restore/save behavior so scheduled runs perform full lint checks, and introduced OS/config-specific cache keys with hash-based inputs to keep restore/save prefixes in sync — reducing missed lint issues and improving build consistency and reliability across environments.

…plicit value (fixes #xxx).

(cherry picked from commit d3cb7823504098b613dbad2461d44066b0bed2d0)

(cherry picked from commit 5972e08345e0ba5f692ca19c89efb205df7eedb2)

(cherry picked from commit 8ab2a05a0a7de46f134bfc42aed4c30b3a087f24)

This reverts commit 5972e08345e0ba5f692ca19c89efb205df7eedb2. (cherry picked from commit 0d3b4108f3985f95dc0eec900413573a81410ec5)

(cherry picked from commit 98ce1de6d33b57def441f00964fc997d58124d53)

(cherry picked from commit ba3843db1db970e6af513b0a478726782a59e0ba)

(cherry picked from commit 3f5c920aa2b29d7efc251b16b233fb99c963936f)

(cherry picked from commit 0d2235023282393cd920750d588137121d7a7c73)

This reverts commit 0d3b4108f3985f95dc0eec900413573a81410ec5. (cherry picked from commit 623d7c3eec613d7427fc03dece143a58fe61bfd2)

(cherry picked from commit d544455a4f8fb92fc3bd3cbfaa63849effbf84eb)

This reverts commit 623d7c3eec613d7427fc03dece143a58fe61bfd2. (cherry picked from commit 3b977078db2845053a45f538c090b0c945a70f7a)

(cherry picked from commit 71fcbd10076355ca44648c663e126ac377a1b00f)

(cherry picked from commit a4d60d70b20360d416bb307e261cc63b0883f223)

(cherry picked from commit 4ed354ab6c613133cb524bd7e79d25cdc5cf352a)

(cherry picked from commit 7e7a7dddfaef6f03748ffe4dda37725bc57c5221)

This reverts commit 40e55ad.

coderabbitai · 2025-10-16T21:33:35Z

Walkthrough

GitHub Actions workflows updated to change lint cache restore/save behavior: scheduled runs no longer restore lint caches (forcing full lint), cache keys made OS- and configuration-specific, and save keys use hashFiles-based prefixes that must stay in sync with restore prefixes.

Changes

Cohort / File(s)	Summary
Lint cache key synchronization & scheduled-run handling `/.github/workflows/clp-artifact-build.yaml`, `/.github/workflows/clp-core-build-macos.yaml`	Reworked cache restore/save logic for lint:check-cpp-static-full: restore is skipped for scheduled runs, restore key prefixes are OS-/config-specific, and save keys now include hashFiles(...) of lint checksum and build outputs. Notes added requiring restore and save key prefixes to remain in-sync.
Environment and execution adjustments `/.github/workflows/clp-artifact-build.yaml`	Ensured lint runs on a cleaned environment (explicit HOME and run-on-image) and updated cache key construction for ubuntu paths to match the new hash-based save key approach.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Scheduler as "Scheduled run"
    participant CI as "GitHub Actions job"
    participant Cache as "actions/cache"
    participant Lint as "lint:check-cpp-static-full"
    rect #E8F0FE
      Note over Scheduler,CI: Scheduled run (cron)
    end
    Scheduler->>CI: trigger job
    CI->>Cache: restore keys (skip for scheduled run)
    Cache-->>CI: no cache restored
    CI->>Lint: run full lint on all files
    Lint-->>CI: lint results
    CI->>Cache: save with hashFiles-based key (cannot rely on restore output)

sequenceDiagram
    autonumber
    participant Push as "Push/PR run"
    participant CI as "GitHub Actions job"
    participant Cache as "actions/cache"
    participant Lint as "lint:check-cpp-static-full"
    Push->>CI: trigger job
    CI->>Cache: restore using OS/config prefix (may match saved hash-prefix)
    Cache-->>CI: cache restored (if keys match)
    alt cache hit
      CI->>Lint: run lint (incremental/fast)
    else cache miss
      CI->>Lint: run full lint
    end
    Lint-->>CI: lint results
    CI->>Cache: save using hashFiles-based key (must match restore prefix format)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

ci: lint:check-cpp-lint-static-full cache not being updated #1419 — Matches the cache key construction and restore/save synchronization fixes for lint:check-cpp-static-full described in this PR.

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The pull request title "fix(ci): Correctly update and restore cache for `lint:check-cpp-lint-static-full`'s generated files (fixes #1419)" directly reflects the core purpose of the changeset. The PR addresses caching issues in the GitHub Actions workflows by making the cache keys unique for save operations (appending a hash), using key prefixes for restore operations, and explicitly setting cache keys per OS and build configuration. The title accurately captures the main fix—correcting how cache is updated and restored for lint check generated files. The title is concise, specific, and clearly communicates the primary change without unnecessary details.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Bill-hbrhbr

Other than we could update the pinned action sha to the latest v4.3.0 release, i don't have other comments. Logic looks correct but there's no way to test until we merge to main.

0057852bfaa89a56745cba8c7296529d2fc39830

Bill-hbrhbr · 2025-10-20T17:00:33Z

For title, how about:

fix(ci): Enable proper cache updates and reuse for C++ static linting task checksums (fixes #1419).

kirkrodrigues · 2025-10-20T18:45:22Z

For title, how about:

fix(ci): Enable proper cache updates and reuse for C++ static linting task checksums (fixes #1419).

How about a multi-line PR title & body:

fix(ci): Correctly update and restore cache for `lint:check-cpp-lint-static-full`'s generated files (fixes #1419):
- Save cache entries using unique key per entry.
- Restore entries using key prefix.
- Don't use outputs from optionally run `restore` task.

kirkrodrigues · 2025-10-20T18:45:40Z

Other than we could update the pinned action sha to the latest v4.3.0 release, i don't have other comments. Logic looks correct but there's no way to test until we merge to main.
0057852bfaa89a56745cba8c7296529d2fc39830

I'd prefer to do this in another PR so that future readers don't think this was a necessary change for this PR.

Bill-hbrhbr · 2025-10-20T18:52:40Z

For title, how about:

fix(ci): Enable proper cache updates and reuse for C++ static linting task checksums (fixes #1419).

How about a multi-line PR title & body:

fix(ci): Correctly update and restore cache for `lint:check-cpp-lint-static-full`'s generated files (fixes #1419):
- Save cache entries using unique key per entry.
- Restore entries using key prefix.
- Don't use outputs from optionally run `restore` task.

Restore latest entries...
Avoid using attributes/outputs

just some suggestions. otherwise lgtm

kirkrodrigues · 2025-10-20T20:36:29Z

Bypassing the remaining workflows since they will take hours to run, they shouldn't fail based on tests on my fork, and the effort of running them will be wasted unless they're run on main.

* feat(webui): Add drawer to display guided query and errors. (y-scope#1421) * docs: Add Slack community invite badge to home page README. (y-scope#1418) * refactor(clp-package): Simplify StrEnum and Path serialization via Annotated serializers. (y-scope#1417) Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves y-scope#1396); Upgrade dependencies for Python components. (y-scope#1405) * chore(docker): Add `--link` flags to COPY/ADD operations for improved build performance (fixes y-scope#1408). (y-scope#1411) * fix(ci): Correctly update and restore cache of `lint:check-cpp-lint-static-full`'s generated files (fixes y-scope#1419): (y-scope#1430) - Save cache entries using unique key per entry. - Restore latest entries using key prefix. - Avoid using outputs from optionally-run `restore` task. * fix(clp-rust-utils): Use AWS SDK default configuration with latest behavior version for S3 client. (y-scope#1445) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * refactor(clp-package): Remove unused `python-dotenv` dependency and related imports (fixes y-scope#1443). (y-scope#1444) * fix(webui): Submit queries that failed ANTLR validation to Presto. (y-scope#1450) * feat(clp-s): Explicitly reject unstructured log inputs during compression. (y-scope#1434) * feat(webui): Show query speed in native search status. (y-scope#1429) * fix(job-orchestration): Make `tag_ids` a required `list[int]` for compatibility with Spider compressor. (y-scope#1453) * feat(clp-mcp-server): Add log viewer links to query results for displaying in LLM output. (y-scope#1454) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * feat(ci): Add tasks for checking and updating Rust lock file (`Cargo.lock`); Add check to GH workflow. (y-scope#1448) * feat(webui): Trigger submit action when pressing Enter on Monaco single line editor. (y-scope#1459) * Add filters. * Update cargo lock. --------- Co-authored-by: davemarco <83603688+davemarco@users.noreply.github.com> Co-authored-by: Abigail Matthews <abigail.v.matthews@gmail.com> Co-authored-by: sitaowang1998 <sitaowang1998@outlook.com> Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com> Co-authored-by: Devin Gibson <gibber9809@users.noreply.github.com> Co-authored-by: hoophalab <200652805+hoophalab@users.noreply.github.com> Co-authored-by: Huangshi Tian <All-less@users.noreply.github.com>

* feat(webui): Add drawer to display guided query and errors. (y-scope#1421) * docs: Add Slack community invite badge to home page README. (y-scope#1418) * refactor(clp-package): Simplify StrEnum and Path serialization via Annotated serializers. (y-scope#1417) Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves y-scope#1396); Upgrade dependencies for Python components. (y-scope#1405) * chore(docker): Add `--link` flags to COPY/ADD operations for improved build performance (fixes y-scope#1408). (y-scope#1411) * fix(ci): Correctly update and restore cache of `lint:check-cpp-lint-static-full`'s generated files (fixes y-scope#1419): (y-scope#1430) - Save cache entries using unique key per entry. - Restore latest entries using key prefix. - Avoid using outputs from optionally-run `restore` task. * fix(clp-rust-utils): Use AWS SDK default configuration with latest behavior version for S3 client. (y-scope#1445) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * refactor(clp-package): Remove unused `python-dotenv` dependency and related imports (fixes y-scope#1443). (y-scope#1444) * fix(webui): Submit queries that failed ANTLR validation to Presto. (y-scope#1450) * feat(clp-s): Explicitly reject unstructured log inputs during compression. (y-scope#1434) * feat(webui): Show query speed in native search status. (y-scope#1429) * fix(job-orchestration): Make `tag_ids` a required `list[int]` for compatibility with Spider compressor. (y-scope#1453) * feat(clp-mcp-server): Add log viewer links to query results for displaying in LLM output. (y-scope#1454) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * feat(ci): Add tasks for checking and updating Rust lock file (`Cargo.lock`); Add check to GH workflow. (y-scope#1448) * feat(webui): Trigger submit action when pressing Enter on Monaco single line editor. (y-scope#1459) * Add filters. * Update cargo lock. * Stupid fix --------- Co-authored-by: davemarco <83603688+davemarco@users.noreply.github.com> Co-authored-by: Abigail Matthews <abigail.v.matthews@gmail.com> Co-authored-by: sitaowang1998 <sitaowang1998@outlook.com> Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com> Co-authored-by: Devin Gibson <gibber9809@users.noreply.github.com> Co-authored-by: hoophalab <200652805+hoophalab@users.noreply.github.com> Co-authored-by: Huangshi Tian <All-less@users.noreply.github.com>

…tatic-full`'s generated files (fixes y-scope#1419): (y-scope#1430) - Save cache entries using unique key per entry. - Restore latest entries using key prefix. - Avoid using outputs from optionally-run `restore` task.

kirkrodrigues added 19 commits October 5, 2025 02:42

fix(ci): Replace undefined cache key for actions/cache/save with ex…

bc99cd9

…plicit value (fixes #xxx).

Add ability to use workflow_dispatch to update the cache.

d905fa0

(cherry picked from commit d3cb7823504098b613dbad2461d44066b0bed2d0)

Change C++ file.

c19c81a

(cherry picked from commit 5972e08345e0ba5f692ca19c89efb205df7eedb2)

clp-artifact-build: Update cache key.

7452825

(cherry picked from commit 8ab2a05a0a7de46f134bfc42aed4c30b3a087f24)

Revert "Change C++ file."

de8ecf4

This reverts commit 5972e08345e0ba5f692ca19c89efb205df7eedb2. (cherry picked from commit 0d3b4108f3985f95dc0eec900413573a81410ec5)

Fix macos workflow.

cb7e00a

(cherry picked from commit 98ce1de6d33b57def441f00964fc997d58124d53)

Try latest actions/cache version.

ddc4c38

(cherry picked from commit ba3843db1db970e6af513b0a478726782a59e0ba)

Debug.

44aaac3

(cherry picked from commit 3f5c920aa2b29d7efc251b16b233fb99c963936f)

Use unique cache key per workflow run.

5edf49f

(cherry picked from commit 0d2235023282393cd920750d588137121d7a7c73)

Reapply "Change C++ file."

ec571cd

This reverts commit 0d3b4108f3985f95dc0eec900413573a81410ec5. (cherry picked from commit 623d7c3eec613d7427fc03dece143a58fe61bfd2)

Debug.

617b933

(cherry picked from commit d544455a4f8fb92fc3bd3cbfaa63849effbf84eb)

Revert "Reapply "Change C++ file.""

1d7817c

This reverts commit 623d7c3eec613d7427fc03dece143a58fe61bfd2. (cherry picked from commit 3b977078db2845053a45f538c090b0c945a70f7a)

Modify C++ file.

0e4f258

(cherry picked from commit 71fcbd10076355ca44648c663e126ac377a1b00f)

Fix macos workflow.

928e423

(cherry picked from commit a4d60d70b20360d416bb307e261cc63b0883f223)

Clean-up.

6b7b412

(cherry picked from commit 4ed354ab6c613133cb524bd7e79d25cdc5cf352a)

Change C++ file.

1ffed90

(cherry picked from commit 7e7a7dddfaef6f03748ffe4dda37725bc57c5221)

Make caches OS-specific.

49d34cc

Change C++ file.

40e55ad

Revert "Change C++ file."

7102129

This reverts commit 40e55ad.

kirkrodrigues requested a review from a team as a code owner October 16, 2025 21:33

kirkrodrigues changed the title ~~fix(ci): Replace undefined cache key for actions/cache/save with explicit value (fixes #1419).~~ fix(ci): Use unique cache key per cache entry to be saved and restore using key prefix (fixes #1419). Oct 16, 2025

kirkrodrigues added 2 commits October 16, 2025 17:38

Merge branch 'main' into fix-clang-tidy-cache

4877480

Touch-ups.

b694692

kirkrodrigues changed the title ~~fix(ci): Use unique cache key per cache entry to be saved and restore using key prefix (fixes #1419).~~ fix(ci): Save cache entries using unique key per entry, and restore entries using key prefix (fixes #1419). Oct 17, 2025

kirkrodrigues requested a review from Bill-hbrhbr October 17, 2025 04:57

Bill-hbrhbr approved these changes Oct 20, 2025

View reviewed changes

kirkrodrigues changed the title ~~fix(ci): Save cache entries using unique key per entry, and restore entries using key prefix (fixes #1419).~~ fix(ci): Correctly update and restore cache for lint:check-cpp-lint-static-full's generated files (fixes #1419): Oct 20, 2025

Merge branch 'main' into fix-clang-tidy-cache

facb397

kirkrodrigues merged commit 6d4545f into y-scope:main Oct 20, 2025
30 checks passed

kirkrodrigues deleted the fix-clang-tidy-cache branch October 20, 2025 20:38

LinZhihao-723 mentioned this pull request Jan 7, 2026

fix(ci): Correctly update and restore cache for lint:cpp-static-check's generated files . y-scope/spider#260

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(ci): Correctly update and restore cache for `lint:check-cpp-lint-static-full`'s generated files (fixes #1419):#1430

fix(ci): Correctly update and restore cache for `lint:check-cpp-lint-static-full`'s generated files (fixes #1419):#1430
kirkrodrigues merged 22 commits into
y-scope:mainfrom
kirkrodrigues:fix-clang-tidy-cache

kirkrodrigues commented Oct 16, 2025 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 16, 2025 •

edited

Loading

Uh oh!

Bill-hbrhbr left a comment

Uh oh!

Bill-hbrhbr commented Oct 20, 2025

Uh oh!

kirkrodrigues commented Oct 20, 2025

Uh oh!

kirkrodrigues commented Oct 20, 2025

Uh oh!

Bill-hbrhbr commented Oct 20, 2025

Uh oh!

kirkrodrigues commented Oct 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

kirkrodrigues commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related issues

Pre-merge checks and finishing touches

Uh oh!

Bill-hbrhbr left a comment

Choose a reason for hiding this comment

Uh oh!

Bill-hbrhbr commented Oct 20, 2025

Uh oh!

kirkrodrigues commented Oct 20, 2025

Uh oh!

kirkrodrigues commented Oct 20, 2025

Uh oh!

Bill-hbrhbr commented Oct 20, 2025

Uh oh!

kirkrodrigues commented Oct 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kirkrodrigues commented Oct 16, 2025 •

edited

Loading

coderabbitai Bot commented Oct 16, 2025 •

edited

Loading