Skip to content

fix(cli): support next_token sync pagination#2818

Merged
mergify[bot] merged 2 commits into
mainfrom
fix/issue-2804-sync-resources
Jun 7, 2026
Merged

fix(cli): support next_token sync pagination#2818
mergify[bot] merged 2 commits into
mainfrom
fix/issue-2804-sync-resources

Conversation

@tmchow

@tmchow tmchow commented Jun 7, 2026

Copy link
Copy Markdown
Owner

Summary

Generated REST sync now recognizes next_token as a cursor request parameter and response cursor field, so multi-page APIs using that token shape no longer fall back to the hardcoded after cursor.

The sync profiler also carries temporal parameter formats into generated CLIs. Date-only lower-bound params such as start_date now receive YYYY-MM-DD, while datetime params keep RFC3339 values.

A generated-output test covers the multi-leaf collection shape from the issue: shared /usercollection/... list paths stay syncable, auth-tagged webhook resources stay out of the default sync set, cursor-backed resources are marked paginated, and the generated runtime helper formats date-only sync params correctly.

Closes #2804

Verification

  • go test ./...
  • scripts/golden.sh verify
  • scripts/verify-generator-output.sh
  • golangci-lint run ./...

@mergify

mergify Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 require-ready-label-and-ci

Wonderful, this rule succeeded.
  • #changes-requested-reviews-by = 0
  • #review-threads-unresolved = 0
  • check-success = build-and-test
  • check-success = generated-test
  • check-success = go-lint
  • check-success = golden
  • check-success = pr-title
  • check-success = test
  • any of:
    • label = ready-to-merge
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release
  • any of:
    • -files ~= ^(\.github/workflows/|\.github/scripts/|scripts/|\.github/CODEOWNERS$)
    • author = tmchow
    • approved-reviews-by = mvanhorn
    • approved-reviews-by = tmchow
    • author = mvanhorn
  • any of:
    • check-success = Greptile Review
    • label = queued
    • check-neutral = Greptile Review
    • check-skipped = Greptile Review
    • head ~= ^mergify/merge-queue/
    • all of:
      • head = release-please--branches--main
      • title ~= ^chore\(main\): release

@greptile-apps

greptile-apps Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR extends the generated sync layer to recognize next_token as a first-class cursor token (both as a request parameter and a response field), and propagates the OpenAPI format hint for since-type parameters so that date-only endpoints receive YYYY-MM-DD values instead of RFC3339 timestamps.

  • next_token cursor support: detectPagination in the parser now classifies next_token/nextToken as a cursor request param; nextFieldCursorNames and pageEnvelopeMetadataKeys are updated to recognize the matching response field, keeping detection and runtime extraction consistent.
  • Date-format-aware since params: A new SinceParamFormat field is threaded from metaFromEndpointsyncableMetaSyncableResource/DependentResource → the generated syncResourceSinceParamFormat switch and formatSyncSinceValue helper, which correctly converts RFC3339 and RFC3339Nano timestamps to YYYY-MM-DD for format: date parameters.
  • Expanded since-param heuristics: isEndpointSinceParamName adds exact-match names (start_date, start_datetime, start_time, from_date, from_datetime) to capture common temporal-filter patterns not previously covered by the substring checks.

Confidence Score: 5/5

Safe to merge. The changes are well-scoped additive extensions with no behavioral regressions on existing sync paths.

All three change areas (next_token cursor detection, SinceParamFormat propagation, and date-format conversion) are self-contained and well-tested. The parse ladder in formatSyncSinceValue handles RFC3339, RFC3339Nano, and pure-date inputs correctly. The SinceParamFormat field is threaded consistently through every copy site in profiler.go and both template loops. The injected end-to-end test compiles and exercises the generated code against a realistic multi-leaf spec.

No files require special attention.

Important Files Changed

Filename Overview
internal/profiler/profiler.go Adds SinceParamFormat field to syncableMeta, SyncableResource, and DependentResource; refactors since-param detection into detectEndpointSinceParamAndFormat; extracts isEndpointSinceParamName helper; correctly propagates the format through metaFromEndpoint, applySyncCandidates, dependentResourceFromEntry, applySpecWalkers, and sortedSyncableResources.
internal/generator/templates/sync.go.tmpl Adds syncResourceSinceParamFormat, formatSyncSinceValue (with RFC3339/RFC3339Nano/date-only parse ladder), and calls them in syncResource/syncDependentResource; extends extractPaginationFromEnvelope cursor key list with next_token/nextToken and pageEnvelopeMetadataKeys accordingly.
internal/openapi/parser.go Adds next_token/nexttoken to the cursor request-param candidates in detectPagination and to nextFieldCursorNames for response-field detection; both changes are consistent with the existing next_cursor handling.
internal/generator/generator_test.go New TestGenerateSyncIncludesMultiLeafCollectionResources test covers multi-leaf collection shape, auth-tag exclusion, cursor detection, and date-format conversion via an injected runtime test compiled and run against the generated CLI.
internal/openapi/parser_test.go Adds a next_token case to TestDetectPaginationPreservesCursorParamCase confirming snake-case variant is classified as cursor type.
internal/profiler/profiler_test.go Adds Format hints to existing since-param test fixtures and asserts the new SinceParamFormat field propagates correctly for both date-time and date formats.
testdata/golden/expected/generate-golden-api/printing-press-golden/internal/cli/sync.go Golden file updated with new syncResourceSinceParamFormat (empty switch — golden spec has no date-only since params), formatSyncSinceValue, next_token/nextToken cursor keys, and the format-conversion call sites in syncResource/syncDependentResource.
testdata/golden/expected/generate-sync-walker-api/sync-walker-golden/internal/cli/sync.go Golden file updated consistently with the same template additions as the other golden outputs.
testdata/golden/expected/generate-tier-routing-api/tier-routing-golden/internal/cli/sync.go Golden file updated consistently with the same template additions as the other golden outputs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[syncResource / syncDependentResource\ncalled with resource name + effectiveSince] --> B{effectiveSince\nnon-empty?}
    B -- No --> E[Skip format step]
    B -- Yes --> C[syncResourceSinceParamFormat\nresource]
    C --> D{SinceParamFormat\ngenerated switch}
    D -- date --> F[formatSyncSinceValue\nvalue, 'date']
    D -- date-time or empty --> G[return value unchanged]
    F --> H{try time.Parse\nRFC3339}
    H -- ok --> I[Format as\n2006-01-02]
    H -- err --> J{try time.Parse\nRFC3339Nano}
    J -- ok --> I
    J -- err --> K{try time.Parse\n2006-01-02}
    K -- ok --> L[return original\nalready a date]
    K -- err --> M[return original\nunparseable fallback]
    I --> N[effectiveSince = YYYY-MM-DD\nsent to API]
    G --> O[effectiveSince = RFC3339\nsent to API]
    L --> N
    M --> O

    style N fill:#d4edda
    style O fill:#d4edda
Loading

Reviews (2): Last reviewed commit: "fix(cli): address sync resource review f..." | Re-trigger Greptile

Comment thread internal/generator/templates/sync.go.tmpl
Comment thread internal/generator/generator_test.go
Comment thread internal/generator/generator_test.go
@tmchow tmchow added the ready-to-merge Allow Mergify to queue and merge this PR when protections pass label Jun 7, 2026
mergify Bot added a commit that referenced this pull request Jun 7, 2026
@mergify mergify Bot added the queued PR is in the Mergify merge queue label Jun 7, 2026
mergify Bot added a commit that referenced this pull request Jun 7, 2026
mergify Bot added a commit that referenced this pull request Jun 7, 2026
@mergify mergify Bot merged commit d0a5d24 into main Jun 7, 2026
31 checks passed
@mergify mergify Bot deleted the fix/issue-2804-sync-resources branch June 7, 2026 22:56
@mergify

mergify Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

This pull request spent 10 minutes 35 seconds in the queue, including 10 minutes 21 seconds running CI.

Required conditions to merge

@mergify mergify Bot removed the queued PR is in the Mergify merge queue label Jun 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Allow Mergify to queue and merge this PR when protections pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generator emits a non-functional sync layer for multi-resource specs (only one resource wired; shipcheck passes anyway)

1 participant