Skip to content

Unify DevIndex Workflows and implement Spider backpressure #9227

@tobiu

Description

@tobiu

The DevIndex currently uses two independent GitHub Action workflows (devindex-spider.yml and devindex-updater.yml) that run on competing 30-minute cron schedules. This leads to intermittent Git merge conflicts when both jobs attempt to modify and push tracker.json simultaneously. Furthermore, uncontrolled Spider discovery can bloat the tracker backlog.

Tasks:

  1. Unify Workflows: Combine the Spider and Updater into a single, sequential pipeline (.github/workflows/devindex-pipeline.yml) that runs hourly. This guarantees atomic commits and zero git conflicts.
  2. Backpressure Valve: Enhance the Spider to check the number of "pending" users (lastUpdate: null) in tracker.json. If the backlog exceeds config.spider.maxPendingUsers (e.g., 2000), the Spider should abort its discovery run to let the Updater catch up.
  3. Restore 3x Loop: With the workflows unified and backpressure in place, the Spider can safely be restored to execute 3 discovery loops per hour instead of 1.
  4. Cleanup: Remove the obsolete individual workflows.

Metadata

Metadata

Assignees

Labels

aibuildBuild system, scripts, and configurationenhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions