-
-
Notifications
You must be signed in to change notification settings - Fork 202
Unify DevIndex Workflows and implement Spider backpressure #9227
Copy link
Copy link
Closed
Labels
aibuildBuild system, scripts, and configurationBuild system, scripts, and configurationenhancementNew feature or requestNew feature or request
Description
The DevIndex currently uses two independent GitHub Action workflows (devindex-spider.yml and devindex-updater.yml) that run on competing 30-minute cron schedules. This leads to intermittent Git merge conflicts when both jobs attempt to modify and push tracker.json simultaneously. Furthermore, uncontrolled Spider discovery can bloat the tracker backlog.
Tasks:
- Unify Workflows: Combine the Spider and Updater into a single, sequential pipeline (
.github/workflows/devindex-pipeline.yml) that runs hourly. This guarantees atomic commits and zero git conflicts. - Backpressure Valve: Enhance the Spider to check the number of "pending" users (
lastUpdate: null) intracker.json. If the backlog exceedsconfig.spider.maxPendingUsers(e.g., 2000), the Spider should abort its discovery run to let the Updater catch up. - Restore 3x Loop: With the workflows unified and backpressure in place, the Spider can safely be restored to execute 3 discovery loops per hour instead of 1.
- Cleanup: Remove the obsolete individual workflows.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
aibuildBuild system, scripts, and configurationBuild system, scripts, and configurationenhancementNew feature or requestNew feature or request