-
-
Notifications
You must be signed in to change notification settings - Fork 202
Implement Scheduled DevIndex Spider Workflow #9140
Copy link
Copy link
Closed
Labels
Description
Create a new GitHub Actions workflow (.github/workflows/devindex-spider.yml) to run the DevIndex Spider on a schedule.
Requirements:
- Schedule: Hourly at minute 30 (
30 * * * *) to offset from the Updater. - Execution: Run the spider command 3 times sequentially in a single job to maximize discovery per hour while balancing API usage.
- Command:
npm run devindex:spider - Data Handling: Commit and push changes to
tracker.json,visited.json, and other resource files. - Safety: Handle
git pull --rebasecorrectly by staging all modified JSON files (apps/devindex/resources/*.json*).
Workflow Steps:
- Checkout
- Setup Node
- Install Dependencies
- Loop: Run Spider x3
- Git Commit & Push (with
git reset --hardsafety)
Reactions are currently unavailable