Skip to content

Implement Scheduled DevIndex Spider Workflow #9140

@tobiu

Description

@tobiu

Create a new GitHub Actions workflow (.github/workflows/devindex-spider.yml) to run the DevIndex Spider on a schedule.

Requirements:

  • Schedule: Hourly at minute 30 (30 * * * *) to offset from the Updater.
  • Execution: Run the spider command 3 times sequentially in a single job to maximize discovery per hour while balancing API usage.
  • Command: npm run devindex:spider
  • Data Handling: Commit and push changes to tracker.json, visited.json, and other resource files.
  • Safety: Handle git pull --rebase correctly by staging all modified JSON files (apps/devindex/resources/*.json*).

Workflow Steps:

  1. Checkout
  2. Setup Node
  3. Install Dependencies
  4. Loop: Run Spider x3
  5. Git Commit & Push (with git reset --hard safety)

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions