Skip to content

[WIP] feat(deployment)!: Add spider scheduler to packaging.#4

Closed
sitaowang1998 wants to merge 433 commits into
junhaoliao:docker-composefrom
sitaowang1998:spider-job-integration
Closed

[WIP] feat(deployment)!: Add spider scheduler to packaging.#4
sitaowang1998 wants to merge 433 commits into
junhaoliao:docker-composefrom
sitaowang1998:spider-job-integration

Conversation

@sitaowang1998

@sitaowang1998 sitaowang1998 commented Oct 9, 2025

Copy link
Copy Markdown

Note

This PR depends on y-scope#1178.

Description

This PR adds spider scheduler to the docker compose packaging. User can choose to use spider for compression orchestration by setting CompressionScheduler.type in configuration file.

This PR:

  • Adds spider database and scheduler configuration.
  • Adds credential validation for spider.
  • Adds a script to setup spider database and the use of it in create database service.
  • Adds a new docker compose file for spider services.
  • Update docker controller to setup environment variables for spider services.
  • Adds docker service to start spider scheduler.
  • Adds a script to start multiple spider workers and the use of it in docker service.
  • Updates compression scheduler to determine which task manager backend to use.

Checklist

  • The PR satisfies the contribution guidelines.
  • This is a breaking change and that has been indicated in the PR title, OR this isn't a
    breaking change.
  • Necessary docs have been updated, OR no docs need to be updated.

Validation performed

  • Compression with celery works.
  • Compression with spider works.
  • GitHub workflows pass.

@coderabbitai

coderabbitai Bot commented Oct 9, 2025

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@sitaowang1998 sitaowang1998 changed the title feat(deployment)!: Add spider scheduler to packaging. [WIP] feat(deployment)!: Add spider scheduler to packaging. Oct 9, 2025
junhaoliao and others added 26 commits October 23, 2025 22:30
… Remove hack in `var/data` conditional mounts (resolves y-scope#1478). (y-scope#1480)
…ured text logs into KV-IR; Update log-surgeon to yscope/log-surgeon@840f262. (y-scope#1460)

Co-authored-by: Bingran Hu <bingran.hu@yscope.com>
Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
Co-authored-by: LinZhihao-723 <zh.lin@mail.utoronto.ca>
… returned from KQL queries. (y-scope#1491)

Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
…rom VCS tracking (fixes y-scope#1422). (y-scope#1499)

Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
…er Compose project to match the CLP package Compose project (fixes y-scope#1496). (y-scope#1500)
…or S3-object and S3-key-prefix compression; Restrict `compress.py` to accept local file paths only. (y-scope#1476)

Co-authored-by: LinZhihao-723 <zh.lin@mail.utoronto.ca>
…r` and `spider_worker`) in the `clp-package` image; Set `mariadb-connector-cpp` build type to `Release` (fixes y-scope#1410). (y-scope#1414)
… using clp-json. (y-scope#1505)

Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
…extAst` into clp-s archives. (y-scope#1468)

Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
- Prevent stale result metadata from updating state.
- Recalculate result table height when switching sql interface.
- Prevent calling toFixed on null value in query speed status.
- Reduce size of results table to remove scroll bar.
… store the log path (instead of the errors) in `status_msg` (fixes y-scope#716). (y-scope#1425)
…che) to avoid failed host resolution of in-container hostname (fixes y-scope#1519). (y-scope#1518)
…vice-level compositions. (y-scope#1522)

Co-authored-by: Junhao Liao <junhao@junhao.ca>
@sitaowang1998 sitaowang1998 deleted the spider-job-integration branch December 5, 2025 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.