build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves #1396); Upgrade dependencies for Python components. by junhaoliao · Pull Request #1405 · y-scope/clp

junhaoliao · 2025-10-10T01:26:05Z

Description

For Python packages clp-package-utils, clp-py-utils, job-orchestration:
- Migrate the package management system from poetry to uv
- Migrate the build system from poetry to hatchling
For Python packages clp-package-utils, clp-py-utils, job-orchestration, integration-tests:
Upgrade dependencies in all Python components and make sure versions of common dependencies match.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Built the package.
Uncommented the contents in build/clp-package/etc/clp-config.yml.
Started the package: sbin/start-clp.sh
Compressed sample logs: sbin/compress.sh ~/samples/hive-24hr
Performed a query in the webui and observed results listed.

Summary by CodeRabbit

Chores
- Migrated packaging/build tooling across components, consolidated dependency declarations and standardized authors metadata; removed obsolete tooling entries.
- Switched CI/task build references to a new component runner.
Tests
- Bumped integration test Python version to 3.10.
Bug Fixes
- Ensured Celery task registration and related job constants are accessible for runtime task registration.
Style
- Minor import ordering and whitespace cleanups for consistency.

…#1396); Require Python >= 3.10 (resolves y-scope#1380); Upgrade dependencies.

coderabbitai · 2025-10-10T01:26:15Z

Walkthrough

Multiple Python components switch build tooling from Poetry to Hatch/uv, several pyproject dependency specifiers and authors metadata are updated, taskfile tasks change from python-component to uv-component, requirements and integration-test Python version are adjusted, and numerous job-orchestration modules receive import reordering or added explicit imports.

Changes

Cohort / File(s)	Summary
Build system & metadata migration `components/clp-package-utils/pyproject.toml`, `components/clp-py-utils/pyproject.toml`, `components/job-orchestration/pyproject.toml`	Move from Poetry to Hatchling/uv: convert dependency declarations to `dependencies` arrays, change `authors` to structured objects, adjust `requires-python` formatting, and update `build-system` to use `hatchling.build`.
Dependency bumps `components/clp-mcp-server/pyproject.toml`	Bump runtime deps: `pymongo` from `>=4.15.1` → `>=4.15.3`; `msgpack` from `>=1.1.1` → `>=1.1.2`.
Task runner migration `taskfile.yaml`	Replace `python-component` tasks with `uv-component` for multiple components; remove internal `python-component` and `component-venv` blocks.
Requirements cleanup `requirements.txt`	Remove `poetry~=1.7` from requirements.
Integration tests Python version `integration-tests/.python-version`	Bump Python version from `3.9` → `3.10`.
Job-orchestration: executor imports & formatting `components/job-orchestration/job_orchestration/executor/compress/celery_compress.py`, `.../compress/compression_task.py`, `.../compress/spider_compress.py`, `.../query/extract_stream_task.py`, `.../query/fs_search_task.py`, `.../query/utils.py`	Add or relocate imports (e.g., import Celery `app`, `CompressionTaskStatus`, spider types, scheduler models); minor spacing/formatting changes; no behavioral changes.
Job-orchestration: scheduler imports & formatting `components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py`, `.../compress/partition.py`, `.../compress/task_manager/celery_task_manager.py`, `.../compress/task_manager/spider_task_manager.py`, `.../query/query_scheduler.py`, `.../query/reducer_handler.py`, `.../scheduler/scheduler_data.py`, `.../scheduler/task_result.py`, `.../scheduler/utils.py`	Normalize or reposition imports (pydantic types, `FileMetadata`, `Job`, etc.), add/remove blank lines, and note one duplicate import; no interface or control-flow changes.

Sequence Diagram(s)

(omitted — changes are import/configuration-only and do not introduce new runtime control flow)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

Migrate Python components from poetry to uv build system #1396 — Implements Poetry→uv/Hatchling migration across components, matching this PR’s build-system and pyproject changes.
Add task for checking/upgrading uv.lock files for all UV components #1381 — Changes Taskfile to uv-component, relevant to UV-managed component and lock file management efforts.

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.
Title Check	✅ Passed	The pull request title "build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves #1396); Upgrade dependencies for Python components." accurately reflects the primary objectives of the changeset. The title clearly identifies the main change—migrating from Poetry to uv and hatchling as the build/packaging backend across multiple Python components—and mentions the secondary objective of upgrading dependencies. The title is concise, specific, and avoids vague or generic phrasing, making it clear to teammates scanning the history that this PR addresses a significant infrastructure modernization alongside dependency updates.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

LinZhihao-723

Are we planning to include uv locks? If so, shall we update tasks for uv lock file checking? #1381

junhaoliao · 2025-10-10T01:31:54Z

Are we planning to include uv locks?

yes

If so, shall we update tasks for uv lock file checking? #1381

i had to write a script to run uv lock --upgrade in different components. i agree that we should add a task (without --upgrade) to save manual efforts. mind if we do it in a separate PR?

LinZhihao-723 · 2025-10-10T01:37:24Z

Are we planning to include uv locks?

yes

If so, shall we update tasks for uv lock file checking? #1381

i had to write a script to run uv lock --upgrade in different components. i agree that we should add a task (without --upgrade) to save manual efforts. mind if we do it in a separate PR?

Sure, just a heads up

junhaoliao · 2025-10-10T02:26:02Z

this empty file has to be added. otherwise, uv build had trouble finding this as a valid Python package.

junhaoliao · 2025-10-10T02:28:36Z

All the changed Python source files in this PR are just the results of running task lint:fix-py.

Yeah I think these changes make sense: before this PR, there's no project name set in the pyproject.toml. Now the imports are separated into "third-party" dependency and the project imports.

junhaoliao · 2025-10-12T21:26:30Z

as discussed offline with @kirkrodrigues , the Python 3.9 -> 3.10 requirement change will be done in a separate PR.

coderabbitai

Actionable comments posted: 3

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 72459fd and 6fe93a1.

⛔ Files ignored due to path filters (5)

components/clp-mcp-server/uv.lock is excluded by !**/*.lock
components/clp-package-utils/uv.lock is excluded by !**/*.lock
components/clp-py-utils/uv.lock is excluded by !**/*.lock
components/job-orchestration/uv.lock is excluded by !**/*.lock
integration-tests/uv.lock is excluded by !**/*.lock

📒 Files selected for processing (3)

components/clp-package-utils/pyproject.toml (1 hunks)
components/clp-py-utils/pyproject.toml (1 hunks)
components/job-orchestration/pyproject.toml (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (18)

GitHub Check: musllinux_1_2-x86_64-static-linked-bins
GitHub Check: package-image
GitHub Check: ubuntu-jammy-dynamic-linked-bins
GitHub Check: musllinux_1_2-x86_64-dynamic-linked-bins
GitHub Check: ubuntu-jammy-static-linked-bins
GitHub Check: manylinux_2_28-x86_64-static-linked-bins
GitHub Check: ubuntu-jammy-lint
GitHub Check: manylinux_2_28-x86_64-dynamic-linked-bins
GitHub Check: centos-stream-9-static-linked-bins
GitHub Check: centos-stream-9-dynamic-linked-bins
GitHub Check: build-macos (macos-14, false)
GitHub Check: build-macos (macos-14, true)
GitHub Check: build-macos (macos-15, true)
GitHub Check: rust-checks (macos-15)
GitHub Check: rust-checks (ubuntu-22.04)
GitHub Check: rust-checks (ubuntu-24.04)
GitHub Check: lint-check (macos-15)
GitHub Check: lint-check (ubuntu-24.04)

coderabbitai · 2025-10-12T21:37:33Z

+authors = [{name = "YScope Inc.", email = "dev@yscope.com"}]
 readme = "README.md"
-
-[tool.poetry.dependencies]
-python = ">= 3.9"
-Brotli = "^1.1.0"
-msgpack = "^1.0.7"
-psutil = "^7.1.0"
-pydantic = "^2.11.9"
-pymongo = "^4.7.2"
-PyYAML = "^6.0.1"
+requires-python = ">=3.9"
+dependencies = [
+    "Brotli>=1.1.0",
+    "msgpack>=1.1.2",
+    "psutil>=7.1.0",
+    "pydantic>=2.12.0",
+    "pymongo>=4.15.3",
+    "PyYAML>=6.0.3",
+]


⚠️ Potential issue | 🟠 Major

Align requires-python with the 3.10 floor.

The PR mandate is to move every package to Python ≥3.10, yet Line 7 still advertises support down to 3.9. This leaves the wheel installable on 3.9, diverging from the documented requirement and increasing support risk. Please bump the constraint here (and ensure matching metadata elsewhere) to enforce the new floor.

-requires-python = ">=3.9" +requires-python = ">=3.10"

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

authors = [{name = "YScope Inc.", email = "dev@yscope.com"}]

readme = "README.md"

[tool.poetry.dependencies]

python = ">= 3.9"

Brotli = "^1.1.0"

msgpack = "^1.0.7"

psutil = "^7.1.0"

pydantic = "^2.11.9"

pymongo = "^4.7.2"

PyYAML = "^6.0.1"

requires-python = ">=3.9"

dependencies = [

"Brotli>=1.1.0",

"msgpack>=1.1.2",

"psutil>=7.1.0",

"pydantic>=2.12.0",

"pymongo>=4.15.3",

"PyYAML>=6.0.3",

]

authors = [{name = "YScope Inc.", email = "dev@yscope.com"}]

readme = "README.md"

requires-python = ">=3.10"

dependencies = [

"Brotli>=1.1.0",

"msgpack>=1.1.2",

"psutil>=7.1.0",

"pydantic>=2.12.0",

"pymongo>=4.15.3",

"PyYAML>=6.0.3",

]

🤖 Prompt for AI Agents

In components/clp-package-utils/pyproject.toml around lines 5 to 15, update the requires-python constraint from ">=3.9" to ">=3.10" to enforce the new Python 3.10 floor; then search the repository for other packaging metadata (other pyproject.toml, setup.cfg, setup.py, or wheel classifiers) and CI matrix entries and update any remaining references to Python 3.9 so they consistently reflect >=3.10.

It was decided that we should do the Python version requirement change in a separate PR to reduce review effort

@junhaoliao, understood! I'll note that the Python 3.10 migration is being handled separately.

Would you like me to create a GitHub issue to track this follow-up work? I can include a reference back to this PR and the discussion.

✏️ Learnings added

Learnt from: junhaoliao PR: y-scope/clp#1405 File: components/clp-package-utils/pyproject.toml:5-15 Timestamp: 2025-10-13T03:24:35.053Z Learning: In the y-scope/clp repository, the Python 3.9 to 3.10 version requirement change was intentionally deferred to a separate PR (after PR #1405) to reduce review effort, as decided in an offline discussion between junhaoliao and kirkrodrigues.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

🧠 Learnings used

Learnt from: junhaoliao PR: y-scope/clp#1242 File: taskfile.yaml:545-547 Timestamp: 2025-08-20T05:38:30.720Z Learning: The python component directories in the y-scope/clp repository are kept very clean and only contain essential files: source code directories, pyproject.toml, poetry.lock, README.md, and the dist/ directory. No cache directories, virtual environments, or other transient files exist in the component directories, so additional exclude patterns beyond "dist/**/*" are unnecessary.

coderabbitai · 2025-10-12T21:37:33Z

+authors = [{name = "YScope Inc.", email = "dev@yscope.com"}]
 readme = "README.md"
-
-[tool.poetry.dependencies]
-python = ">= 3.9"
-boto3 = "^1.35.81"
-# mariadb version must be compatible with libmariadev installed in runtime env.
-# See https://mariadb.com/docs/server/connect/programming-languages/python/install/#Dependencies
-mariadb = "~1.0.11"
-mysql-connector-python = "^8.2.0"
-pydantic = "^2.11.9"
-python-dotenv = "^1.0.1"
-python-Levenshtein = "~0.22"
-sqlalchemy = "~2.0"
-PyYAML = "^6.0.1"
-result = "^0.17.0"
-StrEnum = "^0.4.15"
+requires-python = ">=3.9"
+dependencies = [
+    "boto3>=1.40.49",
+    # mariadb version must be compatible with libmariadev installed in runtime env.
+    # See https://mariadb.com/docs/server/connect/programming-languages/python/install/#Dependencies
+    "mariadb>=1.0.11,<1.1.dev0",
+    "mysql-connector-python>=9.4.0",
+    "pydantic>=2.12.0",
+    "python-dotenv>=1.1.1",
+    "python-Levenshtein>=0.27.1",
+    "sqlalchemy>=2.0.43",
+    "PyYAML>=6.0.3",
+    "result>=0.17.0",
+    "StrEnum>=0.4.15",
+]


⚠️ Potential issue | 🟠 Major

Enforce the Python 3.10 minimum here as well.

Line 7 continues to advertise support for Python 3.9, so users on 3.9 will receive this build despite the intent to drop that runtime. Please update the requirement to reflect the new ≥3.10 baseline.

-requires-python = ">=3.9" +requires-python = ">=3.10"

🤖 Prompt for AI Agents

In components/clp-py-utils/pyproject.toml around lines 5 to 21, the requires-python field still states ">=3.9" while the project now enforces a Python ≥3.10 baseline; update the requires-python value to ">=3.10" (and if present elsewhere in the file or package metadata, make the same change) so published builds correctly require Python 3.10 or newer.

It was decided that we should do the Python version requirement change in a separate PR to reduce review effort

@junhaoliao, understood! That makes sense to separate the Python version requirement change into its own PR.

Would you like me to create a follow-up issue to track updating the requires-python field to ">=3.10" across all Python packages in the repository?

🧠 Learnings used

Learnt from: junhaoliao PR: y-scope/clp#1242 File: taskfile.yaml:545-547 Timestamp: 2025-08-20T05:38:30.720Z Learning: The python component directories in the y-scope/clp repository are kept very clean and only contain essential files: source code directories, pyproject.toml, poetry.lock, README.md, and the dist/ directory. No cache directories, virtual environments, or other transient files exist in the component directories, so additional exclude patterns beyond "dist/**/*" are unnecessary.

coderabbitai · 2025-10-12T21:37:33Z

+authors = [{name = "YScope Inc.", email = "dev@yscope.com"}]
 readme = "README.md"
-
-[tool.poetry.dependencies]
-python = ">= 3.9,<4.0"
-Brotli = "^1.1.0"
-celery = {extras = ["redis"], version = "^5.5.3"}
-# mariadb version must be compatible with libmariadev installed in runtime env.
-# See https://mariadb.com/docs/server/connect/programming-languages/python/install/#Dependencies
-mariadb = "~1.0.11"
-msgpack = "^1.0.7"
-mysql-connector-python = "^8.2.0"
-pika = "^1.3.2"
-pydantic = "^2.11.9"
-pymongo = "^4.7.2"
-PyYAML = "^6.0.1"
-yscope-spider-py="0.1.0"
+requires-python = ">=3.9,<4.0"
+dependencies = [
+    "Brotli>=1.1.0",
+    "celery[redis]>=5.5.3",
+    # mariadb version must be compatible with libmariadev installed in runtime env.
+    # See https://mariadb.com/docs/server/connect/programming-languages/python/install/#Dependencies
+    "mariadb>=1.0.11,<1.1.dev0",
+    "msgpack>=1.1.2",
+    "mysql-connector-python>=9.4.0",
+    "pika>=1.3.2",
+    "pydantic>=2.12.0",
+    "pymongo>=4.15.3",
+    "PyYAML>=6.0.3",
+    "yscope-spider-py==0.1.0",
+]


⚠️ Potential issue | 🟠 Major

Update the Python compatibility floor.

This pyproject still declares Python ≥3.9, contradicting the PR goal of raising the requirement to ≥3.10. Without updating the marker, installers will keep pulling the package on 3.9 runtimes. Please tighten the constraint accordingly (keeping the <4.0 upper bound if that is still intended).

-requires-python = ">=3.9,<4.0" +requires-python = ">=3.10,<4.0"

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

authors = [{name = "YScope Inc.", email = "dev@yscope.com"}]

readme = "README.md"

[tool.poetry.dependencies]

python = ">= 3.9,<4.0"

Brotli = "^1.1.0"

celery = {extras = ["redis"], version = "^5.5.3"}

# mariadb version must be compatible with libmariadev installed in runtime env.

# See https://mariadb.com/docs/server/connect/programming-languages/python/install/#Dependencies

mariadb = "~1.0.11"

msgpack = "^1.0.7"

mysql-connector-python = "^8.2.0"

pika = "^1.3.2"

pydantic = "^2.11.9"

pymongo = "^4.7.2"

PyYAML = "^6.0.1"

yscope-spider-py="0.1.0"

requires-python = ">=3.9,<4.0"

dependencies = [

"Brotli>=1.1.0",

"celery[redis]>=5.5.3",

# mariadb version must be compatible with libmariadev installed in runtime env.

# See https://mariadb.com/docs/server/connect/programming-languages/python/install/#Dependencies

"mariadb>=1.0.11,<1.1.dev0",

"msgpack>=1.1.2",

"mysql-connector-python>=9.4.0",

"pika>=1.3.2",

"pydantic>=2.12.0",

"pymongo>=4.15.3",

"PyYAML>=6.0.3",

"yscope-spider-py==0.1.0",

]

authors = [{name = "YScope Inc.", email = "dev@yscope.com"}]

readme = "README.md"

requires-python = ">=3.10,<4.0"

dependencies = [

"Brotli>=1.1.0",

"celery[redis]>=5.5.3",

# mariadb version must be compatible with libmariadev installed in runtime env.

# See https://mariadb.com/docs/server/connect/programming-languages/python/install/#Dependencies

"mariadb>=1.0.11,<1.1.dev0",

"msgpack>=1.1.2",

"mysql-connector-python>=9.4.0",

"pika>=1.3.2",

"pydantic>=2.12.0",

"pymongo>=4.15.3",

"PyYAML>=6.0.3",

"yscope-spider-py==0.1.0",

]

🤖 Prompt for AI Agents

In components/job-orchestration/pyproject.toml around lines 5 to 21, the requires-python marker still allows Python 3.9 but the PR intends to raise the minimum to 3.10; update the requires-python entry to ">=3.10,<4.0" so installers won't accept Python 3.9 runtimes while preserving the existing upper bound.

It was decided that we should do the Python version requirement change in a separate PR to reduce review effort. Therefore, the requires-python value was unchanged

@junhaoliao, understood! Thanks for clarifying that the Python version bump will be handled in a separate PR. That makes sense for keeping this review focused on the migration to uv + hatchling. I'll skip this suggestion accordingly.

…n pyproject.toml and uv.lock.

LinZhihao-723

Overall lgtm. Only some minor fixes.

LinZhihao-723 · 2025-10-16T20:58:16Z

Shall we defer this?

LinZhihao-723 · 2025-10-16T21:05:06Z

Yeah I think these changes make sense: before this PR, there's no project name set in the pyproject.toml. Now the imports are separated into "third-party" dependency and the project imports.

Bill-hbrhbr

Several general comments:

Each pyproject.toml should follow
https://github.com/y-scope/yscope-dev-utils/tree/main/exports/lint-configs/python
If we really want to make each component into a formal python package, then we should add a src layer. The example setup includes:

the integration tests component (even though it doesn't have actual source files)
log-archival-bench repo (this one doesn't have test files yet)
ystdlib-py

We should probably use the same python version everywhere? idk if mixing up 3.9 and 3.10 makes sense. Or we just defer changing python req in this PR like you said.
The linting process needs to be updated: https://github.com/junhaoliao/clp/blob/cf28cae05f47e32c18ce00cb385ce540cbe9ad43/taskfiles/lint.yaml#L780
But that would probably pollute the scope of this PR. Just need to be aware of this code section that's pending updates.

Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>

junhaoliao · 2025-10-18T02:08:07Z

Thanks for the review. All the points you mentioned are worth doing, but some of them will make the current PR even larger and more difficult to review. Do you mind if we defer some of the items? (feel free to ask the rabbit to create any issue so we don't forget)

please see below for my responses:

Each pyproject.toml should follow
y-scope/yscope-dev-utils@main/exports/lint-configs/python

Do you mean the linting configs should be ported from https://github.com/y-scope/yscope-dev-utils/blob/main/exports/ystdlib-py/pyproject.toml ? I'm aware that they are needed to enable the rules but I intentionally left those out to avoid bloating the PR, which mainly aims to migrate from poetry to uv.

If we really want to make each component into a formal python package, then we should add a src layer. The example setup includes:

the integration tests component (even though it doesn't have actual source files)

log-archival-bench repo (this one doesn't have test files yet)

ystdlib-py

Do you mind if we merge the current PR first, and then open a separate, simple PR to move the sources into the src/ folders during the next development cycle - after the release, once the PRs that modify multiple Python files (like #1178, #1417, and #1425) have been merged?

We should probably use the same python version everywhere? idk if mixing up 3.9 and 3.10 makes sense. Or we just defer changing python req in this PR like you said.

It was @kirkrodrigues 's proposal to defer the Python 3.9 -> 3.10 upgrade to reduce review effort. As a result, the changes to upgrade the min required version were reverted in 6fe93a1 and 464ad89. I believe the Python wheels for clp-package-utils, clp-py-utils and job-orchestration are always built with Python 3.10, with or without the changes in this PR, and therefore there should be no harm if we leave the min required version unchanged.

That said, we should have updated the User Quick Start Guide to say we officially only support Python 3.10 (instead of "3.9 or higher) when we were removing the Jammy Focal image because the libraries are only built for Python 3.10. @coderabbitai create an issue for the incorrect doc

The linting process needs to be updated: junhaoliao/clp@cf28cae/taskfiles/lint.yaml#L780
But that would probably pollute the scope of this PR. Just need to be aware of this code section that's pending updates.

Right, i think we should defer the linter config updates.

coderabbitai · 2025-10-18T02:08:48Z

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Create a GitHub issue for updating Python version documentation

gh issue create \
  --repo y-scope/clp \
  --title "Update User Quick Start Guide to reflect Python 3.10 as the officially supported version" \
  --body "## Background

When the Jammy Focal image was removed, the libraries were only built for Python 3.10. However, the User Quick Start Guide still states support for Python 3.9 or higher, which is inconsistent with the actual build configuration.

## Required Changes

Update the User Quick Start Guide to clearly state that Python 3.10 is the officially supported version, rather than \"3.9 or higher\".

## References

- PR: https://github.com/y-scope/clp/pull/1405
- Comment: https://github.com/y-scope/clp/pull/1405#discussion_r${DISCUSSION_ID}
- Requested by: @junhaoliao" \
  --assignee junhaoliao

Length of output: 42

@junhaoliao, I've created issue #1438 to track the documentation update for Python 3.10 support in the User Quick Start Guide.

Your approach to defer the following items makes perfect sense to keep this PR focused on the core migration from poetry to uv + hatchling:

Linting configs from yscope-dev-utils - Can be added in a follow-up PR
Adding src layer - Good plan to wait until after the release when feat(deployment)!: Migrate package orchestration to Docker Compose (resolves #1177); Temporarily remove support for multi-node deployments. #1178, refactor(clp-package): Simplify StrEnum and Path serialization via Annotated serializers. #1417, and fix(clp-package): Write compression task failure errors to a log, and store the log path (instead of the errors) in status_msg (fixes #716). #1425 are merged to avoid conflicts
Python 3.9 → 3.10 requirement change - Already planned for a separate PR per your discussion with @kirkrodrigues
Linter config updates in taskfiles/lint.yaml - Can be addressed alongside item Can not build on redHat 7 #1

This staged approach will make the reviews more manageable and reduce the risk of conflicts with other in-flight PRs. The current PR already accomplishes the primary objective of migrating the build system and dependency management tooling.

🧠 Learnings used

Learnt from: junhaoliao
PR: y-scope/clp#1405
File: components/clp-package-utils/pyproject.toml:5-15
Timestamp: 2025-10-13T03:24:35.074Z
Learning: In the y-scope/clp repository, the Python 3.9 to 3.10 version requirement change was intentionally deferred to a separate PR (after PR #1405) to reduce review effort, as decided in an offline discussion between junhaoliao and kirkrodrigues.

coderabbitai · 2025-10-18T13:14:43Z

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Create a GitHub issue for updating Python version documentation

gh issue create \
  --repo y-scope/clp \
  --title "Update User Quick Start Guide to reflect Python 3.10 as the officially supported version" \
  --body "## Background

When the Jammy Focal image was removed, the libraries were only built for Python 3.10. However, the User Quick Start Guide still states support for Python 3.9 or higher, which is inconsistent with the actual build configuration.

## Required Changes

Update the User Quick Start Guide to clearly state that Python 3.10 is the officially supported version, rather than \"3.9 or higher\".

## References

- PR: https://github.com/y-scope/clp/pull/1405
- Requested by: @junhaoliao" \
  --assignee junhaoliao

Length of output: 42

@junhaoliao, I've successfully created issue #1439 to track the documentation update for Python 3.10 support in the User Quick Start Guide.

Your staged approach to defer the following items makes perfect sense to keep this PR focused on the core migration from poetry to uv + hatchling:

Linting configs from yscope-dev-utils - Can be added in a follow-up PR
Adding src layer - Good plan to wait until after the release when feat(deployment)!: Migrate package orchestration to Docker Compose (resolves #1177); Temporarily remove support for multi-node deployments. #1178, refactor(clp-package): Simplify StrEnum and Path serialization via Annotated serializers. #1417, and fix(clp-package): Write compression task failure errors to a log, and store the log path (instead of the errors) in status_msg (fixes #716). #1425 are merged to avoid conflicts
Python 3.9 → 3.10 requirement change - Already planned for a separate PR per your discussion with @kirkrodrigues
Linter config updates in taskfiles/lint.yaml - Can be addressed alongside the linting config porting

This pragmatic approach will make the reviews more manageable and reduce the risk of merge conflicts with other in-flight PRs. The current PR already accomplishes the primary objective of migrating the build system and dependency management tooling, which is substantial work on its own.

🧠 Learnings used

Learnt from: junhaoliao
PR: y-scope/clp#1405
File: components/clp-package-utils/pyproject.toml:5-15
Timestamp: 2025-10-13T03:24:35.074Z
Learning: In the y-scope/clp repository, the Python 3.9 to 3.10 version requirement change was intentionally deferred to a separate PR (after PR #1405) to reduce review effort, as decided in an offline discussion between junhaoliao and kirkrodrigues.

# Conflicts: # components/clp-mcp-server/pyproject.toml # components/clp-mcp-server/uv.lock

…; update locks.

LinZhihao-723

Approved as I think getting this PR merged might have higher priority and we should address the linting and project structure problems in future PRs. It would unblock some other PRs if we can switch to uv for Python component management.

For the PR title, how about:

build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves #1396); Upgrade dependencies for Python components.

* feat(webui): Add drawer to display guided query and errors. (y-scope#1421) * docs: Add Slack community invite badge to home page README. (y-scope#1418) * refactor(clp-package): Simplify StrEnum and Path serialization via Annotated serializers. (y-scope#1417) Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves y-scope#1396); Upgrade dependencies for Python components. (y-scope#1405) * chore(docker): Add `--link` flags to COPY/ADD operations for improved build performance (fixes y-scope#1408). (y-scope#1411) * fix(ci): Correctly update and restore cache of `lint:check-cpp-lint-static-full`'s generated files (fixes y-scope#1419): (y-scope#1430) - Save cache entries using unique key per entry. - Restore latest entries using key prefix. - Avoid using outputs from optionally-run `restore` task. * fix(clp-rust-utils): Use AWS SDK default configuration with latest behavior version for S3 client. (y-scope#1445) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * refactor(clp-package): Remove unused `python-dotenv` dependency and related imports (fixes y-scope#1443). (y-scope#1444) * fix(webui): Submit queries that failed ANTLR validation to Presto. (y-scope#1450) * feat(clp-s): Explicitly reject unstructured log inputs during compression. (y-scope#1434) * feat(webui): Show query speed in native search status. (y-scope#1429) * fix(job-orchestration): Make `tag_ids` a required `list[int]` for compatibility with Spider compressor. (y-scope#1453) * feat(clp-mcp-server): Add log viewer links to query results for displaying in LLM output. (y-scope#1454) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * feat(ci): Add tasks for checking and updating Rust lock file (`Cargo.lock`); Add check to GH workflow. (y-scope#1448) * feat(webui): Trigger submit action when pressing Enter on Monaco single line editor. (y-scope#1459) * Add filters. * Update cargo lock. --------- Co-authored-by: davemarco <83603688+davemarco@users.noreply.github.com> Co-authored-by: Abigail Matthews <abigail.v.matthews@gmail.com> Co-authored-by: sitaowang1998 <sitaowang1998@outlook.com> Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com> Co-authored-by: Devin Gibson <gibber9809@users.noreply.github.com> Co-authored-by: hoophalab <200652805+hoophalab@users.noreply.github.com> Co-authored-by: Huangshi Tian <All-less@users.noreply.github.com>

* feat(webui): Add drawer to display guided query and errors. (y-scope#1421) * docs: Add Slack community invite badge to home page README. (y-scope#1418) * refactor(clp-package): Simplify StrEnum and Path serialization via Annotated serializers. (y-scope#1417) Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves y-scope#1396); Upgrade dependencies for Python components. (y-scope#1405) * chore(docker): Add `--link` flags to COPY/ADD operations for improved build performance (fixes y-scope#1408). (y-scope#1411) * fix(ci): Correctly update and restore cache of `lint:check-cpp-lint-static-full`'s generated files (fixes y-scope#1419): (y-scope#1430) - Save cache entries using unique key per entry. - Restore latest entries using key prefix. - Avoid using outputs from optionally-run `restore` task. * fix(clp-rust-utils): Use AWS SDK default configuration with latest behavior version for S3 client. (y-scope#1445) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * refactor(clp-package): Remove unused `python-dotenv` dependency and related imports (fixes y-scope#1443). (y-scope#1444) * fix(webui): Submit queries that failed ANTLR validation to Presto. (y-scope#1450) * feat(clp-s): Explicitly reject unstructured log inputs during compression. (y-scope#1434) * feat(webui): Show query speed in native search status. (y-scope#1429) * fix(job-orchestration): Make `tag_ids` a required `list[int]` for compatibility with Spider compressor. (y-scope#1453) * feat(clp-mcp-server): Add log viewer links to query results for displaying in LLM output. (y-scope#1454) Co-authored-by: Junhao Liao <junhao.liao@yscope.com> * feat(ci): Add tasks for checking and updating Rust lock file (`Cargo.lock`); Add check to GH workflow. (y-scope#1448) * feat(webui): Trigger submit action when pressing Enter on Monaco single line editor. (y-scope#1459) * Add filters. * Update cargo lock. * Stupid fix --------- Co-authored-by: davemarco <83603688+davemarco@users.noreply.github.com> Co-authored-by: Abigail Matthews <abigail.v.matthews@gmail.com> Co-authored-by: sitaowang1998 <sitaowang1998@outlook.com> Co-authored-by: Junhao Liao <junhao@junhao.ca> Co-authored-by: Junhao Liao <junhao.liao@yscope.com> Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com> Co-authored-by: Devin Gibson <gibber9809@users.noreply.github.com> Co-authored-by: hoophalab <200652805+hoophalab@users.noreply.github.com> Co-authored-by: Huangshi Tian <All-less@users.noreply.github.com>

…ackend for Python components (resolves y-scope#1396); Upgrade dependencies for Python components. (y-scope#1405)

junhaoliao added 2 commits October 9, 2025 21:12

chore(clp-package): Migrate fully to uv + hatchling (resolves y-scope…

1f195bf

…#1396); Require Python >= 3.10 (resolves y-scope#1380); Upgrade dependencies.

lint

14c30fa

junhaoliao requested a review from a team as a code owner October 10, 2025 01:26

junhaoliao requested review from Bill-hbrhbr and LinZhihao-723 October 10, 2025 01:26

junhaoliao added 2 commits October 9, 2025 21:27

docs: update required Python version to 3.10 in all docs

92c0961

chore: add uv.lock

0caddbb

LinZhihao-723 reviewed Oct 10, 2025

View reviewed changes

junhaoliao marked this pull request as draft October 10, 2025 01:32

chore: remove poetry from requirements.txt

c93ee6a

add __init__.py to job_orchestration to make it a proper package

cd9aa66

junhaoliao marked this pull request as ready for review October 10, 2025 01:57

junhaoliao requested a review from LinZhihao-723 October 10, 2025 01:57

junhaoliao commented Oct 10, 2025

View reviewed changes

Merge branch 'main' into migrate-uv

72459fd

Revert Python 3.9 -> 3.10 requirement changes; Update uv.lock files

6fe93a1

junhaoliao changed the title ~~chore(clp-package): Migrate fully to uv + hatchling (resolves #1396); Require Python >= 3.10 (resolves #1380); Upgrade dependencies.~~ chore(clp-package): Migrate fully to uv + hatchling (resolves #1396); Upgrade dependencies. Oct 12, 2025

coderabbitai Bot reviewed Oct 12, 2025

View reviewed changes

chore(dependencies): Bump boto3 to 1.40.50 and sqlalchemy to 2.0.44 i…

cf28cae

…n pyproject.toml and uv.lock.

junhaoliao mentioned this pull request Oct 15, 2025

feat(clp-package): Add Python profiling decorator for internal query performance measurement. #1423

Open

3 tasks

LinZhihao-723 requested changes Oct 16, 2025

View reviewed changes

Bill-hbrhbr requested changes Oct 17, 2025

View reviewed changes

junhaoliao and others added 2 commits October 17, 2025 21:33

reorder deps - Apply suggestions from code review

3ef632f

Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>

revert changes in .python-version

464ad89

junhaoliao requested review from Bill-hbrhbr and LinZhihao-723 October 18, 2025 02:08

coderabbitai Bot mentioned this pull request Oct 18, 2025

Update User Quick Start Guide to reflect Python 3.10 as the officially supported version #1438

Closed

junhaoliao marked this pull request as draft October 18, 2025 13:11

coderabbitai Bot mentioned this pull request Oct 18, 2025

Update User Quick Start Guide to reflect Python 3.10 as the officially supported version #1439

Closed

junhaoliao added 2 commits October 18, 2025 09:16

Merge branch 'main' into migrate-uv

19453a8

# Conflicts: # components/clp-mcp-server/pyproject.toml # components/clp-mcp-server/uv.lock

chore: Update dependencies for Pydantic and Brotli to latest versions…

8034e1e

…; update locks.

junhaoliao marked this pull request as ready for review October 18, 2025 13:21

LinZhihao-723 approved these changes Oct 19, 2025

View reviewed changes

junhaoliao changed the title ~~chore(clp-package): Migrate fully to uv + hatchling (resolves #1396); Upgrade dependencies.~~ build(clp-package): Adopt uv + hatchling as the build and packaging backend for Python components (resolves #1396); Upgrade dependencies for Python components. Oct 19, 2025

junhaoliao merged commit 7e1c717 into y-scope:main Oct 19, 2025
31 checks passed

junhaoliao mentioned this pull request Oct 21, 2025

feat(clp-mcp-server): Add log viewer links to query results for displaying in LLM output. #1454

Merged

3 tasks

junhaoliao deleted the migrate-uv branch May 7, 2026 19:46

junhaoliao added a commit to junhaoliao/clp that referenced this pull request May 17, 2026

build(clp-package): Adopt uv + hatchling as the build and packaging b…

2b6f34b

…ackend for Python components (resolves y-scope#1396); Upgrade dependencies for Python components. (y-scope#1405)

Conversation

junhaoliao commented Oct 10, 2025 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Possibly related issues

Pre-merge checks and finishing touches

Uh oh!

LinZhihao-723 left a comment

Choose a reason for hiding this comment

Uh oh!

junhaoliao commented Oct 10, 2025

Uh oh!

LinZhihao-723 commented Oct 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junhaoliao commented Oct 12, 2025

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

LinZhihao-723 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bill-hbrhbr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junhaoliao commented Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 18, 2025

junhaoliao commented Oct 10, 2025 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 10, 2025 •

edited

Loading

coderabbitai Bot Oct 12, 2025 •

edited

Loading

coderabbitai Bot Oct 12, 2025 •

edited

Loading

coderabbitai Bot Oct 12, 2025 •

edited

Loading

Bill-hbrhbr left a comment •

edited

Loading

junhaoliao commented Oct 18, 2025 •

edited

Loading