feat(job-orchestration): Add dataset fields to input configs of compression and search jobs.#839
Conversation
WalkthroughA new constant Changes
Sequence Diagram(s)Not applicable. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
dataset config option to input configs of compression and search jobs.dataset fields to input configs of compression and search jobs.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
components/job-orchestration/job_orchestration/scheduler/job_config.py (1)
25-25: Consider adding docstrings to explain the purpose of the dataset fieldWhile the implementation is correct and meets the requirements, adding docstrings would improve code clarity for future developers who may not be familiar with the dataset feature plan.
class FsInputConfig(BaseModel): type: typing.Literal[InputType.FS.value] = InputType.FS.value - dataset: str = "default" + dataset: str = "default" # Identifier for the dataset this job operates on paths_to_compress: typing.List[str] path_prefix_to_remove: str = None timestamp_key: typing.Optional[str] = NoneSimilar docstrings could be added to the other classes as well.
Also applies to: 33-33, 81-81
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
components/job-orchestration/job_orchestration/scheduler/job_config.py(2 hunks)
🔇 Additional comments (3)
components/job-orchestration/job_orchestration/scheduler/job_config.py (3)
25-25: Implementation looks good!Adding the
datasetfield with a default value of "default" toFsInputConfigensures backward compatibility while introducing the new dataset feature, which aligns with the PR objectives.
33-33: Implementation looks good!Adding the
datasetfield with a default value of "default" toS3InputConfigmaintains consistency with theFsInputConfigimplementation and ensures backward compatibility.
81-81: Implementation looks good!The addition of the
datasetfield toSearchJobConfigcompletes the requirements from the PR objectives and is consistent with the implementation in the input config classes.
|
Caution No docstrings were generated. |
…pression and search jobs. (y-scope#839)
…pression and search jobs. (y-scope#839)
Description
This PR succeeds #819 and covers the following parts of the dataset feature implementation plan:
datasetfields for compression jobs' input configs and search jobs' config.These options will only be used by
clp-supon completion of thisdatasetfeature implementation. When not specified, these options default todefault.Checklist
breaking change.
Validation performed
datasetconfigs neither take effect nor break anything.Summary by CodeRabbit