feat(webui): Add support for compressing unstructured text logs using CLP-JSON. by junhaoliao · Pull Request #1861 · y-scope/clp

junhaoliao · 2026-01-11T22:30:29Z

Description

Add support for compressing unstructured text logs via the WebUI when using the CLP-JSON storage engine.

This feature adds an "Unstructured Logs" checkbox to the compression job form. When enabled, unstructured text logs are converted to JSON before compression, with each log event parsed into timestamp and message fields.

Changes:

Add unstructured boolean option to the compression job creation schema and API
Add "Unstructured Logs" checkbox with tooltip to the CLP-S form items
Pass the unstructured flag through to the compression job config

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

With the checkbox checked, compressed logs from the hive-24hr sample dataset. Errors are reported because the dataset contains empty files which can't be recognized as unstructured text logs files, which is unrelated to the PR.
Performed search in the webui with message:* and observed results were returned.

Summary by CodeRabbit

New Features
- Option to convert unstructured (non‑JSON) logs to JSON during compression via a new checkbox; when enabled, timestamp input is disabled and the conversion takes precedence over timestamp settings.
- Added contextual helper text, tooltip and documentation link explaining unstructured conversion and timestamp behavior.
Style
- Improved tooltip code styling for better readability.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1212888467152666
- https://app.asana.com/0/0/1212888451203393

… CLP-JSON.

…clarity

coderabbitai · 2026-01-11T22:30:40Z

Walkthrough

Adds a "Convert to JSON" (unstructured) checkbox to the compression UI, updates form types and submission payload to include unstructured, extends the shared compression schema, and propagates the flag on the server to set jobConfig.input.unstructured = true. Timestamp key input is disabled when unstructured is enabled.

Changes

Cohort / File(s)	Summary
UI Form & Styling `components/webui/client/src/pages/IngestPage/Compress/ClpSFormItems.tsx`, `components/webui/client/src/pages/IngestPage/Compress/index.module.css`	Adds "Unstructured Logs" checkbox (`valuePropName="checked"`), helper/tooltip text with docs link and note that timestamp key is ignored when enabled; uses `useWatch` to disable Timestamp Key input; adds `.tooltipCode` CSS class.
Form Type & Submission Logic `components/webui/client/src/pages/IngestPage/Compress/index.tsx`	Adds optional `unstructured?: boolean` to `FormValues`; submission includes `unstructured: true` in payload for CLP_S when checked and omits/ignores `timestampKey` in that case.
Shared Schema `components/webui/common/src/schemas/compression.ts`	Extends `CompressionJobCreationSchema` with optional boolean `unstructured`, updating the derived `CompressionJobCreation` shape.
Server Route Handling `components/webui/server/src/routes/api/compress/index.ts`	Reads `unstructured` from request body and sets `jobConfig.input.unstructured = true` when present; retains existing dataset/timestamp handling otherwise.

Sequence Diagram(s)

sequenceDiagram
  participant User as User
  participant Browser as Browser (UI)
  participant API as Server API
  participant Worker as Compression Worker

  User->>Browser: Check "Convert to JSON" and submit form
  Browser->>API: POST /api/compress { ..., unstructured: true }
  API->>API: validate body against CompressionJobCreationSchema
  API->>Worker: enqueue/create job with jobConfig.input.unstructured = true
  Worker->>Worker: run compression job (perform unstructured -> JSON conversion)
  Worker-->>API: job status/result
  API-->>Browser: respond with job creation result

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding WebUI support for compressing unstructured text logs using CLP-JSON.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

# Conflicts: # components/webui/server/src/plugins/app/CompressionJobDbManager/typings.ts

hoophalab

LGTM. Not sure if it makes sense to disable the timestamp input box when "convert to json" is clicked.

Validation:
Unstructured logs are correctly compressed using WebUI.

junhaoliao · 2026-01-19T19:24:55Z

Not sure if it makes sense to disable the timestamp input box when "convert to json" is clicked.

Certainly. Making the changes now

…s enabled and adjust tooltip logic

coderabbitai

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@components/webui/client/src/pages/IngestPage/Compress/ClpSFormItems.tsx`:
- Around line 32-62: The JSX helper UNSTRUCTURED_HELPER_TEXT is using a
defensive fallback (styles["tooltipCode"] || "") for the Typography.Text
className; remove the redundant "|| ''" so the className is just
styles.tooltipCode (or styles["tooltipCode"]) and let React handle
undefined—update both occurrences inside UNSTRUCTURED_HELPER_TEXT (the
Typography.Text nodes) to remove the fallback and avoid forcing an empty string.

coderabbitai · 2026-01-19T20:12:45Z

+const UNSTRUCTURED_HELPER_TEXT = (
+    <>
+        Enable this for non-JSON logs. Each log event will be parsed and converted to JSON with
+        {" "}
+        <Typography.Text
+            className={styles["tooltipCode"] || ""}
+            code={true}
+        >
+            timestamp
+        </Typography.Text>
+        {" and "}
+        <Typography.Text
+            className={styles["tooltipCode"] || ""}
+            code={true}
+        >
+            message
+        </Typography.Text>
+        {" "}
+        fields. See the
+        {" "}
+        <Typography.Link
+            href={"https://docs.yscope.com/clp/main/user-docs/quick-start/clp-json.html#compressing-unstructured-text-logs"}
+            rel={"noopener"}
+            target={"_blank"}
+        >
+            documentation
+        </Typography.Link>
+        {" "}
+        for more details.
+    </>
+);


🧹 Nitpick | 🔵 Trivial

Consider simplifying the className fallback.

The || "" fallback on lines 37 and 44 is defensive but may be unnecessary if the CSS module is properly configured. If the class doesn't exist, styles["tooltipCode"] would return undefined, which React handles gracefully by not applying any class.

Optional simplification

<Typography.Text - className={styles["tooltipCode"] || ""} + className={styles["tooltipCode"]} code={true} >

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

const UNSTRUCTURED_HELPER_TEXT = (

<>

Enable this for non-JSON logs. Each log event will be parsed and converted to JSON with

{" "}

<Typography.Text

className={styles["tooltipCode"] || ""}

code={true}

>

timestamp

</Typography.Text>

{" and "}

<Typography.Text

className={styles["tooltipCode"] || ""}

code={true}

>

message

</Typography.Text>

{" "}

fields. See the

{" "}

<Typography.Link

href={"https://docs.yscope.com/clp/main/user-docs/quick-start/clp-json.html#compressing-unstructured-text-logs"}

rel={"noopener"}

target={"_blank"}

>

documentation

</Typography.Link>

{" "}

for more details.

</>

);

const UNSTRUCTURED_HELPER_TEXT = (

<>

Enable this for non-JSON logs. Each log event will be parsed and converted to JSON with

{" "}

<Typography.Text

className={styles["tooltipCode"]}

code={true}

>

timestamp

</Typography.Text>

{" and "}

<Typography.Text

className={styles["tooltipCode"]}

code={true}

>

message

</Typography.Text>

{" "}

fields. See the

{" "}

<Typography.Link

href={"https://docs.yscope.com/clp/main/user-docs/quick-start/clp-json.html#compressing-unstructured-text-logs"}

rel={"noopener"}

target={"_blank"}

>

documentation

</Typography.Link>

{" "}

for more details.

</>

);

🤖 Prompt for AI Agents

In `@components/webui/client/src/pages/IngestPage/Compress/ClpSFormItems.tsx` around lines 32 - 62, The JSX helper UNSTRUCTURED_HELPER_TEXT is using a defensive fallback (styles["tooltipCode"] || "") for the Typography.Text className; remove the redundant "|| ''" so the className is just styles.tooltipCode (or styles["tooltipCode"]) and let React handle undefined—update both occurrences inside UNSTRUCTURED_HELPER_TEXT (the Typography.Text nodes) to remove the fallback and avoid forcing an empty string.

… CLP-JSON. (y-scope#1861)

junhaoliao added 3 commits December 18, 2025 14:06

feat(webui): Add support for compressing unstructured text logs using…

072ce3a

… CLP-JSON.

Merge remote-tracking branch 'origin/main' into webui-unstructured

04adbe9

feat(webui): Update unstructured logs tooltip and styling for better …

cca7b21

…clarity

junhaoliao requested a review from a team as a code owner January 11, 2026 22:30

junhaoliao requested a review from davemarco January 11, 2026 22:30

junhaoliao and others added 4 commits January 13, 2026 13:49

Merge branch 'main' into webui-unstructured

82151a9

# Conflicts: # components/webui/server/src/plugins/app/CompressionJobDbManager/typings.ts

Merge branch 'main' into webui-unstructured

4757fb3

Merge branch 'main' into webui-unstructured

08ab70e

Merge branch 'main' into webui-unstructured

519d53e

kirkrodrigues requested review from hoophalab and removed request for davemarco January 16, 2026 18:47

Merge branch 'main' into webui-unstructured

9495fc8

junhaoliao added this to the Backlog milestone Jan 19, 2026

hoophalab previously approved these changes Jan 19, 2026

View reviewed changes

junhaoliao and others added 2 commits January 19, 2026 14:34

Merge branch 'main' into webui-unstructured

ace001a

feat(webui): Disable Timestamp Key field when Unstructured Logs i…

34a09fd

…s enabled and adjust tooltip logic

junhaoliao dismissed hoophalab’s stale review via 34a09fd January 19, 2026 20:09

junhaoliao requested a review from hoophalab January 19, 2026 20:09

coderabbitai Bot reviewed Jan 19, 2026

View reviewed changes

hoophalab approved these changes Jan 20, 2026

View reviewed changes

junhaoliao merged commit 97b430f into y-scope:main Jan 21, 2026
21 of 22 checks passed

junhaoliao modified the milestones: Backlog, February 2026, January 2026 Jan 21, 2026

junhaoliao deleted the webui-unstructured branch May 7, 2026 19:46

junhaoliao added a commit to junhaoliao/clp that referenced this pull request May 17, 2026

feat(webui): Add support for compressing unstructured text logs using…

155e7d8

… CLP-JSON. (y-scope#1861)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(webui): Add support for compressing unstructured text logs using CLP-JSON.#1861

feat(webui): Add support for compressing unstructured text logs using CLP-JSON.#1861
junhaoliao merged 10 commits into
y-scope:mainfrom
junhaoliao:webui-unstructured

junhaoliao commented Jan 11, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jan 11, 2026 •

edited

Loading

Uh oh!

hoophalab left a comment

Uh oh!

junhaoliao commented Jan 19, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jan 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

junhaoliao commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Uh oh!

hoophalab left a comment

Choose a reason for hiding this comment

Uh oh!

junhaoliao commented Jan 19, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

junhaoliao commented Jan 11, 2026 •

edited

Loading

coderabbitai Bot commented Jan 11, 2026 •

edited

Loading