Skip to content

docs(api-server): Add guide for how to use the API server.#1684

Merged
hoophalab merged 15 commits into
y-scope:mainfrom
hoophalab:openapidocs
Dec 1, 2025
Merged

docs(api-server): Add guide for how to use the API server.#1684
hoophalab merged 15 commits into
y-scope:mainfrom
hoophalab:openapidocs

Conversation

@hoophalab

@hoophalab hoophalab commented Nov 27, 2025

Copy link
Copy Markdown
Contributor

Description

  1. Add a new sphinx page describing the api server
  2. Add a new docs task that generates openapi.json

Cannot disable try it out feature by the url parameters. The supportedSubmitMethods url has no effect on their demo. Also checked redoc ui. It doesn't look as good as swagger...

Checklist

  • The PR satisfies the contribution guidelines.
  • This is a breaking change and that has been indicated in the PR title, OR this isn't a
    breaking change.
  • Necessary docs have been updated, OR no docs need to be updated.

Validation performed

  1. task docs:serve
  2. The user guide page looks correct
  3. http://localhost:8080/_static/openapi.json is the openapi spec. Cannot verify whether https://petstore.swagger.io/?url=https://docs.yscope.com/clp/main/_static/openapi.json shows correctly yet.

Summary by CodeRabbit

  • New Features

    • Added OpenAPI spec generation for the API server and an automated build step to produce the JSON file.
    • Deployment can toggle the API server via an environment-controlled replica setting.
  • Behaviour

    • API server is automatically disabled for installations using the CLP storage engine; a startup warning is emitted when disabled.
  • Documentation

    • New user guide explaining API server usage, OpenAPI/Swagger access, examples and streaming query workflows; navigation updated.

✏️ Tip: You can customize this high-level summary in your review settings.

@hoophalab hoophalab requested a review from a team as a code owner November 27, 2025 21:39
@coderabbitai

coderabbitai Bot commented Nov 27, 2025

Copy link
Copy Markdown
Contributor

Walkthrough

Adds OpenAPI spec generation (new Rust binary), exposes ApiDoc routes for OpenAPI, integrates generation into docs build, documents API server usage, and adds runtime/config guards to disable the API server for CLP storage engine deployments.

Changes

Cohort / File(s) Summary
OpenAPI generation binary & routes
components/api-server/src/bin/openapi_codegen.rs, components/api-server/src/routes.rs
New Rust binary openapi_codegen that obtains ApiDoc::openapi(), serializes to JSON and writes to a file; ApiDoc re-exported publicly and OpenAPI path metadata (health, query, query_results) added to routes.
Documentation and navigation
docs/src/user-docs/guides-using-the-api-server.md, docs/src/user-docs/guides-overview.md, docs/src/user-docs/index.md
Adds a new user guide for using the API server, inserts guide card in overview, and adds the guide to the Guides toctree.
Docs build task
taskfiles/docs.yaml
Adds api-server-openapi-json task that runs the openapi_codegen binary to emit _static/api-server-openapi.json, integrated into the docs build sequence.
API server enablement & deployment controls
components/clp-package-utils/clp_package_utils/controller.py, components/clp-package-utils/clp_package_utils/scripts/start_clp.py, tools/deployment/package/docker-compose-all.yaml
Sets CLP_API_SERVER_ENABLED=0 when storage engine is CLP; emits runtime warnings in start scripts when storage engine is CLP; Docker Compose api-server service gets deploy.replicas: "${CLP_API_SERVER_ENABLED:-1}".

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • Review openapi_codegen for correct invocation of ApiDoc::openapi(), JSON serialization options, and file-write error handling.
  • Verify pub use api_doc::* and added OpenAPI path attributes correctly expose paths in generated spec.
  • Confirm taskfiles/docs.yaml task variables/paths and integration point in the docs pipeline.
  • Check the CLP storage-engine guard placement and log messages in start_clp.py and controller.py.
  • Validate the Docker Compose deploy.replicas expression behaves as intended when CLP_API_SERVER_ENABLED is 0/1.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: adding documentation and a guide for using the API server.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ba73be9 and f6d3ac3.

📒 Files selected for processing (1)
  • components/clp-package-utils/clp_package_utils/scripts/start_clp.py (2 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1136
File: components/webui/client/src/pages/IngestPage/Details/sql.ts:1-1
Timestamp: 2025-07-29T21:00:07.757Z
Learning: User junhaoliao requested creating a GitHub issue to track server-side SQL query error handling improvements, specifically to prevent uncaught query errors from causing 500 errors to reach the client.
Learnt from: hoophalab
Repo: y-scope/clp PR: 1535
File: components/clp-rust-utils/src/clp_config/package/config.rs:47-61
Timestamp: 2025-11-03T16:17:40.223Z
Learning: In the y-scope/clp repository, the `ApiServer` struct in `components/clp-rust-utils/src/clp_config/package/config.rs` is a Rust-native configuration type and does not mirror any Python code, unlike other structs in the same file (Config, Database, ResultsCache, Package) which are mirrors of Python definitions.
📚 Learning: 2025-08-08T06:59:42.436Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1152
File: components/clp-package-utils/clp_package_utils/scripts/start_clp.py:613-613
Timestamp: 2025-08-08T06:59:42.436Z
Learning: In components/clp-package-utils/clp_package_utils/scripts/start_clp.py, generic_start_scheduler sets CLP_LOGGING_LEVEL using clp_config.query_scheduler.logging_level for both schedulers; compression scheduler should use its own logging level. Tracking via an issue created from PR #1152 discussion.

Applied to files:

  • components/clp-package-utils/clp_package_utils/scripts/start_clp.py
📚 Learning: 2025-10-07T07:54:32.427Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1178
File: components/clp-py-utils/clp_py_utils/clp_config.py:47-47
Timestamp: 2025-10-07T07:54:32.427Z
Learning: In components/clp-py-utils/clp_py_utils/clp_config.py, the CONTAINER_AWS_CONFIG_DIRECTORY constant is intentionally set to pathlib.Path("/") / ".aws" (i.e., `/.aws`) rather than a user-specific home directory. This hardcoded path is part of the container orchestration design.

Applied to files:

  • components/clp-package-utils/clp_package_utils/scripts/start_clp.py
🧬 Code graph analysis (1)
components/clp-package-utils/clp_package_utils/scripts/start_clp.py (1)
components/clp-py-utils/clp_py_utils/clp_config.py (1)
  • StorageEngine (100-102)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
  • GitHub Check: centos-stream-9-static-linked-bins
  • GitHub Check: centos-stream-9-dynamic-linked-bins
  • GitHub Check: manylinux_2_28-x86_64-dynamic-linked-bins
  • GitHub Check: rust-checks
  • GitHub Check: lint-check (ubuntu-24.04)
  • GitHub Check: build (ubuntu-24.04)
  • GitHub Check: build (macos-15)
  • GitHub Check: build-macos (macos-15, true)
  • GitHub Check: build-macos (macos-15, false)
🔇 Additional comments (2)
components/clp-package-utils/clp_package_utils/scripts/start_clp.py (2)

6-6: LGTM!

The StorageEngine import is required for the new storage engine check added below.


73-76: LGTM!

The warning is appropriately placed after all configuration validations pass, providing users with a clear informational message that the API Server is not available when using the CLP storage engine. This aligns with the guard in controller.py that sets CLP_API_SERVER_ENABLED to disable the API server for CLP deployments. The message format with backticks is consistent with the project's logging style.

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Ruff (0.14.6)
components/clp-package-utils/clp_package_utils/scripts/start_clp.py

�[1;31mruff failed�[0m
�[1mCause:�[0m Failed to load extended configuration /tools/yscope-dev-utils/exports/lint-configs/python/ruff.toml (/components/clp-package-utils/pyproject.toml extends /tools/yscope-dev-utils/exports/lint-configs/python/ruff.toml)
�[1mCause:�[0m Failed to read /tools/yscope-dev-utils/exports/lint-configs/python/ruff.toml
�[1mCause:�[0m No such file or directory (os error 2)


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 28fd727 and 57d49dc.

📒 Files selected for processing (6)
  • components/api-server/src/bin/openapi_codegen.rs (1 hunks)
  • components/api-server/src/routes.rs (1 hunks)
  • docs/src/user-docs/guides-api-server.md (1 hunks)
  • docs/src/user-docs/guides-overview.md (1 hunks)
  • docs/src/user-docs/index.md (1 hunks)
  • taskfiles/docs.yaml (2 hunks)
🧰 Additional context used
🧠 Learnings (3)
📚 Learning: 2025-11-03T16:17:40.223Z
Learnt from: hoophalab
Repo: y-scope/clp PR: 1535
File: components/clp-rust-utils/src/clp_config/package/config.rs:47-61
Timestamp: 2025-11-03T16:17:40.223Z
Learning: In the y-scope/clp repository, the `ApiServer` struct in `components/clp-rust-utils/src/clp_config/package/config.rs` is a Rust-native configuration type and does not mirror any Python code, unlike other structs in the same file (Config, Database, ResultsCache, Package) which are mirrors of Python definitions.

Applied to files:

  • docs/src/user-docs/guides-api-server.md
  • components/api-server/src/routes.rs
📚 Learning: 2025-08-25T16:27:50.549Z
Learnt from: davemarco
Repo: y-scope/clp PR: 1198
File: components/webui/server/src/plugins/app/Presto.ts:38-43
Timestamp: 2025-08-25T16:27:50.549Z
Learning: In the CLP webui Presto configuration, host and port are set via package settings (configurable), while user, catalog, and schema are set via environment variables (environment-specific). This mixed approach is intentional - settings are typically set by package and some values don't need to be package-configurable.

Applied to files:

  • docs/src/user-docs/guides-overview.md
📚 Learning: 2025-06-18T20:39:05.899Z
Learnt from: quinntaylormitchell
Repo: y-scope/clp PR: 968
File: docs/src/user-guide/quick-start/overview.md:73-109
Timestamp: 2025-06-18T20:39:05.899Z
Learning: The CLP project team prefers to use video content to demonstrate detailed procedural steps (like tarball extraction) rather than including every step in the written documentation, keeping the docs focused on conceptual guidance.

Applied to files:

  • docs/src/user-docs/guides-overview.md
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (17)
  • GitHub Check: musllinux_1_2-x86_64-dynamic-linked-bins
  • GitHub Check: musllinux_1_2-x86_64-static-linked-bins
  • GitHub Check: package-image
  • GitHub Check: ubuntu-jammy-dynamic-linked-bins
  • GitHub Check: ubuntu-jammy-static-linked-bins
  • GitHub Check: ubuntu-jammy-lint
  • GitHub Check: manylinux_2_28-x86_64-static-linked-bins
  • GitHub Check: centos-stream-9-static-linked-bins
  • GitHub Check: manylinux_2_28-x86_64-dynamic-linked-bins
  • GitHub Check: centos-stream-9-dynamic-linked-bins
  • GitHub Check: build (ubuntu-24.04)
  • GitHub Check: build (macos-15)
  • GitHub Check: build-macos (macos-15, true)
  • GitHub Check: rust-checks
  • GitHub Check: lint-check (macos-15)
  • GitHub Check: build-macos (macos-15, false)
  • GitHub Check: lint-check (ubuntu-24.04)
🔇 Additional comments (2)
docs/src/user-docs/guides-overview.md (1)

36-41: New API Server guide tile is wired correctly.

Link target, title, and description are consistent with the new guides-api-server page and existing card patterns; no issues from a docs IA perspective.

docs/src/user-docs/index.md (1)

62-69: Guide index entry for API Server looks consistent.

The new guides-api-server toctree entry matches the page name and fits logically within the Guides section.

Comment on lines +1 to +18
use std::io::Write;

use anyhow::Result;
use clap::Parser;
use utoipa::OpenApi;

#[derive(Parser)]
#[command(version, about = "Generate openapi.json")]
struct Args {
path: String,
}

fn main() -> Result<()> {
let mut file = std::fs::File::create(Args::parse().path)?;
let api = api_server::routes::ApiDoc::openapi();
write!(file, "{}", api.to_json()?)?;
Ok(())
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

OpenAPI codegen binary is straightforward; you can skip the intermediate String (optional).

The CLI cleanly writes the OpenAPI document to the requested path and integrates well with the docs task. If you ever care about avoiding the extra allocation, you could write JSON directly to the file:

-use std::io::Write;
-
 use anyhow::Result;
@@
 fn main() -> Result<()> {
-    let mut file = std::fs::File::create(Args::parse().path)?;
-    let api = api_server::routes::ApiDoc::openapi();
-    write!(file, "{}", api.to_json()?)?;
+    let file = std::fs::File::create(Args::parse().path)?;
+    let api = api_server::routes::ApiDoc::openapi();
+    serde_json::to_writer(file, &api)?;
     Ok(())
 }

Not required, but slightly simpler and more memory-efficient.

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents
In components/api-server/src/bin/openapi_codegen.rs lines 1-18, avoid creating
an intermediate String for the OpenAPI JSON; instead serialize directly into the
file by replacing the write!(file, "{}", api.to_json()?) call with a direct
serializer like serde_json::to_writer (or serde_json::to_writer_pretty) that
takes &mut file and &api, propagating errors as before.

Comment thread components/api-server/src/routes.rs Outdated
Comment on lines 53 to 69
// Referencing handlers with `super::health` adds an undesired "super" tag to the
// `openapi.json`. Import them here to avoid that.
use super::{__path_health, __path_query, __path_query_results};

#[derive(utoipa::OpenApi)]
#[openapi(info(
title = "API Server",
description = "API Server for CLP",
contact(name = "YScope")
))]
#[openapi(
info(
title = "API Server",
description = "API Server for CLP",
contact(name = "YScope")
),
paths(health, query, query_results)
)]
pub struct ApiDoc;
}
use api_doc::ApiDoc;
pub use api_doc::*;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick | 🔵 Trivial

Tighten the api_doc re-export and align the comment with what’s imported.

The OpenAPI setup looks good: explicitly listing paths(health, query, query_results) and importing the generated __path_* helpers is a clean way to avoid the extra super tag in openapi.json, and it enables the new openapi_codegen binary via ApiDoc::openapi().

Two small polish points:

  1. The comment speaks about referencing handlers via super::health, but the import actually brings in __path_health, __path_query, and __path_query_results. Consider updating the wording so it’s clear you’re importing the generated __path_* functions, not the handlers themselves.

  2. To keep the public surface narrow, it’s safer to re-export just ApiDoc instead of everything from the module:

-mod api_doc {
+mod api_doc {
@@
-    pub struct ApiDoc;
-}
-pub use api_doc::*;
+    pub struct ApiDoc;
+}
+
+pub use api_doc::ApiDoc;

This avoids unintentionally exposing additional items if api_doc grows over time.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// Referencing handlers with `super::health` adds an undesired "super" tag to the
// `openapi.json`. Import them here to avoid that.
use super::{__path_health, __path_query, __path_query_results};
#[derive(utoipa::OpenApi)]
#[openapi(info(
title = "API Server",
description = "API Server for CLP",
contact(name = "YScope")
))]
#[openapi(
info(
title = "API Server",
description = "API Server for CLP",
contact(name = "YScope")
),
paths(health, query, query_results)
)]
pub struct ApiDoc;
}
use api_doc::ApiDoc;
pub use api_doc::*;
// Referencing handlers with `super::health` adds an undesired "super" tag to the
// `openapi.json`. Import them here to avoid that.
use super::{__path_health, __path_query, __path_query_results};
#[derive(utoipa::OpenApi)]
#[openapi(
info(
title = "API Server",
description = "API Server for CLP",
contact(name = "YScope")
),
paths(health, query, query_results)
)]
pub struct ApiDoc;
}
pub use api_doc::ApiDoc;
🤖 Prompt for AI Agents
In components/api-server/src/routes.rs around lines 53 to 69, the comment and
re-export need polishing: update the comment to say you’re importing the
generated __path_* helpers (e.g., __path_health, __path_query,
__path_query_results) rather than the handler functions referenced with
super::health, and change the public re-export to only expose ApiDoc (e.g., pub
use api_doc::ApiDoc;) so the module surface remains minimal and doesn’t
unintentionally export other items.

Comment thread docs/src/user-docs/guides-api-server.md Outdated
Comment thread taskfiles/docs.yaml Outdated
--keep-going \
--builder html \
src "{{.OUTPUT_DIR}}"
- task: "openapi-json"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Ensure openapi-json is rebuilt when API changes, and consider creating the output directory.

Wiring openapi-json into the site pipeline is the right place, but two details are worth tightening:

  1. Staleness risk:
    openapi-json’s sources only include the root taskfile.yaml and this Taskfile, so Task may skip regenerating openapi.json even after changes to the API routes or the openapi_codegen binary. To keep the published spec in sync with the server, consider adding the relevant Rust sources, for example:

    sources:
      - "{{.ROOT_DIR}}/taskfile.yaml"
      - "{{.TASKFILE}}"
      - "{{.ROOT_DIR}}/components/api-server/src/**/*.rs"

    (Or a narrower pattern if you prefer.)

  2. Running task standalone:
    If someone runs task docs:openapi-json directly, {{.G_DOCS_BUILD_DIR}}/_static may not exist yet, causing file creation to fail. You could defensively ensure the parent directory exists:

    cmd: |-
      . "{{.G_RUST_TOOLCHAIN_ENV_FILE}}"
      mkdir -p "$(dirname "{{.OUTPUT_FILE}}")"
      cargo run --bin openapi_codegen -- "{{.OUTPUT_FILE}}"

These changes would make the docs build more robust and keep the OpenAPI spec reliably up to date.

Also applies to: 116-127

🤖 Prompt for AI Agents
In taskfiles/docs.yaml around lines 56 (and similarly 116–127), the openapi-json
task is wired into the site pipeline but its sources are incomplete and the
output directory may not exist when run standalone; update the task's sources to
include the Taskfile, root taskfile, and the API rust sources (e.g.
components/api-server/src/**/*.rs or a narrower pattern) so Task will rerun when
API code or the codegen changes, and modify the task command to ensure the
parent directory of the OUTPUT_FILE is created first (e.g. run mkdir -p on the
dirname of OUTPUT_FILE) before invoking the openapi_codegen binary.

@LinZhihao-723 LinZhihao-723 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong: I realize API server is probably not specific to clp-json. If this is true, we need to update the API document as my comments in the previous PR for doc rewriting were clp-json specific.

Comment thread components/api-server/src/routes.rs Outdated
Comment thread taskfiles/docs.yaml
Comment thread docs/src/user-docs/guides-api-server.md Outdated

The API is defined using the OpenAPI Specification. You can check the API via [Swagger UI][swagger].

[swagger]: https://petstore.swagger.io/?url=https://docs.yscope.com/clp/main/_static/openapi.json

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope there can be a better way to manage this... This hardcodes the path to main, while in a release this should be the version number.
I'm not too familiar with the doc structure. Will leave this to @kirkrodrigues

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a way to make it a little less error-prone. I'll do it in a subsequent PR while modifying a few related things.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewrite this doc to:

  • Add a section to explain how to start the server
  • Using the existing APIs to make a short example (please help me check whether this example looks good to you). I think enumerating what the API lets users do in this doc won't scale as we add more APIs. Adding this example section might be sufficient. We may or may not need to add more example sections in the future release.

Notice that when I'm trying this example, I realize if I query results to a non-existing query like curl -N http://localhost:3001/query_results/1234567 it won't fail and instead exit on success without error message. Is there a way we can exit on failures if the query is invalid (not necessarily in this release)? (or maybe check whether this is my setup issue...

guides-api-server.md

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for polishing the docs! 🙏

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed two commits to:

  • Add a note to notify users that only clp-json is supported at this moment.
  • Move the API reference section before the example section.

@hoophalab

hoophalab commented Nov 28, 2025

Copy link
Copy Markdown
Contributor Author

Notice that when I'm trying this example, I realize if I query results to a non-existing query like curl -N http://localhost:3001/query_results/1234567 it won't fail and instead exit on success without error message. Is there a way we can exit on failures if the query is invalid (not necessarily in this release)? (or maybe check whether this is my setup issue...

I guess we return a 500 internal server error?
curl -N -w "%{http_code}" http://localhost:3001/query_results/1234567 shows 500
Maybe we should also output "500 Internal Server Error" in http body when we return the status code

coderabbitai[bot]

This comment was marked as resolved.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (2)
docs/src/user-docs/guides-api-server.md (2)

5-9: Document the storage-engine requirement for the API server.

The API server is only supported when package.storage_engine is set to clp-s. This constraint is enforced by runtime validation but not mentioned in the documentation. Add a brief note after line 8 stating this requirement and, if helpful, mention where to configure package.storage_engine in etc/clp-config.yaml.


18-55: Fix markdown fence spacing to satisfy markdownlint (MD031).

Fenced code blocks need blank lines before and after them. Add blank lines between list text and each opening fence, and after each closing fence.

Apply these changes:

 1. Submit a search query:
+
    ```shell
    curl -X POST http://localhost:3001/query \
     -H "Content-Type: application/json" \
     -d '{
       "query_string": "*log*",
       "dataset": "default",
       "ignore_case": false,
       "max_num_results": 100,
       "write_to_file": false
     }'
    ```
+
    On success, the server responds with:
+
     ```json
     {
       "query_results_uri": "/query_results/100"
     }
     ```
+
 2. Retrieve search results:
    Use the returned `query_results_uri` to receive search results as an SSE stream:
+
    ```bash
    curl -N http://localhost:3001/query_results/100
    ```
+
    Example streamed output:
+
    ```text
    data: {"timestamp": 1633036800, "message": "Example log message"}

    data: {"timestamp": 1633036810, "message": "Another matched log line"}

    data: {"timestamp": 1633036820, "message": "No logs found" }
    ```
📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d275d62 and 9f3acca.

📒 Files selected for processing (4)
  • components/clp-package-utils/clp_package_utils/controller.py (2 hunks)
  • components/clp-py-utils/clp_py_utils/clp_config.py (1 hunks)
  • docs/src/user-docs/guides-api-server.md (1 hunks)
  • tools/deployment/package/docker-compose-all.yaml (1 hunks)
🧰 Additional context used
🧠 Learnings (7)
📓 Common learnings
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1136
File: components/webui/client/src/pages/IngestPage/Details/sql.ts:1-1
Timestamp: 2025-07-29T21:00:07.757Z
Learning: User junhaoliao requested creating a GitHub issue to track server-side SQL query error handling improvements, specifically to prevent uncaught query errors from causing 500 errors to reach the client.
📚 Learning: 2025-11-10T05:19:56.600Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1575
File: components/clp-py-utils/clp_py_utils/clp_config.py:602-607
Timestamp: 2025-11-10T05:19:56.600Z
Learning: In the y-scope/clp repository, the `ApiServer` class in `components/clp-py-utils/clp_py_utils/clp_config.py` does not need a `transform_for_container()` method because no other containerized service depends on the API server - it's only accessed from the host, so no docker-network communication is expected.

Applied to files:

  • components/clp-package-utils/clp_package_utils/controller.py
  • components/clp-py-utils/clp_py_utils/clp_config.py
📚 Learning: 2025-11-03T16:17:40.223Z
Learnt from: hoophalab
Repo: y-scope/clp PR: 1535
File: components/clp-rust-utils/src/clp_config/package/config.rs:47-61
Timestamp: 2025-11-03T16:17:40.223Z
Learning: In the y-scope/clp repository, the `ApiServer` struct in `components/clp-rust-utils/src/clp_config/package/config.rs` is a Rust-native configuration type and does not mirror any Python code, unlike other structs in the same file (Config, Database, ResultsCache, Package) which are mirrors of Python definitions.

Applied to files:

  • components/clp-package-utils/clp_package_utils/controller.py
  • docs/src/user-docs/guides-api-server.md
  • components/clp-py-utils/clp_py_utils/clp_config.py
📚 Learning: 2025-10-17T19:59:25.596Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1178
File: components/clp-package-utils/clp_package_utils/controller.py:315-315
Timestamp: 2025-10-17T19:59:25.596Z
Learning: In components/clp-package-utils/clp_package_utils/controller.py, worker log directories (compression_worker, query_worker, reducer) created via `mkdir()` do not need `_chown_paths_if_root()` calls because directories are created with the same owner as the script caller. This differs from infrastructure service directories (database, queue, Redis, results cache) which do require explicit ownership changes.

Applied to files:

  • components/clp-package-utils/clp_package_utils/controller.py
📚 Learning: 2025-10-07T07:54:32.427Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1178
File: components/clp-py-utils/clp_py_utils/clp_config.py:47-47
Timestamp: 2025-10-07T07:54:32.427Z
Learning: In components/clp-py-utils/clp_py_utils/clp_config.py, the CONTAINER_AWS_CONFIG_DIRECTORY constant is intentionally set to pathlib.Path("/") / ".aws" (i.e., `/.aws`) rather than a user-specific home directory. This hardcoded path is part of the container orchestration design.

Applied to files:

  • components/clp-package-utils/clp_package_utils/controller.py
📚 Learning: 2024-11-21T15:51:33.203Z
Learnt from: junhaoliao
Repo: y-scope/clp PR: 596
File: components/log-viewer-webui/client/src/api/query.js:16-23
Timestamp: 2024-11-21T15:51:33.203Z
Learning: In `components/log-viewer-webui/client/src/api/query.js`, the `ExtractJsonResp` type definition is accurate as-is and does not require modification. When suggesting changes to type definitions, ensure they align with the server-side definitions, referencing the source code if necessary.

Applied to files:

  • docs/src/user-docs/guides-api-server.md
📚 Learning: 2025-08-17T16:19:24.478Z
Learnt from: Bill-hbrhbr
Repo: y-scope/clp PR: 1100
File: integration-tests/README.md:5-6
Timestamp: 2025-08-17T16:19:24.478Z
Learning: In the CLP project, README.md files can have multiple H1 headings when it makes sense structurally (e.g., "Contributing" as a top-level section alongside the main title). The MD025 markdown linting rule should be ignored in such cases where multiple H1s are intentional.

Applied to files:

  • docs/src/user-docs/guides-api-server.md
🧬 Code graph analysis (1)
components/clp-package-utils/clp_package_utils/controller.py (1)
components/clp-py-utils/clp_py_utils/clp_config.py (2)
  • ApiServer (608-612)
  • StorageEngine (97-99)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (17)
  • GitHub Check: ubuntu-jammy-dynamic-linked-bins
  • GitHub Check: package-image
  • GitHub Check: ubuntu-jammy-static-linked-bins
  • GitHub Check: ubuntu-jammy-lint
  • GitHub Check: musllinux_1_2-x86_64-static-linked-bins
  • GitHub Check: musllinux_1_2-x86_64-dynamic-linked-bins
  • GitHub Check: manylinux_2_28-x86_64-static-linked-bins
  • GitHub Check: manylinux_2_28-x86_64-dynamic-linked-bins
  • GitHub Check: centos-stream-9-dynamic-linked-bins
  • GitHub Check: centos-stream-9-static-linked-bins
  • GitHub Check: build-macos (macos-15, true)
  • GitHub Check: build-macos (macos-15, false)
  • GitHub Check: build (ubuntu-24.04)
  • GitHub Check: build (macos-15)
  • GitHub Check: rust-checks
  • GitHub Check: lint-check (macos-15)
  • GitHub Check: lint-check (ubuntu-24.04)
🔇 Additional comments (4)
docs/src/user-docs/guides-api-server.md (1)

64-64: Consider using a version-aware path for the OpenAPI spec URL.

The swagger-ui link hardcodes main in the docs URL. In versioned releases, this should point to the specific version number instead. This may require updates to the documentation build system to inject version paths dynamically.

tools/deployment/package/docker-compose-all.yaml (1)

507-509: LGTM!

The deploy configuration correctly enables conditional deployment of the API server using the CLP_API_SERVER_ENABLED environment variable, and the comment accurately describes the behaviour.

components/clp-package-utils/clp_package_utils/controller.py (1)

534-536: LGTM!

The guard correctly disables the API server for CLP storage engine deployments, providing defense-in-depth alongside the validation in clp_config.py.

components/clp-py-utils/clp_py_utils/clp_config.py (1)

813-820: Unable to verify automatically — repository access failed. Manual confirmation needed.

The repository could not be cloned to verify that validate_api_server() is actually invoked during startup. Without access to the codebase, I cannot confirm:

  • Whether validate_api_server() is called in initialization/startup code
  • The specific locations and context where this validation is triggered
  • Whether the validation occurs for all relevant code paths

Please confirm manually that validate_api_server() is invoked during configuration initialization before merging this change.

Comment thread components/clp-package-utils/clp_package_utils/controller.py Outdated
Comment thread docs/src/user-docs/guides-overview.md
Comment thread taskfiles/docs.yaml Outdated
Comment thread components/clp-py-utils/clp_py_utils/clp_config.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5401fa1 and ca32df2.

📒 Files selected for processing (1)
  • docs/src/user-docs/guides-using-the-api-server.md (1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: junhaoliao
Repo: y-scope/clp PR: 1136
File: components/webui/client/src/pages/IngestPage/Details/sql.ts:1-1
Timestamp: 2025-07-29T21:00:07.757Z
Learning: User junhaoliao requested creating a GitHub issue to track server-side SQL query error handling improvements, specifically to prevent uncaught query errors from causing 500 errors to reach the client.
Learnt from: hoophalab
Repo: y-scope/clp PR: 1535
File: components/clp-rust-utils/src/clp_config/package/config.rs:47-61
Timestamp: 2025-11-03T16:17:40.223Z
Learning: In the y-scope/clp repository, the `ApiServer` struct in `components/clp-rust-utils/src/clp_config/package/config.rs` is a Rust-native configuration type and does not mirror any Python code, unlike other structs in the same file (Config, Database, ResultsCache, Package) which are mirrors of Python definitions.
📚 Learning: 2025-11-03T16:17:40.223Z
Learnt from: hoophalab
Repo: y-scope/clp PR: 1535
File: components/clp-rust-utils/src/clp_config/package/config.rs:47-61
Timestamp: 2025-11-03T16:17:40.223Z
Learning: In the y-scope/clp repository, the `ApiServer` struct in `components/clp-rust-utils/src/clp_config/package/config.rs` is a Rust-native configuration type and does not mirror any Python code, unlike other structs in the same file (Config, Database, ResultsCache, Package) which are mirrors of Python definitions.

Applied to files:

  • docs/src/user-docs/guides-using-the-api-server.md
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (17)
  • GitHub Check: ubuntu-jammy-lint
  • GitHub Check: ubuntu-jammy-dynamic-linked-bins
  • GitHub Check: ubuntu-jammy-static-linked-bins
  • GitHub Check: package-image
  • GitHub Check: musllinux_1_2-x86_64-static-linked-bins
  • GitHub Check: musllinux_1_2-x86_64-dynamic-linked-bins
  • GitHub Check: manylinux_2_28-x86_64-static-linked-bins
  • GitHub Check: centos-stream-9-dynamic-linked-bins
  • GitHub Check: centos-stream-9-static-linked-bins
  • GitHub Check: manylinux_2_28-x86_64-dynamic-linked-bins
  • GitHub Check: rust-checks
  • GitHub Check: build-macos (macos-15, true)
  • GitHub Check: build-macos (macos-15, false)
  • GitHub Check: lint-check (macos-15)
  • GitHub Check: lint-check (ubuntu-24.04)
  • GitHub Check: build (ubuntu-24.04)
  • GitHub Check: build (macos-15)
🔇 Additional comments (1)
docs/src/user-docs/guides-using-the-api-server.md (1)

25-65: Example flow is sound, but depends on server being pre-running.

The curl example correctly demonstrates the API flow: submit a query to receive a query_results_uri, then fetch results via SSE. However, line 25–26 assumes the server is already running ("Assuming the server is running"). Given the vague startup instructions above, users may be unclear about how to satisfy this prerequisite.

Once you clarify the server startup command/mechanism in the "Starting the API server" section, ensure it's referenced or cross-linked here so users know exactly what to do before attempting these commands.

Comment on lines +12 to +13
CLP starts the API server based on the `api_server` section in `etc/clp-config.yaml`, which includes
a default configuration. You can uncomment and modify this section to override the defaults.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# First, find the file and read the complete context around lines 12-13
echo "=== Reading full file content ==="
cat -n docs/src/user-docs/guides-using-the-api-server.md | head -60

echo -e "\n=== Checking file size ==="
wc -l docs/src/user-docs/guides-using-the-api-server.md

echo -e "\n=== Searching for API server startup references ==="
rg -n "start.*server\|server.*start" --type md --type yaml --type py --type sh -A 2 -B 2

Repository: y-scope/clp

Length of output: 166


🌐 Web query:

CLP API server startup command documentation

💡 Result:

Do you mean a specific "CLP" product? For example:

  • HPE iLO SM CLP (Server Management CLP),
  • NEC/ClusterPRO CLP command-line tools,
  • a vendor-specific "CLP API" (Cloud/Customer/Lifecycle Platform),
    or something else?

Tell me which product or paste a link/error and I’ll fetch the exact startup command docs.


🌐 Web query:

CLP y-scope api_server clp-config.yaml how to start

💡 Result:

Use the CLP package's start script and the package config (etc/clp-config.yml). Quick steps:

  1. Edit /etc/clp-config.yml (or etc/clp-config.yaml) to set api/webui and storage options you need. [1][2]
  2. From the extracted CLP release root, run the bundled start script:
    sbin/start-clp.sh
    If it fails (port conflict or config), adjust etc/clp-config.yml and run again. [1][2]
  3. Open the UI at http://localhost:4000 (or the host/port you set under webui.host/webui.port). [1][3]

References:

  • CLP quick start / starting CLP. [1]
  • CLP configuration (clp-config.yml) / object-storage config. [2]
  • CLP WebUI docs (host/port and running dev server). [3]

Links: [1] CLP clp-json quick-start; [2] Configuring CLP (clp-config); [3] WebUI docs.


Based on my verification through web search, I found that CLP uses sbin/start-clp.sh to start the entire service (including the API server). The documentation at lines 12-13 explains the configuration aspect but does not mention the actual startup command that users need to run.

The original review comment's concern is valid: the section titled "Starting the API server" explains what configuration controls the startup but omits the critical step—the actual command users must execute.

Here is the rewritten review comment:


Document the command to start the API server, not just its configuration.

The section explains that configuration controls API server startup but omits the concrete command users must run. Users need to know to execute sbin/start-clp.sh (or the relevant startup script for their CLP distribution) before attempting the example curl commands in the subsequent section. Currently, lines 12-13 focus on configuration without explaining the action step required to trigger server startup.

🤖 Prompt for AI Agents
In docs/src/user-docs/guides-using-the-api-server.md around lines 12 to 13, the
"Starting the API server" section describes the api_server configuration but
omits the concrete startup command; update the section to explicitly tell users
to run the appropriate startup script (e.g., sbin/start-clp.sh or the
distribution-specific equivalent) before attempting API calls and add a brief
note indicating where to find/modify that script if needed.

@LinZhihao-723 LinZhihao-723 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit issue otherwise lgtm.
For the PR title, how about:

docs(api-server): Add guide for how to use the API server.

Comment thread components/clp-package-utils/clp_package_utils/scripts/start_clp.py Outdated
…lp.py

Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
@hoophalab hoophalab changed the title docs(api-server): Add user guide for CLP API Server. docs(api-server): Add guide for how to use the API server. Dec 1, 2025
@hoophalab hoophalab merged commit a6b373c into y-scope:main Dec 1, 2025
42 of 43 checks passed
@hoophalab hoophalab deleted the openapidocs branch December 2, 2025 18:20
junhaoliao pushed a commit to junhaoliao/clp that referenced this pull request May 17, 2026
)

Co-authored-by: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants