[docs-utils] 3️⃣ New check docs CLI for improved DX by clintandrewhall · Pull Request #249305 · elastic/kibana

clintandrewhall · 2026-01-15T22:06:29Z

Summary

This PR introduces a new check_package_docs CLI that improves the developer experience for validating plugin and package documentation. The new CLI provides a clear separation between building API documentation and checking documentation quality without writing output files.

Note

This PR is a subset of #247688 for ease of review.

This PR was written with Cursor and claude-4.5-opus-high.

Key Changes

New CLI: node scripts/check_package_docs

Validates documentation for plugins/packages without generating output files
Supports --check <any|comments|exports|all> flag to specify validation checks
Allows multiple --check flags to combine checks (e.g., --check any --check comments)
Supports both --plugin and --package flags for filtering by ID

Improved findPlugins API

Refactored to accept separate pluginFilter and packageFilter options
Clearer distinction between filtering by plugin.id (for plugins) and package ID (for packages like @kbn/core)

Deprecation Handling

The --stats flag on build_api_docs is now deprecated
When --stats is used, it automatically routes to the new check_package_docs CLI with a warning

Technical Improvements

Guarded APM initialization to prevent double-start when delegating between CLIs
Added comprehensive unit tests for both build and check CLI flows
Added integration tests for validation exit codes

Usage

# Check all documentation issues for a specific plugin
node scripts/check_package_docs --plugin myPlugin

# Check only missing comments for a package
node scripts/check_package_docs --package @kbn/core --check comments

# Check multiple issue types
node scripts/check_package_docs --plugin myPlugin --check any --check exports

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
Review the backport guidelines and apply applicable backport:* labels.

Release Note

New check_package_docs CLI for validating plugin and package API documentation. The --stats flag on build_api_docs is deprecated in favor of this new dedicated validation command.

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Low risk: The deprecated --stats flag now routes to a different CLI. Existing scripts using --stats will continue to work but with a deprecation warning. No functionality is removed.
Low risk: The findPlugins function signature changed from accepting a single array to an options object. This is an internal API used only within kbn-docs-utils.

elasticmachine · 2026-01-15T22:06:34Z

Pinging @elastic/kibana-operations (Team:Operations)

elasticmachine · 2026-01-15T22:06:37Z

Pinging @elastic/appex-sharedux (Team:SharedUX)

Copilot

Pull request overview

This PR introduces a new check_package_docs CLI that validates API documentation quality without generating output files, improving the developer experience by providing a clear separation between building and checking documentation.

Changes:

Added new check_package_docs CLI with --check flag for validation-only runs
Refactored findPlugins API to accept separate pluginFilter and packageFilter options
Deprecated --stats flag on build_api_docs, routing it to the new check_package_docs CLI

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
scripts/check_package_docs.js	Entry point for the new check CLI
packages/kbn-docs-utils/src/index.ts	Exports the new `runCheckPackageDocsCli` function
packages/kbn-docs-utils/src/check_package_docs_cli.ts	Core implementation of the validation-only CLI
packages/kbn-docs-utils/src/build_api_docs_cli.ts	Added deprecation handling to route `--stats` to check CLI
packages/kbn-docs-utils/src/find_plugins.ts	Refactored to accept separate plugin and package filter options
packages/kbn-docs-utils/src/cli/types.ts	Added `check` and `package` flag types
packages/kbn-docs-utils/src/cli/parse_cli_flags.ts	Enhanced flag parsing to support `--check` and `--package`
packages/kbn-docs-utils/src/cli/tasks/setup_project.ts	Updated to use new filter options structure
packages/kbn-docs-utils/src/utils.ts	Improved missing API items logging message
packages/kbn-docs-utils/src/README.md	Added documentation for the new check CLI
Test files	Comprehensive unit and integration tests for new functionality

packages/kbn-docs-utils/src/check_package_docs_cli.test.ts

packages/kbn-docs-utils/src/check_package_docs_cli.run.test.ts

packages/kbn-docs-utils/src/check_package_docs_cli.test.ts

packages/kbn-docs-utils/src/check_package_docs_cli.run.test.ts

packages/kbn-docs-utils/src/README.md

clintandrewhall · 2026-01-16T02:58:54Z

packages/kbn-docs-utils/src/check_package_docs_cli.ts

+
+const startApm = () => {
+  if (!apm.isStarted()) {
+    initApm(process.argv, rootDir, false, 'check_package_docs_cli');


@jbudz I've split the build_package_docs_cli that included checking docs into two initApm calls. So we should see build_ fall and check_ rise, theoretically.

Is there anything I need to do to enable this change? Is this ok? Wanted a sanity check, given where my meta PR took APM logs last week.

packages/kbn-docs-utils/src/check_package_docs_cli.ts

kowalczyk-krzysztof

Approving SharedUX changes but I'm wondering if getPlugin really needs to exist? The only usage is in checkDependencies and aside from getting a plugin, all it does is adds logging which you do anyway in checkDependencies.

clintandrewhall · 2026-01-27T13:18:18Z

@kowalczyk-krzysztof I remember I split it out for a reason; it's likely for a later phase of the meta PR. I'm AFK, but I'll confirm.

- created new CLI for checking package/plugin documentation - clarifies the DX of building docs vs checking docs - Added --check (any|comments|exports|all) support, plugin/package aliases, and kept --stats delegation with deprecation handling for build and check CLIs; guarded APM init to prevent double-start. - Expanded coverage with CLI flow tests for build and check paths, including stats delegation and validation exit codes.

@kbn/docs-utils: 298 tests passing (100%) Coverage: 93.82% lines, 93.11% statements, 91.95% functions, 88.81% branches

mistic

LGTM. We just need to update the CI jobs for build_api_docs where check was intended before in another PR but the OPS team can do that. Created https://github.com/elastic/kibana-operations/issues/417

clintandrewhall · 2026-01-27T20:59:23Z

Thanks @mistic ! Phase 4 PRs incoming tomorrow.

elasticmachine · 2026-01-27T21:17:28Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: afb287a

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #116 / "before all" hook in "{root}"
[job] [logs] FTR Configs #70 / Agent Builder tools MCP tools creating MCP tools should create an MCP tool by selecting connector and tool

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/docs-utils`	5	4	-1

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`@kbn/docs-utils`	1	2	+1

Unknown metric groups

API count

id	before	after	diff
`@kbn/docs-utils`	5	6	+1

History

💛 Build #386534 was flaky e320d45
💛 Build #382385 was flaky dd97a9b
💔 Build #382360 failed 013c215
💔 Build #382354 failed d9a3b22

…#249305)

clintandrewhall requested a review from a team as a code owner January 15, 2026 22:06

clintandrewhall added release_note:enhancement review release_note:deprecation Team:Operations Kibana-Operations Team backport:skip This PR does not require backporting APIDocs Team:SharedUX Platform AppEx-SharedUX (formerly Global Experience) t// v9.4.0 labels Jan 15, 2026

clintandrewhall requested a review from Copilot January 15, 2026 22:26

Copilot AI reviewed Jan 15, 2026

View reviewed changes

clintandrewhall requested a review from a team as a code owner January 16, 2026 02:54

clintandrewhall commented Jan 16, 2026

View reviewed changes

Ikuni17 reviewed Jan 21, 2026

View reviewed changes

packages/kbn-docs-utils/src/check_package_docs_cli.ts Show resolved Hide resolved

packages/kbn-docs-utils/src/check_package_docs_cli.ts Outdated Show resolved Hide resolved

kowalczyk-krzysztof approved these changes Jan 27, 2026

View reviewed changes

clintandrewhall added 2 commits January 27, 2026 14:22

New and updated tests for code

afb287a

@kbn/docs-utils: 298 tests passing (100%) Coverage: 93.82% lines, 93.11% statements, 91.95% functions, 88.81% branches

clintandrewhall force-pushed the feat/docs-utils/3-check-cli branch from e320d45 to afb287a Compare January 27, 2026 19:25

mistic approved these changes Jan 27, 2026

View reviewed changes

clintandrewhall enabled auto-merge (squash) January 27, 2026 20:59

clintandrewhall merged commit 1b67a4f into elastic:main Jan 27, 2026
16 checks passed

clintandrewhall deleted the feat/docs-utils/3-check-cli branch January 27, 2026 21:16

clintandrewhall mentioned this pull request Jan 29, 2026

[meta] Updating @kbn/docs-utils, add MCP tools #247688

Draft

13 tasks

clintandrewhall changed the title ~~[kbn-docs-utils] Phase 3: New check docs CLI for improved DX~~ [docs-utils] 3️⃣ New check docs CLI for improved DX Jan 29, 2026

hannahbrooks pushed a commit to hannahbrooks/kibana that referenced this pull request Jan 30, 2026

[kbn-docs-utils] Phase 3: New check docs CLI for improved DX (elastic…

47bdf6a

…#249305)

Conversation

clintandrewhall commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Usage

Checklist

Release Note

Identify risks

Uh oh!

elasticmachine commented Jan 15, 2026

Uh oh!

elasticmachine commented Jan 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clintandrewhall Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kowalczyk-krzysztof left a comment

Choose a reason for hiding this comment

Uh oh!

clintandrewhall commented Jan 27, 2026

Uh oh!

mistic left a comment

Choose a reason for hiding this comment

Uh oh!

clintandrewhall commented Jan 27, 2026

Uh oh!

Uh oh!

elasticmachine commented Jan 27, 2026

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

clintandrewhall commented Jan 15, 2026 •

edited

Loading