Skip to content

Commit 239ef29

Browse files
cnasikascnasikas
committed
Merge branch 'alerting_v2' into alerting_v2_streaming
2 parents 12c5067 + 22cf36b commit 239ef29

4,482 files changed

Lines changed: 140238 additions & 51258 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/scout-api-testing/SKILL.md‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -91,12 +91,11 @@ Tests then import `apiTest` from the local fixtures: `import { apiTest } from '.
9191
## Run / debug quickly
9292

9393
- Use either `--config` or `--testFiles` (they are mutually exclusive).
94-
- Run by config: `node scripts/scout.js run-tests --stateful --config <module-root>/test/scout*/api/playwright.config.ts` (or `.../api/parallel.playwright.config.ts` for parallel API runs)
95-
- Run by file/dir (Scout derives the right `playwright.config.ts` vs `parallel.playwright.config.ts`): `node scripts/scout.js run-tests --stateful --testFiles <module-root>/test/scout*/api/tests/my.spec.ts`
96-
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --stateful [--config-dir <configSet>]`, then run Playwright directly: `npx playwright test --config <...> --project local --grep <tag>`.
97-
- `--config-dir` notes:
98-
- `run-tests` auto-detects the custom config dir from `.../test/scout_<name>/...` paths (override with `--config-dir <name>` if needed).
99-
- `start-server` has no Playwright config to inspect, so pass `--config-dir <name>` when your tests require a custom server config.
94+
- Run by config: `node scripts/scout.js run-tests --arch stateful --domain classic --config <module-root>/test/scout*/api/playwright.config.ts` (or `.../api/parallel.playwright.config.ts` for parallel API runs)
95+
- Run by file/dir (Scout derives the right `playwright.config.ts` vs `parallel.playwright.config.ts`): `node scripts/scout.js run-tests --arch stateful --domain classic --testFiles <module-root>/test/scout*/api/tests/my.spec.ts`
96+
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --arch stateful --domain classic [--serverConfigSet <configSet>]`, then run Playwright directly: `npx playwright test --config <...> --project local --grep <tag>`.
97+
- `run-tests` auto-detects custom config sets from `.../test/scout_<name>/...` paths.
98+
- `start-server` has no Playwright config to inspect, so pass `--serverConfigSet <name>` when your tests require a custom config set.
10099
- Debug: `SCOUT_LOG_LEVEL=debug`
101100

102101
## CI enablement

‎.agents/skills/scout-api-testing/references/scout-api-auth.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ const COMMON_HEADERS = {
2525
'elastic-api-version': '2023-10-31', // include for versioned public APIs
2626
};
2727

28-
apiTest.describe('GET /api/my_plugin/foo', { tag: tags.DEPLOYMENT_AGNOSTIC }, () => {
28+
apiTest.describe('GET /api/my_plugin/foo', { tag: tags.deploymentAgnostic }, () => {
2929
let viewer: RoleApiCredentials;
3030

3131
apiTest.beforeAll(async ({ requestAuth }) => {
@@ -55,7 +55,7 @@ const INTERNAL_HEADERS = {
5555
'x-elastic-internal-origin': 'kibana',
5656
};
5757

58-
apiTest.describe('GET /internal/my_plugin/foo', { tag: tags.DEPLOYMENT_AGNOSTIC }, () => {
58+
apiTest.describe('GET /internal/my_plugin/foo', { tag: tags.deploymentAgnostic }, () => {
5959
apiTest('calls internal endpoint', async ({ apiClient, samlAuth }) => {
6060
const { cookieHeader } = await samlAuth.asInteractiveUser('viewer');
6161

‎.agents/skills/scout-best-practices-reviewer/SKILL.md‎

Lines changed: 106 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,117 @@
11
---
22
name: scout-best-practices-reviewer
3-
description: Use when writing and reviewing Scout UI and API test files.
3+
description: Review Scout UI/API tests (including Scout test migrations) for best practices, reuse, and parity.
44
---
55

66
# Scout Best Practices Reviewer
77

88
## Overview
99

10-
- Identify whether the spec is **UI** (`test`/`spaceTest`) or **API** (`apiTest`) and review against the relevant best practices.
11-
- Keep feedback concise and actionable: focus on correctness, flake risk, and CI/runtime impact.
12-
- Report findings ordered by severity and include the matching best-practice heading (from the reference) next to each finding.
10+
Perform a static PR review of Scout UI and API test files (`*.spec.ts`) against Scout best practices and existing Scout abstractions (fixtures, page objects, API helpers). Produce actionable, PR-review-ready feedback that pushes for reuse over one-off implementations.
1311

14-
## References
12+
Important: Do not post GitHub comments unless explicitly stated.
1513

16-
Open only what you need:
14+
### Inputs
1715

18-
- Review checklist (tagging, structure, auth, flake control): `references/scout-best-practices.md`
16+
1. Changed `*.spec.ts` files (and imported helpers/fixtures).
17+
- UI Tests: Use `test` / `spaceTest` (usually in `**/test/scout/ui/**`).
18+
- API Tests: Use `apiTest` (usually in `**/test/scout/api/**`).
19+
2. Neighboring Scout code in the same plugin/solution (existing specs + `test/scout/**/fixtures/**`) to spot reuse opportunities and avoid duplicating helpers.
20+
3. Removed/previous tests (if this is a migration) to verify behavior parity.
21+
4. Scout docs (open only what you need):
22+
- Best practices: `docs/extend/scout/best-practices.md`
23+
- Core concepts & fixtures: `docs/extend/scout/core-concepts.md`, `docs/extend/scout/fixtures.md`
24+
- Reuse surfaces: `docs/extend/scout/page-objects.md`, `docs/extend/scout/api-services.md`
25+
- Type-specific guides: `docs/extend/scout/write-ui-tests.md`, `docs/extend/scout/write-api-tests.md`
26+
- As needed: `docs/extend/scout/api-auth.md`, `docs/extend/scout/browser-auth.md`, `docs/extend/scout/parallelism.md`, `docs/extend/scout/deployment-tags.md`, `docs/extend/scout/a11y-checks.md`, `docs/extend/scout/debugging.md`, `docs/extend/scout/run-tests.md`
27+
28+
## Scope (be comprehensive)
29+
30+
- Don’t limit the review to the diff. Look for duplication and missed reuse by scanning:
31+
- existing Scout specs in the same area (and similar suites elsewhere in the repo)
32+
- available fixtures (`docs/extend/scout/fixtures.md` + local `test/scout/**/fixtures`)
33+
- existing page objects, API services, and fixtures (in `@kbn/scout`, solution Scout packages, and plugin-local `test/scout/**`) before suggesting brand-new helpers
34+
35+
### Quick checklist (details live in `docs/extend/scout/best-practices.md`)
36+
37+
- **Reuse-first**: prefer existing `pageObjects`, fixtures, and `apiServices`; if adding helpers/page objects, place them in the right scope (plugin vs solution vs `@kbn/scout`) and register via fixtures.
38+
- **Fixture boundaries**: `apiClient` for the endpoint under test; `apiServices`/`kbnClient` for setup/teardown only; correct auth + common headers.
39+
- **Correctness**: guardrail assertions before dereferencing response fields; validate contract + side effects; stable error assertions.
40+
- **UI scope**: UI tests should focus on user interactions and rendering; avoid “data correctness” assertions (for example exact API response shapes or exact table cell values) unless the UI behavior depends on them. Prefer Scout API tests (or unit/integration) for data correctness coverage.
41+
- **Isolation**: parallel-safe data and resilient cleanup in hooks; no reliance on file ordering or shared mutable state.
42+
- **RBAC / realism**: minimal permissions (avoid `admin` unless required); space-aware behavior covered or explicitly out of scope.
43+
- **Flake traps**: avoid `waitForTimeout()` and time-based assertions/retries; rely on auto-waiting + explicit readiness signals.
44+
- **Cost**: avoid repeating expensive setup; consider a global setup hook for shared one-time operations.
45+
- **Tags / environment**: validate deployment tags and avoid assumptions that only hold in specific environments.
46+
47+
### Migration parity analysis (required when migration is detected)
48+
49+
- **Detect migration** when the PR removes/changes FTR tests (for example `test/functional/**`, `loadTestFile()`, FTR configs) alongside new/changed Scout specs.
50+
- **If migration is detected**:
51+
- Treat parity gaps as `blocker` unless explicitly de-scoped.
52+
- Confirm the suite is the right **test type** (UI vs API): if the old FTR suite is primarily “data correctness”, prefer migrating it to a Scout API test (or unit/integration) rather than a Scout UI test.
53+
- Build a parity map from old scenarios → new Scout coverage (roles, setup/teardown, assertions, cleanup).
54+
- Call out missing behaviors (including error paths) and recommend exactly where to add coverage.
55+
- Escalate meaningful **Scout vs FTR deltas** when they could change what’s actually being tested, weaken coverage, or increase flake risk. Treat these as parity issues that require action (code change or explicit de-scope/sign-off), and include them in the “Migration parity” output section.
56+
- auth/roles used (e.g., `admin` vs viewer), spaces behavior, and permission realism
57+
- headers/internal origin/REST versioning and any other request shaping differences
58+
- retries and error handling differences (e.g., helper methods with `ignoreErrors`, automatic retries)
59+
- parallelism/isolation differences (worker-scoped fixtures, shared state, cleanup semantics)
60+
- classic vs serverless coverage changes (suite removed from one environment but not the other)
61+
- assertion strength changes (weaker/stronger checks, removal of side-effect validation)
62+
- Verify suite wiring/discovery (new specs are picked up by Scout/Playwright config; no orphaned `loadTestFile()`).
63+
- Ensure any intentional de-scopes are explicit, and that tags/permissions remain equivalent and cloud/serverless compatible where applicable.
64+
- **Output**: include the “Migration parity” section only when action is required; otherwise omit it.
65+
66+
## Output format
67+
68+
Output **only** the applicable sections below. Use headings and lists (**no tables**). Group issues by priority: `blocker` → `major` → `minor` → `nit`. Omit empty priorities.
69+
70+
### 1. Findings
71+
72+
#### Blocker
73+
74+
- **<Concern (use exact checklist heading)> — <short summary>**
75+
- **Explanation**: <1-3 concise, actionable sentences>
76+
- **Evidence**: `<file:line>` (add multiple as needed)
77+
- **Suggested change**: <Specific code edit; include a small snippet if helpful>
78+
79+
#### Major
80+
81+
- **<Concern (use exact checklist heading)> — <short summary>**
82+
- **Explanation**: <...>
83+
- **Evidence**: `<file:line>`
84+
- **Suggested change**: <...>
85+
86+
#### Minor
87+
88+
- **<Concern (use exact checklist heading)> — <short summary>**
89+
- **Explanation**: <...>
90+
- **Evidence**: `<file:line>`
91+
- **Suggested change**: <...>
92+
93+
#### Nit
94+
95+
- **<Concern (use exact checklist heading)> — <short summary>**
96+
- **Explanation**: <...>
97+
- **Evidence**: `<file:line>`
98+
- **Suggested change**: <...>
99+
100+
### 2. Migration parity (only if a test migration is detected and action is required)
101+
102+
Include this section only when the PR removes/changes FTR tests alongside new/changed Scout specs **and** you found at least one parity issue that requires someone to step in (code change or an explicit de-scope/sign-off decision).
103+
Do **not** output an FYI parity map. If everything is equivalent (or differences are clearly benign), omit this section.
104+
105+
#### Blocker / Major / Minor / Nit
106+
107+
- **<Concern (use exact checklist heading)> — <scenario name>**
108+
- **Issue**: <Coverage gap or behavior delta that needs action>
109+
- **Old behavior**: <...>
110+
- **New behavior**: <...>
111+
- **Why it matters**: <1-2 sentences on risk/coverage impact>
112+
- **Suggested fix / decision**: <Required. Either a code change or an explicit de-scope/sign-off the reviewer must confirm.>
113+
- **Evidence**: `<file:line>`
114+
115+
## Follow-up
116+
117+
Offer to generate the updated code, fully incorporating the suggested improvements and resolving any parity gaps.

‎.agents/skills/scout-best-practices-reviewer/references/scout-best-practices.md‎

Lines changed: 0 additions & 108 deletions
This file was deleted.

‎.agents/skills/scout-create-scaffold/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,8 +64,8 @@ Notes:
6464
- Update `.meta` manifests when adding/moving configs or tests:
6565
- `node scripts/scout.js update-test-config-manifests`
6666
- Custom server config sets:
67-
- If you create/use `test/scout_<configSet>`, you typically also need a matching server config under `src/platform/packages/shared/kbn-scout/src/servers/configs/custom/<configSet>`.
68-
- `start-server` requires `--config-dir <configSet>` when using a custom server config set.
67+
- If you create/use `test/scout_<configSet>`, you typically also need a matching server config under `src/platform/packages/shared/kbn-scout/src/servers/configs/config_sets/<configSet>`.
68+
- `start-server` requires `--serverConfigSet <configSet>` when using a custom server config set.
6969

7070
## Path Conventions (Specs)
7171

‎.agents/skills/scout-migrate-from-ftr/SKILL.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,10 @@ Migrate FTR tests to Scout by deciding whether a test should be UI or API, mappi
4848
9. Update Scout manifests (discovery/CI).
4949
- Run `node scripts/scout.js update-test-config-manifests` so `.meta` manifests reflect the new/changed tests and configs.
5050
10. Verify in both stateful and serverless when applicable.
51-
- Use `node scripts/scout.js run-tests --stateful --testFiles <path>` and
52-
`node scripts/scout.js run-tests --serverless=oblt --testFiles <path>` (adjust serverless target).
53-
- If the tests are under `test/scout_<configSet>/...`, `run-tests` auto-detects the server config dir from the Playwright config path (use `--config-dir <configSet>` only to override, or when using `start-server`).
51+
- Use `node scripts/scout.js run-tests --arch stateful --domain classic --testFiles <path>` and
52+
`node scripts/scout.js run-tests --arch serverless --domain observability_complete --testFiles <path>` (adjust the serverless domain).
53+
- If tests are under `test/scout_<configSet>/...`, `run-tests` auto-detects the config set from the Playwright config path.
54+
- If you start servers separately, pass `--serverConfigSet <configSet>` to `node scripts/scout.js start-server ...`.
5455

5556
## Common patterns
5657

‎.agents/skills/scout-ui-testing/SKILL.md‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -104,12 +104,11 @@ test('creates and verifies a dashboard', async ({ pageObjects, page }) => {
104104
## Run / debug quickly
105105

106106
- Use either `--config` or `--testFiles` (they are mutually exclusive).
107-
- Run by config: `node scripts/scout.js run-tests --stateful --config <module-root>/test/scout*/ui/playwright.config.ts` (or `.../ui/parallel.playwright.config.ts` for parallel UI)
108-
- Run by file/dir (Scout derives the right `playwright.config.ts` vs `parallel.playwright.config.ts`): `node scripts/scout.js run-tests --stateful --testFiles <module-root>/test/scout*/ui/tests/my.spec.ts`
109-
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --stateful [--config-dir <configSet>]`, then run Playwright directly: `npx playwright test --config <...> --project local --grep <tag> --headed`.
110-
- `--config-dir` notes:
111-
- `run-tests` auto-detects the custom config dir from `.../test/scout_<name>/...` paths (override with `--config-dir <name>` if needed).
112-
- `start-server` has no Playwright config to inspect, so pass `--config-dir <name>` when your tests require a custom server config.
107+
- Run by config: `node scripts/scout.js run-tests --arch stateful --domain classic --config <module-root>/test/scout*/ui/playwright.config.ts` (or `.../ui/parallel.playwright.config.ts` for parallel UI)
108+
- Run by file/dir (Scout derives the right `playwright.config.ts` vs `parallel.playwright.config.ts`): `node scripts/scout.js run-tests --arch stateful --domain classic --testFiles <module-root>/test/scout*/ui/tests/my.spec.ts`
109+
- For faster iteration, start servers once in another terminal: `node scripts/scout.js start-server --arch stateful --domain classic [--serverConfigSet <configSet>]`, then run Playwright directly: `npx playwright test --config <...> --project local --grep <tag> --headed`.
110+
- `run-tests` auto-detects custom config sets from `.../test/scout_<name>/...` paths.
111+
- `start-server` has no Playwright config to inspect, so pass `--serverConfigSet <name>` when your tests require a custom config set.
113112
- Debug: `SCOUT_LOG_LEVEL=debug`, or `npx playwright test --config <...> --project local --ui`
114113

115114
## CI enablement

0 commit comments

Comments
 (0)