Skip to content

Commit a01e01a

Browse files
jeramysoucyjeramysoucy
authored
Merge branch 'main' into sessionless-profile-retrieval
2 parents a3af1ad + 1f7ec83 commit a01e01a

2,390 files changed

Lines changed: 100258 additions & 44993 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agent/skills/ftr-testing/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: FTR Testing
2+
name: ftr-testing
33
description: Use when creating, updating, debugging, or reviewing Kibana Functional Test Runner (FTR) tests, including test structure, services/page objects, loadTestFile patterns, tags, and how to run FTR locally.
44
---
55

‎.agent/skills/scout-api-testing/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: Scout API Testing
2+
name: scout-api-testing
33
description: Use when creating, updating, debugging, or reviewing Scout API tests in Kibana (apiTest/apiClient/requestAuth/samlAuth/apiServices), including auth choices, response assertions, and API service patterns.
44
---
55

@@ -14,7 +14,7 @@ description: Use when creating, updating, debugging, or reviewing Scout API test
1414
- `x-pack/solutions/search/**` -> `@kbn/scout-search`
1515
- `x-pack/solutions/security/**` -> `@kbn/scout-security`
1616
- Prefer a single top-level `apiTest.describe(...)` per file and avoid nested `describe` blocks; multiple top-level `describe`s are supported, but files get hard to read quickly.
17-
- Tags: add `{ tag: ... }` on the suite (or individual tests) so CI/discovery can select the right deployment target (for example `tags.DEPLOYMENT_AGNOSTIC` or `['@ess']`). Unlike UI tests, API tests don’t currently validate tags at runtime.
17+
- Tags: add `{ tag: ... }` on the suite (or individual tests) so CI/discovery can select the right test target (for example `tags.deploymentAgnostic` or `[...tags.stateful.classic]`). Unlike UI tests, API tests don’t currently validate tags at runtime.
1818
- If the module provides Scout fixtures, import `apiTest` from `<module-root>/test/scout*/api/fixtures` to get module-specific extensions. Importing directly from the module’s Scout package is also fine when you don’t need extensions.
1919
- Browser fixtures are disabled for `apiTest` (no `page`, `browserAuth`, `pageObjects`).
2020

‎.agent/skills/scout-best-practices-reviewer/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: Scout Best Practices Reviewer
2+
name: scout-best-practices-reviewer
33
description: Use when writing and reviewing Scout UI and API test files.
44
---
55

‎.agent/skills/scout-best-practices-reviewer/references/scout-best-practices.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616

1717
- UI tests: tags are required and validated at runtime. Ensure each test has at least one supported tag (typically by tagging the top-level `test.describe(...)` / `spaceTest.describe(...)`).
1818
- API tests: tags aren’t currently validated at runtime, but add `{ tag: ... }` to suites/tests so CI/discovery can select the right deployment target (stateful vs serverless).
19-
- Prefer using `tags.*` constants (for example `tags.DEPLOYMENT_AGNOSTIC`, `tags.ESS_ONLY`) to avoid typos and to align with Scout’s supported tags.
19+
- Prefer using `tags.*` constants (for example `tags.deploymentAgnostic`, `tags.stateful.classic`) to avoid typos and to align with Scout’s supported tags.
2020

2121
## Setup / Teardown (Applies to UI + API)
2222

‎.agent/skills/scout-create-scaffold/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: Scout Create Scaffold
2+
name: scout-create-scaffold
33
description: Generate or repair a Scout test scaffold for a Kibana plugin/package (test/scout*/{api,ui} Playwright configs, fixtures, example specs). Use when you need the initial Scout directory structure; prefer `node scripts/scout.js generate` with flags for non-interactive/LLM execution.
44
---
55

‎.agent/skills/scout-migrate-from-ftr/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: Scout Migrate From FTR
2+
name: scout-migrate-from-ftr
33
description: Use when migrating Kibana Functional Test Runner (FTR) tests to Scout, including decisions about UI vs API tests, mapping FTR services/page objects/hooks to Scout fixtures, and splitting loadTestFile patterns.
44
---
55

‎.agent/skills/scout-ui-testing/SKILL.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
name: Scout UI Testing
2+
name: scout-ui-testing
33
description: Use when creating, updating, debugging, or reviewing Scout UI tests in Kibana (Playwright + Scout fixtures), including page objects, browser authentication, parallel UI tests (spaceTest/scoutSpace), a11y checks, and flake control.
44
---
55

@@ -24,7 +24,7 @@ description: Use when creating, updating, debugging, or reviewing Scout UI tests
2424

2525
## Non-negotiable conventions
2626

27-
- **Tags are required**: Scout validates UI test tags at runtime. Ensure each test has at least one supported tag (typically by tagging the top-level `test.describe(...)` / `spaceTest.describe(...)`, e.g. `tags.DEPLOYMENT_AGNOSTIC`, `tags.ESS_ONLY`, or `tags.PERFORMANCE`).
27+
- **Tags are required**: Scout validates UI test tags at runtime. Ensure each test has at least one supported tag (typically by tagging the top-level `test.describe(...)` / `spaceTest.describe(...)`, e.g. `tags.deploymentAgnostic`, `tags.stateful.classic`, or `tags.performance`).
2828
- **Prefer one suite per file**: keep a single top-level `test.describe(...)` (sequential) or `spaceTest.describe(...)` (parallel) and avoid nested `describe` blocks where possible.
2929
- **UI actions live in page objects**; assertions stay in the spec.
3030
- **Use APIs for setup/teardown**: prefer `apiServices`/`kbnClient`/`esArchiver` in hooks over clicking through the UI.

‎.buildkite/ftr_platform_stateful_configs.yml‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,6 @@ enabled:
102102
- src/platform/test/functional/apps/kibana_overview/config.ts
103103
- src/platform/test/functional/apps/management/config.ts
104104
- src/platform/test/functional/apps/saved_objects_management/config.ts
105-
- src/platform/test/functional/apps/sharing/config.ts
106105
- src/platform/test/functional/apps/status_page/config.ts
107106
- src/platform/test/functional/apps/visualize/group1/config.ts
108107
- src/platform/test/functional/apps/visualize/group2/config.ts
@@ -166,7 +165,6 @@ enabled:
166165
- x-pack/platform/test/alerting_api_integration/spaces_only/tests/actions/config.ts
167166
- x-pack/platform/test/alerting_api_integration/spaces_only/tests/action_task_params/config.ts
168167
- x-pack/platform/test/alerting_api_integration/spaces_only/tests/actions/connector_types/stack/email_recipient_allowlist/config.ts
169-
- x-pack/platform/test/banners_functional/config.ts
170168
- x-pack/platform/test/cases_api_integration/security_and_spaces/config_basic.ts
171169
- x-pack/platform/test/cases_api_integration/security_and_spaces/config_trial.ts
172170
- x-pack/platform/test/cases_api_integration/security_and_spaces/config_trial_common.ts
@@ -294,7 +292,6 @@ enabled:
294292
- x-pack/platform/test/functional_cloud/intercepts.config.ts
295293
- x-pack/platform/test/licensing_plugin/config.public.ts
296294
- x-pack/platform/test/licensing_plugin/config.ts
297-
- x-pack/platform/test/product_intercepts_functional/config.ts
298295
- x-pack/platform/test/reporting_api_integration/reporting_and_security.config.ts
299296
- x-pack/platform/test/reporting_api_integration/reporting_without_security.config.ts
300297
- x-pack/platform/test/reporting_functional/reporting_and_security.config.ts
@@ -380,7 +377,6 @@ enabled:
380377
- x-pack/performance/journeys_e2e/cloud_security_dashboard.ts
381378
- x-pack/performance/journeys_e2e/apm_service_inventory.ts
382379
- x-pack/performance/journeys_e2e/infra_hosts_view.ts
383-
- x-pack/platform/test/custom_branding/config.ts
384380
# configs migrated to the new Kibana architecture
385381
- x-pack/platform/test/api_integration/apis/aiops/config.ts
386382
- x-pack/platform/test/api_integration/apis/cases/config.ts

‎.buildkite/pipeline-utils/ci-stats/pick_test_group_run_order.ts‎

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -341,9 +341,6 @@ export async function pickTestGroupRunOrder() {
341341
timeout_in_minutes: 120,
342342
key: 'jest',
343343
agents: expandAgentQueue('n2-4-spot', 110),
344-
env: {
345-
SCOUT_TARGET_TYPE: 'local',
346-
},
347344
depends_on: JEST_CONFIGS_DEPS,
348345
retry: {
349346
automatic: [
@@ -360,12 +357,10 @@ export async function pickTestGroupRunOrder() {
360357
label: 'Jest Integration Tests',
361358
command: getRequiredEnv('JEST_INTEGRATION_SCRIPT'),
362359
parallelism: integration.count,
363-
timeout_in_minutes: 120,
360+
// TODO: Reduce once we have identified the cause of random long-running tests
361+
timeout_in_minutes: 75,
364362
key: 'jest-integration',
365363
agents: expandAgentQueue('n2-4-spot', 105),
366-
env: {
367-
SCOUT_TARGET_TYPE: 'local',
368-
},
369364
depends_on: JEST_CONFIGS_DEPS,
370365
retry: {
371366
automatic: [
@@ -402,7 +397,6 @@ export async function pickTestGroupRunOrder() {
402397
timeout_in_minutes: 120,
403398
agents: expandAgentQueue(queue, 105),
404399
env: {
405-
SCOUT_TARGET_TYPE: 'local',
406400
FTR_CONFIG_GROUP_KEY: key,
407401
...ftrExtraArgs,
408402
...envFromlabels,

‎.buildkite/pipelines/flaky_tests/pipeline.ts‎

Lines changed: 15 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -35,53 +35,6 @@ if (Number.isNaN(concurrency)) {
3535
const BASE_JOBS = 1;
3636
const MAX_JOBS = 500;
3737

38-
function getScoutConfigGroupType(configPath: string): string | null {
39-
// Match platform paths: x-pack/platform/... or src/platform/...
40-
if (/^(x-pack|src)\/platform\//.test(configPath)) {
41-
return 'platform';
42-
}
43-
// Match solution paths: x-pack/solutions/<solution>/plugins/...
44-
const match = configPath.match(/^x-pack\/solutions\/([^/]+)\/plugins\//);
45-
if (match) {
46-
return match[1];
47-
}
48-
return null;
49-
}
50-
51-
function getScoutServerRunFlags(configPath: string): string[] {
52-
const groupType = getScoutConfigGroupType(configPath);
53-
54-
if (!groupType) {
55-
throw new Error(
56-
`Unable to determine scout config group type from path: ${configPath}. ` +
57-
`Expected path to match platform pattern (x-pack/platform/... or src/platform/...) ` +
58-
`or solution pattern (x-pack/solutions/<solution>/plugins/...)`
59-
);
60-
}
61-
62-
if (groupType === 'platform') {
63-
return ['--stateful', '--serverless=es', '--serverless=oblt', '--serverless=security'];
64-
}
65-
66-
if (groupType === 'workplaceai') {
67-
return ['--serverless=workplace-ai'];
68-
}
69-
70-
const flags = ['--stateful'];
71-
72-
if (groupType === 'observability') {
73-
flags.push('--serverless=oblt');
74-
} else if (groupType === 'security') {
75-
flags.push('--serverless=security');
76-
} else if (groupType === 'search') {
77-
flags.push('--serverless=es');
78-
} else {
79-
throw new Error(`Unknown solution type: ${groupType}.`);
80-
}
81-
82-
return flags;
83-
}
84-
8538
function getTestSuitesFromJson(json: string) {
8639
const fail = (errorMsg: string) => {
8740
console.error('+++ Invalid test config provided');
@@ -163,6 +116,7 @@ function getTestSuitesFromJson(json: string) {
163116
}
164117

165118
const testSuites = getTestSuitesFromJson(configJson);
119+
const hasScoutSuites = testSuites.some((t) => t.type === 'scoutConfig' && t.count > 0);
166120

167121
const totalJobs = testSuites.reduce((acc, t) => acc + t.count, BASE_JOBS);
168122

@@ -194,6 +148,19 @@ steps.push({
194148
if: "build.env('KIBANA_BUILD_ID') == null || build.env('KIBANA_BUILD_ID') == ''",
195149
});
196150

151+
if (hasScoutSuites) {
152+
steps.push({
153+
command: '.buildkite/scripts/steps/test/scout_discover_playwright_configs.sh',
154+
label: 'Discover Scout Playwright configs',
155+
agents: expandAgentQueue('n2-4-spot'),
156+
key: 'scout_playwright_configs',
157+
timeout_in_minutes: 30,
158+
retry: {
159+
automatic: [{ exit_status: '-1', limit: 3 }],
160+
},
161+
});
162+
}
163+
197164
let suiteIndex = 0;
198165
for (const testSuite of testSuites) {
199166
if (testSuite.count <= 0) {
@@ -225,14 +192,12 @@ for (const testSuite of testSuites) {
225192

226193
if (testSuite.type === 'scoutConfig') {
227194
const usesParallelWorkers = testSuite.scoutConfig.endsWith('parallel.playwright.config.ts');
228-
const serverRunFlags = getScoutServerRunFlags(testSuite.scoutConfig);
229195

230196
steps.push({
231197
command: `.buildkite/scripts/steps/test/scout_flaky_configs.sh`,
232198
env: {
233199
SCOUT_CONFIG: testSuite.scoutConfig,
234200
SCOUT_REPORTER_ENABLED: 'true',
235-
SCOUT_SERVER_RUN_FLAGS: serverRunFlags.join('\n'),
236201
},
237202
key: `${TestSuiteType.SCOUT}-${suiteIndex++}`,
238203
label: `${testSuite.scoutConfig}`,
@@ -241,7 +206,7 @@ for (const testSuite of testSuites) {
241206
concurrency_group: process.env.UUID,
242207
concurrency_method: 'eager',
243208
agents: expandAgentQueue(usesParallelWorkers ? 'n2-8-spot' : 'n2-4-spot'),
244-
depends_on: 'build',
209+
depends_on: hasScoutSuites ? ['build', 'scout_playwright_configs'] : 'build',
245210
timeout_in_minutes: 60,
246211
cancel_on_build_failing: true,
247212
retry: {

0 commit comments

Comments
 (0)