Skip to content

Commit 03e7ea3

Browse files
milan-elasticmilan-elastic
committed
Merge branch 'main' of github.com:milan-elastic/integrations into aws-add-dataset-filters-dashboard
2 parents 7325cf7 + 35cd691 commit 03e7ea3

File tree

403 files changed

+19954
-1666
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

403 files changed

+19954
-1666
lines changed

‎.buildkite/scripts/common.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -692,7 +692,7 @@ is_pr_affected() {
692692

693693
echo "[${package}] git-diff: check non-package files"
694694
commit_merge=$(git merge-base "${from}" "${to}")
695-
if git diff --name-only "${commit_merge}" "${to}" | grep -E -v '^(packages/|.github/CODEOWNERS|docs/)' ; then
695+
if git diff --name-only "${commit_merge}" "${to}" | grep -E -v '^(packages/|.github/CODEOWNERS|README.md|docs/)' ; then
696696
echo "[${package}] PR is affected: found non-package files"
697697
return 0
698698
fi

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,7 @@
381381
/packages/universal_profiling_symbolizer @elastic/obs-ds-intake-services
382382
/packages/vectra_detect @elastic/security-service-integrations
383383
/packages/vsphere @elastic/obs-infraobs-integrations
384+
/packages/websocket @elastic/security-service-integrations
384385
/packages/watchguard_firebox @elastic/sec-deployment-and-devices
385386
/packages/websphere_application_server @elastic/obs-infraobs-integrations
386387
/packages/windows @elastic/elastic-agent-data-plane @elastic/sec-windows-platform

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ While this repository contains _sources_ for Elastic Integrations, _built_ Elast
1515
Please review the [Contributing Guide](CONTRIBUTING.md) to learn how to build and develop packages, understand the release procedure and
1616
explore the builder tools.
1717

18+
More information about the CI pipelines that are available in this repostiory [here](./docs/ci_pipelines.md).
19+
1820
## External links
1921

2022
### Package Specification

‎docs/ci_pipelines.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# CI integrations pipelines
2+
3+
This section describes the CI pipelines available in this repository.
4+
5+
Currently, there are five different pipelines:
6+
- https://buildkite.com/elastic/integrations: pipeline in charge of testing all packages using a local Elastic stack. More info at [section](#pull-requests-and-pushes-to-specific-branches).
7+
- https://buildkite.com/elastic/integrations-serverless: pipeline in charge of testing all packages using a Elastic Serverless project. More info at [section](#serverless-pipeline).
8+
- https://buildkite.com/elastic/integrations-publish: pipeline to publish the new versions of packages. More info at [section](#publish-packages).
9+
- https://buildkite.com/elastic/integrations-schedule-daily/: pipeline running every night to test packages in different scenarios. More info at [section](#daily-job).
10+
- https://buildkite.com/elastic/integrations-backport/: pipeline to create backport branches (just from UI). More info at [section](#backport-branches-pipeline).
11+
12+
## Pull Requests and pushes to specific branches
13+
14+
In every push to a Pull Request as well as commits to `main` or `backport-*` branches, this pipeline is triggered automatically: https://buildkite.com/elastic/integrations
15+
16+
This pipeline is in charge of testing the packages with a local Elastic stack.
17+
18+
Special comments that can be added in the Pull Request (by Elastic employees):
19+
- `/test` or `buildkite test this`: a new build is triggered.
20+
- `/test benchmark fullreport`: a new build is triggered creating a full benchmark report (it will be posted as a GitHub comment in the PR).
21+
22+
There are some environment variables that can be added into this pipeline:
23+
- **FORCE_CHECK_ALL**: If `true`, this forces the CI to check all packages even if those packages have no file updated/added/deleted. Default: `false`.
24+
- **STACK_VERSION**: Force the CI steps to spin up a specific Elastic stack version to run the tests. Default: unset.
25+
- **PUBLISH_COVERAGE_REPORTS**: If `true`, it enables reporting coverage reports.
26+
Currently, it is just set for the build triggered with the current major Elastic stack from the daily job. Default: `false`.
27+
28+
These environment variables can be defined in different locations:
29+
- In the [global `env` section](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.yml#L2).
30+
- In the case of the scheduled daily job here:
31+
- [Running tests with previous major Elastic stack](https://github.com/elastic/integrations/blob/5276ef63712f8f2311818770881688870e8422fe/.buildkite/pipeline.schedule-daily.yml#L21).
32+
- [Running tests with current major Elastic stack](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L32).
33+
34+
More details about this CI pipeline:
35+
36+
- Builds running in a Pull Request are canceled if a new commit is pushed to the PR branch.
37+
- Builds running from `main` or `backport-*` branches are finished even if new commits are merged.
38+
- As part of the Pull Requests, there are some benchmark files created and this data is pushed to the PR as a comment.
39+
- This CI pipeline tries to test the minimum packages possible::
40+
- In the Pull Request context, it is checked the files modified/added/deleted in the given PR:
41+
- Those packages with changes (`packages/*`) will be added to the list of packages to be tested.
42+
- If files outside `packages` folder are updated (e.g. `go.mod` , `.buildkite/*`), all packages are going to be tested. There are some exceptions to this, for instance the `.github/CODEOWNERS` file or `.docs/` folder ([files excluded](https://github.com/elastic/integrations/blob/376fc891a1e6c662b4ef1897b118044faf51e7bf/.buildkite/scripts/common.sh#L695)).
43+
- In branches context (`main` or `backport-*`):
44+
- The latest Buildkite build that finished successfully in that branch is retrieved, and all the file changes in the working copy between the changeset of that build and the merged commit are obtained.
45+
- Given all those changes, the packages selected to be tested follow the same rules as in the PR.
46+
- Container logs, as they could contain sensitive information, are uploaded to a private Google Bucket.
47+
- Packages are tested running the Elastic stack with the minimum Kibana version supported according to their manifest (`.conditions.kibana.version`). If a package defines a Kibana version that is not released yet, `elastic-package` will be using the SNAPSHOT version. This can be overridden if the STACK_VERSION variable is defined in the environment.
48+
In the following table, there are some examples:
49+
50+
| STACK_VERSION env | Kibana Condition Package | Released | Elastic stack run |
51+
| :---: | :---: | :---: | :---: |
52+
| "" | `^7.16.0 || ^8.0.0` | Yes | `7.16.0` |
53+
| "" | `^8.12.0` | Yes | `8.12.0` |
54+
| "" | `^8.14.1` | Yes | `8.14.1` |
55+
| "" | `^8.15.0` | No | `8.15.0-SNAPSHOT` |
56+
| `8.14.0` | `^8.13.0` | Yes | `8.14.0` |
57+
58+
If the STACK_VERSION environment variable is defined, just the packages supporting that stack version are tested. For instance:
59+
60+
| STACK_VERSION env | Kibana Condition Package | Tested |
61+
| :---: | :---: | :---: |
62+
| `7.17.0` | `^7.16.0 \|\| ^8.0.0` | Yes |
63+
| `7.17.0` | `^8.12.0` | No |
64+
| `8.12.0` | `^8.13.0` | No |
65+
| `8.14.0` | `^8.13.0` | Yes |
66+
67+
68+
## Publish packages
69+
70+
**Note**: Just available for Employees at Elastic.
71+
72+
For every Pull Request merged onto the `main` branch or any `backport-*` branches,
73+
this pipeline is triggered automatically https://buildkite.com/elastic/integrations-publish to publish new versions
74+
of the packages if any. These new versions are published and made available in https://epr.elastic.co
75+
76+
Environment variables that can be defined in this pipeline:
77+
- **DRY_RUN**: If `true`, packages will not be published. Default: `false`.
78+
- **SKIP_PUBLISHING**: If `true`, not run at all publish procedures. Default: `false`.
79+
80+
These environment variables can be defined:
81+
- At the [global `env` section](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.publish.yml#L3).
82+
- At the [specific buildkite publish step](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.publish.yml#L37).
83+
- In case of the [step from schedule-daily job](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L61)
84+
(it needs to be added the `env` dictionary).
85+
86+
## Serverless Pipeline
87+
88+
**Note**: Just available for Employees at Elastic.
89+
90+
Pipeline in charge of testing all packages with an Elastic Serverless project (https://buildkite.com/elastic/integrations-serverless).
91+
92+
This pipeline follows these steps:
93+
1. Create an Elastic Serverless project of a given type. Currently supported: Observability or Security
94+
1. For each package in the repository:
95+
1. Check if the package is supported in the given Elastic Serverless project.
96+
1. Packages can define which capabilities they need. Depending on those, the package could be available or not in the Serverless project, and therefore they will be tested or not.
97+
- [Capabilities defined for Observability projects](https://github.com/elastic/kibana/blob/40a2bdf651b0eabe5977367ad1b875e7581f0e31/config/serverless.oblt.yml#L145).
98+
- [Capabilities defined for Security capabilities](https://github.com/elastic/kibana/blob/40a2bdf651b0eabe5977367ad1b875e7581f0e31/config/serverless.security.yml#L77).
99+
1. There are some packages that are excluded in the Kibana configuration explicitly:
100+
- [Packages excluded in Observability](https://github.com/elastic/kibana/blob/7c5cf9c76e35925cba7e3bd57cc01d1fffae81a4/config/serverless.oblt.yml#L150).
101+
- [Packages excluded in Security](https://github.com/elastic/kibana/blob/7c5cf9c76e35925cba7e3bd57cc01d1fffae81a4/config/serverless.security.yml#L82).
102+
1. Runs test for all the selected packages:
103+
- Currently, [system tests are not run in Serverless](https://github.com/elastic/integrations/blob/5714f5665bbe3bc29b9e2444c6a94dbc2d5eebe9/.buildkite/scripts/common.sh#L803).
104+
- As in the pipeline for Pull Requests, container logs as they could contain sensitive information, they are uploaded to a private Google Bucket.
105+
1. Deletes the Elastic Serverless project.
106+
107+
Environment variables that can be defined in this pipeline:
108+
- **SERVERLESS_PROJECT**: Serverless project to be created to test packages. Default: observability.
109+
110+
This environment variable can be defined at:
111+
- At the [global `env` section](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.serverless.yml#L3).
112+
- In case of the schedule-daily job:
113+
- [trigger observability tests step](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L46).
114+
- [trigger security test step](https://github.com/elastic/integrations/blob/d6d99792b90838d18844f6df9343bc5f16130666/.buildkite/pipeline.schedule-daily.yml#L56).
115+
116+
117+
## Daily job
118+
119+
**Note**: Just available for Employees at Elastic.
120+
121+
Every night it is configured to run a daily job that will be in charge of testing all packages with different scenarios: https://buildkite.com/elastic/integrations-schedule-daily/
122+
123+
The schedule of this job can be checked [here](https://github.com/elastic/integrations/blob/5714f5665bbe3bc29b9e2444c6a94dbc2d5eebe9/catalog-info.yaml#L93).
124+
125+
The scenarios that are tested in this daily job are:
126+
127+
- Test packages with a local Elastic stack running the previous major version of the stack (7.17.X SNAPSHOT).
128+
- Triggered pipeline: https://buildkite.com/elastic/integrations
129+
- Test packages with a local Elastic stack running the latest major version of the stack (currently 8.X.Y-SNAPSHOT).
130+
- Triggered pipeline: https://buildkite.com/elastic/integrations
131+
- Test packages with an Elastic Serverless Observability project.
132+
- Triggered pipeline: https://buildkite.com/elastic/integration-serverless
133+
- Test packages with an Elastic Serverless Security project.
134+
- Triggered pipeline: https://buildkite.com/elastic/integration-serverless
135+
136+
Those tests that have failed in these scenarios will be reported as GitHub issues notifying the owner teams as defined in `.github/CODEOWNERS` file.
137+
138+
As part of this pipeline, it is also ensured that the latest versions of the packages merged into `main` branch
139+
have been published by triggering the pipeline https://buildkite.com/elastic/integrations-publish.
140+
141+
Each step triggering a new pipeline can be customized through environment variables. Environment variables that can
142+
be used in each pipeline are detailed in the corresponding sections of each pipeline.
143+
144+
145+
## Backport branches pipeline
146+
147+
**Note**: Just available for Employees at Elastic.
148+
149+
Releasing hotfixes from older versions of packages requires creating `backport-*` branches from specific commits in the `main` branch.
150+
In order to help with this task of creating these branches, there exists a pipeline that just can be triggered
151+
from the UI: https://buildkite.com/elastic/integrations-backport/
152+
153+
More information about this pipeline and how to create these hotfixes in:
154+
https://www.elastic.co/guide/en/integrations-developer/current/developer-workflow-support-old-package.html

‎packages/activemq/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: 1.5.0
3+
changes:
4+
- description: Add processor support for broker, queue and topic data streams.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/10409
27
- version: "1.4.0"
38
changes:
49
- description: ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.

‎packages/activemq/data_stream/broker/agent/stream/stream.yml.hbs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,8 @@ tags:
1313
{{/each}}
1414
{{#if ssl}}
1515
{{ssl}}
16+
{{/if}}
17+
{{#if processors}}
18+
processors:
19+
{{processors}}
1620
{{/if}}

‎packages/activemq/data_stream/broker/manifest.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,14 @@ streams:
1919
show_user: false
2020
default:
2121
- activemq-broker
22+
- name: processors
23+
type: yaml
24+
title: Processors
25+
multi: false
26+
required: false
27+
show_user: false
28+
description: >
29+
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the events are shipped. See [Processors](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html) for details.
2230
title: ActiveMQ broker
2331
description: Collect ActiveMQ broker metrics
2432
elasticsearch:

‎packages/activemq/data_stream/queue/agent/stream/stream.yml.hbs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,8 @@ tags:
1313
{{/each}}
1414
{{#if ssl}}
1515
{{ssl}}
16-
{{/if}}
16+
{{/if}}
17+
{{#if processors}}
18+
processors:
19+
{{processors}}
20+
{{/if}}

‎packages/activemq/data_stream/queue/manifest.yml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,14 @@ streams:
1919
show_user: false
2020
default:
2121
- activemq-queue
22+
- name: processors
23+
type: yaml
24+
title: Processors
25+
multi: false
26+
required: false
27+
show_user: false
28+
description: >
29+
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the events are shipped. See [Processors](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html) for details.
2230
title: ActiveMQ queue
2331
description: Collect ActiveMQ queue metrics
2432
elasticsearch:

‎packages/activemq/data_stream/topic/agent/stream/stream.yml.hbs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,8 @@ tags:
1313
{{/each}}
1414
{{#if ssl}}
1515
{{ssl}}
16-
{{/if}}
16+
{{/if}}
17+
{{#if processors}}
18+
processors:
19+
{{processors}}
20+
{{/if}}

0 commit comments

Comments
 (0)