[airflow] Make Airflow package GA by mykola-elastic · Pull Request #15287 · elastic/integrations

mykola-elastic · 2025-09-11T11:50:33Z

Work done in this PR so far (in progress):

bump version to 1.0.0
update ecs version to 8.17.0 [meta] Upgrade integrations to ECS 8.17 #11952
update Readme (+ check that links work)
test with Airflow 2.4.3 and 3.0.6
test with Elastic Stack 8.13, 8.17, 9.0, 9.1
deal with undefined on the integration configuration page
updated system tests (filled metrics log file with 100 rows of data recorded from Airflow 3.0)
remake dashboard

Proposed commit message

See title.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

…manifest.yml

mykola-elastic · 2025-09-15T15:25:36Z

@alaudazzi PR is still in progress, but README can already be reviewed. Can you add reviewing the README file to your task list?

elastic-sonarqube · 2025-09-29T10:45:56Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

alaudazzi

Left a very minor suggestion, otherwise LGTM.

packages/airflow/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

mykola-elastic · 2025-10-06T07:08:07Z

Original dashboard for comparison (BEFORE):

AFTER:

CC @daniela-elastic

mykola-elastic · 2025-10-07T10:19:28Z

Fixed the CI and marked ready for review
CC @ishleenk17

ishleenk17 · 2025-10-08T04:41:46Z

packages/airflow/manifest.yml

 conditions:
  kibana:
-    version: "^8.13.0 || ^9.0.0"
+    version: "^8.17.0 || ^9.0.0"


Why the kibana version bump ?

I used newer Kibanas to remake the dashboard from scratch, so I dropped those older ones. If the 8.13-8.16 needed, I can backport the dashboard (remake it in older kibana).

We also can only do bugfixes to beats 8.17 and newer, if something arises in future SDHs

Do we need to support them?

If there is no specific requirement to upgrade the version, we should be ok with keeping it as it is. So that any future bugs reach as many stacks as possible.

What do you mean by dashboard backport ? You can make dashboard in any stack version, just check that all panels work as it is in older stacks too.

I made and exported the dashboard from 8.17, installing it in 8.13 fails

Remade the dashboard in 8.13. Now it works on all ^8.13 || ^9.0 (verified)

packages/airflow/manifest.yml

ishleenk17 · 2025-10-08T04:46:01Z

packages/airflow/docs/README.md

-## Data streams
+## Compatibility
+
+The Airflow package is tested with Airflow `2.4.0` and `3.0.6`. It should work with any `2.*` and `3.*` versions.


Hoping no backward compatibility breakages here between 2.0 and 3.0 ?

no, I am using only metrics available in 2.0

Are there new metrics added in 3.0 ?

yes, there are few metrics added in some minor versions too

Do we need to support them here ?
Since we are writing we support 3.x but are not supporting the newer metrics ?
This GA activity is also actually doing the service version upgrade activity as well :)
Which is good if we can address the question if we want to include the nwer metrics as well as we are clamining support for 3.x here.

The was no 3.0 when the package was initially created. I remade the dashboard and also tested it with 3.0 too. So, the support means that nothing breaks.
So, is this a phrasing problem?

In service version uograde when we say that we support a newer service version also, there are 2 aspects we check:

Nothing existing breaks

We support any new important metrics that were added between the 2 service versions . In this case 2.0, 3.0.
If we think there are newer metrics added and we are not taking care of that in this PR. We should keep support as 2.0 and add support for new metrics as a task under service version upgrade process.

cc: @lalit-satapathy

if we add new metrics to the dashboard, then the panels using them will break if using older airflows

@mykola-elastic : Since there are newer metrics added in 3.x and we are not taking care of them as part of GA. Lets keep compatibility for 2.x only.
We will move to 3.x as part of the service version upgrade, when we properly adhere to the process and handle the new metrics properly.

updated readme: bcdeda8

ok?

ishleenk17 · 2025-10-08T04:50:17Z

In dashboard, the DAG BagSize, DAG's on the y axis is the DAG count ?

mykola-elastic · 2025-10-08T04:58:19Z

In dashboard, the DAG BagSize, DAG's on the y axis is the DAG count ?

Yes, it is measured in DAGs. I didn't include the word "Count" as it is considered redundant in the new dashboard guidelines

ishleenk17 · 2025-10-09T05:07:12Z

packages/airflow/_dev/build/docs/README.md

-## Data streams
+## Compatibility
+
+The Airflow package is tested with Airflow `2.4.0` and `3.0.6`. It will work with any `2.*` and `3.*` versions, but uses a smaller subset of metrics to be backwards-compatible with Apache Airflow 2.0.


Lets just mention compatibility with 2.x. Lets remove mention of 3.x completely from here.
Smaller subset line will be confusing for users.

So basically use what was existing before in compatibility.

done, "The Airflow package is tested with Airflow 2.4.3. It should work with any 2.* version."

elasticmachine · 2025-10-09T05:31:43Z

💚 Build Succeeded

Buildkite Build
Commit: d65ba1b

History

💚 Build #32485 succeeded 8f39736
💚 Build #32417 succeeded c3ee7b0
💚 Build #32395 succeeded 67184fb
💔 Build #32304 failed f2b07c9
💔 Build #32287 failed ea096fb
💔 Build #32283 failed 6880cec

cc @mykola-elastic

elastic-vault-github-plugin-prod · 2025-10-09T06:02:27Z

Package airflow - 1.0.0 containing this change is available at https://epr.elastic.co/package/airflow/1.0.0/

small restructure of readme

d18380c

mykola-elastic self-assigned this Sep 11, 2025

mykola-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Integration:airflow Airflow labels Sep 11, 2025

mykola-elastic added 4 commits September 11, 2025 14:56

remove ambiguous sentence

231da48

add version 1.0.0

53b67e5

remove "experimental"

138ca10

update ECS to 8.17.0

48bf43c

mykola-elastic force-pushed the airflow-ga branch from 8f13395 to 48bf43c Compare September 11, 2025 11:57

update compatibility section

7ec7ced

mykola-elastic mentioned this pull request Sep 10, 2025

[meta] Upgrade integrations to ECS 8.17 #11952

Open

mykola-elastic changed the title ~~[airflow] Make Airflow package GA~~ [DRAFT] [airflow] Make Airflow package GA Sep 12, 2025

move data_stream parameters from package manifest.yml to data_stream …

4055571

…manifest.yml

mykola-elastic requested a review from alaudazzi September 15, 2025 15:25

mykola-elastic added 2 commits September 25, 2025 14:50

use ttl 60s instead of default (30s)

3c1cac1

Merge branch 'main' into airflow-ga

701ff0e

alaudazzi approved these changes Oct 1, 2025

View reviewed changes

packages/airflow/_dev/build/docs/README.md Outdated Show resolved Hide resolved

mykola-elastic and others added 2 commits October 2, 2025 08:17

apply suggestion (README)

a6404a0

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

fix readme

6849d71

mykola-elastic requested a review from daniela-elastic October 6, 2025 07:13

mykola-elastic added 2 commits October 6, 2025 10:30

update dashboard

a942edc

use real recording from airflow 3.0 for tests

8fb4290

mykola-elastic changed the title ~~[DRAFT] [airflow] Make Airflow package GA~~ [airflow] Make Airflow package GA Oct 6, 2025

mykola-elastic added 2 commits October 6, 2025 11:08

update minimal kibana.version from 8.13 to 8.17

6880cec

update dashboard to support kibana 8.17

ea096fb

mykola-elastic force-pushed the airflow-ga branch from 5b98081 to ea096fb Compare October 6, 2025 08:44

Merge branch 'main' into airflow-ga

f2b07c9

andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Oct 6, 2025

daniela-elastic approved these changes Oct 6, 2025

View reviewed changes

mykola-elastic added 3 commits October 7, 2025 07:41

dashboard: spell out DAG acronym fully the first time

8446b2e

Merge branch 'main' into airflow-ga

bcdf203

update screenshot

67184fb

mykola-elastic marked this pull request as ready for review October 7, 2025 04:46

mykola-elastic requested a review from a team as a code owner October 7, 2025 04:46

mykola-elastic requested a review from ishleenk17 October 7, 2025 08:06

mykola-elastic added 2 commits October 7, 2025 11:42

fix broken view

a39bcf4

remove nonexistent fields

c3ee7b0

ishleenk17 reviewed Oct 8, 2025

View reviewed changes

mykola-elastic requested a review from ishleenk17 October 8, 2025 05:37

mykola-elastic added 2 commits October 8, 2025 11:05

make the dashboard work on Elastic Stack 8.13

8f39736

update README: compatibility

bcdeda8

mykola-elastic force-pushed the airflow-ga branch from 049f7d5 to bcdeda8 Compare October 9, 2025 05:05

ishleenk17 reviewed Oct 9, 2025

View reviewed changes

remove mentions of airflow 3.0

d65ba1b

ishleenk17 approved these changes Oct 9, 2025

View reviewed changes

mykola-elastic merged commit ce98afa into elastic:main Oct 9, 2025
8 checks passed

agithomas pushed a commit to agithomas/integrations that referenced this pull request Oct 30, 2025

[airflow] Make Airflow package GA (elastic#15287)

26e41fc

tehbooom pushed a commit to tehbooom/integrations that referenced this pull request Nov 19, 2025

[airflow] Make Airflow package GA (elastic#15287)

8e03896

Conversation

mykola-elastic commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

mykola-elastic commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elastic-sonarqube bot commented Sep 29, 2025

Quality Gate passed

Uh oh!

alaudazzi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mykola-elastic commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mykola-elastic commented Oct 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mykola-elastic Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mykola-elastic Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ishleenk17 commented Oct 8, 2025

Uh oh!

mykola-elastic commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 9, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 9, 2025

Uh oh!

Reviewers

Assignees

Labels

mykola-elastic commented Sep 11, 2025 •

edited

Loading

mykola-elastic commented Sep 15, 2025 •

edited

Loading

mykola-elastic commented Oct 6, 2025 •

edited

Loading

mykola-elastic Oct 8, 2025 •

edited

Loading

mykola-elastic Oct 9, 2025 •

edited

Loading