Skip to content

Fix JDK matrix pipeline after configurable it split#17461

Merged
dliappis merged 2 commits intoelastic:mainfrom
dliappis:fix-jdk-matrix-pipeline-after-configurable-it-split
Apr 8, 2025
Merged

Fix JDK matrix pipeline after configurable it split#17461
dliappis merged 2 commits intoelastic:mainfrom
dliappis:fix-jdk-matrix-pipeline-after-configurable-it-split

Conversation

@dliappis
Copy link
Copy Markdown
Contributor

@dliappis dliappis commented Apr 1, 2025 •
edited
Loading

Release notes

[rn:skip]

What does this PR do?

PR #17219 introduced configurable split quantities for IT tests, which resulted in broken JDK matrix pipelines (e.g. as seen via the elastic internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.

)

This PR fixes the above problem.

Why is it important/What is the impact to the user?

Restores the functionality of the JDK (Windows/Linux) matrix pipelines.

How to test this PR locally

Linux

BUILDKITE_PIPELINE_NAME="Logstash Linux JDK matrix pipeline" MATRIX_OSES="ubuntu-22.04" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

produces:

Output from generate-steps for Linux 
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
steps:
  - group: ubuntu-22.04/adoptiumjdk_21
    key: ubuntu-22_04_adoptiumjdk_21
    steps:
      - label: Initialize annotation
        key: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        command: |-
          buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 "### Group: ubuntu-22.04 / adoptiumjdk_21
          | **Status** | **Test** |
          | --- | ----|
          "
      - label: Java Unit Test
        key: ubuntu-22_04_adoptiumjdk_21-java-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport ENABLE_SONARQUBE=\"false\"\nci/unit_tests.sh java\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Java Unit Test |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Java Unit Test |\n\"\nfi\n      "
      - label: Ruby Unit Test
        key: ubuntu-22_04_adoptiumjdk_21-ruby-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/unit_tests.sh ruby\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Ruby Unit Test |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Ruby Unit Test |\n\"\nfi\n      "
      - label: Integration Tests - 1/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-1-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 0 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 1/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 1/3 |\n\"\nfi\n      "
      - label: Integration Tests - 2/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-2-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 1 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 2/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 2/3 |\n\"\nfi\n      "
      - label: Integration Tests - 3/3
        key: ubuntu-22_04_adoptiumjdk_21-integration-tests-3-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nci/integration_tests.sh split 2 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | Integration Tests - 3/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | Integration Tests - 3/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 1/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-1-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 0 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 1/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 1/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 2/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-2-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 1 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 2/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 2/3 |\n\"\nfi\n      "
      - label: IT Persistent Queues - 3/3
        key: ubuntu-22_04_adoptiumjdk_21-it-persistent-queues-3-of-3
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nexport FEATURE_FLAG=persistent_queues\nci/integration_tests.sh split 2 3\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | IT Persistent Queues - 3/3 |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | IT Persistent Queues - 3/3 |\n\"\nfi\n      "
      - label: x-pack unit tests
        key: ubuntu-22_04_adoptiumjdk_21-x-pack-unit-test
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/unit_tests.sh\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack unit tests |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack unit tests |\n\"\nfi\n      "
      - label: x-pack integration
        key: ubuntu-22_04_adoptiumjdk_21-x-pack-integration
        depends_on: ubuntu-22.04-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-ubuntu-22.04
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        retry:
          automatic:
            - limit: 3
        command: "\n#!/usr/bin/env bash\nset -euo pipefail\n\n# unset generic JAVA_HOME\nunset JAVA_HOME\n\n# LS env vars for JDK matrix tests\nexport BUILD_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport RUNTIME_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\nexport LS_JAVA_HOME=/opt/buildkite-agent/.java/adoptiumjdk_21\n\nexport PATH=\"/opt/buildkite-agent/.rbenv/bin:/opt/buildkite-agent/.pyenv/bin:$PATH\"\neval \"$(rbenv init -)\"\n\n# temporarily disable immediate failure on errors, so that we can update the BK annotation\nset +eo pipefail\n\nx-pack/ci/integration_tests.sh\n        \nif [[ $$? -ne 0 ]]; then\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-failed: | x-pack integration |\n\"\n  exit 1\nelse\n    buildkite-agent annotate --style=info --context=ubuntu-22_04_adoptiumjdk_21 --append \"| :bk-status-passed: | x-pack integration |\n\"\nfi\n      "

Windows

BUILDKITE_PIPELINE_NAME="Logstash Windows JDK matrix pipeline" MATRIX_OSES="windows-2025" MATRIX_JDKS="adoptiumjdk_21" python3 scripts/jdk-matrix-tests/generate-steps.py | yq .

results in the following output:

# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
steps:
  - group: windows-2025/adoptiumjdk_21
    key: windows-2025_adoptiumjdk_21
    steps:
      - label: Initialize annotation
        key: windows-2025-adoptiumjdk_21-initialize-annotation
        command: |-
          buildkite-agent annotate --style=info --context=windows-2025_adoptiumjdk_21 "### Group: windows-2025 / adoptiumjdk_21
          | **Status** | **Test** |
          | --- | ----|
          "
      - label: Java Unit Test
        key: windows-2025_adoptiumjdk_21-java-unit-test
        depends_on: windows-2025-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-windows-2025
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        artifact_paths:
          - build_reports.zip
        retry:
          automatic:
            - limit: 3
        command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Java Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 java\" -Annotate\n        "
      - label: Ruby Unit Test
        key: windows-2025_adoptiumjdk_21-ruby-unit-test
        depends_on: windows-2025-adoptiumjdk_21-initialize-annotation
        agents:
          provider: gcp
          imageProject: elastic-images-prod
          image: family/platform-ingest-logstash-multi-jdk-windows-2025
          machineType: n2-standard-4
          diskSizeGb: 200
          diskType: pd-ssd
        artifact_paths:
          - build_reports.zip
        retry:
          automatic:
            - limit: 3
        command: ".\\\\.buildkite\\\\scripts\\\\jdk-matrix-tests\\\\launch-command.ps1 -JDK \"adoptiumjdk_21\" -StepNameHuman \"Ruby Unit Test\" -AnnotateContext \"windows-2025_adoptiumjdk_21\" -CIScript \".\\\\ci\\\\unit_tests.ps1 ruby\" -Annotate\n        "

Related issues

PR #17219

@dliappis
Fix JDK matrix pipeline after configurable it split
9bdcd7a 
PR elastic#17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem.
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Apr 1, 2025

This pull request does not have a backport label. Could you fix it @dliappis? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
  • backport-8.x is the label to automatically backport to the 8.x branch.
  • If no backport is necessary, please add the backport-skip label

dliappis
dliappis commented Apr 1, 2025
View reviewed changes
.buildkite/scripts/jdk-matrix-tests/generate-steps.py
def pq_integration_test_parts(self, parts) -> list[JobRetValues]:
return list(map(lambda idx: pq_integration_tests(self, idx+1, parts), range(parts))
def pq_integration_test_parts(self, parts) -> list[partial[JobRetValues]]:
return [partial(self.pq_integration_tests, part=idx+1, parts=parts) for idx in range(parts)]
Copy link
Copy Markdown
Contributor Author

@dliappis dliappis Apr 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(links below are Elastic internal)

Linux tests using this PR: https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/449
Windows tests using this PR: https://buildkite.com/elastic/logstash-windows-jdk-matrix-pipeline/builds/398

@dliappis dliappis requested a review from yaauie April 1, 2025 09:28
@dliappis dliappis self-assigned this Apr 1, 2025
@dliappis dliappis added ci backport-8.17 Automated backport with mergify backport-9.0 Automated backport to the 9.0 branch with mergify backport-8.16 Automated backport with mergify backport-8.x Automated backport to the 8.x branch with mergify backport-8.18 Automated backport with mergify labels Apr 1, 2025
@dliappis dliappis marked this pull request as ready for review April 1, 2025 09:29
@dliappis
Copy link
Copy Markdown
Contributor Author

dliappis commented Apr 1, 2025

For reviewers: please advise regarding backport to 9.0, is this ok?

@jsvd
Copy link
Copy Markdown
Member

jsvd commented Apr 1, 2025

Sorry we missed this @dliappis , @andsel anything we want to take from this PR to add to the merged #17462?

@dliappis
Copy link
Copy Markdown
Contributor Author

dliappis commented Apr 1, 2025

Sorry we missed this @dliappis , @andsel anything we want to take from this PR to add to the merged #17462?

Thanks @jsvd / @andsel for the quick comment!

This purely boils down to preference/readability/more idiomatic Python; both

as things are now after #17462

        return list(map(lambda idx: integration_tests(self, idx+1, parts), range(parts))

or my suggestion

        return [partial(self.integration_tests, part=idx+1, parts=parts) for idx in range(parts)]

are identical in functionality.

Personally I find list comprehensions more concise and readable (as mentioned in the above official Python docs) as well as partial functions also a very readable way to create a new function from an existing one with some parts of the arguments preset, but this is matter of personal preference.

Therefore feel free to close this PR if you prefer.

@andsel
Copy link
Copy Markdown
Member

andsel commented Apr 1, 2025

@jsvd I'm not able to express a preference here, it's matter of returning list of JobRetValues to list of partial functions that return JobRetValues. We could resolve conflicts and accept this. I trust @dliappis Python taste

@elastic-sonarqube
Copy link
Copy Markdown

elastic-sonarqube bot commented Apr 1, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link
Copy Markdown

elasticmachine commented Apr 1, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

cc @dliappis

@dliappis dliappis requested a review from andsel April 7, 2025 13:03
andsel
andsel approved these changes Apr 7, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@andsel andsel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dliappis dliappis merged commit b9469e0 into elastic:main Apr 8, 2025
9 checks passed
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @andsel @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
e909fe1 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @andsel @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
fb7e0ec 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @andsel @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
9d5e282 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @andsel @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
ade0d37 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
mergify bot pushed a commit that referenced this pull request Apr 8, 2025
@dliappis @andsel @mergify
Fix JDK matrix pipeline after configurable it split (#17461)
181c52a 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)
@mergify mergify bot mentioned this pull request Apr 8, 2025
Merged
@dliappis
Copy link
Copy Markdown
Contributor Author

dliappis commented Apr 8, 2025 •
edited
Loading

Merged and successfully tested (elastic internal links: linux / windows)

dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17514)
fcf313a 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17510)
d136e50 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17513)
e0d5464 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17512)
cde5786 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
dliappis added a commit that referenced this pull request Apr 8, 2025
@mergify @dliappis
Fix JDK matrix pipeline after configurable it split (#17461) (#17511)
5f475f0 
PR #17219 introduced configurable split quantities for IT tests, which
resulted in broken JDK matrix pipelines (e.g. as seen via the elastic
internal link:
https://buildkite.com/elastic/logstash-linux-jdk-matrix-pipeline/builds/444

reporting the following error

```
  File "/buildkite/builds/bk-agent-prod-k8s-1743469287077752648/elastic/logstash-linux-jdk-matrix-pipeline/.buildkite/scripts/jdk-matrix-tests/generate-steps.py", line 263
    def integration_tests(self, part: int, parts: int) -> JobRetValues:
    ^^^
SyntaxError: invalid syntax
There was a problem rendering the pipeline steps.
Exiting now.
```
)

This commit fixes the above problem, which was already fixed in #17642, using a more
idiomatic way.

Co-authored-by: Andrea Selva <selva.andre@gmail.com>
(cherry picked from commit b9469e0)

Co-authored-by: Dimitrios Liappis <dimitrios.liappis@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yaauie yaauie yaauie approved these changes

@andsel andsel andsel approved these changes

Assignees

@dliappis dliappis

Labels

backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify backport-8.18 Automated backport with mergify backport-9.0 Automated backport to the 9.0 branch with mergify ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@dliappis @jsvd @andsel @elasticmachine @yaauie