Migration: Jenkins -> Semaphore by Muen342 · Pull Request #1204 · confluentinc/kafka

Muen Zhang (Muen342) · 2024-05-02T13:36:19Z

Migrated jenkins to semaphore

Added Build, Compile, Validations, Publish, Upgrade step
compile-validate runs

./retry_zinc ./gradlew clean publishToMavenLocal build -x test --no-daemon --stacktrace -PxmlSpotBugsReport=true

check-scala-compatibility runs

./retry_zinc ./gradlew clean build -x test --no-daemon --stacktrace -PxmlSpotBugsReport=true -PscalaVersion=2.12

publishStep was migrated to

 ./gradlewAll -PmavenUrl=$mavenUrl -PkeepAliveMode=session uploadArchives` after setting mavenUrl to the vault secret of ` v1/ci/kv/gradle/artifactory_snapshots_settings` or `v1/ci/kv/gradle/artifactory_preview_release_settings

migrated downstream build to

sem-trigger -p $project -b $DOWNSTREAM_BRANCH_NAME -f .semaphore/semaphore.yml

where DOWNSTREAM_BRANCH_NAME is set with set_downstream_branch.sh

migrated the unit and integration tests to

'./gradlew 
                unitTest integrationTest --no-daemon --stacktrace --continue 
                -PtestLoggingEvents=started,passed,skipped,failed -PmaxParallelForks=4 
                -PignoreFailures=true -PmaxTestRetries=1 -PmaxTestRetryFailures=5'

uploaded results with

shopt -s globstar && test-results publish --name Test-Suite --trim-output-to 1024 --omit-output-for-passed **/build/test-results/**/TEST-*.xml

Service bot already ran with codeartifact in service.yml
https://semaphore.ci.confluent.io/jobs/b6fa3e0e-f2e3-4fae-b569-add5a54078a3#L112

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

cla-assistant · 2024-05-02T13:36:34Z

All committers have signed the CLA.

santhoshct · 2024-05-10T06:30:30Z

+    processing: parallel
+global_job_config:
+  env_vars:
+    - name: NO_OF_TEST_WORKER_NODES


why do you all of these env vars? It seems all are copied from ce-kafka. you can only use whichever you want and remove the rest

I'll remove the ones that aren't required

santhoshct · 2024-05-10T06:31:57Z

+              if [ "$NANO_VERSION" = "true" ] && [ "$RELEASE_JOB" = "false" ] && [ "$ENABLE_PUBLISH_ARTIFACTS" = "true" ]; then
+                . ci-tools ci-update-version
+              fi
+            - make compile-validate


do you need these checks. double check again with existing jenkins code.

In jenkins, it does do this, I just changed the command in the makefile to the exact same as was in the jenkinsfile
https://github.com/confluentinc/kafka/pull/1204/files#diff-e6ffa5dc854b843b3ee3c3c28f8eae2f436c2df2b1ca299cca1fa5982e390cf8L80-L81

same for check scala
https://github.com/confluentinc/kafka/pull/1204/files#diff-e6ffa5dc854b843b3ee3c3c28f8eae2f436c2df2b1ca299cca1fa5982e390cf8L72-L73

for CI update version, it's migrated from this
https://github.com/confluentinc/kafka/pull/1204/files#diff-e6ffa5dc854b843b3ee3c3c28f8eae2f436c2df2b1ca299cca1fa5982e390cf8L66-L68

santhoshct · 2024-05-10T06:32:09Z

+            - make compile-validate
+            - make check-scala-compatibility
+            - |
+              if [ "$PUBLISH" = "true" ] && [ "$SEMAPHORE_GIT_REF_TYPE" != "pull-request" ] && [ "$ENABLE_PUBLISH_ARTIFACTS" = "true" ]; then \


ensure these conditions are proper and relevant.

yes, it maps directly to

if (config.publish) { stage("Publish to artifactory") { if (!config.isReleaseJob && !config.isPrJob) { ciTool("ci-push-tag ${env.WORKSPACE} kafka") } if (config.isDevJob) { publishStep('artifactory_snapshots_settings') } else if (config.isPreviewJob) { publishStep('artifactory_preview_release_settings') } } }

from jenkinsfile

santhoshct · 2024-05-10T06:32:29Z

+              echo "PUBLISH: $PUBLISH, RELEASE_JOB: $RELEASE_JOB, SEMAPHORE_GIT_REF_TYPE: $SEMAPHORE_GIT_REF_TYPE, ENABLE_DOWNSTREAM_TRIGGER: $ENABLE_DOWNSTREAM_TRIGGER"
+              if [ "$PUBLISH" = "true" ] && [ "$RELEASE_JOB" = "false" ] && [ "$SEMAPHORE_GIT_REF_TYPE" != "pull-request" ] && [ "$ENABLE_DOWNSTREAM_TRIGGER" = "true" ]; then
+                for project in $DOWNSTREAM_PROJECTS; do
+                  sem-trigger -p $project -b $DOWNSTREAM_BRANCH_NAME -f .semaphore/semaphore.yml


is this required for open source kafka?

yes, here it shows that it uses the downstream build command
https://github.com/confluentinc/kafka/pull/1204/files#diff-e6ffa5dc854b843b3ee3c3c28f8eae2f436c2df2b1ca299cca1fa5982e390cf8L99-L109

santhoshct · 2024-05-13T05:55:07Z

+              echo $(( ($(date +%s%3N) - $SEMAPHORE_JOB_CREATION_TIME * 1000) )) > duration.txt
+              artifact push workflow duration.txt --destination $SEMAPHORE_JOB_INDEX-test-duration-$SEMAPHORE_WORKFLOW_ID || true
+
+  - name: Gradle Build


suggest making build as first block then followed by test.

santhoshct

LGTM

* migrated jenkins to semaphore * moved build to be before test in pipeline

Muen Zhang (Muen342) force-pushed the CPKAFKA-9988 branch from 7d97323 to f47416c Compare May 2, 2024 20:27

Muen Zhang (Muen342) changed the title ~~first test~~ Migrated jenkins to semaphore May 10, 2024

Muen Zhang (Muen342) force-pushed the CPKAFKA-9988 branch from 4b7e315 to c7a03c4 Compare May 10, 2024 05:47

Muen Zhang (Muen342) marked this pull request as ready for review May 10, 2024 05:47

Muen Zhang (Muen342) requested review from a team as code owners May 10, 2024 05:47

Muen Zhang (Muen342) force-pushed the CPKAFKA-9988 branch from c7a03c4 to 1a604a8 Compare May 10, 2024 06:08

santhoshct reviewed May 10, 2024

View reviewed changes

migrated jenkins to semaphore

822724a

Muen Zhang (Muen342) force-pushed the CPKAFKA-9988 branch from 1a604a8 to 822724a Compare May 10, 2024 15:14

Muen Zhang (Muen342) requested a review from santhoshct May 10, 2024 15:17

Muen Zhang (Muen342) changed the title ~~Migrated jenkins to semaphore~~ Migration: Jenkins -> Semaphore May 10, 2024

santhoshct reviewed May 13, 2024

View reviewed changes

moved build to be before test in pipeline

9976d61

Muen Zhang (Muen342) requested a review from santhoshct May 13, 2024 13:17

santhoshct approved these changes May 13, 2024

View reviewed changes

Parag Badani (pbadani) approved these changes May 13, 2024

View reviewed changes

Muen Zhang (Muen342) merged commit aaca303 into master May 13, 2024

Muen Zhang (Muen342) deleted the CPKAFKA-9988 branch May 13, 2024 14:08

Muen Zhang (Muen342) added a commit that referenced this pull request Jun 26, 2024

Migration: Jenkins -> Semaphore (#1204)

417d2a6

* migrated jenkins to semaphore * moved build to be before test in pipeline

Muen Zhang (Muen342) added a commit that referenced this pull request Jun 26, 2024

Migration: Jenkins -> Semaphore (#1204)

90a3216

* migrated jenkins to semaphore * moved build to be before test in pipeline

Muen Zhang (Muen342) added a commit that referenced this pull request Jun 26, 2024

Migration: Jenkins -> Semaphore (#1204)

66538ef

* migrated jenkins to semaphore * moved build to be before test in pipeline

Conversation

Muen Zhang (Muen342) commented May 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Committer Checklist (excluded from commit message)

Uh oh!

cla-assistant Bot commented May 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

santhoshct left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Muen Zhang (Muen342) commented May 2, 2024 •

edited

Loading

cla-assistant Bot commented May 2, 2024 •

edited

Loading