Skip to content

IBM Z: Run DFLTCC tests on the self-hosted builder#1032

Merged
Dead2 merged 1 commit intozlib-ng:developfrom
iii-i:s390x-self-hosted
Oct 22, 2021
Merged

IBM Z: Run DFLTCC tests on the self-hosted builder#1032
Dead2 merged 1 commit intozlib-ng:developfrom
iii-i:s390x-self-hosted

Conversation

@iii-i
Copy link
Copy Markdown
Member

@iii-i iii-i commented Aug 9, 2021

Hello,

I've installed Github Actions Runner on the zlib-ng01 z15 VM. Currently it points to my zlib-ng fork, but you should be able to easily reconfigure it to point to zlib-ng itself using the instructions that are part of this PR. With this in place, we'll have better coverage of DFLTCC code in zlib-ng CI.

Best regards,
Ilya


  • Use the self-hosted builder instead of ubuntu-latest.
  • Drop qemu-related settings from DFLTCC configurations.
  • Install codecov only for the current user, since the self-hosted
    builder runs under a restricted non-root account.
  • Use actions/checkout@v2 for configure checks, since for some reason
    actions/checkout@v1 cannot find git on the self-hosted builder.
  • Update the testing section of the DFLTCC README.
  • Add the infrastructure code for the self-hosted builder.

@codecov
Copy link
Copy Markdown

codecov bot commented Aug 9, 2021

Codecov Report

Merging #1032 (37265e8) into develop (071e4da) will increase coverage by 2.13%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #1032      +/-   ##
===========================================
+ Coverage    79.44%   81.57%   +2.13%     
===========================================
  Files           85       85              
  Lines         8663     8667       +4     
  Branches      1400     1401       +1     
===========================================
+ Hits          6882     7070     +188     
+ Misses        1233     1057     -176     
+ Partials       548      540       -8     
Flag Coverage Δ
macos_clang 70.19% <ø> (+0.34%) ⬆️
macos_gcc 68.48% <ø> (ø)
ubuntu_clang 75.42% <ø> (ø)
ubuntu_clang_debug 75.54% <ø> (ø)
ubuntu_clang_inflate_allow_invalid_dist 75.15% <ø> (ø)
ubuntu_clang_inflate_strict 75.36% <ø> (ø)
ubuntu_clang_mmap 75.39% <ø> (ø)
ubuntu_clang_msan 75.42% <ø> (ø)
ubuntu_clang_pigz 36.16% <ø> (ø)
ubuntu_clang_pigz_no_optim 40.92% <ø> (ø)
ubuntu_clang_pigz_no_threads 35.66% <ø> (ø)
ubuntu_clang_reduced_mem 75.68% <ø> (ø)
ubuntu_gcc 74.82% <ø> (ø)
ubuntu_gcc_aarch64 76.71% <ø> (ø)
ubuntu_gcc_aarch64_compat_no_opt 74.49% <ø> (ø)
ubuntu_gcc_aarch64_no_acle 76.16% <ø> (ø)
ubuntu_gcc_aarch64_no_neon 76.48% <ø> (ø)
ubuntu_gcc_armhf 76.71% <ø> (ø)
ubuntu_gcc_armhf_compat_no_opt 74.45% <ø> (ø)
ubuntu_gcc_armhf_no_acle 76.99% <ø> (ø)
ubuntu_gcc_armhf_no_neon 77.27% <ø> (ø)
ubuntu_gcc_armsf 76.72% <ø> (ø)
ubuntu_gcc_armsf_compat_no_opt 74.45% <ø> (ø)
ubuntu_gcc_compat_no_opt 75.44% <ø> (ø)
ubuntu_gcc_compat_sprefix 74.73% <ø> (ø)
ubuntu_gcc_mingw_i686 0.00% <ø> (ø)
ubuntu_gcc_mingw_x86_64 0.00% <ø> (ø)
ubuntu_gcc_no_avx2 76.78% <ø> (ø)
ubuntu_gcc_no_ctz 77.67% <ø> (ø)
ubuntu_gcc_no_ctzll 77.34% <ø> (ø)
ubuntu_gcc_no_pclmulqdq 73.56% <ø> (ø)
ubuntu_gcc_no_sse2 76.04% <ø> (ø)
ubuntu_gcc_no_sse4 73.91% <ø> (ø)
ubuntu_gcc_o3 74.03% <ø> (ø)
ubuntu_gcc_osb 76.81% <ø> (ø)
ubuntu_gcc_pigz 36.50% <ø> (ø)
ubuntu_gcc_pigz_aarch64 38.60% <ø> (-0.07%) ⬇️
ubuntu_gcc_ppc 74.51% <ø> (ø)
ubuntu_gcc_ppc64 77.55% <ø> (ø)
ubuntu_gcc_ppc64le 76.01% <ø> (ø)
ubuntu_gcc_ppc_no_power8 77.31% <ø> (ø)
ubuntu_gcc_s390x 78.64% <ø> (+2.81%) ⬆️
ubuntu_gcc_sparc64 78.58% <ø> (ø)
ubuntu_gcc_sprefix 74.51% <ø> (ø)
win64_gcc ∅ <ø> (∅)
win64_gcc_compat_no_opt ∅ <ø> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
chunkset.c 54.16% <0.00%> (-4.93%) ⬇️
arch/s390/crc32-vx.c 100.00% <0.00%> (ø)
inflate.c 88.27% <0.00%> (+0.11%) ⬆️
arch/s390/dfltcc_common.c 90.00% <0.00%> (+0.34%) ⬆️
deflate_medium.c 87.97% <0.00%> (+0.63%) ⬆️
test/minideflate.c 60.55% <0.00%> (+1.11%) ⬆️
deflate_stored.c 96.55% <0.00%> (+1.14%) ⬆️
deflate.c 75.00% <0.00%> (+1.51%) ⬆️
uncompr.c 81.39% <0.00%> (+2.32%) ⬆️
compress.c 91.89% <0.00%> (+2.70%) ⬆️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 071e4da...37265e8. Read the comment docs.

@mtl1979
Copy link
Copy Markdown
Collaborator

mtl1979 commented Aug 9, 2021

Fails CI...

@iii-i
Copy link
Copy Markdown
Member Author

iii-i commented Aug 9, 2021

@mtl1979 That's expected - sorry, I should've been more explicit about this. The new builder is currently linked to my fork of zlib-ng, not to the main repo. A token is needed in order to link it to the main repo, which only the admin (@Dead2 I guess?) has access to. So he should either share the token with me, so that I could reconfigure the builder, or, preferably, reconfigure the builder himself - not only because I'm lazy, but also to make sure that the instructions I wrote are adequate.

@mtl1979
Copy link
Copy Markdown
Collaborator

mtl1979 commented Aug 9, 2021

@iii-i Nobody can review the full pull request until @Dead2 makes the necessary changes... For controversial changes, at least two long-time developers need to review.

@iii-i
Copy link
Copy Markdown
Member Author

iii-i commented Sep 8, 2021

@Dead2 ping :-) Could you take a look please?

@Dead2
Copy link
Copy Markdown
Member

Dead2 commented Sep 8, 2021

@iii-i I have not forgotten about this (or other PRs), it is just an extremely hectic period for me currently. Past 6 weeks I have barely been able to do any relaxing at all, and this will probably continue for a couple more weeks due to finishing up on the stuff currently going on plus we are moving offices starting next week. Hopefully things will be back to normal soon, and I can start doing the things I actually want to do again. 😄

In the mean time, anything small and approved by others I will probably merge, but anything bigger or things I think needs some extra scrutiny/testing from me, I am unfortunately unable to make the time for. Very sorry about that.

@iii-i iii-i force-pushed the s390x-self-hosted branch from 3de6604 to a4d9161 Compare October 18, 2021 22:52
@iii-i
Copy link
Copy Markdown
Member Author

iii-i commented Oct 18, 2021

* Use the self-hosted builder instead of ubuntu-latest.
* Drop qemu-related settings from DFLTCC configurations.
* Install codecov only for the current user, since the self-hosted
  builder runs under a restricted non-root account.
* Use actions/checkout@v2 for configure checks, since for some reason
  actions/checkout@v1 cannot find git on the self-hosted builder.
* Update the testing section of the DFLTCC README.
* Add the infrastructure code for the self-hosted builder.
@iii-i iii-i force-pushed the s390x-self-hosted branch from a4d9161 to 37265e8 Compare October 21, 2021 14:07
@Dead2
Copy link
Copy Markdown
Member

Dead2 commented Oct 21, 2021

@iii-i Looks like it works now. So, is it ready to merge?

@iii-i
Copy link
Copy Markdown
Member Author

iii-i commented Oct 21, 2021

@Dead2 Yes, I think so.

Regarding installing updates on the builder, once QEMU 6.2 is released, I will configure auto-updating https://github.com/multiarch/qemu-user-static there and send a corresponding PR.

@Dead2 Dead2 merged commit c762db1 into zlib-ng:develop Oct 22, 2021
Dead2 added a commit that referenced this pull request Dec 20, 2021
 - Fix hangs on macOS #1031
 - Fix minideflate write buffers being overwritten #1060
 - Fix deflateBound and compressBound returning too small size estimates #1071
 - Fix build problems when building outside of source dir #1049
 - Fix build problems on arm2-7 #1030
 - Fixed some compile warnings #1020 #1036 #1037 #1048
 - Improved posix memalign support #888
 - Improvements to testing #637 #1026 #1032 #1035 #1051 #1056 #1063 #1067
 - Improvements for integration into other projects #1022 #1042
 - Code style fixes #637 #1040 #1050 #1075
Dead2 added a commit that referenced this pull request Dec 20, 2021
 - Fix hangs on macOS #1031
 - Fix minideflate write buffers being overwritten #1060
 - Fix deflateBound and compressBound returning too small size estimates #1071
 - Fix build problems when building outside of source dir #1049
 - Fix build problems on arm2-7 #1030
 - Fixed some compile warnings #1020 #1036 #1037 #1048
 - Improved posix memalign support #888
 - Improvements to testing #637 #1026 #1032 #1035 #1051 #1056 #1063 #1067
 - Improvements for integration into other projects #1022 #1042
 - Code style fixes #637 #1040 #1050 #1075
@Dead2 Dead2 mentioned this pull request Dec 20, 2021
Dead2 added a commit that referenced this pull request Dec 20, 2021
 - Fix hangs on macOS #1031
 - Fix minideflate write buffers being overwritten #1060
 - Fix deflateBound and compressBound returning too small size estimates #1071
 - Fix build problems when building outside of source dir #1049
 - Fix build problems on arm2-7 #1030
 - Fixed some compile warnings #1020 #1036 #1037 #1048
 - Improved posix memalign support #888
 - Improvements to testing #637 #1026 #1032 #1035 #1051 #1056 #1063 #1067
 - Improvements for integration into other projects #1022 #1042
 - Code style fixes #637 #1040 #1050 #1075
Dead2 added a commit that referenced this pull request Dec 20, 2021
 - Fix hangs on macOS #1031
 - Fix minideflate write buffers being overwritten #1060
 - Fix deflateBound and compressBound returning too small size estimates #1049 #1071
 - Fix incorrect function declaration warning #1080
 - Fix build problems when building outside of source dir #1049
 - Fix build problems on arm2-7 #1030
 - Fixed some compile warnings #1020 #1036 #1037 #1048
 - Improved posix memalign support #888
 - Improvements to testing #637 #1026 #1032 #1035 #1049 #1051 #1056 #1063 #1067
 - Improvements for integration into other projects #1022 #1042
 - Code style fixes #637 #1040 #1050 #1075
Dead2 added a commit that referenced this pull request Dec 20, 2021
 - Fix hangs on macOS #1031
 - Fix minideflate write buffers being overwritten #1060
 - Fix deflateBound and compressBound returning too small size estimates #1049 #1071
 - Fix incorrect function declaration warning #1080
 - Fix build problems when building outside of source dir #1049
 - Fix build problems on arm2-7 #1030
 - Fixed some compile warnings #1020 #1036 #1037 #1048
 - Improved posix memalign support #888
 - Improvements to testing #637 #1026 #1032 #1035 #1049 #1051 #1056 #1063 #1067 #1079
 - Improvements for integration into other projects #1022 #1042
 - Code style fixes #637 #1040 #1050 #1075
Dead2 added a commit that referenced this pull request Dec 24, 2021
 - Fix hangs on macOS #1031
 - Fix minideflate write buffers being overwritten #1060
 - Fix deflateBound and compressBound returning too small size estimates #1049 #1071
 - Fix incorrect function declaration warning #1080
 - Fix build problems when building outside of source dir #1049
 - Fix build problems on arm2-7 #1030
 - Fixed some compile warnings #1020 #1036 #1037 #1048
 - Improved posix memalign support #888
 - Improvements to testing #637 #1026 #1032 #1035 #1049 #1051 #1056 #1063 #1067 #1079
 - Improvements for integration into other projects #1022 #1042
 - Code style fixes #637 #1040 #1050 #1075
@Dead2
Copy link
Copy Markdown
Member

Dead2 commented Jan 29, 2022

@iii-i QEMU 6.2 has been released, and GHA emailed me that we should update our runner yesterday. Any chance of getting that updated?

I am also contacting RedHat about getting a subscription so we can get system updates on that VM.. OIC, there are warnings but it is actually bypassed

@iii-i
Copy link
Copy Markdown
Member Author

iii-i commented Jan 31, 2022

I'll send a PR with the GHA upgrade (it's strange that the runner does not receive upgrade notifications from the server, but oh well).
With QEMU, we'll have to wait for a new https://github.com/multiarch/qemu-user-static release.

@Dead2
Copy link
Copy Markdown
Member

Dead2 commented Jan 31, 2022

@iii-i I don't think this is the reason it does not self-update for us, but the following was mentioned in the mail from Github.

Normally, self-hosted runners will auto-update during startup. When used in conjunction with the --ephemeral flag,
runner versions prior to 2.283.3 had an issue that would cause startup to fail during this auto-update process.
To ensure minimal disruption to your Actions workflows, we temporarily disabled automatic runner upgrades
for ephemeral runners 

I did not see any ephemeral flag in our config though, and I assume we would not need to use it anyway.

@iii-i
Copy link
Copy Markdown
Member Author

iii-i commented Jan 31, 2022

Ah, I see! We actually do use it. It's needed because it helps ensuring that we start with a clean slate each time.

@Dead2
Copy link
Copy Markdown
Member

Dead2 commented Jan 31, 2022

Mystery solved 💯
The mail also says they will re-enable auto-updates in March or something, so we should be good then 👍

@iii-i iii-i deleted the s390x-self-hosted branch April 20, 2022 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants