Use Maistra proxy based on CentOS Stream 8 in integration tests by jewertow · Pull Request #609 · maistra/istio

jewertow · 2022-09-09T21:38:54Z

Istio has been using its own tool to build container images for some time. This tool is docker-builder.
Unfortunately docker-builder does not support RUN command, so we cannot install iptables and openssl packages in proxyv2 with RUN dnf install ....

To work around this limitation, I built an image with necessary packages, pushed to my registry and used it as the base image for proxyv2.

Image quay.io/jewertow/base-maistra-proxyv2:2.3 was built from the following Dockerfile:

FROM quay.io/centos/centos:stream8

RUN dnf -y upgrade --refresh --nobest && \
    dnf -y install iptables iproute openssl && \
    dnf -y clean all

I think that it does not make sense to extend docker-builder to support RUN, because it's too complicated (if possible)
and we just have to build our official base image and use it as I did with my image.

Update:

I figured out that we can disable building container images with crane by setting ISTIO_DOCKER_BUILDER=docker, so I updated images to use CentOS Stream 8 as a base image and added image app_sidecar_centos_stream_8 to test VMs.

Signed-off-by: Jacek Ewertowski jewertow@redhat.com

jwendell

Not sure about this.

I have experimented with this a week ago: #604 and I think it worked...

Also, please align with @bmangoen and hisr PR: #606

jewertow · 2022-09-12T10:13:18Z

Your PR didn't work. Look at the log - everything failed. You can find there the following warnings:

2022-09-02T13:29:02.519537Z	warn	dockerfile	Skipping RUN: [dnf -y upgrade --refresh --nobest &&     dnf -y install iptables iproute openssl &&     dnf -y clean all]
2022-09-02T13:29:02.521035Z	warn	dockerfile	Skipping RUN: [dpkg -i /tmp/istio-sidecar.deb && rm /tmp/istio-sidecar.deb]

And as you can see in one of istio-init logs, iptables-restore did not exist in the proxy container.

jwendell · 2022-09-12T11:01:18Z

hmm, I see. Is it possible to install the libssl1 in the [upstream] base image? If yes, then we don't need a special Dockerfile based on centos. We can use the same upstream mechanism without any changes.

In the past we needed this image based on centos because the envoy binary built on centos was not compatible with the libc present on Ubuntu. It looks like it's not the case anymore. I was able to run our Envoy on this base image (which is derived from Ubuntu). The problem now is that our Envoy relies on the presence of the /lib64/libssl.so.1.1 library, which is provided by the Ubuntu package libssl1, which is not installed by default.

If there's no way to have this package installed in the base image (and since it's not possible to have RUN apt install anymore), I suggest you add a new target as a dependency for the make docker target .i.e, that will run first, that will create a new base image, let's say, maistra-base which is merely the Dockerfile you created this custom base image. Then, for every run, we first create this custom base image, and proxy is based on that. In other words, we would use your solution, but baking it at build-time instead of relying in a pre-built image hosted somewhere.

jewertow · 2022-09-14T11:50:23Z

/test gencheck

jewertow · 2022-09-14T11:50:31Z

/test integration-servicemesh

jewertow · 2022-09-14T11:50:42Z

/test integration-telemetry

Signed-off-by: Jacek Ewertowski <jewertow@redhat.com>

jewertow · 2022-09-15T15:52:24Z

@jwendell thanks for your advice. I was trying to run tests on Ubuntu, but currently it provides libssl.so.3.0 by default. I was trying to find a workaround to install OpenSSL 1.1.1, e.g. using repositories from older releases, but I had some other issues, so I gave up in favor of bringing an image based on CentOS Stream 8.

jwendell · 2022-09-15T16:10:49Z

pkg/test/framework/components/echo/config.go

+	Rockylinux8   VMDistro = "Centos8"
+	CentosStream8 VMDistro = "CentosStream8"
+
+	DefaultVMDistro = CentosStream8


Is there any reason to have a centos8 image?

Nope, it is not used, but we don't use other images as well. I didn't remove them to not increase the scope of this PR.

jwendell · 2022-09-15T16:11:16Z

prow/integ-suite-kind.sh


 export FAST_VM_BUILDS=true
-export ISTIO_DOCKER_BUILDER=crane
+export ISTIO_DOCKER_BUILDER=docker


I guess this is the change that allows you have have RUN commands in the proxy dockerfile?

Yes, you're right. As you can see, I updated the PR description and mentioned about this variable.

jwendell · 2022-09-15T16:12:25Z

tests/integration/security/egress_gateway_origination_test.go

 // TestSimpleTlsOrigination test SIMPLE TLS mode with TLS origination happening at Gateway proxy
 // It uses CredentialName set in DestinationRule API to fetch secrets from k8s API server
 func TestSimpleTlsOrigination(t *testing.T) {
+	// https://issues.redhat.com/browse/OSSM-755


nit: Put a FIXME: prefix to those comments? Perhaps it makes it easier for us in the future to locate them?

jwendell · 2022-09-15T16:15:17Z

tests/integration/security/egress_gateway_origination_test.go

 // It uses CredentialName set in DestinationRule API to fetch secrets from k8s API server
 func TestSimpleTlsOrigination(t *testing.T) {
+	// https://issues.redhat.com/browse/OSSM-755
+	t.Skip("https://github.com/istio/istio/0")


I think the format is https://github.com/istio/istio/issues/0

Thanks, I fixed it.

Signed-off-by: Jacek Ewertowski <jewertow@redhat.com>

jewertow · 2022-09-15T18:38:34Z

tests/integration/telemetry/stats/prometheus/wasm/stats_wasm_filter_test.go

 func TestMain(m *testing.M) {
 	framework.NewSuite(m).
+		// FIXME: https://issues.redhat.com/browse/OSSM-2000
+		Skip("https://github.com/istio/istio/issues/0").


It turned out that we have to disable whole suite. It works on my local machine, but does not on prow. I was trying to find out what's wrong there, but it seems to be a more complicated issue. I will solve it later.

yxun · 2022-09-15T18:48:34Z

pilot/docker/Dockerfile.proxyv2


+# hadolint ignore=DL3041
+RUN dnf -y upgrade --refresh --nobest && \
+    dnf -y install iptables iproute openssl && \


Do we need to install a specific versoin of openssl or just the latest one ?

We need OpenSSL 1.1.1 and our istio-proxy expects to find libssl.so.1.1. In CentOS Stream 8 the package openssl is exactly this one that we expect. I don't think that this package will be replaced by another version, because the next one is 3.0.0 which provides breaking API and ABI and upgrading from 1.1.1 to 3.0.0 is a big challenge, so it will probably not happen.

jewertow · 2022-09-15T19:52:12Z

/retest

jewertow · 2022-09-15T20:32:52Z

/test unit

* OSSM-2378: Use Maistra proxy CentOS Stream 8 img For integration tests. Based on (#609) * Build container images with docker Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Use maistra proxy and CentOS Stream 8 as a base image for proxyv2 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Test VMs using CentOS Stream 8 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip failing security tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip tests suite telemetry.prometheus.wasm Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * fix(tests): enables fixed integration tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Jacek Ewertowski <jewertow@redhat.com>

* OSSM-2378: Use Maistra proxy CentOS Stream 8 img For integration tests. Based on (maistra#609) * Build container images with docker Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Use maistra proxy and CentOS Stream 8 as a base image for proxyv2 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Test VMs using CentOS Stream 8 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip failing security tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip tests suite telemetry.prometheus.wasm Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * fix(tests): enables fixed integration tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Jacek Ewertowski <jewertow@redhat.com>

* OSSM-2378: Use Maistra proxy CentOS Stream 8 img (#713) * OSSM-2378: Use Maistra proxy CentOS Stream 8 img For integration tests. Based on (#609) * Build container images with docker Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Use maistra proxy and CentOS Stream 8 as a base image for proxyv2 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Test VMs using CentOS Stream 8 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip failing security tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip tests suite telemetry.prometheus.wasm Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * fix(tests): enables fixed integration tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Jacek Ewertowski <jewertow@redhat.com> * Use our proxy (#801) Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * OSSM-5958: Remove wasm extensions (#937) * Do not rely on WASM extensions Since maistra/proxy#253 * Remove wasm extensions from EnvoyFilters Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Disable tests for WASM extensions Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Remove commented code Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> --------- Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Jonh Wendell <jwendell@redhat.com> * Disable failing tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Fix TestTrustDomainValidation - change expected TLS error to make it work with OpenSSL Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> --------- Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Bartosz Majsak <bartosz.majsak@gmail.com> Co-authored-by: Jonh Wendell <jonh.wendell@redhat.com> Co-authored-by: Jonh Wendell <jwendell@redhat.com>

* OSSM-2378: Use Maistra proxy CentOS Stream 8 img (maistra#713) * OSSM-2378: Use Maistra proxy CentOS Stream 8 img For integration tests. Based on (maistra#609) * Build container images with docker Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Use maistra proxy and CentOS Stream 8 as a base image for proxyv2 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Test VMs using CentOS Stream 8 Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip failing security tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Temporarily skip tests suite telemetry.prometheus.wasm Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * fix(tests): enables fixed integration tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Jacek Ewertowski <jewertow@redhat.com> * Use our proxy (maistra#801) Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * OSSM-5958: Remove wasm extensions (maistra#937) * Do not rely on WASM extensions Since maistra/proxy#253 * Remove wasm extensions from EnvoyFilters Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Disable tests for WASM extensions Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Remove commented code Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> --------- Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Jonh Wendell <jwendell@redhat.com> * Disable failing tests Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> * Fix TestTrustDomainValidation - change expected TLS error to make it work with OpenSSL Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> --------- Signed-off-by: Jacek Ewertowski <jewertow@redhat.com> Co-authored-by: Bartosz Majsak <bartosz.majsak@gmail.com> Co-authored-by: Jonh Wendell <jonh.wendell@redhat.com> Co-authored-by: Jonh Wendell <jwendell@redhat.com>

maistra-bot added the size/S label Sep 9, 2022

jwendell reviewed Sep 12, 2022

View reviewed changes

maistra-bot added size/M size/L and removed size/S size/M labels Sep 13, 2022

jewertow changed the title ~~Use maistra proxy and custom base image built on top of CentOS Stream 8~~ Use Maistra proxy built on top of CentOS Stream 8 Sep 13, 2022

jewertow changed the title ~~Use Maistra proxy built on top of CentOS Stream 8~~ Use Maistra proxy based on CentOS Stream 8 in integration tests Sep 15, 2022

jewertow added 3 commits September 15, 2022 14:52

Build container images with docker

28ed155

Signed-off-by: Jacek Ewertowski <jewertow@redhat.com>

Use maistra proxy and CentOS Stream 8 as a base image for proxyv2

aeba3ac

Signed-off-by: Jacek Ewertowski <jewertow@redhat.com>

Test VMs using CentOS Stream 8

747b338

Signed-off-by: Jacek Ewertowski <jewertow@redhat.com>

jewertow force-pushed the use-maistra-proxy branch from 71cd890 to e3c2abb Compare September 15, 2022 15:44

jewertow requested review from bmangoen, jwendell, luksa and yxun September 15, 2022 15:52

jwendell reviewed Sep 15, 2022

View reviewed changes

jewertow added 2 commits September 15, 2022 20:32

Temporarily skip failing security tests

2cacf03

Signed-off-by: Jacek Ewertowski <jewertow@redhat.com>

Temporarily skip tests suite telemetry.prometheus.wasm

15a4435

Signed-off-by: Jacek Ewertowski <jewertow@redhat.com>

jewertow force-pushed the use-maistra-proxy branch from e3c2abb to 15a4435 Compare September 15, 2022 18:33

jewertow commented Sep 15, 2022

View reviewed changes

jewertow requested a review from jwendell September 15, 2022 18:38

yxun reviewed Sep 15, 2022

View reviewed changes

jwendell approved these changes Sep 15, 2022

View reviewed changes

jewertow added the okay to merge label Sep 15, 2022

maistra-bot merged commit 739e326 into maistra:maistra-2.3 Sep 15, 2022

bartoszmajsak mentioned this pull request Dec 21, 2022

OSSM-2378: Use Maistra proxy CentOS Stream 8 img #713

Merged

11 tasks

yxun mentioned this pull request May 22, 2024

[maistra-2.6] Rebase building OpenSSL-based proxy image #1008

Closed

12 tasks

Conversation

jewertow commented Sep 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jwendell left a comment

Choose a reason for hiding this comment

Uh oh!

jewertow commented Sep 12, 2022

Uh oh!

jwendell commented Sep 12, 2022

Uh oh!

jewertow commented Sep 14, 2022

Uh oh!

jewertow commented Sep 14, 2022

Uh oh!

jewertow commented Sep 14, 2022

Uh oh!

jewertow commented Sep 15, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jewertow commented Sep 15, 2022

Uh oh!

jewertow commented Sep 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jewertow commented Sep 9, 2022 •

edited

Loading