@whummer @thrau I am having a heck of a time with two things. 1) getting the elastic search plugins to install without hanging for 20+ minutes. 2) getting the tests to run on arm in circleCI.

I'm wondering if the choice of alpine as base is causing us some pain with so much python and musl. Maybe it's causing compile issues on Arm? What's interesting is the tests will run locally on my M1 mac build so maybe an available CPU/memory issue? I'm open to thoughts.

Hi @chezsmithy - thanks again so much for your hard work on this PR. We'd be happy to jump in and help out with the remaining issues (e.g., test failures). Do you think you could give us push access to your fork, so we can collaborate on your branch? Thanks! 🙏

Hi @chezsmithy - thanks again so much for your hard work on this PR. We'd be happy to jump in and help out with the remaining issues (e.g., test failures). Do you think you could give us push access to your fork, so we can collaborate on your branch? Thanks! 🙏

You should have access now. I've added in a new Dockerfile called Dockerfile.complete-debian. It's based off of Debian buster python 3.8. Build times are greatly improved for arm. Also, it has stabilized my test runs. Likely this is due to having glibc included. I wonder if it might make sense to use this rather than alpine as the base? Thoughts?

@whummer I've setup the full run. After the move to Debian it's all green locally and in ci, except the same Elasticsearch tests still failing only in ci. They run just fine locally so I could use some help debugging those. I'm getting close to needing to integration test an actual push to docker. Going to need to simulate that by changing the image tags perhaps to avoid actually deploying images.

Thank you so much for crunching through this PR @chezsmithy , 🙏 and apologies for the delay on our end - unfortunately we got swamped with a few tasks the last couple of days. Will carve out some time later today and tomorrow to review your changes and see how we can best integrate them into our e2e pipeline.

Please do let us know if we can help out with testing the Docker push - if we could ensure that we're using a different testing tag for now, that would be great.. 👍 Thanks!

@chezsmithy - can we please also give @alexrashed push access to your branch, he'll also help out with reviewing and incorporating the changes into our e2e flow. Thanks! 🙏

I've got the Elasticsearch plug-ins to install now, but the tests still seem to fail. Root cause looks like this in the test logs:

tests/integration/test_elasticsearch.py::ElasticsearchTest::test_create_existing_domain_causes_exception
-------------------------------- live log setup --------------------------------
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:53 waiting for initialization lock
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:55 initialization lock acquired in 0.00 seconds
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:206 creating elasticsearch domain test_es_domain_1
INFO localstack.services.es.es_api:es_api.py:107 starting <class 'localstack.services.es.cluster.ProxiedElasticsearchCluster'> on localhost:4571
[2021-11-02 05:08:08 +0000] [17] [INFO] Running on http://0.0.0.0:4571 (CTRL + C to quit)
INFO hypercorn.error:logging.py:90 Running on http://0.0.0.0:4571 (CTRL + C to quit)
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:214 asserting created state of domain test_es_domain_1 (state = False)
INFO localstack.services.es.cluster:cluster.py:163 starting elasticsearch: su localstack -c '/opt/code/localstack/localstack/infra/elasticsearch/bin/elasticsearch -E http.port=44339 -E http.publish_port=44339 -E transport.port=0 -E network.host=127.0.0.1 -E http.compression=false -E path.data="/opt/code/localstack/localstack/infra/elasticsearch/data" -E path.repo="/tmp/localstack/es_backup" -E xpack.ml.enabled=false' with env {'ES_JAVA_OPTS': '-Xms200m -Xmx600m', 'ES_TMPDIR': '/opt/code/localstack/localstack/infra/elasticsearch/tmp'}
INFO localstack.services.es.cluster:cluster.py:172 elasticsearch process ended <--- Looks like the process died
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:214 asserting created state of domain test_es_domain_1 (state = False)
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:214 asserting created state of domain test_es_domain_1 (state = False)
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:214 asserting created state of domain test_es_domain_1 (state = False)
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:214 asserting created state of domain test_es_domain_1 (state = False)
INFO tests.integration.test_elasticsearch:test_elasticsearch.py:214 asserting created

What's odd is if I try the same test using the same docker file locally on my M1 Mac, the same tests pass. Must be something environmental on the circleci runners?

Hi @chezsmithy!
Thanks for giving me the permissions to push on your fork and thanks for all the great work! That's quite some heavy lifting with the different archs, qemu, buildx,...

I started working on this on Friday. First thing was to fix the base image build's ES plugin installations (which was broken due to changes to their webserver's cipher suite, which in turn weren't supported in our stripped down JRE). This actually had nothing to do with this PR, but was basically a prerequisite to get started.

Currently I'm working on getting the multiarch alpine base image build working for me, because we would actually try to stick with alpine if that's possible. We really want to keep things as small and fast as possible, and therefore we really want to avoid the additional 310 MB of the Debian image.

We could also short-circuit in our community Slack to join forces if you would like to.

Hi @chezsmithy!

Thanks for giving me the permissions to push on your fork and thanks for all the great work! That's quite some heavy lifting with the different archs, qemu, buildx,...

I started working on this on Friday. First thing was to fix the base image build's ES plugin installations (which was broken due to changes to their webserver's cipher suite, which in turn weren't supported in our stripped down JRE). This actually had nothing to do with this PR, but was basically a prerequisite to get started.

Currently I'm working on getting the multiarch alpine base image build working for me, because we would actually try to stick with alpine if that's possible. We really want to keep things as small and fast as possible, and therefore we really want to avoid the additional 310 MB of the Debian image.

We could also short-circuit in our community Slack to join forces if you would like to.

Thanks! Not many changes in the end, but it's got a few moving parts to it. It might be worth considering Debian, just for the benefit of your python code build times. On alpine, with musl it appears may of the wheels need to be built from scratch extending the docker build time. With Debian and glibc only a few need to build making it much faster. I have both docker files so you can compare and see. Love to work with you on slack.

Hey, please see my comment here for an update: #2550 (comment)

we cleaned up the history in this PR, and there are a few kinks which i'm currently fixing. We should then be good to prepare the release for monday!

build is green :-) we will prepare the 0.13 release now, and will merge the PR manually hopefully on monday