Allow setting non-standard AWS Glue Registry endpoints for testing#5502
Conversation
| topicMetrics.getNumberOfPollAuthErrors().increment(); | ||
| Thread.sleep(10000); | ||
| } catch (RecordDeserializationException e) { | ||
| LOG.error("Deserialization error while doing poll()", e); |
There was a problem hiding this comment.
I originally had a problem where I didn't set the AWS_SECRET_ACCESS_KEY in the data-prepper docker image and it was failing to connect to the instance of motoserver I was running. The helpful error message in the stack trace wasn't being displayed as it was about the 3rd "caused-by" down in the trace. I had to add this error message to see it.
I don't think this is ideal so I wanted some guidance on how you want me to handle this error so that it provides a meaningful message to a user for correcting things.
Change set allows using the kafka.schema.registry_url setting to set non-standard endpoints when using a mock or fake AWS glue registry (opensearch-project#5377). Signed-off-by: Shane Schisler <shane@schisler.net>
370ee4c to
a8a32b7
Compare
|
I see that the pipeline currently has some accepted failing tests based on previous merges. I briefly looked into the failing tests and they didn't make a lot of sense based on the changes I made. Before I go too deep troubleshooting the Kafka plugin integration test failures, could someone let me know if these are expected and accepted as failing as well. |
…pensearch-project#5502) Change set allows using the kafka.schema.registry_url setting to set non-standard endpoints when using a mock or fake AWS glue registry (opensearch-project#5377). Signed-off-by: Shane Schisler <shane@schisler.net> Co-authored-by: Shane Schisler <shane@schisler.net>
…pensearch-project#5502) Change set allows using the kafka.schema.registry_url setting to set non-standard endpoints when using a mock or fake AWS glue registry (opensearch-project#5377). Signed-off-by: Shane Schisler <shane@schisler.net> Co-authored-by: Shane Schisler <shane@schisler.net> Signed-off-by: George Chen <qchea@amazon.com>
…pensearch-project#5502) Change set allows using the kafka.schema.registry_url setting to set non-standard endpoints when using a mock or fake AWS glue registry (opensearch-project#5377). Signed-off-by: Shane Schisler <shane@schisler.net> Co-authored-by: Shane Schisler <shane@schisler.net>
…pensearch-project#5502) Change set allows using the kafka.schema.registry_url setting to set non-standard endpoints when using a mock or fake AWS glue registry (opensearch-project#5377). Signed-off-by: Shane Schisler <shane@schisler.net> Co-authored-by: Shane Schisler <shane@schisler.net>
…pensearch-project#5502) Change set allows using the kafka.schema.registry_url setting to set non-standard endpoints when using a mock or fake AWS glue registry (opensearch-project#5377). Signed-off-by: Shane Schisler <shane@schisler.net> Co-authored-by: Shane Schisler <shane@schisler.net>
…pensearch-project#5502) Change set allows using the kafka.schema.registry_url setting to set non-standard endpoints when using a mock or fake AWS glue registry (opensearch-project#5377). Signed-off-by: Shane Schisler <shane@schisler.net> Co-authored-by: Shane Schisler <shane@schisler.net> Signed-off-by: mamol27 <mamol27@yandex.ru>
|
@seschis this is breaking existing behavior. "Previously this setting was ignored and the code would only connect to the standard aws glue registry endpoint." -- this is incorrect. We do use this setting. I need to fix this immediately. I will add another option under |
Description
Add support for the kafka.schema.registry_url setting for the aws_glue schema registry. Previously this setting was ignored and the code would only connect to the standard aws glue registry endpoint. This change allows for running a local aws glue registry using something like localstack or motoserver so that local integration testing can occur. We noticed this problem when setting up AWS OSIS and being unable to locally perform integration testing in our application. Closes #5377
Issues Resolved
Resolves #5377
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.