fix(tests): stabilize AuthzITs and some other cleanups by k-wall · Pull Request #3103 · kroxylicious/kroxylicious

k-wall · 2026-01-06T18:41:28Z

Type of change

Select the type of your PR

Bugfix

Description

The AuthzITs are unstable for me on my main work machine. We are also seeing the same failures in CI.

I am running this command:

mvn verify -Dit.test='*AuthzIT' -DskipUTs  -Dfailsafe.failIfNoSpecifiedTests=false -pl kroxylicious-integration-tests -am

The issues I'm seeing:

Side-effect assertion sporadically failing

The side effect assertions are failing. For instance, in DeleteTopicsAuthzIT I see failures like so:

org.opentest4j.AssertionFailedError: 
expected: []
 but was: ["eve-topic", "alice-topic"]
Expected :[]
Actual   :["eve-topic", "alice-topic"]
at io.kroxylicious.proxy.filter.authorization.AuthzIT.verifyApiEqivalence(AuthzIT.java:588)
	at io.kroxylicious.proxy.filter.authorization.AuthzIT$Equivalence.verifyBehaviour(AuthzIT.java:239)
	at io.kroxylicious.proxy.filter.authorization.DeleteTopicsAuthzIT.shouldEnforceAccessToTopics(DeleteTopicsAuthzIT.java:292)

I think this is down the the eventually consistent nature of Kafka. Just because you've say, just created a topic, doesn't give you the guarantee that an admin observation will observe it.

Proxied/reference response mismatches

I'm seeing some fails like this:

org.opentest4j.AssertionFailedError: [Expect proxied response to be the same as the reference response with equivalent AuthZ] 
expected: 
  {"alice"="{
    "throttleTimeMs" : 0,
    "responses" : [ {
      "name" : "alice-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "bob-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "eve-topic",
      "errorCode" : 29,
      "topicId" : null
    } ]
  }", "bob"="{
    "throttleTimeMs" : 0,
    "responses" : [ {
      "name" : "alice-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "bob-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "eve-topic",
      "errorCode" : 29,
      "topicId" : null
    } ]
  }", "eve"="{
    "throttleTimeMs" : 0,
    "responses" : [ {
      "name" : "alice-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "bob-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "eve-topic",
      "errorCode" : 29,
      "topicId" : null
    } ]
  }"}
 but was: 
  {"alice"="{
    "throttleTimeMs" : 0,
    "responses" : [ {
      "name" : "alice-topic",
      "errorCode" : 0,
      "topicId" : null
    }, {
      "name" : "bob-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "eve-topic",
      "errorCode" : 29,
      "topicId" : null
    } ]
  }", "bob"="{
    "throttleTimeMs" : 0,
    "responses" : [ {
      "name" : "alice-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "bob-topic",
      "errorCode" : 0,
      "topicId" : null
    }, {
      "name" : "eve-topic",
      "errorCode" : 29,
      "topicId" : null
    } ]
  }", "eve"="{
    "throttleTimeMs" : 0,
    "responses" : [ {
      "name" : "alice-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "bob-topic",
      "errorCode" : 29,
      "topicId" : null
    }, {
      "name" : "eve-topic",
      "errorCode" : 29,
      "topicId" : null
    } ]
  }"}
<Click to see difference>


	at io.kroxylicious.proxy.filter.authorization.AuthzIT.verifyApiEqivalence(AuthzIT.java:583)
	at io.kroxylicious.proxy.filter.authorization.AuthzIT$Equivalence.verifyBehaviour(AuthzIT.java:241)
	at io.kroxylicious.proxy.filter.authorization.DeleteTopicsAuthzIT.shouldEnforceAccessToTopics(DeleteTopicsAuthzIT.java:292)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.Optional.ifPresent(Optional.java:178)

This again is an eventually consistency issue. The test harness creates the topic (alice-topic etc), and then the test proceeds to perform an operation on the topic

Additional Context

Why are you making this pull request?

Checklist

Please go through this checklist and make sure all applicable tasks have been done

PR raised from a fork of this repository and made from a branch rather than main.
Write tests
Update documentation
Make sure all unit/integration tests pass
Make sure all Sonarcloud warnings are addressed or are justifiably ignored.
If applicable to the change, trigger the system test suite. Make sure tests pass.
If applicable to the change, trigger the performance test suite. Ensure that any degradations to performance numbers are understood and justified.
Ensure the PR references relevant issue(s) so they are closed on merging.
For user facing changes, update CHANGELOG.md (remember to include changes affecting the API of the test artefacts too).

NOTE: You must be a member of @kroxylicious/developers to trigger the system test and performance test suites. If you are not part of this group, comment on the PR requesting a trigger, tagging @kroxylicious/developers.

pom.xml

piotrpdev · 2026-01-07T14:19:17Z

...-integration-tests/src/test/java/io/kroxylicious/proxy/filter/authorization/KafkaDriver.java

-                apiKeys,
-                prettyJsonString(responseConverter.writer().apply(response, apiVersion)),
-                cluster.name());
+        LOG.atDebug().setMessage("{} {} response: {} << {}")


Why change to debug here instead of using .atInfo() to match the previous behavior?

In my opinion, these tests were writing too much output. I just want to see the names of test classes which is quick to eyeball when the tests run cleanly. When the test fails, I want the test to produce a good assertion message. I can then turn up logging if I need to to investigate the failure.

If tests spew war-and-peace on the happy path, that's disruptive for me.

piotrpdev

LGTM

tombentley · 2026-01-07T23:01:19Z

@k-wall sorry, but I've only just seen this after merging #3109 and opening #3110. Neither #3106 nor #3107 showed any indication that you were looking at them. I think we need to be more diligent about assigning issues to people, and referencing issues in PRs intended to address them.

kill some unused code. Signed-off-by: Keith Wall <kwall@apache.org>

…ually consistent nature of kafka Signed-off-by: Keith Wall <kwall@apache.org>

…sponse to ease reasoning about the differences Signed-off-by: Keith Wall <kwall@apache.org>

k-wall · 2026-01-08T12:42:20Z

@k-wall sorry, but I've only just seen this after merging #3109 and opening #3110. Neither #3106 nor #3107 showed any indication that you were looking at them. I think we need to be more diligent about assigning issues to people, and referencing issues in PRs intended to address them.

No problem. I'd been talking about the issue for a couple of days on Slack, so I'd assumed that everyone might have noticed. But we have too many channels for everyone to monitor everything!

… to the broker before commencing the test Signed-off-by: Keith Wall <kwall@apache.org>

Signed-off-by: Keith Wall <kwall@apache.org>

k-wall · 2026-01-08T16:12:36Z

...ious-integration-tests/src/test/java/io/kroxylicious/proxy/filter/authorization/AuthzIT.java

            return scenario.clobberResponse(cl, node);
        });
-        assertThat(mapValues(proxiedResponsesByUser,
+        assertThatJson(mapValues(proxiedResponsesByUser,


This gives a much more human readable diff when the assertion fails.

k-wall · 2026-01-08T16:14:11Z

...ious-integration-tests/src/test/java/io/kroxylicious/proxy/filter/authorization/AuthzIT.java

-                    if (!scenario.needsRetry(r)) {
-                        break;
-                    }
-                    try {


The sleep didn't seem to be achieving anything.

tombentley · 2026-01-08T19:57:56Z

etc/checkstyle-custom_checks.xml

-<!--        </module>-->
+        <module name="IllegalImport">
+            <!-- we shouldn't be relying on class shaded into TC for dependencies -->
+            <property name="illegalPkgs" value="org.testcontainers.shaded"/>


Thank you for this!

...-integration-tests/src/test/java/io/kroxylicious/proxy/filter/authorization/KafkaDriver.java

…roxy/filter/authorization/KafkaDriver.java Signed-off-by: Tom Bentley <tombentley@users.noreply.github.com>

sonarqubecloud · 2026-01-08T20:27:29Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

why: in kroxylicious#3103 666c64d we changed the AuthzIT tests to ensure that the topics were in a consistent state before the starting the tests. The AbstractAuthzEquivalenceIT tests have their own cluster prep code, so didn't benefit from this change meaning those tests could still fail owning to lack of consistency. This change pulls out a common base class and refactors both AuthzIT and AbstractAuthzEquivalenceIT to use it. Signed-off-by: Keith Wall <kwall@apache.org>

why: in #3103 666c64d we changed the AuthzIT tests to ensure that the topics were in a consistent state before the starting the tests. The AbstractAuthzEquivalenceIT tests have their own cluster prep code, so didn't benefit from this change meaning those tests could still fail owning to lack of consistency. This change pulls out a common base class and refactors both AuthzIT and AbstractAuthzEquivalenceIT to use it. Signed-off-by: Keith Wall <kwall@apache.org>

k-wall requested a review from a team as a code owner January 6, 2026 18:41

k-wall changed the title ~~chore(tests): stabilize authzit~~ chore(tests): stabilize AuthzITs Jan 6, 2026

k-wall marked this pull request as draft January 6, 2026 18:41

k-wall changed the title ~~chore(tests): stabilize AuthzITs~~ fix(tests): stabilize AuthzITs Jan 6, 2026

k-wall force-pushed the stablise-tests branch from 7a0257e to 73a2908 Compare January 7, 2026 09:44

k-wall changed the title ~~fix(tests): stabilize AuthzITs~~ fix(tests): stabilize AuthzITs (part 1) Jan 7, 2026

k-wall marked this pull request as ready for review January 7, 2026 11:05

k-wall added the test Relates to testing label Jan 7, 2026

k-wall mentioned this pull request Jan 7, 2026

DeleteRecordsAuthzIT sometimes fails with a sporadic NOT_LEADER_OR_FOLLOWER #3106

Closed

piotrpdev reviewed Jan 7, 2026

View reviewed changes

piotrpdev approved these changes Jan 7, 2026

View reviewed changes

k-wall added 3 commits January 8, 2026 12:26

chore(tests): pipe down! make authz it less chatty at info.

ef58823

kill some unused code. Signed-off-by: Keith Wall <kwall@apache.org>

fix(tests): query for side-effects in a loop to account for the event…

7582011

…ually consistent nature of kafka Signed-off-by: Keith Wall <kwall@apache.org>

fix(tests): use json-unit-assertj to compare the proxied/reference re…

9889c6c

…sponse to ease reasoning about the differences Signed-off-by: Keith Wall <kwall@apache.org>

k-wall force-pushed the stablise-tests branch from c30eb5d to 5562bfe Compare January 8, 2026 12:29

piotrpdev mentioned this pull request Jan 8, 2026

systemd's DefaultLimitMEMLOCK=8M is not enough for IOUringIT test to pass #2802

Closed

k-wall added 2 commits January 8, 2026 16:08

fix(tests): use admin to confirm for test topics are completely known…

af2ff75

… to the broker before commencing the test Signed-off-by: Keith Wall <kwall@apache.org>

chore(imports): clean up some stray shaded imports

8c96d4f

Signed-off-by: Keith Wall <kwall@apache.org>

k-wall force-pushed the stablise-tests branch from 07d30e4 to 8c96d4f Compare January 8, 2026 16:08

k-wall changed the title ~~fix(tests): stabilize AuthzITs (part 1)~~ fix(tests): stabilize AuthzITs and some other cleanups Jan 8, 2026

k-wall commented Jan 8, 2026

View reviewed changes

tombentley approved these changes Jan 8, 2026

View reviewed changes

Update kroxylicious-integration-tests/src/test/java/io/kroxylicious/p…

b9f0c2e

…roxy/filter/authorization/KafkaDriver.java Signed-off-by: Tom Bentley <tombentley@users.noreply.github.com>

tombentley enabled auto-merge (rebase) January 8, 2026 20:09

tombentley merged commit b5cba98 into kroxylicious:main Jan 8, 2026
24 checks passed

SamBarker mentioned this pull request Jan 11, 2026

Test failures due to RLIMIT_MEMLOCK #3120

Closed

k-wall mentioned this pull request Feb 4, 2026

fix(tests): stablise MetadataAuthzEquivalenceIT #3249

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(tests): stabilize AuthzITs and some other cleanups#3103

fix(tests): stabilize AuthzITs and some other cleanups#3103
tombentley merged 6 commits intokroxylicious:mainfrom
k-wall:stablise-tests

k-wall commented Jan 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

piotrpdev Jan 7, 2026

Uh oh!

k-wall Jan 8, 2026

Uh oh!

piotrpdev left a comment

Uh oh!

tombentley commented Jan 7, 2026

Uh oh!

k-wall commented Jan 8, 2026

Uh oh!

k-wall Jan 8, 2026

Uh oh!

k-wall Jan 8, 2026

Uh oh!

tombentley Jan 8, 2026

Uh oh!

Uh oh!

sonarqubecloud bot commented Jan 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

k-wall commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Description

Side-effect assertion sporadically failing

Proxied/reference response mismatches

Additional Context

Checklist

Uh oh!

Uh oh!

piotrpdev Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

k-wall Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

piotrpdev left a comment

Choose a reason for hiding this comment

Uh oh!

tombentley commented Jan 7, 2026

Uh oh!

k-wall commented Jan 8, 2026

Uh oh!

k-wall Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

k-wall Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

tombentley Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Jan 8, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

k-wall commented Jan 6, 2026 •

edited

Loading