Skip to content

libnetwork/ipam: fix racy, flaky unit test#45019

Merged
thaJeztah merged 1 commit intomoby:masterfrom
corhere:libnet/fix-ipam-flaky-test
Feb 23, 2023
Merged

libnetwork/ipam: fix racy, flaky unit test#45019
thaJeztah merged 1 commit intomoby:masterfrom
corhere:libnet/fix-ipam-flaky-test

Conversation

@corhere
Copy link
Contributor

@corhere corhere commented Feb 16, 2023

TestRequestReleaseAddressDuplicate gets flagged by go test -race because the same err variable inside the test is assigned to from multiple goroutines without synchronization, which obscures whether or not there are any data races in the code under test.

Trouble is, the test depends on the data race to exit the loop if an error occurs inside a spawned goroutine. And the test contains a logical concurrency bug (not flagged by the Go race detector) which can result in false-positive test failures. Because a release operation is logged after the IP is released, the other goroutine could reacquire the address and log that it was reacquired before the release is logged.

- What I did

Fixed up the test so it is no longer subject to data races or false-positive test failures, i.e. flakes.

- How I did it

  • Hardcoded the number of loop iterations instead of terminating as soon as the address pool is exhausted
  • Changed the order of operations for logging to the ips slice to err on the side of false negatives instead of false positives

- How to verify it

$ go test -race -count=5 -run TestRequestReleaseAddressDuplicate ./libnetwork/ipam

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

TestRequestReleaseAddressDuplicate gets flagged by go test -race because
the same err variable inside the test is assigned to from multiple
goroutines without synchronization, which obscures whether or not there
are any data races in the code under test.

Trouble is, the test _depends on_ the data race to exit the loop if an
error occurs inside a spawned goroutine. And the test contains a logical
concurrency bug (not flagged by the Go race detector) which can result
in false-positive test failures. Because a release operation is logged
after the IP is released, the other goroutine could reacquire the
address and log that it was reacquired before the release is logged.

Fix up the test so it is no longer subject to data races or
false-positive test failures, i.e. flakes.

Signed-off-by: Cory Snider <csnider@mirantis.com>
Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah merged commit ca2fe68 into moby:master Feb 23, 2023
@corhere corhere deleted the libnet/fix-ipam-flaky-test branch February 27, 2023 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: lib network TestRequestReleaseAddressDuplicate

4 participants