Skip to content

Ensure all integration daemon logging happens before test exit#39197

Merged
thaJeztah merged 1 commit intomoby:masterfrom
djsweet:log-daemon-exit-before-tests-finish
May 11, 2019
Merged

Ensure all integration daemon logging happens before test exit#39197
thaJeztah merged 1 commit intomoby:masterfrom
djsweet:log-daemon-exit-before-tests-finish

Conversation

@djsweet
Copy link
Contributor

@djsweet djsweet commented May 9, 2019

As of Go 1.12, the testing package panics if a goroutine logs to a
testing.T after the relevant test has completed. This was not
documented as a change at all; see the commit
95d06ab6c982f58b127b14a52c3325acf0bd3926 in the Go repository for the
relevant change.

At any point in the integration tests, tests could panic with the
message "Log in goroutine after TEST_FUNCTION has completed". This was
exacerbated by less direct logging I/O, e.g. running make test with
its output piped instead of attached to a TTY.

The most common cause of panics was that there was a race condition
between an exit logging goroutine and the StopWithError method:
StopWithError could return, causing the calling test method to return,
causing the testing.T to be marked as finished, before the goroutine
could log that the test daemon had exited. The fix is simple: capture
the result of cmd.Wait(), then log, then send the captured
result over the Wait channel. This ensures that the message is
logged before StopWithError can return, blocking the test method
so that the target testing.T is not marked as finished.

Signed-off-by: Daniel Sweet danieljsweet@icloud.com

- What I did
Ensured internal/test/daemon logged exit before the controlling test completed.

- How I did it
Logged daemon process completion before sending result of d.cmd.Wait() to the d.Wait channel.

- How to verify it
Run make test | cat several times; ideally not foregrounded, ideally on a slow system, ideally in a VM; this would eventually fail for me after 3 repeated attempts.

- Description for the changelog

Always log daemon exit before each integration test finishes

As of Go 1.12, the `testing` package panics if a goroutine logs to a
`testing.T` after the relevant test has completed. This was not
documented as a change at all; see the commit
95d06ab6c982f58b127b14a52c3325acf0bd3926 in the Go repository for the
relevant change.

At any point in the integration tests, tests could panic with the
message "Log in goroutine after TEST_FUNCTION has completed". This was
exacerbated by less direct logging I/O, e.g. running `make test` with
its output piped instead of attached to a TTY.

The most common cause of panics was that there was a race condition
between an exit logging goroutine and the `StopWithError` method:
`StopWithError` could return, causing the calling test method to return,
causing the `testing.T` to be marked as finished, before the goroutine
could log that the test daemon had exited. The fix is simple: capture
the result of `cmd.Wait()`, _then_ log, _then_ send the captured
result over the `Wait` channel. This ensures that the message is
logged before `StopWithError` can return, blocking the test method
so that the target `testing.T` is not marked as finished.

Signed-off-by: Daniel Sweet <danieljsweet@icloud.com>
Copy link
Member

@cpuguy83 cpuguy83 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@yongtang yongtang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov
Copy link

codecov bot commented May 11, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@a4d6993). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #39197   +/-   ##
=========================================
  Coverage          ?   37.05%           
=========================================
  Files             ?      612           
  Lines             ?    45448           
  Branches          ?        0           
=========================================
  Hits              ?    16842           
  Misses            ?    26321           
  Partials          ?     2285

@thaJeztah thaJeztah merged commit 3998dff into moby:master May 11, 2019
@thaJeztah
Copy link
Member

Thanks!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants