Skip to content

Fix DockerTrustSuite SetUpTest#19471

Merged
thaJeztah merged 1 commit intomoby:masterfrom
thaJeztah:fix-19393-increase-timeout
Jan 20, 2016
Merged

Fix DockerTrustSuite SetUpTest#19471
thaJeztah merged 1 commit intomoby:masterfrom
thaJeztah:fix-19393-increase-timeout

Conversation

@thaJeztah
Copy link
Copy Markdown
Member

This hopefully makes DockerTrustSuite.SetUpTest less flaky.

Increased the number of attempts to check if Notary is available before giving up.

fixes #19393 (fingers crossed)

This hopefully makes DockerTrustSuite.SetUpTest
less flaky.

Increased the number of attempts to check
if Notary is available before giving up.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought more attempts was better than a longer "sleep", so that we don't make the tests take longer if it does come up fast enough

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. But even after 20 attempts, that is still a 200 ms timeout. I'd suggest raising it even further. It is very possible for I/O or resource starvation to make launching a service take more than 200 ms.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's 10 * 20 * 20 = 4000 milliseconds = 4 seconds

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aaronlehmann let's give this a try; we can raise it further if it's still flaky. If it is, we may want to temporarily add some logging to see how long it actually takes, sgty?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 seconds should be sufficient.

@thaJeztah
Copy link
Copy Markdown
Member Author

Janky, why are you failing.. java.lang.IllegalStateException

@tiborvass
Copy link
Copy Markdown
Contributor

LGTM

@thaJeztah
Copy link
Copy Markdown
Member Author

Lol, and gccgo is failing on another flaky test 😢 #19368

@MHBauer
Copy link
Copy Markdown
Contributor

MHBauer commented Jan 20, 2016

LGTM. Worth a shot.

@thaJeztah
Copy link
Copy Markdown
Member Author

It's green, so lets merge!

thaJeztah added a commit that referenced this pull request Jan 20, 2016
@thaJeztah thaJeztah merged commit cd1ed85 into moby:master Jan 20, 2016
@thaJeztah thaJeztah deleted the fix-19393-increase-timeout branch January 20, 2016 05:58
@thaJeztah thaJeztah mentioned this pull request Jan 20, 2016
12 tasks
@StefanScherer
Copy link
Copy Markdown
Contributor

@thaJeztah I'll let Jenkins test this with the DockerTrustSuite enabled on ARM.

@thaJeztah
Copy link
Copy Markdown
Member Author

@StefanScherer thanks! As mentioned, we can still tweak this, my primary concern was to get the test at least a bit more reliable :-)

@StefanScherer
Copy link
Copy Markdown
Contributor

@thaJeztah Sure!

But look, DockerTrustSuite works on ARM:

09:15:39 PASS: docker_cli_build_test.go:5831: DockerTrustSuite.TestBuildContextDirIsSymlink 3.846s
09:15:45 SKIP: docker_cli_create_test.go:332: DockerTrustSuite.TestCreateWhenCertExpired (Currently changes system time, causing instability)
09:15:51 SKIP: docker_cli_pull_trusted_test.go:65: DockerTrustSuite.TestPullWhenCertExpired (Currently changes system time, causing instability)
09:15:57 SKIP: docker_cli_run_test.go:3200: DockerTrustSuite.TestRunWhenCertExpired (Currently changes system time, causing instability)
09:16:12 PASS: docker_cli_build_test.go:5779: DockerTrustSuite.TestTrustedBuild 9.048s
09:16:18 PASS: docker_cli_build_test.go:5810: DockerTrustSuite.TestTrustedBuildUntrustedTag 0.717s
09:16:32 PASS: docker_cli_create_test.go:282: DockerTrustSuite.TestTrustedCreate    7.526s
09:16:52 PASS: docker_cli_create_test.go:359: DockerTrustSuite.TestTrustedCreateFromBadTrustServer  13.584s
09:17:05 PASS: docker_cli_create_test.go:319: DockerTrustSuite.TestTrustedIsolatedCreate    6.406s
09:17:16 PASS: docker_cli_pull_trusted_test.go:35: DockerTrustSuite.TestTrustedIsolatedPull 5.467s
09:17:29 PASS: docker_cli_pull_trusted_test.go:177: DockerTrustSuite.TestTrustedOfflinePull 6.398s
09:17:40 PASS: docker_cli_pull_trusted_test.go:14: DockerTrustSuite.TestTrustedPull 5.648s
09:17:55 PASS: docker_cli_pull_trusted_test.go:205: DockerTrustSuite.TestTrustedPullDelete  9.150s
09:18:14 PASS: docker_cli_pull_trusted_test.go:93: DockerTrustSuite.TestTrustedPullFromBadTrustServer   12.909s
09:18:20 SKIP: docker_cli_pull_trusted_test.go:147: DockerTrustSuite.TestTrustedPullWithExpiredSnapshot (Currently changes system time, causing instability)
09:18:31 PASS: docker_cli_push_test.go:203: DockerTrustSuite.TestTrustedPush    4.832s
09:18:40 PASS: docker_cli_push_test.go:243: DockerTrustSuite.TestTrustedPushWithDeprecatedEnvPasswords  3.494s
09:18:51 PASS: docker_cli_push_test.go:222: DockerTrustSuite.TestTrustedPushWithEnvPasswords    4.913s
09:19:04 PASS: docker_cli_push_test.go:299: DockerTrustSuite.TestTrustedPushWithExistingSignedTag   7.386s
09:19:16 PASS: docker_cli_push_test.go:279: DockerTrustSuite.TestTrustedPushWithExistingTag 5.417s
09:19:22 SKIP: docker_cli_push_test.go:371: DockerTrustSuite.TestTrustedPushWithExpiredSnapshot (Currently changes system time, causing instability)
09:19:28 SKIP: docker_cli_push_test.go:397: DockerTrustSuite.TestTrustedPushWithExpiredTimestamp (Currently changes system time, causing instability)
09:20:11 PASS: docker_cli_push_test.go:255: DockerTrustSuite.TestTrustedPushWithFailingServer   37.082s
09:20:22 PASS: docker_cli_push_test.go:351: DockerTrustSuite.TestTrustedPushWithIncorrectDeprecatedPassphraseForNonRoot 5.328s
09:20:33 PASS: docker_cli_push_test.go:329: DockerTrustSuite.TestTrustedPushWithIncorrectPassphraseForNonRoot   5.369s
09:20:47 PASS: docker_cli_push_test.go:423: DockerTrustSuite.TestTrustedPushWithReleasesDelegation  7.828s
09:20:55 PASS: docker_cli_push_test.go:267: DockerTrustSuite.TestTrustedPushWithoutServerAndUntrusted   1.849s
09:21:11 PASS: docker_cli_run_test.go:3146: DockerTrustSuite.TestTrustedRun 10.244s
09:21:33 PASS: docker_cli_run_test.go:3238: DockerTrustSuite.TestTrustedRunFromBadTrustServer   15.162s
09:21:42 PASS: docker_cli_create_test.go:303: DockerTrustSuite.TestUntrustedCreate  3.202s
09:21:51 PASS: docker_cli_pull_trusted_test.go:49: DockerTrustSuite.TestUntrustedPull   3.195s
09:22:00 PASS: docker_cli_run_test.go:3178: DockerTrustSuite.TestUntrustedRun   3.337s

I'll send a PR for that 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky test: DockerTrustSuite.SetUpTest (more apparent on ARM?)

6 participants