Skip to content

resolver timeout fixes#2369

Merged
AkihiroSuda merged 3 commits intomoby:masterfrom
tonistiigi:resolver-timeout-fixes
Sep 21, 2021
Merged

resolver timeout fixes#2369
AkihiroSuda merged 3 commits intomoby:masterfrom
tonistiigi:resolver-timeout-fixes

Conversation

@tonistiigi
Copy link
Copy Markdown
Member

closes #2367
closes #2088

hosts mutex is called on initialization, meaning GetResolver might
block if it is in the middle of auth exchange. This is currently bad
in the case where the Job initialization needs to register a name before
timeout is reached.

Authorizer stores the current session.Group so if it is
overwritten for another resolver it means that session might
have been dropped and authentication will fail. I haven't been able
to reproduce this issue so not 100% it fixes the case but even if it is
a different race it looks incorrect anyway.

For the Job registration, we might want to do some follow-up change
that avoids running things before job ID is registered.

@maxlaverse

hosts mutex is called on initialization, meaning `GetResolver` might
block if it is in the middle of auth exchange. This is currently bad
in the case where Job initialization needs to register a name before
timeout is reached.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Authorizer stores the current session.Group so if it is
overwritten for another resolver it means that session might
have been dropped and authentication will fail.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>

func (jl *Solver) Get(id string) (*Job, error) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Second)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tonistiigi this is to be above the 5 seconds timeout of the sessionManager.Any() ?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These shouldn't be related. One should never wait for another.

@AkihiroSuda AkihiroSuda merged commit 44891f4 into moby:master Sep 21, 2021
@tonistiigi tonistiigi mentioned this pull request Oct 1, 2021
jmacelroy pushed a commit to okteto/buildkit that referenced this pull request Oct 19, 2021
closes moby#2367
closes moby#2088

hosts mutex is called on initialization, meaning GetResolver might
block if it is in the middle of auth exchange. This is currently bad
in the case where the Job initialization needs to register a name before
timeout is reached.

Authorizer stores the current session.Group so if it is
overwritten for another resolver it means that session might
have been dropped and authentication will fail. I haven't been able
to reproduce this issue so not 100% it fixes the case but even if it is
a different race it looks incorrect anyway.

For the Job registration, we might want to do some follow-up change
that avoids running things before job ID is registered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

error: failed to authorize: no active session for <session-id>: context deadline exceeded Error "no such job"

3 participants