Skip to content

avoid reauthentication loops#1746

Merged
jtopjian merged 3 commits intogophercloud:masterfrom
sapcc:avoid-reauth-loop
Oct 17, 2019
Merged

avoid reauthentication loops#1746
jtopjian merged 3 commits intogophercloud:masterfrom
sapcc:avoid-reauth-loop

Conversation

@majewsky
Copy link
Copy Markdown
Contributor

It does not make sense to reauthenticate multiple times for a single request. When the backing service returns 401 for a brand-new token, reauthenticating again will just send us into an infinite loop.

Closes #1745.

majewsky added a commit to sapcc/limes that referenced this pull request Oct 15, 2019
@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Oct 15, 2019

Build failed.

It does not make sense to reauthenticate multiple times for a single
request. When the backing service returns 401 for a brand-new token,
reauthenticating again will just send us into an infinite loop.

Closes gophercloud#1745.
@majewsky
Copy link
Copy Markdown
Contributor Author

Okay, so this got way worse than I expected.

The test failure is in go test ./testing -run TestConcurrentReauth. ProviderClient.Reauthenticate has several places right now where it unlocks the reauth mutex before locking it again. This allows concurrent goroutines to glitch through the reauthentication without the token having changed at all.

I tried to replace the hasReauthenticated flag with a counter so that I can permit more than one reauthentication per Request call. It turns out that I need to allow 250 (!) reauthentications per Request call to make the test pass mostly reliably. (With a reduced number of concurrent requests, that number goes even higher.)

There is definitely something wrong with Reauthenticate(). I have some ideas and will poke around a bit.

The old implementation has several places where the reauth mutex is
unlocked before being locked again. This allows concurrent goroutines to
glitch through reauthentication without the token having changed at all.

Since mutexes and esp. RW mutexes are incredibly hard to get right, I
propose this alternate implementation that coordinates concurrent
Reauthenticate calls with channels.
@majewsky
Copy link
Copy Markdown
Contributor Author

Okay, so this PR also got way bigger than I expected. I threw away most of the mutexes in type reauthlock and moved to an implementation that uses channels to coordinate concurrent Reauthenticate calls. This implementation is way more readable and more idiomatic, and also passes all the tests on the first try.

@coveralls
Copy link
Copy Markdown

coveralls commented Oct 15, 2019

Coverage Status

Coverage decreased (-0.3%) to 76.995% when pulling 610c593 on sapcc:avoid-reauth-loop into 0c7b9d1 on gophercloud:master.

@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Oct 15, 2019

Build succeeded.

@jtopjian
Copy link
Copy Markdown
Contributor

@majewsky Thanks so much for looking into this.

I think I understand the implementation - it looks like you have implemented correctly what the removed code was trying to achieve. I can also confirm the existing tests are passing, even doing a count of 50 rounds. And -race isn't reporting any issues, either.

Is it possible to add an additional unit test to reproduce the issue you discovered?

@majewsky
Copy link
Copy Markdown
Contributor Author

I added a testcase. You can verify that the testcase detects the fixed bug by removing the && !state.hasReauthenticated check from provider_client.go:432.

@theopenlab-ci
Copy link
Copy Markdown

theopenlab-ci bot commented Oct 16, 2019

Build succeeded.

Copy link
Copy Markdown
Contributor

@jtopjian jtopjian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's give this a shot :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ProviderClient.Request should not perform multiple reauthentications in a row

3 participants