Skip to content

Fix race conditions when printing current status of tasks#399

Merged
mrnugget merged 3 commits into
mainfrom
mrn/fix-race-conditions
Dec 1, 2020
Merged

Fix race conditions when printing current status of tasks#399
mrnugget merged 3 commits into
mainfrom
mrn/fix-race-conditions

Conversation

@mrnugget

Copy link
Copy Markdown
Contributor

This fixes https://github.com/sourcegraph/sourcegraph/issues/16195 by doing two things:

  1. It threads every update and read of *TaskStatus in executor through a mutex to provide a consistent view of all []*TaskStatus. While we had a sync.Map in place it didn't protect us from race conditions, since we passed around pointers to TaskStatus freely: the executor goroutines would happily write to a *TaskStatus and the campaignProgressPrinter.PrintStatus would read []*TaskStatus while they're being updated. The sync.Map was useless in practice. So instead of that, I added a mutex that's used when updating a *TaskStatus and when reading all []*TaskStatus when printing the status.
  2. It uses a 1-capacity semaphore to control access to the campaignProgressPrinter through the PrintStatuses callback: only one execution at a time and if another execution is already in progress it returns.

@mrnugget mrnugget requested a review from a team November 30, 2020 14:11

@eseliger eseliger left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM, I trust you that it fixes the underlying issue :)

Comment thread internal/campaigns/executor.go
Comment thread internal/campaigns/executor.go

@LawnGnome LawnGnome left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for fixing this bug! (I'll spare us all the Rust evangelism here.)

@mrnugget

mrnugget commented Dec 1, 2020

Copy link
Copy Markdown
Contributor Author

LGTM. Thanks for fixing this bug! (I'll spare us all the Rust evangelism here.)

I know :| But to be fair to Go, what we're doing here is communicating between two parts by sharing memory, instead of sharing by communicating. (This is orthogonal to Rust's memory checks that would've probably avoided this, but a thought I had yesterday)

@mrnugget mrnugget merged commit 1ec0535 into main Dec 1, 2020
@mrnugget mrnugget deleted the mrn/fix-race-conditions branch December 1, 2020 10:13
scjohns pushed a commit that referenced this pull request Apr 24, 2023
* Replace sync.Map with locked access around TaskStatuses

* Avoid PrintStatuses being called concurrently

* Add changelog entry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

src-cli: race conditions in campaignProgressPrinter

3 participants