Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

Executors: add general dequeue endpoint#51892

Merged
sanderginn merged 50 commits into
mainfrom
rc/executor-general-dequeue
May 16, 2023
Merged

Executors: add general dequeue endpoint#51892
sanderginn merged 50 commits into
mainfrom
rc/executor-general-dequeue

Conversation

@sanderginn

@sanderginn sanderginn commented May 12, 2023

Copy link
Copy Markdown
Contributor

This PR adds a first implementation of a general dequeue endpoint that can accept multiple queues.

Improvements

Due to generics, it's pretty difficult to not break this up into queue specific components due to the record types handled by each queue. This means there's still a bunch of reproduced code in the current implementation.

It currently imlements ServeHTTP. Once https://github.com/sourcegraph/sourcegraph/issues/51656 is added, it might make sense to turn this into a new interface.

Demo

Only batch change jobs

batches.mp4

Only codeintel jobs

codeintel.mp4

Both queue jobs

both.queues.mp4

Test plan

  • Unit tests
  • Demo

@cla-bot cla-bot Bot added the cla-signed label May 12, 2023
@sanderginn sanderginn changed the title Rc/executor general dequeue Executors: add general dequeue endpoint May 12, 2023
@sanderginn sanderginn requested a review from a team May 12, 2023 19:34
@sourcegraph-bot

sourcegraph-bot commented May 12, 2023

Copy link
Copy Markdown
Contributor

Codenotify: Notifying subscribers in CODENOTIFY files for diff 2f9fb8e...1b158f7.

Notify File(s)
@efritz enterprise/cmd/frontend/internal/executorqueue/handler/BUILD.bazel
enterprise/cmd/frontend/internal/executorqueue/handler/handler.go
enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go
enterprise/cmd/frontend/internal/executorqueue/handler/multihandler_test.go
enterprise/cmd/frontend/internal/executorqueue/queuehandler.go
enterprise/internal/executor/types/http.go
enterprise/internal/executor/types/job.go
@eseliger enterprise/cmd/frontend/internal/executorqueue/handler/BUILD.bazel
enterprise/cmd/frontend/internal/executorqueue/handler/handler.go
enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go
enterprise/cmd/frontend/internal/executorqueue/handler/multihandler_test.go
enterprise/cmd/frontend/internal/executorqueue/queuehandler.go

Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated

@Piszmog Piszmog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job! Just some questions and nits, but looks good!

I do wonder if we should wrap the logic ServeHTTP in something like handler.wrapHander. There are a lot of exit points in the code that require return. So it could be easy to forget a return when handling an error. Something like handler.wrapHander could make it so an error is returned that way we can have confidence that the logic ends with the error.

But not a big deal right now

Comment thread enterprise/internal/executor/types/job.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/queuehandler.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
@sanderginn sanderginn requested a review from Piszmog May 15, 2023 15:30

@Piszmog Piszmog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! Just a couple more comments

Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
Comment thread enterprise/cmd/frontend/internal/executorqueue/handler/multihandler.go Outdated
@sanderginn

Copy link
Copy Markdown
Contributor Author

@Piszmog thanks for the last review, merging with the last changes. I've added some demo videos as well.

@sanderginn sanderginn merged commit 205c869 into main May 16, 2023
@sanderginn sanderginn deleted the rc/executor-general-dequeue branch May 16, 2023 12:36
cesrjimenez pushed a commit that referenced this pull request May 17, 2023
- Closes https://github.com/sourcegraph/sourcegraph/issues/50613

This PR adds a first implementation of a general dequeue endpoint that
can accept multiple queues.

### Improvements
Due to generics, it's pretty difficult to not break this up into queue
specific components due to the record types handled by each queue. This
means there's still a bunch of reproduced code in the current
implementation.

It currently imlements `ServeHTTP`. Once
https://github.com/sourcegraph/sourcegraph/issues/51656 is added, it
might make sense to turn this into a new interface.

## Demo
### Only batch change jobs

https://github.com/sourcegraph/sourcegraph/assets/2979513/ebe6844c-0fb9-4207-a3ce-f651a1a48631

### Only codeintel jobs

https://github.com/sourcegraph/sourcegraph/assets/2979513/af027eed-6950-4a55-b94c-73847f31ddb8

### Both queue jobs

https://github.com/sourcegraph/sourcegraph/assets/2979513/255c8716-15cb-44e4-9241-f5e28d8d8c7e


## Test plan
- [x] Unit tests
- [x] Demo

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

---------

Co-authored-by: Randell Callahan <piszmogcode@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

General Dequeue Endpoint

4 participants