Extract all queries to drivers, nuke sqlc, pro-specific executable #379

bgentry · 2025-07-15T03:55:01Z

Since it was first prototyped, the UI implemented its own separate sqlc stack independent of the main River library. This has caused a number of issues with keeping the two in sync and resulted in a lot of redundant sqlc-related setup in this project.

We've also had to bundle bits of Pro-related queries and schema into this library's Go backend to facilitate features like workflows and concurrency limiting in the UI. This setup has made it difficult or impossible to properly support certain Pro features (workflow cancellation) without having to fully reimplement them in the UI's backend, and there's a limit to how reliably we can override things like JobRetry for sequences because the UI is typically built + distributed against the OSS river without Pro-specific overrides. We can't take dependencies on Pro-specific types without making it a hard dependency of this library or via hacky solutions like build flags, but even then Go modules would expect the pro modules to always be available.

To resolve this, this PR proofs out a pattern where there's a separate riverproui module and executable which is able to register its own additional endpoints and features to riverui's API. This separate module has a dependency on riverqueue.com/riverpro while avoiding it on the rest of the project. This allows the riverpro package to leverage its own client, pilot, and driver types and to expose those via the riverui Go backend over JSON. The frontend has been updated to check for the appropriate flags from the backend before assuming the Pro APIs are available and attempting to call them.

The UI has also been reworked to be driver-agnostic instead of only supporting riverpgxv5. As of now this is structured as a hard breaking change. ⚠️

As a nice bonus, there's now an npm run dev:pro to launch the Pro UI + backend, alongside the preexisting npm run dev. This makes it trivial to develop against either scenario, and I was able to fix at least one major bug thanks to this. I think it'd be helpful to set up visual testing of key pages for both of these in the future.

Pairs with riverqueue/river#983.

TODOs

proof out CI + build infra for riverproui
dedicated Docker registry for Pro users to access Pro-specific UI images in case they want to just run one without embedding into an existing app
comment out go.mod replaces and instead point at merged Integrate riverui queries into driver system, add Client.Schema() river#983 commit
Changelog

brandur

This piece of the trifecta looks great! Eliminates a ton of the fairly duplicative stuff that we had in River UI and gives us a path for it to potentially operate on other database engines too (I don't think anyone's asked for this so far, but it'd be nice to at least keep if possible).

cmd/riverproui/main.go

bgentry · 2025-07-27T15:40:32Z

example_test.go

 	// Create the River UI server. This server implements http.Handler and can be
 	// mounted in an HTTP mux
-	server, err := riverui.NewServer(&riverui.ServerOpts{
-		Client:  client,
+	server, err := riverui.NewServer(riverui.NewEndpoints(&riverui.EndpointsOpts[pgx.Tx]{
+		Client: client,
+	}), &riverui.ServerOpts{
 		DevMode: true, // Use the live filesystem—don't use this outside tests
-		DB:      dbPool,
 		Logger:  logger,
 		Prefix:  "/riverui", // Mount the UI under /riverui path
 	})


Not liking this so far, both from naming but also from the multiple constructors and opts structs. I ran through a few examples below to illustrate alternatives, though the other options I can come up with will require keeping several structs aligned and copying between them properly.

// Example 1: a separate endpoints bundle, separate endpoints and server opts // structs. This is what's currently implemented. server, err := riverui.NewServer( riverui.NewEndpoints(&riverui.EndpointsOpts{ Client: client, JobListHideArgsByDefault: true, }), &riverui.ServerOpts{ Logger: logger, Prefix: "/riverui", // Mount the UI under /riverui path }, ) if err != nil { panic(err) } server, err := riverui.NewServer( riverproui.NewEndpoints(&riverproui.EndpointsOpts{ Client: client, JobListHideArgsByDefault: true, }), &riverui.ServerOpts{ Logger: logger, Prefix: "/riverui", // Mount the UI under /riverui path }, ) if err != nil { panic(err) } // Example 2: same as example 1, but with unified opts structs. Server opts would // need to be surfaced by the endpoints bundle. server, err := riverui.NewServer( riverui.NewEndpoints(&riverui.EndpointsOpts{ Client: client, JobListHideArgsByDefault: true, Logger: logger, Prefix: "/riverui", // Mount the UI under /riverui path }), ) if err != nil { panic(err) } server, err := riverui.NewServer( riverproui.NewEndpoints(&riverproui.EndpointsOpts{ Client: client, JobListHideArgsByDefault: true, Logger: logger, Prefix: "/riverui", // Mount the UI under /riverui path }), ) if err != nil { panic(err) } // Example 3: no "endpoint bundle", just two different constructors and opts structs: server, err := riverui.NewServer( &riverui.ServerOpts{ Client: client, JobListHideArgsByDefault: true, Logger: logger, Prefix: "/riverui", // Mount the UI under /riverui path }, ) if err != nil { panic(err) } server, err := riverproui.NewServer( &riverproui.ServerOpts{ Client: client, JobListHideArgsByDefault: true, Logger: logger, Prefix: "/riverui", // Mount the UI under /riverui path }, ) if err != nil { panic(err) }

Yeah, seems tricky for sure. Proposal seems pretty good.

Just looking at this again, I think there is a chance that all the alternatives mentioned above are too fixated on the original design that proposes multiple structs with multiple option structs. Here's what I'd propose an alternative: one "flat" server with options struct that takes a "flat" endpoints bundle:

server, err := riverui.NewServer(&riverui.ServerOpts{ DevMode: true, // Use the live filesystem—don't use this outside tests Endpoints: riverui.NewEndpoints(client, nil), Logger: logger, Prefix: "/riverui", // Mount the UI under /riverui path }) if err != nil { panic(err) }

Notably:

The endpoints bundle becomes an options property instead of a position parameter. I think this is fine because ServerOpts is required anyway. Endpoints just becomes a required property on a required struct.

client becomes a position parameter of NewEndpoints instead of being in the options struct. This is because client is always required, but other options will rarely be required, making it ergonomic to leave options as nil most of the time (i.e. riverui.NewEndpoints(client, nil)).

The one non-test option in the endpoint options migratest to server options instead:

uiServer, err := riverui.NewServer(&riverui.ServerOpts{ DevMode: devMode, Endpoints: createBundler(client), JobListHideArgsByDefault: jobListHideArgsByDefault, LiveFS: liveFS, Logger: logger, Prefix: pathPrefix, })

The server sends endpoints-specific options down using a new Configure function:

opts.Endpoints.Configure(&apibundle.EndpointBundleOpts{ JobListHideArgsByDefault: opts.JobListHideArgsByDefault, })

I think that simplifies/prettifies things quite a bit with no disadvantages I can see. Thoughts?

Here's the complete diff for reference:

diff --git a/cmd/riverui/main.go b/cmd/riverui/main.go index 18bf156..53fd400 100644 --- a/cmd/riverui/main.go +++ b/cmd/riverui/main.go @@ -16,11 +16,8 @@ func main() { func(dbPool *pgxpool.Pool) (*river.Client[pgx.Tx], error) { return river.NewClient(riverpgxv5.New(dbPool), &river.Config{}) }, - func(client *river.Client[pgx.Tx], opts *riveruicmd.BundleOpts) apibundle.EndpointBundle { - return riverui.NewEndpoints(&riverui.EndpointsOpts[pgx.Tx]{ - Client: client, - JobListHideArgsByDefault: opts.JobListHideArgsByDefault, - }) + func(client *river.Client[pgx.Tx]) apibundle.EndpointBundle { + return riverui.NewEndpoints(client, nil) }, ) } diff --git a/example_test.go b/example_test.go index 07a3957..b821fc5 100644 --- a/example_test.go +++ b/example_test.go @@ -10,7 +10,6 @@ import ( "os" "strings" - "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" "riverqueue.com/riverui" @@ -52,12 +51,11 @@ func ExampleNewServer() { // Create the River UI server. This server implements http.Handler and can be // mounted in an HTTP mux - server, err := riverui.NewServer(riverui.NewEndpoints(&riverui.EndpointsOpts[pgx.Tx]{ - Client: client, - }), &riverui.ServerOpts{ - DevMode: true, // Use the live filesystem—don't use this outside tests - Logger: logger, - Prefix: "/riverui", // Mount the UI under /riverui path + server, err := riverui.NewServer(&riverui.ServerOpts{ + DevMode: true, // Use the live filesystem—don't use this outside tests + Endpoints: riverui.NewEndpoints(client, nil), + Logger: logger, + Prefix: "/riverui", // Mount the UI under /riverui path }) if err != nil { panic(err) diff --git a/handler.go b/handler.go index 8bcbb90..98a0fbc 100644 --- a/handler.go +++ b/handler.go @@ -32,32 +32,37 @@ type endpointsExtensions interface { } type EndpointsOpts[TTx any] struct { - Client *river.Client[TTx] - JobListHideArgsByDefault bool // Tx is an optional transaction to wrap all database operations. It's mainly // used for testing. Tx *TTx } type endpoints[TTx any] struct { - opts *EndpointsOpts[TTx] + bundleOpts *apibundle.EndpointBundleOpts + client *river.Client[TTx] + opts *EndpointsOpts[TTx] } -func NewEndpoints[TTx any](opts *EndpointsOpts[TTx]) apibundle.EndpointBundle { +func NewEndpoints[TTx any](client *river.Client[TTx], opts *EndpointsOpts[TTx]) apibundle.EndpointBundle { return &endpoints[TTx]{ - opts: opts, + client: client, + opts: opts, } } +func (e *endpoints[TTx]) Configure(bundleOpts *apibundle.EndpointBundleOpts) { + e.bundleOpts = bundleOpts +} + func (e *endpoints[TTx]) Validate() error { - if e.opts.Client == nil { + if e.client == nil { return errors.New("client is required") } return nil } func (e *endpoints[TTx]) MountEndpoints(archetype *baseservice.Archetype, logger *slog.Logger, mux *http.ServeMux, mountOpts *apiendpoint.MountOpts, extensions map[string]bool) []apiendpoint.EndpointInterface { - driver := e.opts.Client.Driver() + driver := e.client.Driver() var executor riverdriver.Executor if e.opts.Tx == nil { executor = driver.GetExecutor() @@ -66,11 +71,11 @@ func (e *endpoints[TTx]) MountEndpoints(archetype *baseservice.Archetype, logger } bundle := apibundle.APIBundle[TTx]{ Archetype: archetype, - Client: e.opts.Client, + Client: e.client, DB: executor, Driver: driver, Extensions: extensions, - JobListHideArgsByDefault: e.opts.JobListHideArgsByDefault, + JobListHideArgsByDefault: e.bundleOpts.JobListHideArgsByDefault, Logger: logger, } @@ -95,7 +100,9 @@ func (e *endpoints[TTx]) MountEndpoints(archetype *baseservice.Archetype, logger // ServerOpts are the options for creating a new Server. type ServerOpts struct { // DevMode is whether the server is running in development mode. - DevMode bool + DevMode bool + Endpoints apibundle.EndpointBundle + JobListHideArgsByDefault bool // LiveFS is whether to use the live filesystem for the frontend. LiveFS bool // Logger is the logger to use logging errors within the handler. @@ -137,11 +144,11 @@ type Server struct { } // NewServer creates a new Server that serves the River UI and API. -func NewServer(bundle apibundle.EndpointBundle, opts *ServerOpts) (*Server, error) { - if bundle == nil { +func NewServer(opts *ServerOpts) (*Server, error) { + if opts.Endpoints == nil { return nil, errors.New("endpoints is required") } - if err := bundle.Validate(); err != nil { + if err := opts.Endpoints.Validate(); err != nil { return nil, err } @@ -152,6 +159,10 @@ func NewServer(bundle apibundle.EndpointBundle, opts *ServerOpts) (*Server, erro return nil, err } + opts.Endpoints.Configure(&apibundle.EndpointBundleOpts{ + JobListHideArgsByDefault: opts.JobListHideArgsByDefault, + }) + prefix := cmp.Or(strings.TrimSuffix(opts.Prefix, "/"), "") frontendIndex, err := fs.Sub(FrontendIndex, "dist") @@ -206,11 +217,11 @@ func NewServer(bundle apibundle.EndpointBundle, opts *ServerOpts) (*Server, erro } extensions := map[string]bool{} - if withExtensions, ok := bundle.(endpointsExtensions); ok { + if withExtensions, ok := opts.Endpoints.(endpointsExtensions); ok { extensions = withExtensions.Extensions() } - endpoints := bundle.MountEndpoints(baseservice.NewArchetype(opts.Logger), opts.Logger, mux, &mountOpts, extensions) + endpoints := opts.Endpoints.MountEndpoints(baseservice.NewArchetype(opts.Logger), opts.Logger, mux, &mountOpts, extensions) var services []startstop.Service diff --git a/handler_test.go b/handler_test.go index 727c131..6fb7992 100644 --- a/handler_test.go +++ b/handler_test.go @@ -25,9 +25,8 @@ func TestNewHandlerIntegration(t *testing.T) { createClient := insertOnlyClient createBundle := func(client *river.Client[pgx.Tx], tx pgx.Tx) apibundle.EndpointBundle { - return NewEndpoints(&EndpointsOpts[pgx.Tx]{ - Client: client, - Tx: &tx, + return NewEndpoints(client, &EndpointsOpts[pgx.Tx]{ + Tx: &tx, }) } @@ -35,8 +34,9 @@ func TestNewHandlerIntegration(t *testing.T) { t.Helper() logger := riverinternaltest.Logger(t) - server, err := NewServer(bundle, &ServerOpts{ + server, err := NewServer(&ServerOpts{ DevMode: true, + Endpoints: bundle, LiveFS: true, Logger: logger, projectRoot: "./", diff --git a/internal/apibundle/api_bundle.go b/internal/apibundle/api_bundle.go index 2f8fff9..361f8cd 100644 --- a/internal/apibundle/api_bundle.go +++ b/internal/apibundle/api_bundle.go @@ -21,7 +21,12 @@ type APIBundle[TTx any] struct { Logger *slog.Logger } +type EndpointBundleOpts struct { + JobListHideArgsByDefault bool +} + type EndpointBundle interface { + Configure(bundleOpts *EndpointBundleOpts) MountEndpoints(archetype *baseservice.Archetype, logger *slog.Logger, mux *http.ServeMux, mountOpts *apiendpoint.MountOpts, extensions map[string]bool) []apiendpoint.EndpointInterface Validate() error } diff --git a/internal/riveruicmd/riveruicmd.go b/internal/riveruicmd/riveruicmd.go index 5a3f48e..febdd58 100644 --- a/internal/riveruicmd/riveruicmd.go +++ b/internal/riveruicmd/riveruicmd.go @@ -26,7 +26,7 @@ type BundleOpts struct { JobListHideArgsByDefault bool } -func Run[TClient any](createClient func(*pgxpool.Pool) (TClient, error), createBundle func(TClient, *BundleOpts) apibundle.EndpointBundle) { +func Run[TClient any](createClient func(*pgxpool.Pool) (TClient, error), createBundle func(TClient) apibundle.EndpointBundle) { ctx := context.Background() logger := slog.New(getLogHandler(&slog.HandlerOptions{ @@ -88,7 +88,7 @@ type initServerResult struct { uiServer *riverui.Server // River UI server } -func initServer[TClient any](ctx context.Context, logger *slog.Logger, pathPrefix string, createClient func(*pgxpool.Pool) (TClient, error), createBundler func(TClient, *BundleOpts) apibundle.EndpointBundle) (*initServerResult, error) { +func initServer[TClient any](ctx context.Context, logger *slog.Logger, pathPrefix string, createClient func(*pgxpool.Pool) (TClient, error), createBundler func(TClient) apibundle.EndpointBundle) (*initServerResult, error) { if !strings.HasPrefix(pathPrefix, "/") || pathPrefix == "" { return nil, fmt.Errorf("invalid path prefix: %s", pathPrefix) } @@ -127,15 +127,13 @@ func initServer[TClient any](ctx context.Context, logger *slog.Logger, pathPrefi return nil, err } - bundler := createBundler(client, &BundleOpts{ + uiServer, err := riverui.NewServer(&riverui.ServerOpts{ + DevMode: devMode, + Endpoints: createBundler(client), JobListHideArgsByDefault: jobListHideArgsByDefault, - }) - - uiServer, err := riverui.NewServer(bundler, &riverui.ServerOpts{ - DevMode: devMode, - LiveFS: liveFS, - Logger: logger, - Prefix: pathPrefix, + LiveFS: liveFS, + Logger: logger, + Prefix: pathPrefix, }) if err != nil { return nil, err diff --git a/internal/riveruicmd/riveruicmd_test.go b/internal/riveruicmd/riveruicmd_test.go index 89de6e4..bb52bf6 100644 --- a/internal/riveruicmd/riveruicmd_test.go +++ b/internal/riveruicmd/riveruicmd_test.go @@ -40,11 +40,8 @@ func TestInitServer(t *testing.T) { func(dbPool *pgxpool.Pool) (*river.Client[pgx.Tx], error) { return river.NewClient(riverpgxv5.New(dbPool), &river.Config{}) }, - func(client *river.Client[pgx.Tx], opts *BundleOpts) apibundle.EndpointBundle { - return riverui.NewEndpoints(&riverui.EndpointsOpts[pgx.Tx]{ - Client: client, - JobListHideArgsByDefault: opts.JobListHideArgsByDefault, - }) + func(client *river.Client[pgx.Tx]) apibundle.EndpointBundle { + return riverui.NewEndpoints(client, nil) }, ) require.NoError(t, err)

Amazing, thank you! This was nearly complete, but needed a few more fixes for the riverproui side which I took care of. We're also fully rebased.

Getting close—a few more tweaks to suit the other comments, along with a changelog entry to cover all of this.

bgentry · 2025-07-27T15:41:22Z

riverproui/cmd/riverproui/main.go

The interface here isn't perfect, but it does show that we can share essentially all logic between the two executables.

brandur

Looking good!

authmiddleware/auth.go

brandur · 2025-07-27T22:28:11Z

example_test.go

 	// Create the River UI server. This server implements http.Handler and can be
 	// mounted in an HTTP mux
-	server, err := riverui.NewServer(&riverui.ServerOpts{
-		Client:  client,
+	server, err := riverui.NewServer(riverui.NewEndpoints(&riverui.EndpointsOpts[pgx.Tx]{
+		Client: client,
+	}), &riverui.ServerOpts{
 		DevMode: true, // Use the live filesystem—don't use this outside tests
-		DB:      dbPool,
 		Logger:  logger,
 		Prefix:  "/riverui", // Mount the UI under /riverui path
 	})


Yeah, seems tricky for sure. Proposal seems pretty good.

brandur · 2025-07-27T22:29:24Z

handler.go

 	"io"
 	"io/fs"
 	"log/slog"
 	"net/http"


Might not be the worst idea to rename this file or break it up into multiple files or something. It's not obvious that this is the place that you should look to find API endpoints and Server.

brandur · 2025-07-27T22:30:02Z

handler.go

+	opts *EndpointsOpts[TTx]
+}
+
+func NewEndpoints[TTx any](opts *EndpointsOpts[TTx]) apibundle.EndpointBundle {


Given Endpoints is such a user-facing construct, it should probably get a little more documentation explaining what this is and maybe with basic usage information.

Took care of this here: #402

handler.go

handler_test.go

brandur

Cool! Looking good.

bgentry · 2025-08-16T16:17:51Z

@brandur I added a changelog here, though if we want to go ahead with the handler rename it'll need some tweaking. Let's also make any tweaks to it as a follow up so I can get this merged and further prep for release.

A mistake was made in #379 by including internal types in user-facing structs and as arguments, specifically by exposing things from the `internal/apibundle` package like `EndpointsBundle`. To rectify this, move user-facing types into a top-level `uiendpoints` package (not internal) which now includes a `Bundle` and `BundleOpts` type that can be referenced both from `riverui` and from packages in `internal` without creating a dependency loop.

* put user-facing bundle in uiendpoints A mistake was made in #379 by including internal types in user-facing structs and as arguments, specifically by exposing things from the `internal/apibundle` package like `EndpointsBundle`. To rectify this, move user-facing types into a top-level `uiendpoints` package (not internal) which now includes a `Bundle` and `BundleOpts` type that can be referenced both from `riverui` and from packages in `internal` without creating a dependency loop. * group riverqueue.com/riverui imports separately * don't build Docker on pull_request *

…able (#379)

bgentry requested a review from brandur July 15, 2025 03:55

brandur reviewed Jul 24, 2025

View reviewed changes

cmd/riverproui/main.go Show resolved Hide resolved

bgentry force-pushed the bg-pro-integration branch 2 times, most recently from 20b39b7 to d150c18 Compare July 27, 2025 15:37

bgentry commented Jul 27, 2025

View reviewed changes

bgentry force-pushed the bg-pro-integration branch from d150c18 to a09144c Compare July 27, 2025 15:45

bgentry requested a review from brandur July 27, 2025 15:46

brandur reviewed Jul 27, 2025

View reviewed changes

bgentry force-pushed the bg-pro-integration branch 21 times, most recently from f82e808 to 198aaa1 Compare August 3, 2025 20:53

brandur approved these changes Aug 14, 2025

View reviewed changes

bgentry mentioned this pull request Aug 16, 2025

Use schema-based testing #396

Closed

use riverpro v0.16.0-rc.2 prerelease

34e59d3

bgentry force-pushed the bg-pro-integration branch 2 times, most recently from 0649671 to d1862b4 Compare August 16, 2025 15:33

bgentry added 4 commits August 16, 2025 10:34

docker publishing pipeline for riverproui

c2c01a5

feedback fixes

4edf69d

mimic makefile test commands from main river

a35d39e

changelog

4e7ecbc

bgentry force-pushed the bg-pro-integration branch 6 times, most recently from c32ccb9 to 7c06e46 Compare August 16, 2025 16:10

lint fixes

f352202

bgentry force-pushed the bg-pro-integration branch from 7c06e46 to f352202 Compare August 16, 2025 16:14

bgentry merged commit 828af7a into master Aug 16, 2025
34 of 37 checks passed

bgentry deleted the bg-pro-integration branch August 16, 2025 16:29

bgentry mentioned this pull request Aug 16, 2025

recommend riverproui switch on workflow, concurrency CTAs #401

Merged

This was referenced Aug 16, 2025

put user-facing bundle in uiendpoints #402

Merged

Remove __riverUiBasePath and __riverUiAssetUrl global functions #382

Merged

bgentry mentioned this pull request Aug 20, 2025

Workflow cancellation #407

Merged

bgentry added a commit that referenced this pull request Sep 28, 2025

fixup! Extract all queries to drivers, nuke sqlc, pro-specific execut…

bd2e8d5

…able (#379)

Extract all queries to drivers, nuke sqlc, pro-specific executable #379

Extract all queries to drivers, nuke sqlc, pro-specific executable #379

Uh oh!

Conversation

bgentry commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODOs

Uh oh!

brandur left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brandur left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandur left a comment

Choose a reason for hiding this comment

Uh oh!

bgentry commented Aug 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bgentry commented Jul 15, 2025 •

edited

Loading