Skip to content

Event broadcast based cache invalidation#3001

Merged
vigoo merged 47 commits intomainfrom
deploy-event-broadcast
Mar 18, 2026
Merged

Event broadcast based cache invalidation#3001
vigoo merged 47 commits intomainfrom
deploy-event-broadcast

Conversation

@vigoo
Copy link
Copy Markdown
Contributor

@vigoo vigoo commented Mar 17, 2026

Resolves #2431

This PR introduces a new, extensible way how registry service change events can be propagated to other services for cache invalidation purposes. This directly solves one invocation hot-path performance issue but I applied it in other places as well.

The basic architecture is the following:

  • Registry service exports a gRPC stream of events
  • Worker service / worker executor nodes connect (and maintain the connection via reconnects etc) a single connection to an arbitrary registry service node to receive such events, and react to them by invalidating some caches
  • Because in production deployments there can be arbitrary number of load-balanced registry nodes, worker service nodes and executors, the registry service nodes also need to make sure that each node broadcasts every event not just the ones that are directly happening on a call to that particular node. This is done through a notification interface that, when running a single node (in local golem) is a simple in-process notify, and when running with postgres backend is implemented through the database with pg_notify/listen
  • The events are persisted transactionally to an events table, and old events got periodically purged

@vigoo vigoo marked this pull request as ready for review March 17, 2026 14:47
deployment_hash: deployment_revision.hash,
};
// Query domains and record registry change event in the same transaction
let domains = Self::query_domains_for_environment(tx, environment_id).await?;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Domain registrations do not change as part of a deployment, but rather in a separate crud api. This should be checked there.

golem.common.EnvironmentId environment_id = 1;
uint64 deployment_revision = 2;
// Union of old and new domains — includes removed domains for cache invalidation.
repeated string domains = 3;
Copy link
Copy Markdown
Contributor

@mschuwalow mschuwalow Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not part of the deployment / should be separate event

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I took "atomic deployment" too literally, i guess :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything that is part of the manifest is atomic. There is just certain admin stuff that is handled separately / has its own lifecycle and behaves in a special way.

Right now those are:

  • domains
  • security schemes
  • secrets


// Streamed to worker-service subscribers when any registry state changes.
// Each event carries exactly one payload variant.
message RegistryInvalidationEvent {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will also need security scheme changes for the worker service / http api

@vigoo vigoo merged commit ce4a6e9 into main Mar 18, 2026
29 checks passed
@vigoo vigoo deleted the deploy-event-broadcast branch March 18, 2026 09:17
@github-actions github-actions bot locked and limited conversation to collaborators Mar 18, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add event based cache invalidation

3 participants