Skip to content

Experimental feature: Durable Object Facets#4123

Merged
kentonv merged 16 commits intomainfrom
kenton/facets
Jun 17, 2025
Merged

Experimental feature: Durable Object Facets#4123
kentonv merged 16 commits intomainfrom
kenton/facets

Conversation

@kentonv
Copy link
Member

@kentonv kentonv commented May 9, 2025

This is a new experimental Durable Objects feature. It requires the --experimental CLI flag and experimental compat flag to use. I fully expect that we may end up changing the design once we've played with it, and that's fine. This isn't intended to become generally available any time soon.

Feature Specification

Multifaceted Durable Objects are Durable Objects (DOs) which are composed of multiple "facets" implemented by different isolates. Each facet is written like a regular DO class, but all the facets run together on the same machine to implement the DO. Each facet has its own storage in the form of a SQLite database, but these databases are all stored together as one logical object.

A multifaceted DO has a "main" facet which implements its public interface. This works exactly like a regular Durable Object. The main facet's implementation can call out to other "facets" as needed, using ctx.facets:

// The main interface to manage facets is on `ctx`.
let facets = this.ctx.facets;

// Each facet has a name, which identifies its respective slice of storage. Names are hierarchical:
// if the main facet creates a facet called "foo", and that facet in turn creates a facet called
// "bar", the latter facet's true name is "foo/bar". A facet cannot directly access its siblings,
// unless the common parent facet chooses to pass references explicitly.
//
// Names are limited to 128 characters and cannot contain control characters nor `/`.
let name = "foo";

// To use a facet, pass the name to `this.ctx.facets.get()`.
//
// The second parameter specifies information needed to start up the facet. If the facet is already
// running, and matches the given parameters, the already-running instance is returned. If the
// parameters have changed, then the existing instance will shut down and will reload using the
// new parameters.
//
// Like with regular Durable Objects, there is no explicit "create" operation for a facet. It is
// implicitly created when first used, and is implicitly deleted if it shuts down with nothing left
// in storage.
let facet = this.ctx.facets.get(name, {
  // The class to use. This is a facet binding, which points to a class that may be defined
  // in a different Worker. As usual, ctx.exports can be used to refer to other classes
  // exported by the current worker. Dynamic Dispatch can also look up facets using
  // `getFacetClass`.
  class: this.env.FOO_FACET_CLASS,

  // The value that the facet sees in `ctx.id`. This can be a real Durable Object ID or a plain
  // string. In this example, the parent uses the facet name as the ID, but this may or may not
  // make sense depending on the use case. If not specified, defaults to inheriting the parent's
  // ID.
  id: name
});

// `facet` acts like a DO stub, except all calls are local.
await facet.someRpcMethod();

// `facets.abort()` forcefully aborts the facet immediately. No further code will execute in the
// facet until it is started again. The next call to `facets.get()` is guaranteed to call the
// callback and start a new instance. `reason` is thrown by any outstanding or future RPC calls
// on existing stubs pointing into the facet.
//
// This also transitively aborts all children of the facet.
this.ctx.facets.abort(name, reason);

// Deleting a facet aborts the facet if it is running and then deletes its underlying storage. This
// applies transitively to all children as well.
this.ctx.facets.delete(name);

// Note that `storage.deleteAll()` deletes all facets in addition to regular storage.
this.ctx.storage.deleteAll();

Notes:

  • When any running facet becomes "broken", the entire actor breaks and will restart. There is currently no mechanism to catch errors in the parent, though one might be added in the future. As a special exception, if a facet becomes broken because a parent used facets.abort() or facets.delete() on it, this does not break the entire actor, only the specific facet.

Storage Details

In workerd, a DO's main database has always been stored in a file called <actor-id>.sqlite. Each non-root facet is stored in a separate file <actor-id>.<facet-id>.sqlite, where <facet-id> is a small integer assigned to each facet. An index of facet IDs is maintained in a separate file, <actor-id>.facets. This file contains a simple capnp-encoded representation of the facet tree, covering at least all facets for which files exist on disk. This index enables us to avoid encoding facet names into file names, and makes it possible to list and delete facets without performing a full directory listing (which may contain files for unrelated actors).

In production (not yet implemented), facets will only be supported when using SRS to store actor data (not the old storage backend). Each facet is stored as a separate SRS "lane". Lane names are prefixed by their facet path, which is the list of facet names leading from the root to the specific facet, separated by / characters.

@kentonv kentonv requested a review from MellowYarker May 9, 2025 21:53
@kentonv kentonv requested review from a team as code owners May 9, 2025 21:53
@github-actions
Copy link

github-actions bot commented May 9, 2025

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

@G4brym
Copy link
Member

G4brym commented May 19, 2025

any chance we could get a "facet clone" method?
in theory since all the facets storage are on the same metal, it could "just" run a cp <actor-id>.1.sqlite <actor-id>.2.sqlite and then boot the facet, right?

maybe something along these lines:

const facetA = this.ctx.facets.get('name-a');

const facetB = this.ctx.facets.clone('name-b', facetA);
// or
const facetB = this.ctx.facets.clone('name-b', 'name-a');

Copy link
Contributor

@MellowYarker MellowYarker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pretty cool! It'll be interesting to see what we can build on top of Facets.

@kentonv
Copy link
Member Author

kentonv commented Jun 13, 2025

Just rebased... but it was a pretty ugly rebase due to clashes with the new containers feature. Wish I'd landed this earlier, ugh.

@kentonv kentonv force-pushed the kenton/facets branch 2 times, most recently from d0a4031 to ec2c840 Compare June 13, 2025 21:31
@kentonv
Copy link
Member Author

kentonv commented Jun 13, 2025

(and another rebase because I hadn't rebased all the way to main yet and there were MORE conflicts (my own fault))

@kentonv kentonv force-pushed the kenton/facets branch 2 times, most recently from 9ed463b to 88338af Compare June 13, 2025 22:28
@kentonv
Copy link
Member Author

kentonv commented Jun 13, 2025

Urgh and that version still didn't compile when merged with main so I rebased again and added a commit to fix the problems.

@kentonv
Copy link
Member Author

kentonv commented Jun 14, 2025

Rebased again to pick up #4332 to fix CI failure, hopefully.

No that had nothing to do with the CI failure. I was mislead by the word "snapshot" but it's a totally different type of snapshot.

@kentonv
Copy link
Member Author

kentonv commented Jun 14, 2025

The check-snapshot build is actually passing, but then failing to post a comment saying it passed. Can ignore.

Just pushed a tweak to work around Windows path-parsing behavior in the test.

@MellowYarker
Copy link
Contributor

Will take another pass at this tomorrow morning.

kentonv added 10 commits June 17, 2025 14:33
…lass.

The goal is to make `ActorNamespace` independent of `WorkerService`, although this doesn't get all the way there yet.

The reason we want `ActorNamespace` to be independent is that multifaceted actors may contain instances of classes from multiple services.
…d scheduler.

The actor storage should actually originate from the namespace's parent service and NOT from the service that hosts the actor class.

The alarm scheduler is a server-wide singleton so it doesn't matter where we get it from. Might as well come with the storage.

This eliminates dependencies on the ActorClass's service for any purpose other than constructing the Actor object.
This is a binding that points to a DO class _without_ a storage namespace attached. It's more like a service binding than a DO namespace binding.

DO class bindings will be used with facets.
With facets, each facet could have a different class, so we can't get this from the parent namespace anymore.
There's not yet any API to instantiate them. This is just setting up the scaffolding.
From the design doc:

```js
// The main interface to manage facets is on `ctx`.
let facets = this.ctx.facets;

// Each facet has a name, which identifies its respective slice of storage. Names are hierarchical:
// if the main facet creates a facet called "foo", and that facet in turn creates a facet called
// "bar", the latter facet's true name is "foo/bar". A facet cannot directly access its siblings,
// unless the common parent facet chooses to pass references explicitly.
//
// Names are limited to 128 characters and cannot contain control characters nor `/`.
let name = "foo";

// To use a facet, pass the name to `this.ctx.facets.get()`.
//
// The second parameter specifies information needed to start up the facet. If the facet is already
// running, and matches the given parameters, the already-running instance is returned. If the
// parameters have changed, then the existing instance will shut down and will reload using the
// new parameters.
//
// Like with regular Durable Objects, there is no explicit "create" operation for a facet. It is
// implicitly created when first used, and is implicitly deleted if it shuts down with nothing left
// in storage.
let facet = this.ctx.facets.get(name, {
  // The class to use. This is a facet binding, which points to a class that may be defined
  // in a different Worker. As usual, ctx.exports can be used to refer to other classes
  // exported by the current worker. Dynamic Dispatch can also look up facets using
  // `getFacetClass`.
  class: this.env.FOO_FACET_CLASS,

  // The value that the facet sees in `ctx.id`. This can be a real Durable Object ID or a plain
  // string. In this example, the parent uses the facet name as the ID, but this may or may not
  // make sense depending on the use case. If not specified, defaults to inheriting the parent's
  // ID.
  id: name
});

// `facet` acts like a DO stub, except all calls are local.
await facet.someRpcMethod();

// `facets.abort()` forcefully aborts the facet immediately. No further code will execute in the
// facet until it is started again. The next call to `facets.get()` is guaranteed to call the
// callback and start a new instance. `reason` is thrown by any outstanding or future RPC calls
// on existing stubs pointing into the facet.
//
// This also transitively aborts all children of the facet.
this.ctx.facets.abort(name, reason);

// Deleting a facet aborts the facet if it is running and then deletes its underlying storage. This
// applies transitively to all children as well.
this.ctx.facets.delete(name);

// Note that `storage.deleteAll()` deletes all facets in addition to regular storage.
this.ctx.storage.deleteAll();
```
This commit was assisted by Claude Code. I wrote the header file, then prompted it to fill in the implementation and test. There was a lot of iteration on the details, culminating in me heavily editing the implementation by hand. I'm not sure if it saved time.
Each actor namespace has always had its own directory on disk, but previously it did not create a kj::Directory for it. Instead, a single SqliteDatabase::Vfs was created for the parent of all the actor storage directories, and each sqlite file was opened by path that included the actor namespace directory.

That had two problems:
* It's ugly that an ActorNamespace could open files outside its own storage, and just has to promise to always use the correct path to its storage.
* We actually didn't give the ActorNamespace a kj::Directory at all, just to the single SqliteDatabase::Vfs. But we now want to open other kinds of files in this directory, so we need a kj::Directory. Let's just actually open the directory specific to the namespace, and create a separate Vfs per-namespace.
kentonv added 6 commits June 17, 2025 14:33
Not going to try to rebase these back into the history, too complicated.
This wasn't a direct merge conflict but the latest rebase doesn't build. This fixes it but I don't want to figure out how to push this back into the correct spot in the history.
@kentonv
Copy link
Member Author

kentonv commented Jun 17, 2025

I just pushed a rebase (to get the fix for check-snapshot), but this PR thread page doesn't seem to have noticed the push. https://github.com/cloudflare/workerd/tree/kenton/facets does show it. I dunno what's going on. Maybe it'll catch up later.

@kentonv kentonv merged commit 712d88f into main Jun 17, 2025
31 of 35 checks passed
@kentonv kentonv deleted the kenton/facets branch June 17, 2025 22:35
@dmaretskyi
Copy link

As an external developer building on top of cloudflare, what's the benefit of using facets instead of just groups of Durable Objects? Is it so they are forced to run on the same machiene which gurantees low-latency RPCs between them?

@kentonv
Copy link
Member Author

kentonv commented Jun 29, 2025

@dmaretskyi

As an external developer building on top of cloudflare, what's the benefit of using facets instead of just groups of Durable Objects? Is it so they are forced to run on the same machiene which gurantees low-latency RPCs between them?

That's one reason.

The other reason is resource / namespace management. If you have a "group" of several objects working together, where each one runs different code, you now have to have several different namespaces in which those objects live, and each object has to have an ID, etc. You also need a convention for how these objects get cleaned up when they are no longer needed.

With facets, there's only one namespace needed -- the one for the root facet. Deleting the root facet deletes all the children.

This advantage is particularly acute when combined with #4383, dynamic worker loading: You are dynamically loading code for a one-off worker that defines a Durable Object class. Without facets, the only way to actually run that Durable Object would be to provision a whole namespace for it. Dynamically loading workers makes sense, but dynamically creating DO namespaces seems wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants