Skip to content

Implement hub artifact storage with OCI semantics and S3 backend #390

@marccampbell

Description

@marccampbell

Summary

Add a hub-owned artifact storage layer that gives ElasticClaw one storage foundation for checkpoints, volumes, logs, caches, and future large artifacts.

The design should use OCI-style artifact semantics at the ElasticClaw layer and support S3-compatible object storage as a backend. Local filesystem storage should remain available for development and simple deployments.

Design goals

  • Store blobs by content digest.
  • Store manifests that describe typed artifacts.
  • Support refs/tags for named artifact versions, such as volumes/workspace/name:latest.
  • Support custom media types for ElasticClaw artifact kinds.
  • Verify blob and manifest digests.
  • Allow backend implementations for local disk and S3-compatible object stores.
  • Keep permissions, retention, and GC roots in the hub control plane.

Initial artifact store interface

The first version should expose a small internal interface rather than requiring a complete public OCI registry implementation on day one:

type ArtifactStore interface {
    PutBlob(ctx context.Context, r io.Reader) (digest string, size int64, err error)
    GetBlob(ctx context.Context, digest string) (io.ReadCloser, error)
    PutManifest(ctx context.Context, manifest []byte) (digest string, err error)
    GetManifest(ctx context.Context, digest string) ([]byte, error)
    ResolveRef(ctx context.Context, repo, tag string) (digest string, error)
    Tag(ctx context.Context, repo, tag, digest string) error
}

The exact API can change during implementation, but the important boundary is that checkpoints and volumes should not talk directly to S3, local files, or ad hoc blob tables.

Backends

  • Local disk backend for development and single-node installs.
  • S3-compatible backend for durable object storage.
  • Optional later adapter for a full OCI registry or ORAS-compatible endpoints.

Media type direction

Define ElasticClaw media types for future users:

  • application/vnd.elasticclaw.checkpoint.v1+json
  • application/vnd.elasticclaw.volume.v1+json
  • application/vnd.elasticclaw.volume.layer.v1.tar+zstd

Acceptance criteria

  • Hub has an internal artifact store abstraction.
  • Local disk backend is implemented.
  • S3-compatible backend is implemented or clearly scaffolded behind config.
  • Artifacts are content-addressed by digest.
  • Refs/tags can point to manifest digests.
  • Hub metadata can record artifact refs/digests without embedding large blobs in relational tables.
  • Follow-up checkpoint and volume features can build on this API.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions