Skip to content

Storage Backends

Jump to bottom
Refringe edited this page Mar 8, 2026 · 1 revision

Storage Backends

Anchor LFS supports two storage backends: local filesystem (default) and S3-compatible object storage.

Local Filesystem

The local backend stores LFS objects on disk in the configured data_directory. This is the default and requires no additional configuration.

Configuration

[options]
data_directory = "./data"

# No [storage] block needed, or explicitly:
[storage]
backend = "local"

Object Layout

Objects are stored using a two-level directory sharding scheme to avoid filesystem performance issues with large numbers of files in a single directory:

data/
├── <endpoint>/
│   ├── ab/
│   │   ├── cd/
│   │   │   └── abcdef0123456789...  (full OID as filename)
│   │   └── ef/
│   │       └── abef...
│   └── 12/
│       └── 34/
│           └── 1234...
├── locks/
│   └── <endpoint>/
│       └── locks.json
└── signing.key

The first two characters of the OID form the first directory level, the next two form the second level, and the full OID is the filename.

Upload Process

  1. The object is written to a temporary file in the same directory
  2. SHA-256 hash is computed during the write and verified against the declared OID
  3. If the hash matches, the temporary file is atomically renamed to the final path
  4. If the hash does not match, the temporary file is deleted and the upload is rejected

Disk Space

The local backend checks available disk space before accepting uploads. If insufficient space is available, the upload is rejected with an appropriate error. The server also detects and reports ENOSPC (disk full) conditions during writes.

When to Use Local Storage

  • Small to medium deployments
  • Single-server setups
  • Development and testing
  • When you want minimal dependencies

S3-Compatible Storage

The S3 backend stores objects in any S3-compatible object storage service, including AWS S3, Cloudflare R2, MinIO, and others.

Configuration

[storage]
backend = "s3"
s3_bucket = "my-lfs-bucket"
s3_region = "us-east-1"

Full S3 Options

Option Default Description
s3_bucket (required) S3 bucket name
s3_region (required) AWS region (e.g., us-east-1, eu-west-1)
s3_endpoint (AWS default) Custom endpoint URL for S3-compatible services
s3_access_key_id (AWS credential chain) Access key ID. If omitted, uses the default AWS credential chain (IAM roles, environment variables, shared credentials file, etc.)
s3_secret_access_key (AWS credential chain) Secret access key. If omitted, uses the default AWS credential chain.
s3_prefix lfs/ Object key prefix. All objects are stored under this prefix.
s3_presigned_urls true Generate presigned URLs for direct client-to-S3 transfers. When disabled, the server proxies all transfers.
s3_force_path_style false Use path-style addressing (s3.amazonaws.com/bucket) instead of virtual-hosted-style (bucket.s3.amazonaws.com). Required by some S3-compatible services.

S3 Object Layout

Objects use the same two-level sharding as local storage, prefixed with the configured s3_prefix:

lfs/<endpoint>/<oid[0:2]>/<oid[2:4]>/<full-oid>

Presigned URLs

When s3_presigned_urls is enabled (the default), the server generates presigned URLs during batch negotiation. Clients then upload to and download from S3 directly, bypassing the Anchor LFS server for data transfer. This significantly reduces bandwidth and load on the server.

When disabled, the Anchor LFS server proxies all transfers. Clients upload to and download from the server, which then forwards data to/from S3.

Provider-Specific Examples

AWS S3

[storage]
backend = "s3"
s3_bucket = "my-lfs-bucket"
s3_region = "us-east-1"
# Credentials via IAM role or environment variables

Cloudflare R2

[storage]
backend = "s3"
s3_bucket = "my-lfs-bucket"
s3_region = "auto"
s3_endpoint = "https://ACCOUNT_ID.r2.cloudflarestorage.com"
s3_access_key_id = "your-r2-access-key"
s3_secret_access_key = "your-r2-secret-key"

MinIO

[storage]
backend = "s3"
s3_bucket = "lfs"
s3_region = "us-east-1"
s3_endpoint = "https://minio.example.com"
s3_access_key_id = "minioadmin"
s3_secret_access_key = "minioadmin"
s3_force_path_style = true

DigitalOcean Spaces

[storage]
backend = "s3"
s3_bucket = "my-lfs-space"
s3_region = "nyc3"
s3_endpoint = "https://nyc3.digitaloceanspaces.com"
s3_access_key_id = "your-spaces-key"
s3_secret_access_key = "your-spaces-secret"

Environment Variable Overrides

All S3 settings can be overridden via environment variables (useful for keeping credentials out of config files):

export ANCHOR_LFS_STORAGE_BACKEND=s3
export ANCHOR_LFS_S3_BUCKET=my-lfs-bucket
export ANCHOR_LFS_S3_REGION=us-east-1
export ANCHOR_LFS_S3_ACCESS_KEY_ID=your-key
export ANCHOR_LFS_S3_SECRET_ACCESS_KEY=your-secret

When to Use S3 Storage

  • Large deployments with many or very large objects
  • Multi-server or highly available setups
  • When you want offsite or redundant storage
  • When you want to reduce bandwidth through your server (using presigned URLs)

Data That Always Uses Local Storage

Regardless of the storage backend, the following data is always stored locally in the data_directory:

  • File locks (locks/<endpoint>/locks.json): Lock state for each endpoint
  • Signing key (signing.key): Auto-generated HMAC signing key

Ensure the data_directory volume is persistent even when using S3 storage.

Next Steps

  • Configuration: Full configuration reference
  • Docker: Docker deployment with storage volumes
  • Security: Integrity verification and signed URLs

Clone this wiki locally