Skip to content

Troubleshooting

Refringe edited this page Mar 8, 2026 · 1 revision

Troubleshooting

Common issues and their solutions when running Anchor LFS.

Server Won't Start

"failed to load configuration"

The server can't find or parse the config file.

  • Ensure config.toml exists in the working directory, or set ANCHOR_LFS_CONFIG to the correct path
  • Check for TOML syntax errors (mismatched quotes, missing brackets)
  • Run the server from the directory containing config.toml

"no endpoints configured"

At least one [[endpoints]] block is required in the configuration file.

"endpoint missing name" / "missing endpoint path"

Every endpoint must have both name and endpoint fields set.

"duplicate endpoint path"

Two endpoints have the same endpoint path. Each endpoint must have a unique path.

"path is reserved"

The endpoint path conflicts with a reserved server path (e.g., /health). Choose a different path.

"URL is not a github.com URL; set github_api_url for GitHub Enterprise"

When using a non-github.com repository URL with authentication = "github", you must also set github_api_url to the GitHub Enterprise API endpoint.

"signing_key must be at least 32 bytes (64 hex characters)"

The configured signing key is too short. Generate a proper key:

openssl rand -hex 32

"storage backend requires s3_bucket" / "requires s3_region"

When using S3 storage, both s3_bucket and s3_region are required.

Authentication Issues

401 Unauthorised on All Requests

  • Verify the endpoint's authentication setting is correct
  • Ensure the GitHub PAT hasn't expired
  • Check that the PAT has the required repository permissions
  • For private endpoints, both downloads and uploads require authentication

403 Forbidden on Upload

  • The GitHub PAT needs push (write) permission on the repository
  • Verify the url field in the endpoint matches the GitHub repository the token has access to

Authentication Works Intermittently

  • Check auth_cache_ttl as cached results may be stale. Try setting it to "0s" temporarily to disable caching
  • Verify the GitHub API is accessible from the server (check network/firewall rules)

GitHub Enterprise Authentication Fails

  • Ensure github_api_url includes the trailing slash: https://github.example.com/api/v3/
  • Verify the server can reach the GitHub Enterprise instance
  • Check that the PAT was issued by the correct GitHub Enterprise instance

Transfer Issues

Uploads Fail with "hash mismatch"

The uploaded file's SHA-256 hash doesn't match the declared OID. This usually indicates:

  • File corruption during transfer
  • A proxy or middleware is modifying the request body
  • The client computed the hash incorrectly

Uploads Rejected as Too Large

The file exceeds max_upload_size (default: 5 GiB). Increase the limit in configuration:

[options]
max_upload_size = 10737418240  # 10 GiB

Also check your reverse proxy configuration. Nginx defaults to a 1 MB body size limit:

client_max_body_size 0;  # Disable limit

Downloads Return 403 Forbidden

  • The signed URL may have expired. Check url_expiry in your configuration.
  • The signing key may have changed between when the URL was generated and when it was used (e.g., server restart without persisted key, or different key across instances)

Timeouts on Large Files

If transfers time out, check your reverse proxy timeout settings. Anchor LFS itself has no fixed read/write timeouts, but your proxy might:

Nginx:

proxy_read_timeout 3600s;
proxy_send_timeout 3600s;

Apache:

ProxyTimeout 3600

507 Insufficient Storage

The disk is full or has insufficient space. Free up disk space or switch to S3 storage.

Rate Limiting

429 Too Many Requests

You've exceeded the rate limit. The response includes a Retry-After header indicating when to retry.

To increase the limit:

[options]
rate_limit_tokens = 50000
rate_limit_window = "24h"

S3 Storage Issues

"creating S3 storage adapter" Error

  • Verify s3_bucket and s3_region are set correctly
  • Check that credentials are valid (either explicit or via the AWS credential chain)
  • For custom endpoints, ensure s3_endpoint is a valid URL

Access Denied on S3 Operations

  • Verify the IAM user/role has s3:GetObject, s3:PutObject, s3:DeleteObject, and s3:HeadObject permissions on the bucket
  • Check the bucket policy allows access from the server
  • For presigned URLs, the client needs direct network access to the S3 endpoint

Presigned URLs Not Working

If clients can't access presigned URLs:

  • Ensure the S3 bucket is accessible from the client network
  • Check CORS configuration on the bucket if clients are web-based
  • Try disabling presigned URLs to proxy through the server: s3_presigned_urls = false
  • For S3-compatible services, try enabling path-style addressing: s3_force_path_style = true

URL and Proxy Issues

"base_url is not set" Warning

Warning

Always set base_url in production. Without it, the server relies on X-Forwarded-* headers to construct transfer URLs, increasing the attack surface. A misconfigured or missing reverse proxy could allow clients to influence the generated URLs. See Security for details.

Set base_url in your configuration for production deployments:

[options]
base_url = "https://lfs.example.com"

Without it, the server derives URLs from X-Forwarded-Proto and X-Forwarded-Host headers, which must be set by your reverse proxy.

Incorrect URLs in Batch Responses

If action URLs in batch responses have the wrong scheme (http instead of https) or hostname:

  1. Set base_url in the configuration (recommended)
  2. Or ensure your reverse proxy sets X-Forwarded-Proto and X-Forwarded-Host correctly

Request Buffering Causes Timeouts

Some reverse proxies buffer the entire request body before forwarding. For large LFS transfers, disable request buffering:

Nginx:

proxy_request_buffering off;
proxy_buffering off;

Logging and Debugging

Enable Debug Logging

[options]
additional_logging = true

This switches from JSON logging to human-readable console output at debug level, showing all request details.

Request Correlation

All LFS responses include an X-Request-ID header. Use this to correlate client requests with server log entries.

Check Server Health

curl http://localhost:5420/health

Expected response:

{"status":"ok","version":"v1.0.0"}

File Locking Issues

Lock Owned by Wrong User

Lock ownership is tied to the authenticated user's GitHub username. If using authentication = "none", ownership defaults to the Basic Auth username or "anonymous".

Stale Locks

If a lock was left behind (e.g., user forgot to unlock), another user can force-unlock:

git lfs unlock --force path/to/file

Next Steps

Clone this wiki locally