feat(API): reject writes when disk space drops below 512 MiB

TheLastCicada · TheLastCicada · commit 613336d34116 · 2026-05-15T13:44:38.000-07:00
SQLite returns SQLITE_FULL mid-transaction when the underlying
filesystem runs out of space, which can corrupt the WAL and leave the
V1/V2 databases inconsistent. Add a defensive guard that periodically
inspects free space on the cadt/v1 and cadt/v2 data directories and:

- rejects POST/PUT/PATCH writes with HTTP 507 Insufficient Storage when
  free space drops below 512 MiB (sized for the existing 100 MB
  filestore upload limit + typical WAL growth during batch operations)
- emits a debounced log warning when free space drops below 1 GiB so
  operators have advance notice before writes are rejected
- always permits GET reads and DELETE so operators can free space
  without restarting the service

Status (severity, freeBytes, threshold) is exposed under the diskSpace
field on /health, /v1/health, and /v2/health for monitoring. /health
uses a non-blocking peek of the cached statfs result so liveness probes
never wait on a slow filesystem (a synchronous statfs round trip can
trip the default k8s livenessProbe timeout precisely when /health most
needs to respond).

Severity transitions (ok-&gt;warn, warn-&gt;block, etc.) drive a single log
line each. The earlier time-windowed debounce was vulnerable to /health
scrape traffic refreshing the timestamp and silencing the
operator-facing block alert when a real write first hit the threshold.

Mirrors the frsize ?? bsize byte math used by the existing /diagnostics
endpoint so the two views agree on free space.

Thresholds are intentionally not user-configurable; the values are
sized to the existing upload limits and lowering them risks the
SQLITE_FULL the guard is designed to prevent.
diff --git a/README.md b/README.md
@@ -44,6 +44,15 @@ CADT and Chia system usage will depend on many factors, including how busy the b
 
 ARM and x86 systems are supported.  While Windows, MacOS, and all versions of Linux are supported, Ubuntu Linux is the recommended operating system as it is used most in testing and our internal hosting.
 
+#### Disk space guard
+
+CADT defensively rejects write requests when the filesystem holding the V1/V2 SQLite databases drops below a low-space threshold, so a full disk cannot corrupt the database mid-transaction. Operators should monitor for this and free space (or expand the volume) before it triggers.
+
+* When free space falls below **1 GiB** (2³⁰ bytes), CADT logs a warning. Writes are still served.
+* When free space falls below **512 MiB** (2²⁰ × 512 bytes), CADT rejects `POST`, `PUT`, and `PATCH` requests with HTTP `507 Insufficient Storage` and logs an error. `GET` reads and `DELETE` requests are still served so operators can free space without restarting the service. Note that a very large `DELETE` (e.g., a cascading delete that touches many rows) writes to the SQLite WAL during the transaction; if the disk is critically low even an allowed `DELETE` may run out of space before the next checkpoint, so prefer freeing files outside the database first.
+* Log lines are emitted only on **severity transitions** (`ok` → `warn`, `warn` → `block`, recovery to `ok`, etc.) — not on every observation — so monitoring scrapes against `/health` cannot drown out the alert when free space first drops below a threshold.
+* Current status is exposed under the `diskSpace` field on the `/health`, `/v1/health`, and `/v2/health` endpoints for monitoring. The field is `null` until the first probe completes, otherwise an object with `severity` (`ok` / `warn` / `block` / `unknown`), `freeBytes` (the lowest free-space figure observed across the V1/V2 data directories, or `null` when every probe failed), and the `blockBytes` / `warnBytes` thresholds. The 507 response body itself only contains `{message, error: "INSUFFICIENT_DISK_SPACE", success: false}` — exact byte counts are reserved for `/health` so anonymous callers (the disk-space gate runs before the `CADT_API_KEY` check) cannot enumerate operational state.
+
 ### Linux
 
 A binary file that can run on all Linux distributions on x86 hardware can be found for each tagged release named `cadt-linux-x64-<version>.zip`.  This zip file will extract to the `cadt-linux-64` directory by default, where the `cadt` file can be executed to run the API.
diff --git a/src/middleware.js b/src/middleware.js
@@ -21,6 +21,14 @@ import { OrganizationsV2 } from './models/v2/index.js';
 import { logger } from './config/logger.js';
 import { sendReadOnlyError } from './utils/read-only-response.js';
 import { getRateLimitRetryAfterSeconds } from './utils/rate-limit.js';
+import {
+  checkDiskSpace,
+  peekDiskSpaceStatus,
+  refreshDiskSpaceStatus,
+  logDiskSpaceStatus,
+  buildInsufficientDiskSpaceError,
+  isWriteRejectedByDiskGuard,
+} from './utils/disk-space.js';
 
 const { USE_SIMULATOR } = getConfig().APP;
 
@@ -183,6 +191,35 @@ app.use(async function (req, res, next) {
       return sendReadOnlyError(res);
     }
 
+    // Defensive disk-space guard. SQLite (the V1/V2 backing store) returns
+    // SQLITE_FULL mid-transaction when the underlying filesystem runs out
+    // of space, which can corrupt the WAL and leave the DB inconsistent.
+    // Reject POST/PUT/PATCH below the hardcoded BLOCK_BYTES threshold
+    // (see src/utils/disk-space.js) so reads stay available and operators
+    // can free space via DELETE without restarting. Runs after READ_ONLY
+    // (configuration wins over transient state) but before the wallet/
+    // datalayer RPC checks below so we don't pay an RPC cost on a request
+    // we already know we can't persist.
+    if (isWriteRejectedByDiskGuard(req.method)) {
+      let diskStatus = null;
+      try {
+        diskStatus = await checkDiskSpace();
+        logDiskSpaceStatus(diskStatus);
+      } catch (diskErr) {
+        // Fail open: a bug in the guard itself must not take down writes.
+        // computeStatus() swallows per-directory statfs errors and the
+        // dir-resolution helpers don't throw, so reaching this branch
+        // means something unexpected happened (e.g., getChiaRoot blew up
+        // because os.homedir() failed under a misconfigured PID-1).
+        logger.error(
+          `disk-space-guard: unexpected check failure: ${diskErr.message}`,
+        );
+      }
+      if (diskStatus && diskStatus.severity === 'block') {
+        return res.status(507).json(buildInsufficientDiskSpaceError());
+      }
+    }
+
     await assertChiaNetworkMatchInConfiguration();
     await assertDataLayerAvailable();
     if (req.method !== 'GET') {
@@ -507,9 +544,34 @@ app.use(async function (req, res, next) {
 });
 
 app.get('/health', (req, res) => {
+  // Surface disk-space status so monitoring can alert before writes get
+  // blocked. Use the non-blocking peek (cached value only) and kick off
+  // an async refresh in the background — synchronously awaiting statfs
+  // here can cause k8s liveness probes (default 1 s timeout) to fail
+  // when the filesystem is slow or wedged, which is exactly when /health
+  // most needs to stay responsive.
+  let diskSpace = null;
+  try {
+    const status = peekDiskSpaceStatus();
+    if (status) {
+      diskSpace = {
+        severity: status.severity,
+        freeBytes: status.freeBytes,
+        blockBytes: status.blockBytes,
+        warnBytes: status.warnBytes,
+      };
+      // Drive the debounced log from /health too so a mostly-idle CADT
+      // (no writes in flight) still surfaces warn/block transitions.
+      logDiskSpaceStatus(status);
+    }
+    refreshDiskSpaceStatus();
+  } catch (err) {
+    logger.debug(`disk-space-guard: /health peek failed: ${err.message}`);
+  }
   res.status(200).json({
     message: 'OK',
     timestamp: new Date().toISOString(),
+    diskSpace,
   });
 });
 
diff --git a/src/routes/v1/index.js b/src/routes/v1/index.js
@@ -3,6 +3,12 @@
 import express from 'express';
 import { getWalletHealthResponse } from '../wallet-health.js';
 import { getConfig } from '../../utils/config-loader';
+import {
+  peekDiskSpaceStatus,
+  refreshDiskSpaceStatus,
+  logDiskSpaceStatus,
+} from '../../utils/disk-space.js';
+import { logger } from '../../config/logger.js';
 const V1Router = express.Router();
 
 import {
@@ -19,9 +25,30 @@ import {
 } from './resources';
 
 V1Router.get('/health', (req, res) => {
+  // Non-blocking: use cached disk-space status only and trigger an async
+  // refresh. Synchronously awaiting statfs would risk timing out k8s
+  // liveness probes when the filesystem is slow. See bare /health in
+  // src/middleware.js for full rationale.
+  let diskSpace = null;
+  try {
+    const status = peekDiskSpaceStatus();
+    if (status) {
+      diskSpace = {
+        severity: status.severity,
+        freeBytes: status.freeBytes,
+        blockBytes: status.blockBytes,
+        warnBytes: status.warnBytes,
+      };
+      logDiskSpaceStatus(status);
+    }
+    refreshDiskSpaceStatus();
+  } catch (err) {
+    logger.debug(`disk-space-guard: /v1/health peek failed: ${err.message}`);
+  }
   res.status(200).json({
     message: 'V1 API is running',
     timestamp: new Date().toISOString(),
+    diskSpace,
   });
 });
 
diff --git a/src/routes/v2/index.js b/src/routes/v2/index.js
@@ -28,13 +28,40 @@ import { OrganizationsV2Router } from './resources/organizations-v2.js';
 import { AuditV2Router } from './resources/audit-v2.js';
 import { OfferV2Router } from './resources/offer-v2.js';
 import { FilestoreV2Router } from './resources/filestore-v2.js';
+import {
+  peekDiskSpaceStatus,
+  refreshDiskSpaceStatus,
+  logDiskSpaceStatus,
+} from '../../utils/disk-space.js';
+import { logger } from '../../config/logger.js';
 
 const V2Router = express.Router();
 
 V2Router.get('/health', (req, res) => {
+  // Non-blocking: use cached disk-space status only and trigger an async
+  // refresh. Synchronously awaiting statfs would risk timing out k8s
+  // liveness probes when the filesystem is slow. See bare /health in
+  // src/middleware.js for full rationale.
+  let diskSpace = null;
+  try {
+    const status = peekDiskSpaceStatus();
+    if (status) {
+      diskSpace = {
+        severity: status.severity,
+        freeBytes: status.freeBytes,
+        blockBytes: status.blockBytes,
+        warnBytes: status.warnBytes,
+      };
+      logDiskSpaceStatus(status);
+    }
+    refreshDiskSpaceStatus();
+  } catch (err) {
+    logger.debug(`disk-space-guard: /v2/health peek failed: ${err.message}`);
+  }
   res.status(200).json({
     message: 'V2 API is running',
     timestamp: new Date().toISOString(),
+    diskSpace,
   });
 });
 
diff --git a/src/utils/disk-space.js b/src/utils/disk-space.js
diff --git a/tests/v2/integration/disk-space-guard.spec.js b/tests/v2/integration/disk-space-guard.spec.js
