Respect preserveSchedule on forced transient failures

When applyJobResult is invoked from a manual force run (src/cron/service/ops.ts:479-490 passes preserveSchedule: true), transient errors now always write nextRunAtMs = backoffNext and ignore the preserved natural schedule. That means a forced run of a long-interval recurring every job can now trigger an unexpected automatic retry within ~30s after a network blip, whereas the prior behavior kept the original cadence/anchor for force runs.

Useful? React with 👍 / 👎.

retryConfig.backoffMs fetched but silently ignored

resolveRetryConfig is called to get retryConfig, and retryConfig.retryOn is correctly forwarded to isTransientCronError. However, retryConfig.backoffMs is never passed to errorBackoffMs, so any per-deployment backoff schedule configured via cronConfig.retry.backoffMs is silently ignored for recurring jobs. The one-shot path at line 385 correctly forwards retryConfig.backoffMs:

const backoff = errorBackoffMs(consecutive, retryConfig.backoffMs);

The recurring path should do the same:

Without this, cronConfig.retry.backoffMs only takes effect for at jobs, not cron/every jobs, which is inconsistent and likely unintended.

This is a comment left during a code review.
Path: src/cron/service/timer.ts
Line: 419

Comment:
**`retryConfig.backoffMs` fetched but silently ignored**

`resolveRetryConfig` is called to get `retryConfig`, and `retryConfig.retryOn` is correctly forwarded to `isTransientCronError`. However, `retryConfig.backoffMs` is never passed to `errorBackoffMs`, so any per-deployment backoff schedule configured via `cronConfig.retry.backoffMs` is silently ignored for recurring jobs. The one-shot path at line 385 correctly forwards `retryConfig.backoffMs`:

```typescript
const backoff = errorBackoffMs(consecutive, retryConfig.backoffMs);
```

The recurring path should do the same:

```suggestion
      const backoff = errorBackoffMs(job.state.consecutiveErrors ?? 1, retryConfig.backoffMs);
```

Without this, `cronConfig.retry.backoffMs` only takes effect for `at` jobs, not `cron`/`every` jobs, which is inconsistent and likely unintended.

How can I resolve this? If you propose a fix, please make it concise.

Addressed in `7a98b18f5`: recurring error path now takes backoff schedule from `resolveRetryConfig(state.deps.cronConfig).backoffMs` when `cron.retry.backoffMs` is set (same reference as one-shot). Full default ladder still used when unset. Also merged latest `openclaw/main` into this branch to refresh CI.

Backoff escalation test creates fresh job instances instead of reusing one

Each runError(n) call creates a fresh CronJob and manually pre-sets consecutiveErrors before calling applyJobResult. This means the test does not exercise the actual counter accumulation path across repeated failures on the same job — it only verifies that errorBackoffMs returns a larger value for larger inputs.

A more realistic regression test would apply errors to the same job instance in sequence:

it("uses increasing backoff on consecutive transient errors for a daily job", () => {
  const state = makeState();
  const job = makeCronJob("0 10 * * *");

  const recordNext = () => {
    applyJobResult(state, job, {
      status: "error",
      error: "econnreset",
      startedAt: NOW_MS,
      endedAt: ENDED_AT,
    });
    return job.state.nextRunAtMs!;
  };

  const first = recordNext();   // consecutiveErrors → 1, 30s backoff
  const second = recordNext();  // consecutiveErrors → 2, 60s backoff
  const third = recordNext();   // consecutiveErrors → 3, 5min backoff

  expect(second).toBeGreaterThan(first);
  expect(third).toBeGreaterThan(second);
  expect(third).toBeLessThanOrEqual(ENDED_AT + 5 * 60_000);
});

This also implicitly validates that consecutiveErrors increments correctly rather than relying on manually supplied values.

This is a comment left during a code review.
Path: src/cron/service.issue-46341-recurring-transient-retry.test.ts
Line: 145-168

Comment:
**Backoff escalation test creates fresh job instances instead of reusing one**

Each `runError(n)` call creates a fresh `CronJob` and manually pre-sets `consecutiveErrors` before calling `applyJobResult`. This means the test does not exercise the actual counter accumulation path across repeated failures on the same job — it only verifies that `errorBackoffMs` returns a larger value for larger inputs.

A more realistic regression test would apply errors to the same job instance in sequence:

```typescript
it("uses increasing backoff on consecutive transient errors for a daily job", () => {
  const state = makeState();
  const job = makeCronJob("0 10 * * *");

  const recordNext = () => {
    applyJobResult(state, job, {
      status: "error",
      error: "econnreset",
      startedAt: NOW_MS,
      endedAt: ENDED_AT,
    });
    return job.state.nextRunAtMs!;
  };

  const first = recordNext();   // consecutiveErrors → 1, 30s backoff
  const second = recordNext();  // consecutiveErrors → 2, 60s backoff
  const third = recordNext();   // consecutiveErrors → 3, 5min backoff

  expect(second).toBeGreaterThan(first);
  expect(third).toBeGreaterThan(second);
  expect(third).toBeLessThanOrEqual(ENDED_AT + 5 * 60_000);
});
```

This also implicitly validates that `consecutiveErrors` increments correctly rather than relying on manually supplied values.

How can I resolve this? If you propose a fix, please make it concise.

Keep recurring retries on full backoff ladder

Passing retryConfig.backoffMs into the recurring error path changes the default ladder for recurring jobs from [30s, 60s, 5m, 15m, 60m] to only the first three entries, because resolveRetryConfig falls back to DEFAULT_BACKOFF_SCHEDULE_MS.slice(0, 3). After the 3rd consecutive failure, recurring jobs now retry every 5 minutes instead of backing off to 15/60 minutes, which increases retry pressure for short-interval jobs and weakens the existing retry-storm protection.

Useful? React with 👍 / 👎.

Honor custom retry backoff during startup catch-up

Using cronConfig.retry.backoffMs here changes recurring jobs’ persisted nextRunAtMs, but the restart gate in isErrorBackoffPending still computes backoff with the default ladder (errorBackoffMs(consecutiveErrors)), so the two paths diverge. In a configured environment with a longer custom backoff (for example 2h), a gateway restart after 30s+ can treat backoff as expired and planStartupCatchup (with allowCronMissedRunByLastRun) may run the cron job before the configured cooldown and before its stored nextRunAtMs, which breaks operator-configured retry pacing after restarts.

Useful? React with 👍 / 👎.

Decouple recurring transient checks from one-shot retryOn config

The recurring error path now calls isTransientCronError(..., retryConfig.retryOn), so any custom cron.retry.retryOn setting will also gate recurring retries. This is a regression for deployments that tuned cron.retry for one-shot jobs only (as documented in src/config/schema.help.ts:1199), because a config like retryOn: ["rate_limit"] will make recurring network/timeout failures skip the new backoff retry behavior and wait until the next natural schedule again. Please keep recurring transient classification independent of one-shot retry config (or update the config contract everywhere).

Useful? React with 👍 / 👎.

Align restart backoff gating with recurring backoff config

Using cron.retry.backoffMs for recurring errors here changes persisted nextRunAtMs, but startup replay gating still uses isErrorBackoffPending with the default ladder only. In a config like cron.retry.backoffMs: [7200000], a cron job that failed transiently can be replayed on restart (via missed-slot logic) well before its configured 2-hour cooldown and before its stored nextRunAtMs, because the gate considers backoff elapsed after the default 30s. This breaks operator-configured retry pacing specifically after restarts.

Useful? React with 👍 / 👎.

Addressed in 52a71a93a: startup missed-slot gating now reuses the same recurring backoff schedule as applyJobResult, so custom cron.retry.backoffMs prevents premature replay after restart. Added a restart-catchup regression test for the custom-backoff case, and targeted cron tests passed.

Classify built-in timeout errors as transient

The new recurring retry branch now depends on isTransientCronError(result.error), but the scheduler’s own timeout failures are emitted as "cron: job execution timed out" (from timeoutErrorMessage()), which does not match the current timeout transient pattern (timeout|etimedout). In practice, timed-out recurring jobs (for example a daily job that hits the execution timeout) are treated as permanent errors and wait for the next natural schedule instead of taking the intended backoff retry path, so the timeout case in this fix is not actually covered.

Useful? React with 👍 / 👎.

Addressed in 18f0891690: updated TRANSIENT_PATTERNS.timeout to /(timeout|timed\\s+out|etimedout)/i so "cron: job execution timed out" from timeoutErrorMessage() is correctly classified as transient. Added regression test for the exact internal message.