Production AI Automation Notes #2: Token Budgets \u2014 per-step, per-pipeline, per-day enforcement for LLM pipelines

Last tested: May 2026 — Claude Opus 4.7, OpenRouter, draftcat v0.1

Practitioner notes on running LLM pipelines in production. Follow @renezander030 for the next entry.

This is Production AI Automation Notes #2. The series:

• #1 Agent Approval Gates: why agents shouldn't reach side effects directly
• #2 Token Budgets (you are here): per-step, per-pipeline, per-day enforcement
• #3 Agentic Task System: durable memory for long-running agents
• #4 Driving CapCut from LLM agents: structured tool calls into a non-API surface

If the gate trips, the pipeline returns the error and the engine logs [pipeline:NAME][step:NAME] BUDGET_BLOCKED: .... No half-spent step, no retry storm.

A single daily cap is what most people start with. It fails for three reasons that show up in real traffic.

per-step exists to catch single-prompt blowups. A skill that normally returns 400 tokens occasionally produces 6,000 because a user pasted a transcript into an "adjust this draft" channel. Without a per-step cap, one bad input chews 20× the planned budget before the response even reaches your validator.

per-pipeline exists to catch stuck loops. Multi-step pipelines that retry-on-validation-fail can re-enter the same step many times in a single run. The daily counter sees this happen slowly. The per-pipeline counter sees it inside the run, where you can still abort cheaply.

per-day exists to protect the credit card. Provider bills are settled monthly, but the dashboard updates in near-real-time. A per-day cap is the difference between "we noticed at 9am" and "we noticed when the invoice arrived." Set it just above your expected peak, not at your monthly ceiling.

The three tiers compose. The pre-flight check runs against the daily limit using the per-step request size, so a single oversized step that would push the daily total over the line is rejected before the network call.

type ModelConfig struct {
    Model     string  `yaml:"model"`
    MaxTokens int     `yaml:"max_tokens"`
    CostIn    float64 `yaml:"cost_per_1k_input"`
    CostOut   float64 `yaml:"cost_per_1k_output"`
}

type BudgetConfig struct {
    PerStepTokens     int `yaml:"per_step_tokens"`
    PerPipelineTokens int `yaml:"per_pipeline_tokens"`
    PerDayTokens      int `yaml:"per_day_tokens"`
}

type BudgetTracker struct {
    tokensUsedToday    int
    tokensUsedPipeline int
    dayStart           time.Time
}

Three counters, three configs. ModelConfig carries the per-call ceiling and the cost coefficients so the engine can convert tokens to dollars without re-reading the provider's pricing page. BudgetTracker is the in-process state the engine consults before every AI step.

This is the function that returns BUDGET_BLOCKED. Paste the error string into your repo search if you're wondering why a pipeline went silent.

func (b *BudgetTracker) check(limit int, requested int) error {
    if b.dayStart.Day() != time.Now().Day() {
        b.tokensUsedToday = 0
        b.dayStart = time.Now()
    }
    if b.tokensUsedToday+requested > limit {
        return fmt.Errorf("BUDGET_BLOCKED: daily token limit %d would be exceeded (used: %d, requested: %d)", limit, b.tokensUsedToday, requested)
    }
    return nil
}

The day rollover is intentionally cheap: a single date comparison, no scheduler, no cron, no external clock. The first call after midnight server-local time resets the counter. If your service runs across timezones, set the server clock to the timezone whose midnight you want as the budget boundary.

Wiring into the pipeline loop, the check happens before every step of type ai:

case "ai":
    // Budget pre-flight
    if err := budget.check(cfg.Budgets.PerDayTokens, cfg.Budgets.PerStepTokens); err != nil {
        log.Printf("[pipeline:%s][step:%s] %s", pipeline.Name, step.Name, err)
        return err
    }
    // ... resolve skill, build prompt, call LLM

The check uses per_step_tokens as the requested amount and per_day_tokens as the limit. The pipeline aborts before the HTTP call, so you don't pay for a request you're going to throw away.

Pre-flight uses the configured per-step cap as a worst-case estimate. After the call returns, you record what the model actually used:

func (b *BudgetTracker) record(tokens int) {
    b.tokensUsedToday += tokens
    b.tokensUsedPipeline += tokens
}

Both counters move together. The call site pulls the real numbers from the provider response and feeds them in along with the cost:

resp, err := callLLM(aiCtx, cfg, role, prompt)
if err != nil {
    return fmt.Errorf("[step:%s] LLM call failed: %w", step.Name, err)
}

// Record token usage
budget.record(resp.InputTokens + resp.OutputTokens)
log.Printf("[pipeline:%s][step:%s] model=%s tokens=%d+%d cost=$%.4f latency=%dms",
    pipeline.Name, step.Name, resp.Model,
    resp.InputTokens, resp.OutputTokens, resp.CostUSD, resp.LatencyMs)

Cost is computed inside callLLM from resp.InputTokens, resp.OutputTokens, and the model's cost_per_1k_input / cost_per_1k_output. Logging both the token split and the dollar amount on every step gives you a per-pipeline cost ledger in plain log output, which is the cheapest observability you can buy.

Straight from a working pipeline:

models:
  gpt-4o-mini:
    model: openai/gpt-4o-mini
    max_tokens: 1024
    cost_per_1k_input: 0.00015
    cost_per_1k_output: 0.0006

budgets:
  per_step_tokens: 2048
  per_pipeline_tokens: 10000
  per_day_tokens: 100000

timeouts:
  ai_call: 30s
  operator_approval: 4h
  pipeline_total: 5m

The per_step_tokens: 2048 value is intentionally larger than max_tokens: 1024. The step cap covers input plus output worst-case, while max_tokens only caps output. Setting per-step equal to max-output is a common misconfiguration that throws false BUDGET_BLOCKED errors on long input prompts.

A few honest limitations so you can decide whether to extend it.

• Single-process tracker. Counters live in memory. Two engine instances running the same config will each enforce their own budget independently. For multi-instance deployments, back the counters with Redis or a small Postgres table keyed on (day, pipeline).
• Daily counter resets on restart. A service restart at 14:00 zeroes the today counter. If you restart often, persist tokensUsedToday and dayStart to disk on every record().
• No per-pipeline persistence. tokensUsedPipeline is a field on the same tracker; the example engine resets it implicitly between runs. If you want a hard per-pipeline cap that survives mid-run crashes, store the running total alongside the pipeline's state row.
• No cost-based cap. The check is in tokens, not dollars. If you mix expensive and cheap models in one pipeline, the daily token limit may permit a dollar overrun. Add a parallel costUsedToday counter and check it against a per_day_usd config value.

Each of these is a deliberate trade for a small dependency surface. The whole guardrail is ~40 lines of Go and survives every model swap.

• draftcat (Go, MIT) at github.com/renezander030/draftcat
  • main.go: BudgetConfig, BudgetTracker, check, record, pipeline pre-flight in the AI step branch
  • config.yaml: working budgets + models + timeouts example

Clone and grep for BUDGET_BLOCKED to land on the enforcement point in 5 seconds.

If you ship LLM pipelines in production, drop a comment with your config: model mix, per-day cap, what triggered the last BUDGET_BLOCKED, and what you'd change. The interesting data is in the asymmetries: cap-too-low producing false positives, cap-too-high catching nothing, model swaps that broke the per-step assumption.

Production AI Automation Notes:

• #1 Agent Approval Gates
• #3 Agentic Task System
• #4 Driving CapCut from LLM agents

Pipeline engineering deep dives:

• YAML pipeline config: the surrounding config shape
• Skill file format: how prompts get pulled into steps
• Pipeline engine internals: the step loop in detail
• Env-driven config loader: secrets + overrides
• Performance monitor for LLM pipelines: latency + cost dashboards

2026-05-25: initial publish