Long LLM jobs on Cloudflare Workflows: per-step retries and a budget guard

An astrology app where AI writes the interpretation has a simple contract: birth date, time and place in, a long personal report out. The paid report is a long LLM job. Every buyer gets about eight chapters, and chapters take a while: the model adapter waits up to 120 seconds for a single call. The buyer pays, then waits for an emailed link. The flow puts that wait at about two to three minutes. In that window the model API can answer 429 or 5xx, the network can drop, and the mail provider can have an outage. A retry that starts from scratch runs chapters that were already fine and pays for them again. Parallel chapters add a second trap: they can all pass a budget check at the same moment. The app runs on Cloudflare Workflows with per-step retries and a budget guard. This note is about how those two pieces fit together.

Why a workflow, not a worker

The previous version had a pg-boss worker meant for a self-managed VPS. It ran end to end on my machine and never shipped, because deploying it meant a VPS, DNS and server administration. This version is TanStack Start on Cloudflare Workers, with Cloudflare Workflows for generation and Postgres on Supabase.

A durable workflow step is a named piece of work whose result the engine saves. If a run is interrupted and resumed, steps that already finished return their saved result instead of running again. In Workflows each step.do() result is persisted, so one step per chapter acts as a checkpoint. After a crash, finished chapters are read back, not re-run or re-billed. Workflows replaced pg-boss and the separate worker.

I didn’t need an agent framework on top. The pipeline is chained prompts in a fixed order, not agents. Promise.all over step.do() already gives fan-out and join. A separate queue was off the list too: Workflows covers what Redis and BullMQ would have done.

I checked the bill as well. Workflows on the Paid plan included 500,000 steps a month and charged $0.80 per 100,000 beyond that, with retries not counted. At 100 orders a month the app would use about 0.4% of the included steps. I wrote down when to look again: past about 20 steps per report, or about 30,000 reports a month.

One step per chapter

The report Workflow is a short list of steps:

1. Load, then save the chart onto the report. It goes on early, so the reader’s figures survive the 90-day personal-data purge, even for a report that fails partway.
2. Seed the checkpoint that marks which chapters are done.
3. Write the synthesis. Every chapter reads it, so all chapters describe the same person in one voice. It is never delivered.
4. Fan out: one step.do() per chapter, in parallel through Promise.all.
5. Seal. A report seals only when every chapter is done. Otherwise it is marked failed.
6. Deliver the email in its own step, so a mail-provider outage can’t fail a report that has already sealed.

The fan-out has roughly this shape:

// a sketch, not the app's code
await Promise.all(
  notDone.map((key) =>
    step.do(`chapter-${key}`, () => writeChapter(key)),
  ),
);

Two guarantees keep a paid report from being generated twice. The Workflows engine caches each step by a deterministic name, chapter-{key}. A new instance reads the stored checkpoint and regenerates only the chapters not marked done. The instance id is report-{reportId}, and it doubles as the idempotency key. A replayed Stripe webhook that finds the instance already there counts as success.

An empty parse marks a chapter failed, not done. Each step opens and closes its own database connection, because connections don’t survive hibernation between steps. Leaked pools had already exhausted my local Postgres (“too many clients”).

The same checkpoint feeds the waiting screen. An SSE stream re-reads it every 1,200 ms and sends a frame only when the snapshot changes. It closes after ten minutes at most, and the client reconnects. I didn’t use the Workflow instance status API for progress. It reports steps instead of chapters, and it would duplicate a checkpoint that already exists.

Where the retries live

The first layer is inside the model call. The adapter makes up to three attempts. It retries 429, 5xx and network errors, and it doesn’t retry any other 4xx. A 429 or a 5xx usually means “not now”, so another try can work. Another 4xx usually means the request itself is wrong, and the same request gets the same answer. Each call is also capped at 8,000 output tokens.

The second layer is the step boundary. It decides what a retry costs. A retried report doesn’t start over. The synthesis is reused verbatim, and only the chapters not marked done are generated again.

Resume has its own test. It runs a partial report through the same database functions and engine. The engine reaches the model through a port, so the test plugs in one that throws if it is asked to regenerate a done chapter or the synthesis. Only the missing chapters get requested, and the kept chapters stay byte-identical. The full pipeline also ran end to end against wrangler dev on fixtures: paid, Workflow, 8 of 8 chapters sealed, reader rendered.

One place gets no retries on purpose. The only script that can make a paid Anthropic call refuses to run without an explicit --i-will-spend-money flag. It makes one attempt per key.

The budget guard

The guard is called BudgetGuard. It journals the cost of every LLM call per report and refuses new generation above the monthly cap. Each call reserves its estimated cost before it starts, so parallel calls can’t all pass the check at once. Prices come from a rate table in code: dollars per million tokens for input, output, cache write and cache read.

Reserve before the fan-out

That reservation doesn’t reach across steps. Each chapter step has its own guard, and it can’t see its siblings’ reservations. This is a check-then-act race: several workers read the same shared total, and each one acts on a number that is already stale. So one pre-flight runs before the parallel chapters start. It reserves the worst case once against the cap: chapters remaining times the maximum chapter cost.

export function assertFanOutAffordable(input: {
  readonly chaptersRemaining: number;
  readonly maxChapterCostUsd: number;
  readonly spentThisMonthUsd: number;
  readonly monthlyCapUsd: number;
}): void {
  const addedWorstCaseUsd = input.chaptersRemaining * input.maxChapterCostUsd;
  const worstCaseUsd = input.spentThisMonthUsd + addedWorstCaseUsd;
  if (addedWorstCaseUsd > 0 && worstCaseUsd > input.monthlyCapUsd) {
    throw new BudgetExceeded(worstCaseUsd, input.monthlyCapUsd);
  }
}

Price the worst case high

The worst case has to be the real one. An unknown or mistyped model id is priced at a $15/$75 per-million-token penalty tier. At that tier one chapter’s worst case is about $0.64, which is 28% above the flat $0.5 per-call estimate. A flat constant would under-reserve for exactly the model nobody meant to configure.

Other nets

If the budget variable is unset, the code falls back to a $20 monthly cap. That is a second net under the hard spend limit set in the Anthropic console. The real model adapter runs only when the fixtures flag is off and an API key is present. Any non-falsy flag value forces fixtures, typos included. A deployment with the flag off and no key refuses to generate. LLM tests run on recorded fixtures, and a guard test shows the CI path cannot construct the real adapter. The end-to-end harness boots two dev servers, one with a normal budget and one with a strict one.

The admin shows cost per order next to a re-run action, and cost per product and month from the llm_calls table. Duplicate payments were a leak of their own. Before the duplicate-payment policy, 7 orders on one run produced 7 reports at 7 times the model cost. Now a second session for the same run and product is recorded as a duplicate and creates no report. The buyer gets the existing link by email.

What I’d keep

• One durable step per unit of paid work, with a deterministic name, so a retry reads finished work back instead of paying for it again.
• Retries next to the failure: 429, 5xx and network errors retried inside the call, other 4xx not retried.
• A reservation before every call, plus one worst-case reservation before any parallel fan-out.
• Unknown model ids priced at a penalty rate, so a typo makes the estimate bigger, not smaller.
• Safe defaults: fixtures unless the flag is off and a key is present, and a cap in code under the console limit.