Guardrails as skills: no backtest reported until checks pass

Agents love a good backtest. One of my models showed +112% in its backtest and −13.9% live. It had passed CPCV, a Monte Carlo permutation test and a blind test. The evaluation was contaminated before any of them saw it: the model had dropped 88% of its signals, the ones whose book was dead at exit. I wrote a Claude Code skill that won’t let agents report a backtest until data validation, bias guards and multiple-testing corrections pass. Each rule in it is taken from a real failure, and that model was one of them. That is what I mean by guardrails as skills: the rules live in one skill file, and the checks live in code the skill tells agents to run.

One skill file, four steps in order

This note expands the Guardrails as Skills card in my agent systems. The skill is a single SKILL.md in my user-level skills folder. Its trigger description marks it “MANDATORY before reporting any positive backtest result”. The agent’s job, it says, “is not to find edge” but “to stop yourself from believing a lie”. Its default assumption: any positive result is an artifact until proven otherwise.

Then it fixes the order:

Data validation rejects corrupt book data at ingest: crossed, frozen or dust.
The guard checks survivorship, phantom quotes, frozen quotes and concentration.
SPA and StepM run after any search over N candidates.
CPCV runs only if steps 1 to 3 passed.

Step 1 is not hypothetical. In the raw data, 9.06% of seconds had crossed books made by quotes of 1 to 8 shares, and they fabricated false edges. In one research round, 99.6% of “value” trades fired on fictional crossed books. The ETL applies a $10-per-level dust filter, and fills are simulated only on rows marked book_valid.

The order is the point. CPCV re-tests a model across many train and test splits; a permutation test compares its result with results on shuffled data. Both take the backtest itself as given. The skill says steps 1 and 2 “don’t exist as a product”. The guard’s header adds that five open-source data-validation and trading/ML libraries were searched, and none has the concept of an outcome that is missing because the trade lost.

The skill tells agents to run the guard, not retype it. This is the call, verbatim from the skill:

import sys; sys.path.insert(0, 'research/hunt')
from guard import guard
guard(df, y='y_up', exit_col='exit_bid', sig='signal', mid='mid',
      ask='ask', ask_next='ask_t1', ask_age='ask_age_s',
      dates=dates, pnl=pnl)          # raises GuardError if contaminated

A failing check raises GuardError, flags the backtest as contaminated and tells the agent not to run CPCV or SPA, because the evaluation is wrong before it reaches them. The guard is tested both ways. Run directly, it reproduces the four failures on synthetic data, then runs a clean control with real edge that has to pass. In a TypeScript port, writing the survivorship test showed that unfilled trades never got a settle value, so the unfilled bucket stayed empty and the guard “would have reported ‘clean’ on contaminated data forever”.

Four failures the guard knows by name

guard.py ties each check to a named case. The thresholds below are the ones in its code.

Survivorship

Never drop_nulls on anything downstream of the outcome. Compare the outcome rate of the rows you kept with the rows you dropped. If the rates differ, the rows are missing not at random, and deleting them changes the answer. When the book is dead at exit, hold to settlement instead of deleting the row. The guard fails on a gap above 0.05 and ignores drop rates under 1%. Its named case is y120, the model from the opening.

Phantom quotes

A phantom signal predicts the quote, not the settlement. In the skill’s example, a book signal made +5.8% per dollar at latency 0 and −2.1% at one second. The win rate barely moved, from 51.6% to 51.7%. The fill price did, from 0.477 to 0.513. The guard fails when the signal’s correlation with the next ask move is more than twice its correlation with the settlement, unless the signal barely moves the quote (an absolute correlation under 0.05). The named case, qi10, correlated 0.208 with the quote and 0.026 with the settlement.

Frozen quotes

A stale quote is a price that may no longer be on offer. In one close-zone strategy, 43% of the “good” signals had an ask that had not moved in over 120 seconds, and roughly 40% of its apparent edge was fiction. The guard fails when more than 10% of signals have an ask older than 120 seconds. If quote age cannot be measured, the skill says to report it as a risk, not a pass.

Concentration

A mean can be carried by a few lucky days. The skill asks for the median day and the share of positive days, never only the mean, and fails a result when its top three days make more than 50% of the profit. The guard also fails when the median day is not positive. The named case: strategies where the top three days made 95% of the profit.

Every trial counts

Test enough ideas and some will look good by chance: at a 5% significance level, about 1 in 20 strategies with no edge will pass anyway. Multiple-testing correction raises the bar to account for every test that was run. SPA, a test for superior predictive ability, asks whether the best of many candidates beats a benchmark once the whole search is counted. StepM returns which candidates, if any, survive.

The skill makes SPA mandatory after any search, and a consistent p above 0.05 means nothing beats the benchmark. The case the skill cites: 600 trials of a parameter search produced a “winner” with an expected value of +511% per dollar and “40/40 positive out-of-sample”. SPA gave p = 0.62. The skill’s smell tests are blunter: EV above 50% per dollar means a broken evaluation, and with n under 50, “you have nothing”.

Every report has to state the median day, days positive, top-3 concentration, the latency curve, n and the configurations tried. The skill calls one sentence its most important:

this is 3 bets, it means nothing yet.

n_trials must count every configuration ever tried, including the hand-written ones. The sign has to be right too: the SPA implementation the skill uses takes losses, not profits, so you pass −pnl. The wrong sign “silently rejects real edge and accepts noise”, and the skill adds: “This bug shipped once already.” judge_fixed.py documents it. A synthetic oracle with 86% positive days had scored p = 0.49 and failed; with the correct sign it scored p = 0.000. Every SPA p-value reported until then in that family was meaningless.

The research workspace adds the other half: hypotheses pre-registered, every trial counted. When I set it up, I asked for a constitution built from the skill’s rules. Before anyone touches outcome data, a PREDECLARED file fixes the hypothesis, entry and exit rules, costs, pass and fail thresholds and the full grid. The hypothesis must name who is on the other side of the trade and why they lose. After every Phase 1 report, a skeptic agent must recompute the headline numbers from raw data. Since constitution version 1.1.0, each PREDECLARED file must be committed on its own, before any result exists: a skeptic showed that a hash alone was not enough.

An earlier round pre-declared 96 rules before any backtest result and stated: “That is n_trials for SPA/StepM. No other rule is reported as a strategy.” On the trading stack, agent swarms tested about 1,500 hypotheses under pre-registered rules. The honest result: the market is efficient after fees. The rest of that work is in the prediction-market trading stack case file.

The skills need a guard too

The same workspace holds 1,096 curated agent skills behind activation profiles and a static security scanner. Four are quarantined. A profile script activates a subset, such as core, crypto or math, by symlinking it into .claude/skills/, which Claude Code reads at session start.

The scanner never executes what it scans. It looks for code that runs a remote payload, reaches the network, reads secrets, hides what it does or tells the agent to act behind the user’s back. It also says plainly: “A static scan cannot prove a skill is safe.” The four quarantined skills cannot load. One installs an hourly cron against a live exchange profile. One opens remote access through public tunnels using an SSH key. Two read a secret from the OS keychain and send it to a third-party API.

Skills that do load are still advice, not fact, and every report must list them that way. Fee and API facts must come from official docs, never from skills or model memory. The constitution holds my own skill to that: it notes that quant-backtest-rigor hardcodes a fee rate of 0.07, “true only for crypto”.

What I’d keep

Rules written from real failures, each tied to a named case.
Data checks first, and no CPCV or SPA on a backtest that failed them.
Checks shipped as code the agent imports, not prose it retypes.
Guards tested both ways: known failures must fail, clean data with real edge must pass.
Every trial counted, and every pre-registration committed before the first result.

Guardrails as skills: no backtest gets reported until the checks pass