The prompt, not the model: what blind LLM-as-judge tests showed

An astrology app I worked on turns a birth date, time and place into a long personal report. AI writes the interpretation. Text is easy to get from a model. Text that does not read as machine-written is the hard part. When the output does read that way, there are two obvious suspects: the model and the prompt. Both change the output, so it is easy to blame the wrong one. The model mix in the app came from blind LLM-as-judge tests. Their finding: the prompt, not the model, made the text sound AI-written.

This note covers the prompt side of that system and the checks around it.

What a blind LLM-as-judge test is

LLM-as-judge means asking a language model to evaluate text against written criteria. The judge scores a single text or picks the better of two. In a blind test, the judge is not told which model wrote which text. That keeps the model’s name out of the verdict. I use judges outside this app too. My Claude Code workflows run parallel builders, judges, critics and adversarial verifiers.

The finding is about how the text sounds. It also points to where a fix belongs. If the prompt makes the text sound AI-written, swapping the model leaves the cause in place. The fix has to go into the prompt. That does not make the model irrelevant, and I come back to it below.

A blind test asks for the opposite discipline on the builder’s side. You have to know exactly which model wrote each text. The script that writes the free readings was once found using the default model configured in the Claude Code CLI. It now pins Claude Sonnet 5 by name.

What the models are allowed to do

The report pipeline is a fixed chain of prompts rather than a set of agents. Claude Opus drafts each chapter, a GPT model critiques it with quotes, and Opus repairs only the flagged spans.

None of the models computes the chart. The chart facts are computed in code from a Swiss Ephemeris service, so the model interprets and never invents. Planetary positions and degrees are never written by a model. They are injected as data blocks when the report renders. The fixed system prompt says the same thing in words: no degrees, no minutes, no planetary positions, because the model interprets and does not calculate. Signs and house names reach it as words, never as numbers. When the birth time is missing, the prompt forbids invented data.

That leaves the model one job: interpretation. A synthesis call runs before the chapters. Every chapter reads it, so all chapters describe the same person in one voice. The synthesis is never delivered, and a retry reuses it verbatim. Each request is laid out the same way for prompt caching: the fixed system prompt, then the synthesis, then the chapter task.

For the daily reading, a research note proposed a different blind test, read by people instead of a judge model. The same aspects would be written by Claude Haiku 4.5 and Claude Sonnet 5, then compared without knowing which is which, for under $0.50 in total. The code defaults the daily reading to Sonnet 5. A comment there records the choice of safety on Romanian nuance over Haiku.

What the prompts say about sound

The report, daily and free-text prompts share one voice guide. It merges eight research passes into a single source.

Patterns to avoid

The prompt behind the free readings names the patterns that make text read as AI-written. These are its lines, copied as they are:

- NO em-dashes or en-dashes ("—", "–"). Use a period, a comma, or two dots. If you
  reach for a dash, it is almost always a period. This is non-negotiable.
- Do NOT write like AI. Avoid: the tricolon of effect ("nu doar X, ci Y si Z"),
  "in esenta", "in cele din urma", "merita mentionat", "este important sa",
  hedging stacks ("poate, s-ar putea, eventual"), every paragraph the same length,
  every sentence the same length, rhetorical questions, "esti o persoana care".
- Vary sentence length on purpose. A short sentence after a long one lands.

In English, the Romanian examples read roughly as “not just X, but Y and Z”, “in essence”, “in the end” and “worth mentioning”. The last two are “it is important to” and “you are a person who”. The hedging stack is “maybe, it might, possibly”. A tricolon is a run of three parallel phrases used for rhythm. A hedging stack piles several qualifiers onto one claim. The dash rule comes back below, as code.

Targets

The voice guide also sets numbers:

• reading level: grade 6 to 7;
• sentence length: about 15 words on average, with a hard cap around 20 to 25;
• Flesch reading ease: 60 to 70 or higher;
• paragraphs: 2 to 4 sentences on a phone screen.

Flesch reading ease scores text from sentence length and syllables per word. A higher score means easier text.

The fixed system prompt also asks for flowing paragraphs without bullet lists. It requires comma-below ș and ț explicitly. The prompt file cites research in which uppercase Ș and Ț were wrong in 5.4% of cases when the prompt did not ask for them.

Output format is part of the prompt too. In the report engine, model prose never goes through a JSON schema. The project’s rules cite a measured quality drop when prose goes into JSON, even with perfect escaping. A response cut off by max_tokens still leaves usable markdown. Truncated JSON does not. A deterministic parser turns that markdown into typed blocks afterwards.

Specificity and real readers

Two adversarial research passes concluded that flat text and generic statements that fit anyone work the same way: the reader supplies the meaning. So the prompts aim for specificity the reader cannot deny, plus one real open question, instead of a “wow”. One prompt change told the model to vary its first line. A check on 6 charts with opposite temperaments then produced six different openings.

Real readers tested the voice too. Three Romanian readers took part: one aged 24, one aged 41 and a skeptical engineer of 33. All three stopped scrolling and said the Romanian “sounds like a person, not AI”. The objections they shared (planet jargon, a transparent pay hook, flattery) became generator rules.

Checks around the prompt

The app’s 1,728 free readings were written by an agent team: a writer, an editor, blind judges and an automated gate. That is one reading per Sun, Moon and Ascendant combination, 12 × 12 × 12. The writer is a local script. It keeps a resumable queue of trios and runs parallel claude -p calls, four by default, on the Claude Code subscription instead of the paid API.

The prompt fixes the shape of each reading: a level-2 heading, then exactly four prose blocks, with no em or en dashes. A trio is marked done only after its file passes a gate. Anything else stays todo and is retried. This is the dash check in that gate, verbatim:

  const allText = blocks.map((b) => b.text ?? '').join(' ');
  if (/[—–]/.test(allText)) return 'contains an em-dash or en-dash';
  return null; // valid

The prompt asks for no dashes, and the code checks anyway. A rule this mechanical does not need a judge.

In the report engine, content prompts are versioned data. Editors move them from draft to test to publish, while the chapter set, order and schemas stay in code. A published prompt overrides the seeded chapter prompt at generation time. At most one version per chapter is live, and the code default runs when nothing is published. Prompt text that editors can change in the database has its less-than signs escaped, so it cannot close the prompt’s XML tags and be read as instructions.

Each report stamps the flow and prompt versions that produced it. With that stamp, any report traces back to the prompt version behind it. The launch runbook adds a test for sameness: if more than 60% of the text for two different charts is interchangeable, the prompts get rewritten.

What I’d keep

• Judge blind, so the verdict is about the text and not the model’s name.
• Pin the model by name in every script that writes text. If the CLI picks the model, a comparison can quietly test the wrong one.
• Name the AI patterns in the prompt, then check the mechanical ones, like dashes, in code.
• Compute the facts in code and leave the model only the interpretation.
• Version prompts like code, and stamp every output with the prompt version that wrote it.