I Built the Eval First. It Said the Model Wasn't Needed Yet.

The Setup

Readwider is a product I'm building for indie authors who swap newsletter mentions: promote my book in your newsletter and I'll promote yours. The record of a swap is a captured copy of the newsletter that went out, plus a few facts read off it: the date, whether the tracking link was in it, whether the cover appeared. All three are deterministic. Exact string equality for the link, image correlation for the cover.

The fourth fact needed a model: what did the send actually say about the book? Find the passage, quote it word for word, or say honestly that there weren't any words about it. The product rule is strict: the quote is shown to the author whose book it is, and "we couldn't find words about it" must never read as "your partner didn't do it".

Before writing the model call, I wrote down what would count as it working. That decision turned out to be the whole story.

The Question Nobody Asks First: Why Not Grep?

The honest objection arrived early. We're looking for a book title in a newsletter. Why is that an LLM task?

The answer I gave was reasonable: a title match finds where the title appears, not the words about the book. A sender writes "Rowan's new novel" without the title. A title appears in a backlist footer that isn't about anything. Two books share one send and the sentence about one runs into the other. All true. All untested.

So instead of arguing, I made the baseline a first-class participant in the eval: an instance that satisfies the same interface the models do, answers by finding the first passage containing the title and quoting the sentence around it, and costs nothing. Same seventeen cases, same validator, same scoring. If the models couldn't beat it, the LLM wasn't earning its place.

The Eval, and Why Those Metrics

The task is extractive question answering with abstention. That decomposes into two decisions that fail differently, so they're scored apart:

• → Abstention (mention present or absent). Two cells of the confusion matrix matter, and they aren't symmetric. A real mention reported as absent is the expensive error, because of what it implies about the partner. So I report mention recall and mention precision separately, and never accuracy: most sends do mention the book, so accuracy is the majority class in disguise.
• → The span. The question is where, not how well written. Character-level span F1 against a gold string, the extractive-QA convention. Not exact match, which fails a boundary one clause wide. Not ROUGE, because the output is constrained to be verbatim, so lexical overlap with the source is trivially 1. Not an LLM judge, because the gold is an offset pair and === answers it.

No calibration and no AUC: the model emits no probability, there's no threshold to sweep, and the product never shows a score. The pass line went into the runner's header before the first run: mention recall 1.0, mention precision 1.0, mean span F1 at or above 0.8.

Seventeen labelled cases, built from seven real captured newsletters across two email providers: nine mentions over three titles, eight abstentions, each tagged with a slice (single-book mention, mention in a two-book send, absent with another book present). Small. Honest about being small.

First Run: The Baseline Wins

That's not a disappointing result. It's the eval working on its first run. The alternative, shipping the hosted model because it felt like the right tool, would have cost money per capture to achieve what a string comparison achieves for free, on every case I could actually test.

The Eval Found Two Bugs Before I Could Blame the Model

The first run didn't look like the table above. It looked like this: Haiku at 0.78 mention recall, two cases "rejected" by the validator because the quote wasn't a substring of the chosen passage. Fabrication, I assumed. The model made up words.

Then I looked at the per-case rows instead of the aggregate. Both failures were from the same email provider, and both quotes were real sentences from the newsletter. The provider's editor had split those paragraphs across two blocks mid-sentence: "...a novel about a house on the Norfolk coast" / "that keeps every story ever told inside it...". My text projection treated each block as a passage. The model quoted the sentence a reader would see. The validator refused it for crossing a boundary I'd drawn.

Fix: a paragraph that doesn't end a sentence continues into the next. Deterministic, provider-neutral, one rule. Haiku went to 1.0 recall.

Then every F1 dropped. Including the baseline's, which doesn't read a prompt and can't have gotten worse at anything.

The labels had been cut at the old passage boundaries. When the projection learned to join the sentence, a model quoting the whole sentence was being penalised for being right. The fix wasn't to move the labels toward the models' answers, which is how an eval stops measuring. It was to write down one labelling rule, the whole sentence about the book, as rendered, punctuation included, and relabel under it. F1 went from 0.71 to 0.94 for Haiku and from 0.72 to 0.97 for the baseline, for the same answers.

The Prompt Change I Was Sure About

The local model had two failures. One was a tidied apostrophe, refused by the validator, which is the validator working and not a prompt problem. The other was more interesting: asked about The Lantern House in a newsletter about two other books by the same author, it claimed a mention. Author-for-book conflation. The prompt never said to match the title, not the author.

So I wrote v2: one sentence added, versioned so its hash changed, run against every instance with the v1 rows kept as the control. The adoption rule was declared before the run: the targeted case flips, no instance loses mention recall, F1 moves less than one case's worth.

Before writing that down, one more check. With seventeen cases, one case is 6% of the set. If the model varied run to run by one case, the "regression" would be noise. So I ran v1 a second time and diffed it case by case against the first: zero of seventeen differed. Temperature 0 through Ollama was deterministic on these inputs. The loss was the sentence's, and the rejection was clean.

The Detour That Wasn't About the Model Either

The local model's first two calls timed out at twenty seconds with zero tokens. My hypothesis was a cold load. Then the second call also timed out, and ollama ps showed the model resident at 13 GB with a 262,144-token context window on a 16 GB machine, split between CPU and GPU. It was also a thinking model, and it was spending the whole output budget reasoning before producing any JSON.

Neither knob was reachable through the OpenAI-compatible endpoint I'd written the adapter against. It honoured a JSON schema, which I'd verified, but ignored think: false and the context size. The vendor's native API took both. Same model, after: 26 tokens, two seconds warm, the same quote as the hosted model.

"OpenAI-compatible" is a claim about wire format, not about control. The knobs that make a small model usable tend to live outside the subset. Three probes and fifteen minutes settled it; guessing would have been a day of "the local model is too slow".

What the Model Call Sits Behind

None of the above was possible with a bare API call. The pieces that made it possible are small, and each one produced a finding this week:

• → A thin provider-neutral contract. One method: text in, text out, usage, or a named failure (timeout, unreachable, refused, empty). Three instances behind it: a scripted fake for the test suite, the hosted model, the local one. The baseline is a fourth, which is what let the eval run all of them with one flag.
• → Deterministic work before the model. The capture becomes numbered passages; the model selects one and copies words from it rather than recalling text. A closed choice is far easier for a small model than free generation, and it makes fabrication structurally impossible rather than merely caught.
• → A validator in code. The quote must be a substring of the passage the model chose. Not of the whole text: that closes the loophole where a model picks passage 0 and quotes passage 2. A refused answer is a named outcome, not an exception, and the ladder moves to the next instance on exactly that.
• → A trace per call with no text in it. Model, prompt hash, tokens, latency, cost at the price on file, outcome. A model with no price on file costs null, never zero. The prompt hash is what makes a v2 row impossible to confuse with a v1 row.

What it deliberately isn't: an agent harness. No memory, no tools, no loop. This is one call with one answer, and bolting a memory layer onto a stateless task is the same mistake as importing an abstraction from a different problem. Knowing which components don't apply is part of the job.

What It Tells the Operation

The feature is wired end to end: capture a send, run the ladder, store the quote with its provenance, show it as a fourth sentence under the capture, record the trace. Locally it works. In production, the model keys are deliberately unset, so an author sees exactly what they saw before.

That's the operational decision the eval licenses, and it's a cheap one to revisit. Two paths forward, both decided by numbers rather than taste: label the cases only a model can get right and re-run, wiring the hosted key if a model beats the baseline on them; or ship the baseline itself as the first rung, for free, with a model added behind it only when the eval says it's worth it.

Before Adding a Model Call

1. Is the deterministic alternative a row in your eval table? If not, you don't know whether the model is earning its place, and you won't until it is.
2. Are your metrics the decision's metrics? Asymmetric costs want precision and recall kept apart. A number that never reaches a surface doesn't need calibration. A gold that's an offset doesn't need a judge.
3. When a number moves, did the control move too? If it did, the problem is in the labels or the pipeline. Fix that before touching the model or the prompt.
4. Did you declare the adoption rule and measure the noise floor before the prompt iteration? One change per iteration, previous version kept, same version run twice. Otherwise a one-case effect is a coin flip you've named.
5. Can the eval be re-run from a committed file? Cases, results per model and prompt hash, and a baseline. If the record lives in a dashboard, the next person can't check your claim.

The Takeaway

I set out to add a model to a product and ended up with a feature that's tested, traced, and switched off, because the thing I built first was the instrument that could tell me it wasn't needed yet. Every finding this week came from that instrument: the baseline tie, the split-sentence bug, the label boundaries, the rejected prompt, the variance check. The model call is forty lines behind a contract. The eval is the work.

If a hiring manager asks what I built, the honest sentence isn't "an LLM feature". It's "an eval that showed the model wasn't justified yet, and the plumbing that made it possible to say so against three instances with one flag".