Blog

I added the first LLM call to a product with the eval and a deterministic baseline in place before the model. On the real cases, grep tied the hosted model. The eval also found two pipeline bugs I'd have blamed on the model, and rejected a prompt change after I measured run-to-run variance.

Prompting a model to 'ask questions instead of answering' sounds like it should produce a Socratic tutor. Two recent preprints show why that assumption doesn't hold up, and what actually works instead.

OATutor is an open-source adaptive tutor built around Bayesian Knowledge Tracing and built-in A/B testing. Two studies run on it, about fifteen months apart, show why that infrastructure matters. They also show how a small pilot's tidy result stopped holding up at scale.

I tuned the formula constants until every spec gate passed. The pipeline still produced an empty high-priority bucket. The win wasn't a metric; it was recognizing the spec's two requirements were quietly antagonistic.

I refined a sentiment classification prompt twice trying to clear an 85% accuracy target. Both attempts regressed. The win came from changing what I asked the model for, not how I asked for it.

I expected a pre-trained embedding model to be neutral on dating compatibility. Instead, it was systematically wrong, scoring incompatible pairs higher than compatible ones. Here's why that made fine-tuning more interesting.

I built two RAG pipelines. In one, adding a reranker lowered MRR by 0.070. In the other, it raised MRR by 0.056. Same technique, opposite results.