Findings from building AI systems. Evidence-based, not opinion-driven.
Prompting a model to 'ask questions instead of answering' sounds like it should produce a Socratic tutor. Two recent preprints show why that assumption doesn't hold up, and what actually works instead.
OATutor is an open-source adaptive tutor built around Bayesian Knowledge Tracing and built-in A/B testing. Two studies run on it, about fifteen months apart, show why that infrastructure matters. They also show how a small pilot's tidy result stopped holding up at scale.
I tuned the formula constants until every spec gate passed. The pipeline still produced an empty high-priority bucket. The win wasn't a metric; it was recognizing the spec's two requirements were quietly antagonistic.
I refined a sentiment classification prompt twice trying to clear an 85% accuracy target. Both attempts regressed. The win came from changing what I asked the model for, not how I asked for it.
I expected a pre-trained embedding model to be neutral on dating compatibility. Instead, it was systematically wrong, scoring incompatible pairs higher than compatible ones. Here's why that made fine-tuning more interesting.
I built two RAG pipelines. In one, adding a reranker dropped accuracy by 7%. In the other, it improved accuracy by 7%. Same technique, opposite results.