Building a Socratic AI Tutor from Scratch: How LLMs Learn to Ask Better Questions Than They Answer

Isn't "Ask Questions, Don't Answer" Enough?

A Socratic tutor is one that helps you reach an answer by asking you questions, instead of just telling you the answer. The idea is old (it's named after a teaching style from ancient Greece), but building one out of a language model looks, at first glance, like a one-line fix. You write a system prompt: "Never give the direct answer. Always respond with a guiding question instead." The model already knows how to answer questions well. You're not asking it to do anything hard, just to change its output shape.

That's the obvious plan. It's also not what the evidence supports.

Does Asking Questions Make It Socratic?

A 2025 preprint, Discerning Minds or Generic Tutors?, looks at exactly this gap. Prompting a model to end its responses with a question produces the form of Socratic teaching: there's a question mark at the end of the sentence. It doesn't guarantee the function: actually tracking what a specific student does and doesn't understand, and adjusting the next question to fit.

It's the difference between a support agent who asks "how can I help?" and one who actually listens to the answer. Both produce a question. Only one produces a conversation that goes anywhere.

So What Actually Makes a Tutor Adaptive?

A separate 2025 preprint, Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues, took a different approach entirely. Instead of prompting a model to ask questions, the authors trained one (an open-source Llama 3.1 8B model, using a preference-optimization method) directly on a single target: does the student answer correctly on the next turn.

Nobody told the model to ask questions. It was only ever rewarded for the student's next answer being right.

The authors' own qualitative read of the transcripts: the trained model was more likely to ask a question, where other methods (including GPT-4o) more often just told the student what to do next. Nobody built that in. It came out of optimizing for outcomes alone.

Why Would Optimizing for Outcomes Produce Questions?

The preprint doesn't say. The authors describe the question-asking as an emergent behavior and stop there. They don't offer a mechanism, and neither this post nor the paper should be read as claiming one has been proven.

Here's a plausible explanation, offered as inference rather than as a finding: a model that's rewarded only for whether the student gets the next answer right can't earn that reward by simply stating the answer itself, because stating the answer doesn't test whether the student can produce it on their own, so pulling the student's own reasoning out first, by asking rather than telling, becomes the strategy that actually pays off under that specific reward.

That's one sentence doing a lot of work on purpose. It's the kind of causal chain that's easy to lose if it's cut into three short ones. Treat it as a reasonable guess, not as something the research established.

Takeaway

If you're building or evaluating something that's supposed to teach by asking rather than telling:

• → Prompting for the form isn't the same as testing for it. A system prompt that says "ask, don't tell" will produce questions. It won't, on its own, tell you whether those questions actually track what the student understands.
• → The stronger result came from optimizing outcomes, not style. The tutor that actually improved at eliciting correct answers was never told to ask questions. It was trained against the outcome that mattered, and the questioning behavior followed.
• → If you can't train, at least evaluate against outcomes. Without the ability to optimize a model directly, the closest substitute is testing whether it closes the loop with a student who's clearly understood something, rather than auditing whether it phrased its last response as a question.