OATutor Was Built to Make Experiments Cheap to Rerun. Here's What Happened When Researchers Did.

Isn't a Statistically Significant Pilot Enough?

OATutor is an open-source adaptive tutoring platform built by the Computational Approaches to Human Learning Research (CAHLR) lab at UC Berkeley's School of Education. It runs entirely as a client-side React app, uses Bayesian Knowledge Tracing (BKT) to estimate what a student has and hasn't mastered, and ships with three OpenStax algebra textbooks worth of curated problems (Elementary, Intermediate, and College Algebra). It's MIT-licensed and deployable to GitHub Pages without a backend (Pardos, Tang, Anastasopoulos, Sheel & Zhang, CHI '23).

The feature that matters for this post isn't the tutoring itself, it's the built-in A/B testing: OATutor randomly assigns students to conditions and logs every interaction, so a research team can stand up a controlled experiment without building study infrastructure from scratch. That's supposed to make it cheap to run an experiment, get an answer, and move on.

In 2023, Zachary Pardos and Shreya Bhandari used exactly that infrastructure to ask a timely question: can ChatGPT write tutoring hints as good as a human tutor's? The pilot had 77 participants split across two algebra topics. The result looked clean: human-tutor-authored hints produced statistically significant learning gains. ChatGPT's hints didn't. About 70% of ChatGPT's hints passed a manual quality check, but the ones that shipped weren't enough to move the needle the way the human-authored hints did (a 2023 preprint, not yet peer-reviewed).

A statistically significant result from a real study, with real students, is normally where you stop and treat the question as settled.

What Happened When the Study Got Rerun at Scale?

In 2024, Pardos and Bhandari published a follow-up in PLOS ONE, a peer-reviewed journal, using OATutor as the delivery platform again (Pardos & Bhandari, 2024). This time: 274 participants (after exclusions), four subject areas instead of two (Elementary Algebra, Intermediate Algebra, College Algebra, Statistics), three conditions (ChatGPT hints, human-tutor hints, no-help control), and a pre-test/acquisition/post-test structure.

Against the no-help control, ChatGPT's gain was statistically significant (p = 0.011). Human tutor's gain, despite being a similar size, wasn't (p = 0.087). And directly against each other, ChatGPT and human-tutor hints were not statistically distinguishable (p = 0.416).

The pilot's clean, one-directional result, human beats ChatGPT, didn't reappear. The bigger study couldn't separate the two conditions from each other, and if anything the one condition that cleared the bar against doing nothing was ChatGPT, not the human-authored hints.

Why Would a Bigger Study Erase the Gap?

Neither paper frames this as a single unified causal story, so what follows is inference, not something either paper states outright. Two concrete differences between the studies stand out. First, sample size: 77 participants across two topics versus 274 across four. A 77-person study split into three-way comparisons has less power to detect a real but modest difference, and more room for a between-group gap to look larger and cleaner than it actually is. Second, hint quality control: the follow-up added a self-consistency technique to catch and filter bad ChatGPT-generated hints before they reached a student, cutting the rate of low-quality hints from 32% (no mitigation) to nearly 0% for algebra topics and 13% for statistics.

Put together, a plausible read is that the pilot's clean win for human-authored hints was partly a small-sample result and partly an artifact of unfiltered ChatGPT output dragging down that condition's apparent effectiveness, rather than a stable fact about what ChatGPT-generated hints can do. Treat that as a reasonable guess, not as something either paper established.

So What Does This Say About OATutor Itself?

The reversal is only visible because both studies ran on the same open platform. OATutor's built-in A/B testing already handles random assignment and interaction logging: when a student visits the tutor, a condition is assigned automatically and every subsequent action is logged under it (Pardos et al., CHI '23). The second study didn't require rebuilding tutoring software, a content library, or an experiment pipeline from scratch. It required writing a bigger study and pointing it at the same infrastructure.

That's the explicit design goal, not something read into the system after the fact. The CHI '23 paper states the aim directly: let a researcher run an experiment on OATutor, then "make their entire end-to-end experimental framework, content, and platform available as a Github link in a publication for others to replicate, compare to, and build off of." The same paper compares OATutor against six other open-source tutor-like projects and finds it's the only one supporting content authoring, an adaptive algorithm, and A/B testing all three. This pair of studies is a case of that infrastructure being used for precisely what it's for: catching a result that didn't hold up, on a second, better-powered pass.

Takeaway

If you're evaluating AI-generated tutoring content, or reading someone else's pilot study about it:

• → A clean p-value from a small pilot isn't the final answer. The 77-person study had a statistically significant, one-directional result. It didn't survive being rerun at more than three times the scale.
• → Beating a control isn't the same as beating the alternative. ChatGPT hints cleared the bar against no help; human-tutor hints, despite a similar-sized gain, didn't. Neither cleared the bar against the other. Read all three comparisons, not just the one that's significant.
• → Cheap replication is what catches this. The reversal was only visible because rerunning the study didn't require rebuilding the tutor. If your evaluation setup makes a rerun expensive, budget for the fact that your first result might not survive one.