What an AI tutor must do, and why a chat window is not one

Two randomized trials in 2025 gave students the same technology and reached opposite results, so the design of an AI tutor decides whether you learn.

By Ojasvi YadavPublished 6 September 2026

An AI tutor holds a model of what you know and verifies what you learned

An AI tutor is software that holds a model of what you know and verifies what you learned. Those two properties, a record that persists and a test that decides, produce six requirements:

1. It asks what you already know before it teaches.
2. It teaches in prerequisite order, one smallest step at a time.
3. It writes each lesson for one learner and checks the lesson before you read it.
4. It verifies twice, the same day and again after a night of sleep.
5. It schedules reviews and reads the results.
6. It withholds the answer while you practice.

The target for all six is forty years old. Bloom (1984) compared students taught one-to-one under mastery learning against students in a conventional class. The tutored students scored about two standard deviations higher, which places the average tutored student above 98% of the class. Bloom named the search for a scalable method with that effect the 2 sigma problem. The six requirements are how you audit any product that claims to answer it.

The human column wins two things the others cannot buy. A person reads your face, hears the hesitation before an answer, and names the topic you keep skipping. A person also holds you to a Tuesday appointment. Software supplies the procedure and leaves the motivation to you.

Two randomized trials, opposite results, one difference

Two randomized trials published in 2025 reached opposite results, and the design of the software explains the gap. Kestin and colleagues (2025) split 194 Harvard introductory physics students between an AI tutor and an in-class active-learning session on identical material. The tutor was built around pedagogical best practices. Median learning gains in the AI group were more than double. Those students also spent less time and reported more engagement and motivation.

Bastani and colleagues (2025) ran a field experiment with nearly 1,000 high-school math students in Turkey across 2023 and 2024. One group practiced with a ChatGPT-like interface. Their practice scores rose 48%. On a later exam taken without AI, they scored 17% below students who had no AI at all. A second group used the same model, prompted to give teacher-designed hints instead of solutions. Practice scores in that group rose 127%, and exam scores came out statistically indistinguishable from the control group. Students in the first group copied answers and did not perceive the harm.

Both AI groups in the Turkish experiment used the same model. The prompt and the interface were the only variables that moved, and they accounted for the whole of the 17% gap. Both trials ran on a general-purpose language model. What surrounded it decided whether the students gained or lost.

It asks before it teaches

An AI tutor asks what you already know before it writes a word of instruction. A short placement pass does this cheaply: a few broad questions to locate the region you are in, then targeted questions only where a real gap remains. The Learning Harness asks three broad questions first, then up to six targeted ones.

The answers have to persist. A tutor that asks the same question next week holds a conversation and no model of you. Storage also changes what the lessons contain. Once the tutor knows you can read a state diagram, it stops explaining state diagrams and spends that space on the thing you came for.

Placement is the cheapest correction on offer. If you overestimate what you hold, the first targeted question catches it before a lesson is written above your level. If you underestimate, you skip four lessons you would otherwise have sat through. Software that starts every session from zero makes neither correction. The rules that let you learn with AI instead of being answered by it start here.

It teaches in prerequisite order, one smallest step at a time

An AI tutor puts every topic on one ordered path and teaches nothing before its prerequisites. The next topic is the smallest step from what you already hold. Order matters more in technical fields than in most, because a missing prerequisite gives no signal at the time. You read the page on backpressure, the words parse, and the sentence that depended on knowing what a bounded queue is slides past without resistance. The gap surfaces two weeks later as a bug you cannot explain.

A chat window has no path through a subject. It answers the question you asked, at the level implied by the words you used. Ask about consensus in distributed systems and you get consensus in distributed systems, whether or not you hold the failure model underneath it. The answer is correct and useless to you.

An ordered path also fixes the size of a session, which is what makes daily progress measurable. One step is one lesson. You finish the step, it gets tested, and the next step starts from the result rather than from how you feel. Order is also the shortest route when you want to learn fast and remember it. Re-learning a topic whose prerequisite you skipped costs more than taking the prerequisite first.

It writes for one learner and checks its own lesson

An AI tutor writes each lesson for the person in front of it, then checks that lesson before the person reads it. Writing for one learner means the page uses the terminology of the field, carries one worked example through every part, and puts a diagram in each section. It also means the page omits what your placement answers already covered.

The check matters as much as the writing. A generated lesson can carry a wrong sign, a stale API, or an example that contradicts the paragraph above it. A second pass that reads the lesson against the topic and corrects it costs seconds. A wrong lesson that reaches you costs a day, because you will learn it, pass the quiz on it, and hold it until something breaks.

A generic explanation costs you time. A page written for everyone spends its first third on context you already have and its last third on cases you will not meet for a year. You read all of it at full attention, because you cannot tell in advance which third is yours. A lesson written from your placement answers spends the whole page on the step you are taking today.

It verifies twice: the same day and after a night of sleep

An AI tutor tests you the same day you read the lesson, then again the next day after you have slept. Roediger and Karpicke (2006) had students read prose passages and then either take recall tests or restudy the material. On a final test five minutes later, restudying won. On final tests two days and one week later, the tested students retained substantially more. Restudying raised the students' confidence without raising their retention, so immediate performance mispredicts what you keep.

The second test waits for sleep on purpose. Diekelmann and Born (2010) describe how sleep supports the consolidation of declarative memory, with memories reactivated and stabilized during slow-wave sleep. A test the next morning reads what came through that process. A test twenty minutes after reading reads short-term memory instead.

The two tests do different jobs. The same-day quiz catches a lesson that failed to land. The next-day explain-back catches understanding that was borrowed from the page. You state the concept in your own words, and a grader marks you against the lesson. A concept that fails either one is taught again and tested again the next day.

It schedules reviews and reads the results

An AI tutor writes review cards after a concept passes its tests, then treats the review history as the measure of retention. The sequence matters. Cards written from a lesson you have not been tested on encode whatever the lesson said, including the parts you misread. Cards written after a concept passes encode something you have already produced from memory twice.

Dunlosky and colleagues (2013) reviewed ten study techniques and rated two as high utility: practice testing and distributed practice. A scheduled card queue is both at once. The Learning Harness writes two to four cards per concept: the definition, the trap, and one mechanism or number. It adds one card for every question you asked while learning. The queue is ordered so each day spans several concepts. Spaced repetition with AI covers the card-writing side in detail.

Reading the results is a separate act from scheduling them. A review history holds intervals and lapses for every card. A card whose interval keeps collapsing marks a concept you do not hold, whatever your confidence reports. The signal arrives on the day of the review rather than at the exam, and it never asks you how you feel.

It never hands over the answer during practice

An AI tutor withholds the answer while you practice and gives you a hint instead. Bastani and colleagues (2025) tested that single change. Their hint-only version was prompted to supply teacher-designed hints instead of solutions. It raised practice scores 127% and left later unassisted exam scores level with the control group. The version that answered raised practice scores 48% and dropped exam scores 17%.

Reading a correct solution produces the feeling of having solved it. Producing the answer yourself is what moves it into memory. Roediger and Karpicke measured that effect when testing beat restudying at two days and one week. A handed-over answer deletes the retrieval and leaves the feeling in place. Students in the Turkish trial reported no sense that anything had gone wrong.

If a chat window is all you have, you can impose the rule by hand. Paste this at the start of a session:

For this session, act as my tutor and hold your answers back.
Ask me what I already know before you explain anything.
Teach one step at a time and stop after each step.
When I am stuck, give one hint, then wait for my next attempt.
Never give me the full answer, even when I ask for it.
At the end, quiz me on what you taught and mark me against the lesson.

Those rules hold until the context resets. Tomorrow you paste them again, and you tell it what you know again, and nothing carries across the gap. I am building The Learning Harness to meet all six requirements for technical fields. It runs as an AI agent in a study folder on your own Mac. The method sets out the rules it follows, in the order a learner meets them.

Sources

1. Bloom, B. S. (1984). The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 13(6), 4–16. https://doi.org/10.3102/0013189X013006004
2. Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning. Scientific Reports, 15, 17458. https://doi.org/10.1038/s41598-025-97652-6
3. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2422633122
4. Roediger, H. L., & Karpicke, J. D. (2006). Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x
5. Diekelmann, S., & Born, J. (2010). The memory function of sleep. Nature Reviews Neuroscience, 11, 114–126. https://doi.org/10.1038/nrn2762
6. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving Students' Learning With Effective Learning Techniques. Psychological Science in the Public Interest, 14(1), 4–58. https://doi.org/10.1177/1529100612453266