What AI *can't* do in the classroom

When Khan Academy launched Khanmigo, I was quite impressed, more for what it could become than for what it already was, and I myself had already referenced Sal Khan's TED Talk here, so I think I have some responsibility to update what I think on the subject, especially because Khanmigo's story has been unfolding in a way that is quite revealing about how we tend to project onto tools a capacity for transformation that they simply do not have on their own.

Collage by Valerie Chiang for Edutopia, iStock (3)

Khanmigo was created through a partnership between Khan Academy and OpenAI that Josh Tyrangiel describes in the book AI for Good as a Las Vegas-style marriage between two completely mismatched partners, and even so it has been a non-event for most students so far. Sal Khan himself said as much publicly:

"for a lot of students, it was a non-event. They just didn't use it much."

What interests me in this story isn't Khanmigo's underwhelming flight itself, but what it points to. Khanmigo cannot perceive an actual student, doesn't know when to start a conversation with sensitivity or when to end it, doesn't build relationship, doesn't notice disengagement. I don't think this is an engineering problem that some future version will solve, it's a structural limitation of what the tool is.

A little over a month ago, Stanford released a review of the state of research on AI in K-12 education, and the number that caught my attention most was this: out of more than 800 academic papers on the subject, only 20 produce causal evidence strong enough to say something reliable about what AI actually does to students and teachers. And these 20 studies show a fairly consistent pattern that is, at the very least, uncomfortable: when students have access to the tool during a task, performance improves, but when access is removed and the assessment is independent, the gains frequently disappear, and in some cases performance is worse than that of students who never used the tool at all.

The most promising results, curiously, are on the teachers' side rather than the students'. Teachers with access to ChatGPT spent about 30% less time planning lessons with no reduction in quality, and there's one specific finding I find especially interesting: AI seems to be more useful for less experienced teachers, which carries considerable equity implications in a system where the most vulnerable teachers tend to work in the most underserved schools.

I've worked in education for over twenty years, and what all this evidence suggests to me is that AI has more potential when it frees up the teacher to do what AI cannot do than when it tries to do what the teacher does, because education's problem has never been a lack of content or a lack of exercises, it's a lack of qualified, present human attention, and that is something no tool is going to replace.