2w ago

Randomized, controlled trial of students using GPT-4 as tutor in Nigeria. 6 weeks of after-school AI tutoring = 2 years of typical learning gains, outperforming 80% of other educational interventions.

blogs.worldbank.org From chalkboards to chatbots: Transforming learning in Nigeria, one prompt at a time

"AI helps us to learn, it can serve as a tutor, it can be anything you want it to be, depending on the prompt you write," says Omorogbe Uyiosa, known as "Uyi" by his friends, a student from the Edo Boys High School, in Benin City, Nigeria. His school was one of the beneficiaries of a pilot that used...

You're viewing a single thread.

21 comments

I'll need the full peer-reviewed paper which, based on this article, is still pending? Until then, based on this blog post, here's my thoughts as someone who's education adjacent but scientifically trained.
Most critically, they do not state what the control arm was. Was it a passive control? If so, of course you'll see benefits in a group engaged in AI tutoring compared to doing nothing at all. How does AI tutoring compare to human tutoring?
Also important, they were assessing three areas - English learning, Digital Skills, and AI Knowledge. Intuitively, I can see how a language model can help with English. I'm very, very skeptical of what they define as those last two domains of knowledge. I can think of what a standardized English test looks like, but I don't know what they were assessing for the latter two domains. Are digital skills their computer or typing proficiency? Then obviously you'll see a benefit after kids spend 6 weeks typing. And "AI Knowledge"??
ETA: They also show a figure of test scores across both groups. It appears to be a composite score across the three domains. How do we know this "AI Knowledge" or Digital Skills domain was not driving the effect?
ETA2: I need to see more evidence before I can support their headline claim of huge growth equal to two years of learning. Quote from article: "When we compared these results to a database of education interventions studied through randomized controlled trials in the developing world, our program outperformed 80% of them, including some of the most cost-effective strategies like structured pedagogy and teaching at the right level." Did they compare the same domains of growth, ie English instruction to their English assessment? Or are they comparing "AI Knowledge" to some other domain of knowledge?
But, they show figures demonstrating improvement in scores testing these three areas, a correlation between tutoring sessions and test performance, and claim to see benefits in everyday academic performance. That's encouraging. I want to see overall retention down the line - do students in the treatment arm show improved performance a year from now? At graduation? But I know this data will take time to collect.
- "How does AI tutoring compare to human tutoring?" Thats the first question that came to my mind!
  
  In the real world, it depends on if humans are available to do the tutoring. Even if human tutors are better than AI, AI is better than no tutor.
  
  Personally, I don't think the evidence has established an AI tutor is better than no tutor. Maybe for English, certainly not for Math, Science, History, Art, or any other subjects.
  LLMs can be wrong - so can teachers, but I'd bet dollars to donuts the LLM is wrong more often.
  
  This is true. I was just thinking in the case of the study, but /any/ help when needed is better than no help.
  
  Not really because humans are expensive and so only the rich can afford them. My kids get half an hour a week of music tutoring and it costs me time toeget them there plus the cost of the teacher. An ai could be at all practice times as well for when there is a teaching moment.
  
  Unfortunately, I don't think we're close to getting an AI music tutor. From a practical standpoint, a large language model can't interpret music.
  From an ethos standpoint, allowing AI to train a musician breaks my heart.
  
  That is a good - but different - point.
  Though a large part of tutoring music is just "play that again" said in a way that the music is played again.
  
  I could see a world where an AI teaches the mechanics of music, but is not relied on for evaluation of the students imagination in applying the mechanics.
  When I was in art classes I wanted the class to teach me the techniques I did not know, but I was usually disappointed.
  Like my life drawing classes would be me drawing images, and then getting judged on /what I already knew how to do/.
  What I felt I would have benefited from was more along the lines of "here is three different shading techniques and how they can be used" or "here are three ways to use {oil | acrylic | water color} paints you've not used before."
  I always had ideas of images to create, what I could have benefited from was intros to more tools and ways to use those tools.
  I could see an AI being able to do that for students of the arts.
- They just discovered that kids learn about AI and computers by them using AI and computers.

21 comments