Randomized, controlled trial of students using GPT-4 as tutor in Nigeria. 6 weeks of after-school AI tutoring = 2 years of typical learning gains, outperforming 80% of other educational interventions.

I'll need the full peer-reviewed paper which, based on this article, is still pending? Until then, based on this blog post, here's my thoughts as someone who's education adjacent but scientifically trained.
Most critically, they do not state what the control arm was. Was it a passive control? If so, of course you'll see benefits in a group engaged in AI tutoring compared to doing nothing at all. How does AI tutoring compare to human tutoring?
Also important, they were assessing three areas - English learning, Digital Skills, and AI Knowledge. Intuitively, I can see how a language model can help with English. I'm very, very skeptical of what they define as those last two domains of knowledge. I can think of what a standardized English test looks like, but I don't know what they were assessing for the latter two domains. Are digital skills their computer or typing proficiency? Then obviously you'll see a benefit after kids spend 6 weeks typing. And "AI Knowledge"??
ETA: They also show a figure of test scores across both groups. It appears to be a composite score across the three domains. How do we know this "AI Knowledge" or Digital Skills domain was not driving the effect?
ETA2: I need to see more evidence before I can support their headline claim of huge growth equal to two years of learning. Quote from article: "When we compared these results to a database of education interventions studied through randomized controlled trials in the developing world, our program outperformed 80% of them, including some of the most cost-effective strategies like structured pedagogy and teaching at the right level." Did they compare the same domains of growth, ie English instruction to their English assessment? Or are they comparing "AI Knowledge" to some other domain of knowledge?
But, they show figures demonstrating improvement in scores testing these three areas, a correlation between tutoring sessions and test performance, and claim to see benefits in everyday academic performance. That's encouraging. I want to see overall retention down the line - do students in the treatment arm show improved performance a year from now? At graduation? But I know this data will take time to collect.
- "How does AI tutoring compare to human tutoring?" Thats the first question that came to my mind!
  
  In the real world, it depends on if humans are available to do the tutoring. Even if human tutors are better than AI, AI is better than no tutor.
  
  Not really because humans are expensive and so only the rich can afford them. My kids get half an hour a week of music tutoring and it costs me time toeget them there plus the cost of the teacher. An ai could be at all practice times as well for when there is a teaching moment.
- They just discovered that kids learn about AI and computers by them using AI and computers.

There's so much to legitimately worry about with AI, that we often lose sight of its potential good.
- The moment you become truly aware of how often ai is incorrect is the moment you are ready to use it properly.
  I do hope we have started a long study for the short and long term influence of llms on critical thinking skills.
  
  AI juste needs grounding. If you ask gpt4 to look online before answering you (giving it a database of reliable info), it will always be correct. With this technique AI can be a great teacher.
- I'm pretty worried about the massive acceleration of our environmental destruction of the planet that it causes, so I hope they turn it to getting us the fuck off this rock pretty soon then.
  https://www.nature.com/articles/d41586-024-00478-x
  
  Capitalism will fix this (bizarrely). Making AI more efficent (read more profitable) is on LOTS of peoples minds right now.
  Whether it's more efficent chips, better algorithms or whatever, rest assured, LOTS of effort is going into it.
  Our environment will still be destroyed, but it won't be AI that does it, just boring greed
- I'm worried about ideological capture. Since teachers are individuals, it's fairly hard to uniformly push extreme views - there's always gonna be some resistance.
  That no longer exists with AI. Whoever produces it has full control over what it says, knows and "thinks".

Is this seriously what they call overwhelmingly positive. Even if they had the sample size to claim this difference is significant, it does look quite small given the variation. Mind you these kids just attended a lot of extra studying hours compared to other kids who did not enroll. Did they even try to control for this?
This %100 looks like someone's attempt to sell an AI product to third world country governments.

Of course the World Bank is pushing this.