Controversy Surrounds Research on ChatGPT's Impact on Student Learning in Schools
On a Fresh Note: Assessing the Veracity of AI's Impact on Education
Remember the buzz around Microsoft and Carnegie Mellon Study that suggested AI was dimming our intel? A more recent study, unaltered by hype, by researchers Jin Wang and Wenxiang Fan, appears to counter the eye-catching headlines. Here's the lowdown on why we used appears - read on for the deets.
The duo's study, published in May 2025, has sparked significant interest as it delves into the rapidly growing interest in AI tools for education. It also tackles issues surrounding screen time, diversity, and the quality of learning.
Yet, harmonious agreement is scarce on what's published. For instance, Ilkka Tuomi, Chief Scientist at Meaning Processing Ltd., expresses reservations about the study's methodology.
Let's begin with a peek at the video below before we scrutinize the study and Tuomi's critical views.
- 1 AI and Learning Outcomes - Valid Claims?
- 2 Misleading Applications of Hedges' g
- 3 Core Concerns with the Meta-Analytic Structure
- 4 Risk of Misinterpretation - What's g = 0.867, Really?
- 5 Misguiding Assumptions by Context
- 5.1 Best Use in STEM and Skill-Based Courses?
- 5.2 High Impact in Problem-Based Learning?
- 5.3 Ideal Duration - 4-8 Weeks?
- 6 Support for Higher-Order Thinking?
- 7 Why These Effect Sizes Aren't Reliable
- 8 The Danger of Regularizing Education Without Rigor
AI and Learning Outcomes - Valid Claims?
A recent meta-analysis by Jin Wang and Wenxiang Fan - studying 51 studies on the use of ChatGPT in education - was shared as evidence that AI bolsters learning outcomes. The findings, published in Nature: Humanities and Social Sciences Communications, have drawn attention from educators and policymakers eager to explore AI in classrooms.
But as the study gained traction, voices of dissent emerged. Among the most vocal is Tuomi, who questions the study's methodological integrity. His concerns expose underlying issues with how evidence is selected, analyzed, and communicated in the rapidly evolving AI-in-education research landscape.
Below, find Tuomi's opening comments, posted on LinkedIn. Spoiler alert: he isn't holding back. That's why I've called these studies junk.
The claims follow up the next day, offering more context. Then, another update reveals Tuomi's concerns in greater detail.
Misleading Applications of Hedges' g
Wang & Fan use Hedges' g - a popular statistical tool - to calculate effect sizes across 51 studies evaluating ChatGPT's impact on student outcomes. While Hedges' g supports standardization and cross-comparison of diverse study results, its application in this meta-analysis fails to address core validity issues - instead, it obscures them.
To clarify any potential confusion, the mathematical use of Hedges' g holds water. The interpretative problem lies in employing it across weak or heterogeneous studies without quality filtering.
Core Concerns with the Meta-Analytic Structure
Wang & Fan do not report on whether included studies were peer-reviewed, randomized, or sufficiently powered. The study skips simple quality screening or scoring methods. This oversight is crucial: the aggregation of effect sizes from studies of varying rigor without quality weighting damages the validity of overall findings.
Moreover, Wang & Fan neglect to explore heterogeneity – despite it being high. A structurally sound meta-analysis would analyze differences in context, methodology, or sample characteristics to understand where effects differ and why. Wang & Fan, however, present averaged effect sizes as if they sum up a consistent phenomenon.
Risk of Misinterpretation - What's g = 0.867, Really?
The headline claim of the study is that ChatGPT significantly improves student learning performance with a large effect size (g = 0.867). Yet, these impressive figures carried by questionable foundations.
For example, some included studies appear to:
- Lack control groups
- Use teacher-made or unvalidated assessments
- Have treatment groups with a scant number of participants
- Come from low-credibility journals with sparse peer review
A value like g = 0.867, calculated across such assorted studies, may look precise but reveals little about genuine, real-world effects.
Misguiding Assumptions by Context
1. Best Use in STEM and Skill-Based Courses?
The study indicates:
- STEM subjects: g = 0.737
- Skills courses: g = 0.874
These figures are presented as if AI consistently strengthens structured learning. Nonetheless, the inconsistency across studies (e.g., task type, assessment format, duration) raises concerns that these results may stem from cherry-picked findings or publication bias rather than genuine insights.
2. High Impact in Problem-Based Learning?
The reported g = 1.113 in problem-based learning is the study's highest figure. But it fails to provide insight regarding how many studies fall into this category, how PBL was defined, or whether comparison groups followed similar pedagogical models. Once again, the result is tempting yet analytically weak.
3. Ideal Duration - 4-8 Weeks?
The study suggests that interventions lasting 4-8 weeks yielded the best results (g = 0.999). Yet, no statistical justification is provided for why this time frame matters. It is more likely that longer exposure allowed students more time to adapt, while shorter durations could represent pilot programs or single-session trials, which are not directly comparable.
Support for Higher-Order Thinking?
Wang & Fan report a moderate effect on critical thinking (g = 0.457) and recommend AI as a tutor (g = 0.945) or with scaffolds like Bloom's taxonomy to enhance outcomes. While plausible, these suggestions rely on uncategorized and unverified implementations. Time will tell if AI can genuinely promote higher-order thinking skills.
The Danger of Regularizing Education Without Rigor
To put it simply, the Wang & Fan (2025) study showcases the risks of entangling educational interventions without meticulousness. The use of Hedges' g delivers the illusion of accuracy, but the absence of methodological control, inconsistent definitions, and absent quality thresholds yield findings with little practical relevance.
Educators, policymakers, and researchers must approach these findings with caution and demand higher standards in AI-integrated education where transparency, theory, and replicability have equal importance as effect size.
Source: Enrichment Data
The reliability of the claims made in the meta-analysis by Jin Wang and Wenxiang Fan regarding the impact of AI, specifically ChatGPT, on student outcomes is subject to significant scrutiny. Their study, which analyzed 51 studies on ChatGPT in education, suggests that AI can improve learning outcomes, marking a shift from questioning whether AI should be used in classrooms to how it should be effectively integrated[1] . However, criticism from experts like Ilkka Tuomi, Chief Scientist at Meaning Processing Ltd., highlights several methodological concerns:
- Use of Hedges’ g: The study's use of Hedges’ g, a measure of effect size, has been questioned. Critics argue that the application of this metric may not accurately reflect the actual impact of AI on student outcomes[1] .
- Misinterpretation Risks: The effect size reported in the study (g = 0.867) may be misleading without proper context. Critics suggest that such a figure could be interpreted in various ways, potentially leading to misinterpretations about the actual benefits of AI in education[1] .
- Lack of Methodological Control: There are concerns about the absence of sufficient methodological controls in the meta-analysis. This lack of control could lead to biased results, as different studies may have varying methodologies and quality standards[1] .
- Contextual Generalizations: The study's conclusions may overgeneralize the impact of AI across different educational contexts. Educational settings vary significantly, and what works in one context may not apply universally[1] .
- Support for Higher-Order Thinking: Some critics question whether the study adequately supports the claim that AI enhances higher-order thinking skills. While AI can facilitate certain cognitive tasks, its role in promoting deeper, more complex thinking is still a topic of debate[1] .
In summary, while the study by Jin Wang and Wenxiang Fan suggests positive outcomes from using ChatGPT in education, the reliability of these claims is compromised by methodological concerns and potential misinterpretations. Further research with robust methodologies is needed to fully assess the impact of AI on student outcomes.
1. The concerns raised by Ilkka Tuomi, Chief Scientist at Meaning Processing Ltd., question the validity of the study by Jin Wang and Wenxiang Fan, which suggests AI can bolster learning outcomes.2. Tuomi's criticism focuses on the study's use of Hedges’ g, a measure of effect size, claiming that it may not accurately reflect the real impact of AI on student outcomes.3. The danger of misinterpretation arises from the calculated effect size (g = 0.867), which may be misleading without proper context.4. The study's conclusions, according to critics, could yield biased results due to the lack of methodological control across diverse studies.5. Critics also argue that the study may overgeneralize the impact of AI across different educational contexts, limiting its usefulness and practical relevance.
