Tag: Feedback

Leveraging artificial intelligence to predict young learners’ online learning engagement

By Zia Hassan, Center for Research and Reform in Education, Johns Hopkins University

With many schools rushing to adopt Generative AI, it is important to consider the real learning gains (or lack thereof) that these tools offer. A 2023 study by Pardos & Bhandari examined the use of AI-generated hints as a scaffolding mechanism with Algebra students.

Seventy-seven participants (high school graduates selected via Amazon’s MTURK system) were assigned to a control group (which provided human-generated hints) or an experimental group (which provided AI-generated hints). The researchers wanted to learn the rate of “low quality” AI-generated hints, as well as if the hints produced learning gains compared to the control group. The questions from the lesson were fed, verbatim, to ChatGPT in order to generate the hints. Quality checks were performed manually to ensure that all AI-generated hints were correct and showed the proper steps. This was then contrasted with the control group, whose hints were generated by undergraduate tutors. Pre and post tests were administered to check for learning gains between the two groups.

The results showed that 70% of the hints generated by ChatGPT were considered to be good quality, and that there was a statistically significant learning gain in the control group. A major limitation of the study is that the researchers did not prompt the AI to use any scaffolding strategies. Therefore, the quality of the hints between groups not only differed by human or AI creator, but also by pedagogical theory. Human tutors were probably more likely to employ Vygotsky-esque scaffolds, while ChatGPT was more likely to provide an immediate answer. Future work could improve upon the prompts used in this study and create a multi-tiered approach with less consequential hints being revealed at first.

Examining the effects of AI assistance on student agency in higher education

By Feifei Wang, the Chinese University of Hong Kong

While AI-powered learning technologies are increasingly used to automate and support learning activities, often with positive outcomes, their impact on student agency is under-explored. Student agency refers to students’ capacity to actively regulate learning actions, make responsible decisions, and navigate various learning contexts, which is essential for lifelong learning. A recent randomized controlled experiment explored the impact of AI assistance on student agency in higher education, addressing three research questions: Do students learn from AI assistance? After an initial period of time, can AI assistance be replaced with self-monitoring checklists? Would complementing AI assistance with self-monitoring checklists enhance student performance?

The study involved 1625 undergraduate students across 10 courses from various disciplines in 2020. During the initial four-week period, students provided peer-reviewed comments to each other, guided by AI features to enhance their feedback. Over the following four-week period, they were divided into four groups: a non-AI-assisted group, an AI-assisted group, a self-monitoring group without AI assistance, and a self-monitoring group with AI assistance. The study used six measures to evaluate student agency from different perspectives of students’ reviews: rate of reviews that needed revision, similarity to previous comments, relatedness to reviewed resources, review length, time spent on reviews, and helpfulness ratings from other reviewers.

Results showed that AI assistance significantly improved the quality of students’ reviews, but the influence declined after AI assistance was removed. This suggests that while AI can effectively scaffold learning, students tend to rely on it rather than learn from it. Additionally, after using AI assistance for some time, students can still benefit from self-monitoring checklists even without AI assistance. However, combining AI assistance with self-regulation strategies did not lead to significant improvement in student performance. The authors attribute the insignificant improvement to two possible reasons. First, when supports of varying strengths interact, the stronger one may overshadow or diminish the impact of the weaker one. Second, learners have limited cognitive resources, which can be overwhelmed if the cognitive load exceeds their capacity, so the higher load from AI assistance might have reduced their capacity to effectively use self-monitoring checklists. The authors concluded that while AI-powered learning technologies present many benefits, they should be used with caution, taking into account pedagogical factors and meticulously balancing potential benefits against possible drawbacks.

Effects of feedback in technology-rich learning environments

By Winnie Tam, Centre for University and School Partnership, The Chinese University of Hong Kong

Technology-rich learning environments (TREs) integrate new technologies and media to enhance information resources and tools, such as intelligent tutoring systems, virtual reality, and educational learning games. A meta-analysis conducted by Cai and colleagues examined the impact of feedback on academic performance within TREs. The included studies required both an experimental group and a control group, with the experimental group receiving one type of feedback while the control group either received no feedback or a different type of feedback. The analysis encompassed 182 effect sizes from 61 studies, with the majority published between 2010 and 2021. In comparison to the no feedback condition, the feedback group had a medium positive effect (+0.44) on academic performance. The type of feedback served as a significant moderator, with elaborate feedback having a stronger effect than feedback solely indicating if answers were correct. Specifically, the most effective feedback was explanation feedback (+0.69), which detailed why responses were right or wrong, followed by metacognitive feedback (+0.52), which related to the process of monitoring and regulating learning, followed by prompt feedback (+0.39), which provided information such as examples, tips, or demonstrations..

Given the rapid development of technology in education, this meta-analysis provides a timely update on feedback within this research area.

Six recommendations on how to effectively use feedback to improve students’ learning

By Carmen Pannone, University of Cagliari, Italy

Offering valuable feedback is essential for educators to encourage student advancement and enrich learning. Effective feedback helps tackle misconceptions and narrow the distance between a student’s current level and desired goals. However, inadequately provided feedback can have adverse consequences and impede progress. Teacher feedback is critical for enhancing student accomplishments, but identifying the most efficient forms of guidance remains a challenge.

The Education Endowment Foundation published a report containing six recommendations for teachers to support students’ learning through feedback. These recommendations are the result of integrating empirical research findings and the expertise of academics and practitioners. Each recommendation starts with a vignette, illustrating common challenges faced by teachers, includes case studies of feedback practice to represent current approaches, and suggests techniques and ideas that might work based on the evidence and the panel’s expertise.

The first three recommendations act as the main guiding principles: (1) establish the foundation for effective feedback through high quality instruction and formative assessment; (2) provide well-timed feedback that emphasizes progress in learning; (3) create a plan for students to receive and apply feedback, including time and opportunities for utilization. Two recommendations suggest teachers carefully consider the delivery method, whether to provide a (4) written or (5) verbal feedback, according to purpose and time-efficiency. The last recommendation is about (6) developing a school policy that emphasizes and illustrates the principles of effective feedback.

The report can be highly valuable for teachers, offering them a guide on how to provide feedback in ways that are most likely to have a positive impact on students.