ValiAssist: Aligning AI Programming Feedback with Educational Goals

Paper: ValiAssist: Scaling LLM-generated Programming Feedback Validation by Demonstration

Authors: Xiaohang Tang, Rehema Abulikemu, Peng Gao, Yan Chen

Executive Summary

Generative AI has shown potential in providing programming feedback at scale, yet a significant challenge exists in ensuring this feedback aligns with the specific educational goals of instructors. Often, instructors find it difficult to articulate precise validation criteria for the AI-generated feedback before they have had the opportunity to review actual examples and adjust any misalignments.

To address this issue, the researchers introduced ValiAssist, a system designed to enhance validation through demonstration. This system allows instructors to derive reusable validation logic from their revisions of AI-generated feedback. Essentially, instructors can refine the feedback criteria based on their insights and apply these adjustments to other student submissions.

In a comparative study involving 16 instructors, ValiAssist significantly improved the alignment of feedback with teaching objectives. Instructors using this system reported that 91% of the feedback generated was aligned with their pedagogical intent, compared to just 57% from the traditional approach. Additionally, ValiAssist more than doubled the accuracy of identifying and correcting problematic feedback criteria, achieving 77% accuracy against a mere 32% in the conventional setting.

The findings from this study indicate that validation by demonstration can effectively transform instructors' intuitive judgments about individual cases into clear and reusable criteria. This process not only enhances the quality of AI-generated feedback but also clarifies the conditions under which these judgments are applicable or not.

By using ValiAssist, educators could improve their ability to provide constructive feedback, ultimately benefiting students' learning experiences. The research suggests that tools like ValiAssist have the potential to streamline the integration of AI in educational contexts, making it easier for instructors to harness the technology in a way that supports their teaching goals.

Academic Abstract

Generative AI can create programming feedback at scale, but validating its alignment with instructors' pedagogical goals and strategies remains challenging. Instructors may struggle to fully specify validation criteria upfront, as these often emerge through inspecting concrete examples and revising misaligned feedback. We present ValiAssist, a validation-by-demonstration system that infers reusable validation logic from instructors' revisions of AI-generated feedback and allows instructors to inspect and refine that logic as it is applied across other student submissions. In a comparative study with 16 instructors, ValiAssist helped instructors create and refine validation criteria, producing feedback that better aligned with their pedagogical intent (91% vs. 57%) and more than doubling the accuracy of correcting problematic criteria (77% vs. 32%). Our findings suggest that validation by demonstration can help instructors turn judgments they know how to make on individual cases into explicit, reusable logic, while progressively clarifying where those judgments do and do not generalize.