Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most
Published workshop paper
Tahreem Yasir; Wenbo Li; Sam Gilson; Sutapa Tithi; Xiaoyi Tian; Tiffany Barnes
Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026), pp. 819–840
Citation year: 2026.
Summary
Can LLM tutoring agents recognize when students need feedback? This study evaluates seven LLM feedback agents on 10,836 solution–feedback pairs in propositional logic. The agents handled optimal steps more reliably than incorrect or valid but suboptimal steps, and accurate diagnosis did not consistently produce actionable feedback.
What this study can tell us
The evaluation concerns feedback performance on the studied dialogue data. It does not directly measure how much students learned from the agents. Recommendations for combining approaches should be read as design implications.
Abstract
Effective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) but untested for LLM-based tutors. As LLMs are increasingly explored as conversational complements to ITS, evaluating their diagnostic precision is essential. We present a benchmark of seven LLM feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution-feedback pairs and three feedback conditions. Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most. These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits. Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness. Our findings suggest that LLMs are better suited for hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended scaffolding and dialogue.
The abstract as deposited with the publisher record.
Citation
@inproceedings{yasir2026confirming,
title = {{Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most}},
author = {Tahreem Yasir and Wenbo Li and Sam Gilson and Sutapa Tithi and Xiaoyi Tian and Tiffany Barnes},
year = {2026},
booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
pages = {819--840},
doi = {10.18653/v1/2026.bea-1.56},
url = {https://sgilson7.github.io/papers/yasir2026confirming/}
}