# Reliability of Claude Code as an Agentic AI Grading System for Handwritten University Exams

DevFeed: [Reliability of Claude Code as an Agentic AI Grading System for Handwritten University Exams](<https://devfeed.tech/articles/reliability-of-claude-code-as-an-agentic-ai-grading-system-for-handwritten-university-exams-128272.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.13217>)

Author: Rayed AlGhamdi

Published: 2026-11-11T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Artificial Intelligence](<https://devfeed.tech/sources/arxiv-artificial-intelligence.md>)

Topics: [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [Uncertainty quantification LLMs](<https://devfeed.tech/topics/uncertainty-quantification-llms.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [claude](<https://devfeed.tech/tags/claude.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [claude-opus](<https://devfeed.tech/tags/claude-opus.md>), [coding](<https://devfeed.tech/tags/coding.md>), [computer-graphics](<https://devfeed.tech/tags/computer-graphics.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [correlation](<https://devfeed.tech/tags/correlation.md>), [course](<https://devfeed.tech/tags/course.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-cy](<https://devfeed.tech/tags/cs-cy.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [design](<https://devfeed.tech/tags/design.md>), [diagnostic](<https://devfeed.tech/tags/diagnostic.md>), [domain](<https://devfeed.tech/tags/domain.md>), [drawing](<https://devfeed.tech/tags/drawing.md>), [error](<https://devfeed.tech/tags/error.md>), [exams](<https://devfeed.tech/tags/exams.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [university](<https://devfeed.tech/tags/university.md>)

## AI overview

In this study, Claude Code showed strong agreement with one course expert when grading handwritten computer graphics exams: total-score correlation was 0.97, with a mean absolute error of 0.85 marks and a small 0.68-mark positive bias. It also had high consistency across grading sessions (ICC = 0.96), and feedback was coded as accurate in 92.8% of diagnostic and 97.8% of confirmation cases. The study covered 60 consenting students, using separate samples for score agreement and test-retest analysis; because one expert supplied the benchmark, it measures agreement with that expert rather than inter-rater reliability. Drawing questions were an outlier, and recurring errors involved shallow keyword matching, ambiguity in corrected multiple-choice selections, and attention drift on diagrams. The authors conclude the system may assist with objective and short-form items under safeguards, but instructor judgment remains necessary for boundary cases and drawing items.

## Source excerpt

arXiv:2611.13217v1 Announce Type: new Abstract: This study examines the reliability of Claude Code, an agentic deployment of the Claude Opus 4.6 large language model, as a grading system for handwritten university exams. Four research questions addressed agreement with expert human grading on total scores (RQ1), agreement across five question types from multiple choice to evaluative judgement (RQ2), consistency across two independent grading sessions one week apart (RQ3), and the accuracy of AI-generated feedback (RQ4). Claude Code graded the 20-mark mid-term papers of 60 consenting students in a Saudi computer graphics course in a single agentic session, guided by three rubric files. Reliability was estimated on disjoint samples: 20 papers for RQ1, RQ2 and RQ4, and 15 papers for RQ3. Analyses used Pearson correlation, mean absolute error, Bland-Altman analysis, weighted Cohen kappa, intraclass correlation, and a four-category coding scheme separating diagnostic from confirmation feedback. The author, as the sole subject-matter expert for the course, provided the human benchmark, so the design assesses agreement with a domain-expert reference rather than inter-rater reliability. AI grading agreed strongly with human grading on totals (r = 0.97; MAE = 0.85 marks) with a small but significant AI-positive bias of 0.68 marks, showed excellent test-retest consistency (ICC = 0.96), and produced accurate feedback (92.8% diagnostic; 97.8% confirmation). Drawing items were the outlier on three of four measures. Three failure modes recurred: keyword matching without conceptual depth, selection-state ambiguity on corrected multiple-choice answers, and region-of-attention drift on hand-drawn diagrams. The findings suggest that AI grading at this capability level can assist instructors on objective and short-form items under structural safeguards but is not yet ready to replace instructor judgement at boundary cases or on drawing items.