Evaluating open-ended high-stakes exams with LLMs: Alignment between ChatGPT-4o and human grading in high- and low-resource languages

Large language models (LLMs) are increasingly used for grading written responses, yet large-scale benchmarks against human expert evaluation remain scarce, especially across languages with different resource levels. This study evaluates ChatGPT-4o using a reranked retrieval-augmented generation (RAG) framework to grade Finland’s national high-stakes matriculation examination from 1,016 students’ open-ended responses. We examine GPT-4o’s alignment with official grades, recognition of grading-relevant keywords, and the effect of translating responses from a low-resource language (Finnish) into a high-resource language (English). Using descriptive statistics and correlation analyses, results show that GPT-4o’s grades on a 0–15 scale closely matched human evaluations: 75% of scores were within ±2 points of official grades, with only 3% severe outliers. Translating responses into English improved alignment to 85%. While the model generally identified relevant keywords effectively, occasional misinterpretations of contextual usage reduced grading reliability in a few cases. Overall, the findings demonstrate both the promise and current limitations of LLM-based assessment. There is a substantial potential to use LLMs as a supplementary grading tools, particularly in high-resource languages, but they do not yet match the consistency or interpretative depth of human expert evaluators. The study underlines the need for human oversight, rigorous validation, and careful consideration of language effects when deploying LLMs in high-stakes educational assessment.