GPT-5o-mini hallucinates medical residency applicant grades

Summary

Dear Applicants, Programs, and Medical Schools,Since the opening of the residency recruitment season, we’ve heard your feedback on the Cortex Core Clerkship Transcript Normalization feature. As a physician-founded organization born out of our lived experience as applicants and program leaders, we understand the many emotions tied to the start of the season. We believe it is our duty and responsibility to communicate openly with all stakeholders about how our tools are used and how they may impact application review.What We’ve HeardOver the past week, while the grades on transcripts and corresponding application documents have been accurate and fully accessible, we’ve received a small number of reports from the community about inaccuracies in a subset of automation-extracted grades displayed within Cortex.In each case:Once confirmed, we corrected the extracted grade immediately.Where the extracted grade was accurate, we provided feedback and guidance to the reporting program or school about its interpretation and the extraction methodology.Because Cortex is now offered complimentary to all ERAS-participating programs through the AAMC-Thalamus collaboration, many programs are using it for the first time this season. We recognize the novelty of this feature may also be contributing to questions and uncertainty.How the Tool WorksCortex makes accessible all application data including the medical school transcript, Medical Student Performance Evaluation (MSPE), and all other documents in their raw, original, and unaltered format. Cortex uses automated extraction (optical character recognition (OCR) and natural language processing (NLP)) to parse clerkship grades from medical school transcripts.The software then generates reference reports including grades, percentiles, and graphical distributions allowing reviewers a structured way to contextualize grades for efficiency and comparison within and across medical schools.First released in 2020, the tool is updated each recru...