Texas' annual reading test adjusted difficulty yearly, masking improvement

Summary

Texas children’s performance on an annual reading test was basically flat from 2012 to 2021, even as the state spent billions of additional dollars on K-12 education. I recently did a peer-reviewed deep dive into the test design documentation to figure out why the reported results weren’t showing improvement. I found the flat scores were at least in part by design. According to policies buried in the documentation, the agency administering the tests adjusted their difficulty level every year. As a result, roughly the same share of students failed the test over that decade regardless of how objectively better they performed relative to previous years. From 2008 to 2014, I was a bilingual teacher in Texas. Most of my students’ families hailed from Mexico and Central America and were learning English as a new language. I loved seeing my students’ progress. Yet, no matter how much they learned, many failed the end-of-year tests in reading, writing and math. My hunch was that these tests were unfair, but I could not explain why. This, among other things, prompted me to pursue a Ph.D. in education to better understand large-scale educational assessment. Ten years later, in 2024, I completed a detailed exploration of Texas’s exam, currently known as the State of Texas Assessments of Academic Readiness, or STAAR. I found an unexpected trend: The share of students who correctly answered each test question was extraordinarily steady across years. Where we would expect to see fluctuation from year to year, performance instead appears artificially flat. The STAAR’s technical documents reveal that the test is designed much like a norm-referenced test – that is, assessing students relative to their peers, rather than if they meet a fixed standard. In other words, a norm-referenced test cannot tell us if students meet key, fixed criteria or grade-level standards set by the state. In addition, norm-referenced tests are designed so that a certain share of students always fail, becau...