Audit: Colorado school accountability system ‘reasonable and appropriate’

Colorado’s school accountability system provides a “reasonable and appropriate basis” for measuring school performance, and most schools and districts receiving extra help and guidance from the state show some improvement, an audit of the system found.

Education Commissioner Katy Anthes said the results suggest the system doesn’t need an overhaul — so long as legislators still believe in the focus and intent of the accountability law. 

Supporters of test-based accountability echoed that belief and said they were pleased to see that schools targeted for intervention often showed improvement. Critics of the accountability system — who had hoped to use the audit to press for bigger changes — said it was simplistic and failed to answer big questions about how to best support all students.

The audit also found that Colorado schools that serve large percentages of students of color and students in poverty on average do worse on state tests. Even in districts with high overall performance, students from those subgroups had lower scores on state tests. 

The audit could not answer the question of whether those students receive lower-quality instruction or face greater barriers to learning due to factors outside the classroom, or both. Nor did the audit look at whether any bias is embedded in the tests that measure student performance.

“Whether you love the accountability system or hate the accountability system, this report will be a Rorschach test for people to reflect their own views,” said Van Schoales, senior policy director at the Keystone Policy Center and a longtime education advocate. His own take is that the system is “mostly fair and valid and doing what it’s designed to do.” 

Colorado’s school accountability system goes back to 2009. The system uses standardized test results, graduation rates, college enrollment, and other factors to rate schools and districts. The state provides extra help to those that receive either of the lowest two ratings. If they don’t improve after several years, the State Board of Education can order more significant changes and remove some decision-making power from superintendents and school boards. 

At the urging of school districts and community groups, Colorado lawmakers ordered the performance audit to determine if the accountability system is working as intended and if it’s helping or harming student performance. 

The Office of the State Auditor hired the Human Resources Research Organization, known as Humrro, to conduct the audit at a cost of $384,000. The audit released Monday is based on data from the Colorado Department of Education and surveys and interviews with school district officials, educators, parents, and interested groups.

The report outlined nine main findings:

• The state’s performance measures are “reasonable and appropriate” to gauge the performance of schools and districts. The audit found that schools that do worse on the accountability system also do worse on state tests. 
• Schools with higher proportions of Hispanic students, students who receive free or reduced-priced lunch, or students with disabilities had worse academic outcomes. Higher proportions of girls in the student body were associated with better academic outcomes.
• High-performing schools still have students from underrepresented groups not meeting expectations in achievement, or in growth. In 82% of the elementary schools that received the highest rating in 2018–19, students with disabilities as a group did not meet expectations in English language arts. Similarly, in 24% of elementary schools with the highest rating in 2018–19, the English learners group did not meet standards.
• The size of a school does matter. Schools with fewer students were more likely to have higher variability in average test scores and larger margins of error. The audit points out that the state tries to correct for this by using three-year averages for small samples.
• Participation rates don’t have a significant effect on school or performance ratings. 
• For high schools, those with more Advanced Placement course offerings or a higher percentage of career and technical education graduates tended to have better student academic outcomes. But simply providing more Advanced Placement courses or career and technical education opportunities may not increase academic achievement, academic growth, or postsecondary and workforce readiness.
• Colorado’s growth measures generally support the state’s objectives, set by law. For example, growth measures show where students are making progress even if they don’t yet meet grade-level expectations. However, the audit notes that all growth models have some inherent unreliability.
• Surveys, which the report says aren’t completely representative, indicate the accountability data is being used to “help inform decision making in support of students’ educational outcomes.” But the report notes that parents need the data to be more understandable and accessible. 
• From 2014-15 to 2018-19, schools with low ratings that used one of the state’s four improvement programs generally improved academics more or declined less than did low-rated schools that didn’t participate in state programs to improve. However, this happened less frequently with math scores. 

Education Commissioner Katy Anthes said she was pleased the audit found the system is working as intended under the original legislation and that schools that were targeted for intervention showed improvement. 

Going forward, she said the state needs to do more to ensure both parents and educators have access to data and understand it. 

She also said the state probably needs to focus more on math instruction. Colorado is in the midst of an ongoing push to improve reading instruction. Improvements in early literacy may lead to improvements in other subject areas, Anthes said, but the state may also need to put more attention on how math is taught. 

Anthes said it’s not simple to determine whether the system is measuring the impact of poverty and racism when it identifies more schools serving low-income students of color as low performing, or whether it’s measuring poor educational services.

“None of the questions are black and white,” she said. “You’re always measuring multiple things, and our system is designed to identify the students that need the most support and prioritize directing our limited resources to those students.” 

The report also pointed out that based on interviews, many people still perceive the accountability system to be punitive rather than supportive. Low ratings make it difficult for districts to hire and retain staff, and often students transfer to other schools, all factors that can make improvement more difficult.

“Rather than a rating system that is positive and focused on learning and helping schools and districts achieve high levels of student academic performance, one stakeholder referred to the performance ratings as being improperly interpreted and treated by families and communities as an Angie’s List or consumer review-type rating,” the report states. 

Schoales noted that most educators surveyed think parents don’t understand state test results or school ratings, and many parents told the auditor they never received a paper copy of their child’s results on state tests. Schools and the state could encourage better communication between teachers and parents about academic performance. 

Oliver Grenham, chief education officer for Westminster Public Schools, said he was disappointed with the report. 

“I found it to be very simplified,” Grenham said. “There wasn’t really any depth. The most pertinent question was, ‘has this made a difference for overall students over time?’ and that was not clearly articulated.”

Westminster is one of the districts that received multiple consecutive low ratings and faced state intervention, but has managed to improve and avoid state orders. Leaders in the district have continued to speak out against the accountability system.

“I was thankful we received funds from the state, but that was really on us to implement our strategies,” Grenham said. 

He said the auditors should have further examined the correlations they found between certain groups of students and low performance. He said he also would have liked to see more discussion on why schools that have more advanced course offerings have better outcomes. He said it may be related to funding and other resources available to schools.

“When you have a small portion of schools beating the odds, but the overall system remains the same, that’s an issue the state needs to address,” Grenham said.

“If anything this might set up another review that will go in depth,” Grenham said. “I don’t think this matched the legislative intent. More work needs to be done there.”

Jen Walmer, state director of Democrats for Education Reform, said the audit supports the idea that the accountability system is accurately identifying schools and districts that need more support and that improved instruction, more access to advanced classes, and other interventions do make a difference for students. 

While the authors of the audit were careful to describe correlation and not causation, Walmer said the finding that low-performing schools generally improved shows that change is possible. Going forward, Walmer said she hopes to see more focus on how to ensure improvement is more widespread and less focus on changing accountability measures.

“What can we do to make sure all kids have what they need, instead of whacking at the mirror because we don’t like what the mirror shows us?” she said.

And Walmer said she does see places where the system would benefit from changes. For example, higher-performing districts should be more accountable for how smaller student subgroups, such as students in poverty, perform. 

She would also like to see a higher bar for schools to get the highest rating — right now, schools can get the highest rating even if they don’t meet performance standards on all the available metrics — and more attention to whether students are making progress fast enough to eventually meet grade-level expectations.

A new on-track measurement that the state is phasing in over the next two years could help, she said.

Read the full report here.

Yesenia Robles is a reporter for Chalkbeat Colorado covering K-12 school districts and multilingual education. Contact Yesenia at [email protected].

Bureau Chief Erica Meltzer covers education policy and politics and oversees Chalkbeat Colorado’s education coverage. Contact Erica at [email protected].