Teacher Evaluations Should Be Apple to Apple

Published on February 26, 2009

Even if the tests used were designed perfectly, the current systems evaluating U.S. teachers based on the ranking of student test scores are fundamentally flawed, said Derek Neal, professor of economics at the College.

“They’re trying to solve a problem that is almost impossible to solve statistically,” Neal said during a Becker Brown Bag Series presentation at Harper Center on February 18. “More importantly than that, the question they’re asking is not even well-posed.”

Most existing performance pay systems uses statistical models that attempt to rank all teachers in a given state or district on a one dimensional performance scale based on the test score results of their students. However, Neal argues that such rankings should be made only among groups of teachers who are teaching students who began the year at similar levels of achievement. Neal argues that teaching honors math may be a different job than teaching remedial math, and therefore it makes no sense to look at student outcomes in an honors class versus a remedial class and ask which teacher performed better. Further, Neal explained that the complicated models often used to answer such hypothetical questions are leaning heavily on many modeling assumptions that are not tested and likely false.

Neal argued that performance pay systems should compare student outcomes among teachers who teach students with comparable levels of baseline achievement. Rewards should be given for doing a better job than other teachers who teach similar students. Further, Neal explained that such a system of relative performance pay can provide correct incentives for teachers without ever tackling the thorny issue of ranking all teachers on a common performance scale. Neal presented a system for implementing relative performance pay and noted that, because his system uses only the ordinal information contained in test results, it is less subject to political manipulation.

“The other thing is that the outcomes are invariant to the scale you use to report the test, so you can’t play with this ex post as a politician and change the scales and weightings to get the distribution and rewards the way you want them politically.”

However, Neal also pointed out that even the best performance pay system might not be as beneficial as systems that foster more competition in the market for education. No place in the developed world are public schools, even those incorporating the use of vouchers creating competition, allowed to compete for teachers with the same freedoms as in other professions, he said. For example, competition defines the types of review, performance evaluation, bonus pay, partnership rules, and other factors law firms use. Neal argued that systems that foster competition among schools for teachers may be the best mechanism for learning what the best ways are to recruit, screen, and pay teachers.

Neal’s presentation provided significant food for thought for first-year students Saboora Bhutta and Sara Kaufman, who are working on a project for Chicago Public Schools, they said. “Right now people are really looking at education incentive systems in the wrong way,” Kaufman said.

“My biggest takeaway was learning more about the different education incentive models that are out there and learning how to think about coming up with a new model,” Bhutta said.

                                                                                                                    — Phil Rockrohr