Cash upfront the way to get teachers to rack up better student test scores, study finds
BY ROSALIND ROSSI Education Reporter email@example.com August 24, 2012 7:20PM
Chicago Heights District 170 office, 30 W. 16th Street Friday, August 24, 2012. | John H. White~Sun-Times
Updated: September 27, 2012 11:12AM
What happens to student achievement if you start the school year by offering teachers end-of-year bonuses for improved student test scores?
But give teachers cash up front on the condition they will lose it if test scores don’t show gains?
Lots of improvement.
That’s what researchers say a study of 150 Chicago Heights teachers found in what some are touting as the first U.S. field experiment to indicate merit pay for teachers can work — if timed properly.
For one school year, the research project by heavyweight economists from the University of Chicago and Harvard University — including the author of the best-seller Freakonomics — turned nine elementary schools in Chicago Heights School District 170 into a teacher testing ground of a behavioral phenomenon called “loss aversion.’’
“It’s a deeply ingrained behavioral trait. .. that all human beings have — this underlying phenomenon that ‘I really, really dislike losses, and I will do all I can to avoid losing something,’ ‘’ said one author, John List, chairman of the University of Chicago’s department of economics.
Chicago Heights teachers given $4,000 upfront on the condition they could lose it — or double it with the right results – produced 2 ½ to 3 ½ times the gains in student math scores as teachers offered up to $8,000 in end-of-year bonuses, the study indicated. Similar findings were not observed in reading, where researchers encountered more technical glitches.
List conceded his sample of about 150 teachers did not establish an “overarching truth” but “I think there needs to be more studies to see how robust my results are.”
“I’d love for all of Chicago Public Schools to replicate my results. That would be great,’’ said List, identified by Forbes Magazine as one of top seven behavioral economists in the world in 2010.
The study emerges amid growing national interest in teacher merit pay and other incentives tied to improved student performance. Chicago Public Schools and Chicago Teachers Union officials are battling over those topics right now in teacher contract talks, with the CTU staunchly opposed to merit pay.
An ‘unhappy’ union
Meanwhile, behind the scenes, in Chicago Heights, drama is bubbling over the study.
The Chicago Heights teachers union is contending Supt. Tom Amadio told them the district would not be identified, said a spokesman for the group’s parent union, the Illinois Federation of Teachers.
“They agreed to let [researchers] collect some data,’’ said IFT spokesman Dave Comerford. “Part of the reason they agreed is that the district and union wouldn’t be identified. . . . They didn’t want to be perceived as supporters of merit pay.’’
Union leaders and members are “unhappy” and feel “somewhat misled,’’ Comerford said.
Plus, Comerford said, Chicago Heights teachers have concerns about the tests that were used and how the study was conducted. They “question its validity,’’ he said.
For his part, Supt. Amadio says he initially saw the study as a “hot potato’’ and “stayed out of it.’’ But “now the results are out and it’s obviously going to become a subject of discussion.’’
The union brought the offer of the $1 million research project to Chicago Heights teachers and 150 of 160 voluntarily agreed to participate, List said. Teachers were divided into roughly three groups of 50 in the fall of 2010, with one given money up front, a second promised end-of-the-year bonuses, and a third “control” group that was offered and received nothing extra.
The study was conducted by List; U of C colleague and Freakonomics author Steven Levitt; Harvard economist and MacArthur genius grant winner Roland Fryer Jr., and University of California-San Diego economist Sally Sadoff. It was funded by the Kenneth and Anne Griffin Foundation, which also has bankrolled other incentive programs in Chicago Heights involving students.
‘I spent that money’
The field experiment also reflects the expanding reach of economists into the world of education. A chapter of Levitt’s Freakonomics, for example, delves into student tests by describing how economic algorithms detected cheating in 5 percent of Chicago Public School classrooms in the 1990s.
One Chicago Heights field experiment volunteer said she initially did not realize that economists — rather than education professors — were conducing the study, although Sadoff said she thought the U of C economics department letterhead was on teacher consent forms.
“The whole study being conducted by the economics department at the University of Chicago has caused a lot of ill feelings in our building,’’ said the teacher, who asked for anonymity. “Why wouldn’t I feel so offended if it were the college of education? . . .
“I feel the college of ed would say, ‘Tell us what you did. Share your knowledge. How did you get your scores up so much, so we can all become better teachers?’ But that’s not the purpose of this study.’’
The teacher said she is against merit pay because “it’s going to have teachers doing anything they can. If that means cheating, some will.’’
Some teachers were clearly worried about losing their upfront bonuses, the teacher said.
“I know a bunch of people who said, ‘I spent that money. My kids better do good because I don’t have that money any more.’ So could that have played a part in why their kids did better? I don’t know.’’
Sadoff said proctors were present during testing to guard against cheating. And the gains on the diagnostic test used in the study, called ThinkLink, also were reflected in higher-stakes state achievement tests that ThinkLink was based on, indicating “the same kind of learning was going on,’’ Sadoff said.
‘Widgets’ vs ‘Adolescents’
The study’s authors found traditional teacher bonus offers in Chicago Heights produced the same lack of impact as seen in similar teacher pay schemes in New York City and Nashville. The only prior field study of a “loss aversion” payment plan, they said, occurred in Nanjing, China, where it improved productivity among factory workers who made and inspected DVD players and other consumer electronics.
Barnett Berry, president of the Center for Teaching Quality, said the study seems to suggest that districts pay “teachers working with children and adolescents’’ in the same way “Chinese factory workers” were paid for “producing widgets.’’
“I think this suggests a dire lack of understanding of the complexities of teaching,’’ Berry said.
Berry raised technical questions about the size of the sample, the gains methodology and the number of ThinkLink questions used to draw conclusions about gains.
Moreover, Berry said, he found the study “offensive” because “economists would never suggest that economists be paid this way. The economists make predictions about the economy. If their predictions don’t turn out, should the economists return their consultant fees?”’
Chicago Teachers Union President Karen Lewis said the study falsely assumes money motivates teachers to teach better.
“Merit pay assumes we are not working hard enough and we have hidden something from the children,’’ said Lewis, who also serves as vice president of the Chicago Heights’ parent union, the IFT. “Like we have a teacher box and we only pulled six things out of our teacher box but if you pay us more we’ll go in the box and pull out more. . . .
“The problem with economists is they don’t understand what intrinsically motivates teachers to do this work — and it is not money,” Lewis said. “The overwhelming majority of teachers are motivated by seeing kids learn. ... There is nothing better than seeing that lightbulb turn on.’’
Executives of another teacher quality group called the new study “dynamite” and the gains it found “monumental.’’
Kate Walsh and Laura Johnson of the National Council on Teacher Quality write on the group’s blog that they have many questions about the study but “the gains were clearly so big that other work is critical.’’
Also showing interest was Chester Finn Jr., president of the Thomas B. Fordham Institute and senior fellow at Stanford University’s Hoover Institution. He said the “up-front-but-you-could-lose-it’’ approach is “both plausible and promising.’’
However, Finn foresaw “all manner of logistical problems’’ trying to implement the pay scheme, including recruiting collection agencies to chase down debt-bound teachers.
“Any number of other approaches might also work, which is why I suggest that trying various versions [of merit or incentive pay] is desirable. [It’s] way premature to lock in on any ‘one best’ version,’’ Finn said.
‘Great’ but ‘now what?’
List said the “money-up-front’’ approach could work if teachers who are paid over 12 months agree to have payroll deductions during June, July and August if they failed to produce gains. But the scheme should be combined with some kind of minimum raise, so “up-front’’ payments would constitute bonuses, he said.
As for the Chicago Heights superintendent, Amadio said “The University of Chicago is very well respected. Based on the data they have and these are the results, I think it’s telling — very telling.”
However, Amadio said, broaching the topic would be easier in flusher times. Teachers would not agree to take a pay cut, he predicted, so any upfront payment would have to be crafted as a bonus.
“It’s a great study and everything — but now what?” Amadio said. “We don’t have the money to do it, nor do I think the teachers would agree to do it.’’
Contributing: Art Golab