You know the old fable about blind men touching different parts of an elephant to determine its true nature. The guy who feels just the ear draws one conclusion, the fellow who holds the trunk draws another, and so on. That’s my reaction to the latest article touting the success of the District of Columbia’s teacher controversial evaluation system, IMPACT, appearing in Education Next, the journal published by Harvard and Stanford whose contents tend to skew right. It’s a clear case of limited examination of the available information. Not to say that the authors are blind, just that they haven’t examined more of the elephant.
The authors, Thomas Dee of Stanford University and James Wyckoff of the University of Virginia, generally praise IMPACT, asserting that it has driven out poor teachers, rewarded successful ones, and helped the DC schools improve.
However, Dee and Wyckoff gloss over and withhold critical information, material that their readers really ought to be made aware of. Just consider their bold assertion that DCPS is “the fastest-improving large urban school system in the United States as measured by the National Assessment of Educational Progress.” Maybe so, but why don’t the authors disaggregate the data and dig a little deeper? Probably because the results contradict their thesis.
Mary Levy and I have disaggregated the data. Here’s one small bite: “Despite small overall increases, minority and low-income scores lag far behind the NAEP’s big-city average, and the already huge achievement gaps have actually widened. From 2007 to 2015, the NAEP reading scores of low-income eighth graders increased just 1 point, from 232 to 233, while scores of non-low-income students (called “others” in NAEP-speak) climbed 31 points, from 250 to 281. Over that same time period, the percentage of low-income students scoring at the “proficient” level remained at an embarrassingly low 8 percent, while proficiency among “others” climbed from 22 percent to 53 percent. An analysis of the data by race between 2007 and 2015 is also discouraging: black proficiency increased 3 points, from 8 percent to 11 percent, while Hispanic proficiency actually declined, from 18 percent to 17 percent. In 2007 the white student population was not large enough to be reported, but in 2015 white proficiency was at 75 percent.”
You got that, right? The so-called Achievement Gap is wider today than when IMPACT was introduced.
Dee and Wyckoff lavish praise on IMPACT’s approach, noting that it has changed and evolved. What they do not tell their readers is the enormous cost of IMPACT. Mary Levy and I ran the numbers:
“Under Rhee and Henderson, spending on non-teaching personnel has swollen dramatically. According to the latest statistics from Census Bureau fiscal reports, DCPS central office spending in 2015 was 9.5 percent of total current expenditures, compared to 1 percent 4 or less in surrounding districts. Today DCPS central offices have one employee for every sixty-four students, a striking change over the pre-Rhee/Henderson era ratio of one to 113 students. Those central office dollars could have been used to provide wraparound social services for children, services that would have allowed teachers to be more effective.
Many of these highly paid non-teachers spend their days watching over teachers in scheduled and unscheduled classroom observations, generally lasting about thirty minutes—not even an entire class meeting. Why so many of these teacher watchers? Because those who subscribe to top-down management do not trust teachers.”
Those excerpts are from Mary’s and my piece in the forthcoming issue of The Washington Monthly, which just published Thomas Toch’s paean to DCPS, an equally misleading essay. Perhaps it’s not a coincidence that Education Next also printed Toch’s piece. It’s all part of a fairly well-designed campaign to convince the world that the top-down, data-driven, test-and-punish approach to fixing schools is just what the doctor ordered. It’s the reform that Democrats for Education Reform and most Republicans favor, despite strong evidence that it does not work.
All three writers–Toch, Wyckoff, and Dee–whitewash or completely ignore the widespread cheating by adults, even though it occurred in more than half of the DC schools. I wrote about that in a recent post and hope you will revisit that sordid story about the flood of wrong answers being changed to right. Threats from the Chancellor to “get the scores up or get out,” erasure parties, and more
I’m sure you know that the fable about the blind men and the elephant is also a joke. In that version, one more blind man is walking behind the elephant, picking up its poop, examining it, and then drawing his own conclusions about the true nature of an elephant. It seems to me that’s where both of these articles, Toch’s and the Dee/Wyckoff piece, belong.
(PS: MY NEW BOOK, ADDICTED TO REFORM: A 12-STEP PROGRAM TO RESCUE PUBLIC EDUCATION, WILL BE PUBLISHED BY THE NEW PRESS IN A FEW DAYS. PLEASE PICK UP A COPY AT YOUR LOCAL BOOKSELLER. THE ELECTRONIC VERSION, AVAILABLE ON AMAZON, INCLUDES HUNDREDS OF MY VIDEOS.)
7 thoughts on “Touching the Elephant”
Yes, they are mostly touching the borders between effectiveness grades. Of course, those on the bubble make sure that the numbers mostly come out good nuf to move from one category to the next. Why would they assume however, that those metrics, in those situations, tell us anything about increased learning. People do their jobs – making the numbers fit in the categories they need to fit into.
At the risk of angering some of my friends, I believe an eval system should mostly remove teachers who do their job badly – based on observing their bad teaching. They wasted a great opportunity for getting rid of bad teachers. Most teachers would have supported them on that. But we and most administrators will always “monkey wrench” an invalid and unreliable system for guesti-mating “effectiveness.”
And that get’s back to your point. The opportunity costs of IMPACT are enormous
Brava, Mary Levy. What endurance!!
But, John Merrow’s summary could have been stronger on the scores with a conclusion explaining the seeming discrepancy: DCPS average scores have improved only because the home resources and increases enrollment of students of highly educated white parents has grown. More than 90% of those students attend and are mostly in the majority at less than 10 of DC’s 150 public and public charter schools.
The scores of students of color, most of them in schools with almost no advantaged peers, continue to stagnate despite a dozen years of intense oversight by overseers beginning with Michelle Rhee. Nor is there evidence that their DCPS schools have improved in other ways.
Harry, Those two excerpts were just a sample, a taste of what’s to come in the next issue of The Washington Monthly. You won’t be disappointed, I promise. It’s all there
[…] I want to suggest to Eliza Shapiro that she read my last two books: The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education (rev., 2016); and Reign of Error: The Hoax of the Privatization Movement and the Danger to America’s Public Schools. She should read Mercedes Schneider on John White in Louisiana and John Merrow on the subject of the D.C. “miracle” that wasn’t. (John Merrow and Mary Levy will have an article in the next issue of the Washington Monthly that takes apart the D.C. “miracle.” but in the meanwhile Shapiro can read this post that Merrow wrote: https://themerrowreport.com/2017/08/08/touching-the-elephant/comment-page-1/ […]
John and Mary, thanks for being faithful and careful stewards of data and for providing numbers that DCPS has either failed or intentionally decided not to share with those of us with a stake in this game. Thanks for continuing to reveal this obsession with testing for what it is and its consequences for what they are. As most of the rest of the country has come to its senses, DCPS continues to pursue this failing strategy. Hopefully at some point the combination of these numbers and frustration from teachers, students, and parents will turn that around.
John, Speaking of elephants, is the one in the room being overlooked? It seems to me that improved results in educational performance by low income students (primarily black and Hispanic in the statistics you quote) does not seem to be dependent on money spent or curriculum delivery methods. Is it possible that part of the cause (possibly a significant part) for the disappointing improvement results among low income students relates to nome environment versus school environment? Obviously, kids spend a lot more time away from school than in it. To what extent is their desire and ability to learn affected by their experiences during the away from school periods? If the scores for the non-low income students are improving following changes in DCPS, those changes seem to be benefiting at least a portion of the school population. Did the research that you and Mary Levy did indicate why the outcomes for the two student groups (low income and non-low income) are so different? If not, a “deeper dive” by the two of you may benefit all of us. Knowing about a situation is good; knowing why that situation developed and what possibly can be done to improve it (especially when it involves kids who really need help) is even better. Improving our country’s educational outcomes has to be a national priority, regardless of who is in office. Thanks for continuing to focus on it and bringing it to our attention. Regards, Marty
Sent from my iPhone
[…] Reblogged from THE MERROW REPORT […]