Before you read further, please picture NEA President Lily Garcia, Secretary of Education Arne Duncan, AFT President Randi Weingarten, and U.S Senator Lamar Alexander sitting around my kitchen table.
What are they doing? They are playing my new socially valuable parlor game, “Multiple Measures.”©
What’s “Multiple Measures”©? Well, it’s the phrase everybody uses when they’re talking about the most reliable way of evaluating schools, teachers and students.’ After all, anybody who knows anything about education understands that a single measure (i.e., a score on a standardized test) cannot accurately capture the complexity of the enterprise.
Unfortunately, the conversation usually stops with that overused phrase, “multiple measures,” and we continue to rely heavily on one measure, scores on standardized tests.
Here’s how to get beyond the talk. Take a pile of plain index cards. Then, invite a few people from differing political camps who are solution-oriented, or at least willing to engage in dialogue, to have a cup of coffee and play “Multiple Measures.”©
Let me demonstrate at my kitchen table with Secretary Duncan, Senator Alexander and Presidents Garcia and Weingarten. First I distribute pencils and index cards to each of them, and then I explain the challenge: “What exactly do you mean when you say “multiple measures”? Your task is to write specific measures on the back of index cards. Each of you must identify three measures, then put the cards, face down, in the middle of the table.”
Mix up the cards, then turn them over. Discuss….
With that awesome foursome, the specific measures might include test scores, teacher attendance, student attendance, participation in Advanced Placement classes,extent of project-based learning, years required to gain tenure, teacher turnover, and who knows what else. A few measures would most likely appear on more than one index card.
Now repeat the process. Twelve more cards. Turn over. Discuss….
Keep playing until you have reached agreement on at least five specific measures.
The principle behind the game is simple: It’s time for us to measure what we value, instead of our current practice of valuing what’s easy to measure.
The goals of my game are threefold: general agreement on multiple ways of assessing schooling; a lowered temperature; and a commitment to work together for the benefit of children and society.
I think everyone reading this should arrange a game of “Multiple Measures”© in your community. Please take pains to include people from the left, right and center, men and women of varying ages, races and occupations. {{1}}
If you’re wondering, “Where can I buy this wonderful game?” I have good news: “Multiple Measures”© is now commercially available…..from me. The basic set, which includes a dozen index cards and four unsharpened pencils, sells for $69.95. The deluxe boxed set, available for $99.95, includes two dozen index cards, six pre-sharpened pencils and a signed photograph of my kitchen table. {{2}}
Multiple games of “Multiple Measures”© in local communities might break the logjam and chip away at the walls {{3}} we’ve built up in our polarized society, and I am excited about that.
I am also excited about making a bundle of dough. I’ve worked in non-profit enterprises for my entire career, and “Multiple Measures”© is my first effort in pure, naked capitalism. Frankly, I hope to make enough money {{4}} to allow me to retire comfortably. I’ve stockpiled multiple cases of index cards and #2 pencils, so I’m ready for the orders to pour in.
But my life-long habits of generosity and public service are hard to break, and so I have decided to donate an unspecified portion {{5}} of the proceeds to the “Save Nebraska’s Whales” fund.
Can’t wait to hear your reports about playing “Multiple Measures”©….and to cash your checks.
—-
[[1]]1. Extensive field studies indicate that four is the ideal number, but you may try other combinations. A basic rule that cannot be tinkered with, however, is “No Shouting!”[[1]]
[[2]]2. The first 100 customers will receive a set of “Multiple Measures”© coasters to protect your kitchen table from coffee stains. Hurry. Act now![[2]]
[[3]]3. This is not an impossible dream. Arne Duncan and Lily Garcia have breakfast together once a month, and perhaps they will play “Multiple Measures”© next time.[[3]]
[[4]]4. I’m not about to cut into my profits by paying for shipping. That’s on you, so add $7.50 to the tab.[[4]]
[[5]]5. When I decide on the portion,I will let you know.[[5]]
The Merrow Report 
John, I think you’re on the right track to push for specificity around those multiple measures, but I would add that there’s a difference between agreeing what to look for and what it means when we see it. As long as policy makers continue to see teachers as a management problem rather than a resource, many of those multiple measures are subject to misinterpretation or misuse. For example, regarding teacher absences – if you find a certain school or district has a higher rate, then certainly we want to address that. But don’t assume the absenteeism is a contributing cause rather than a symptom of problems in the school. In some cases, it may be appropriate to adjust policies around the use of leave, but in other cases, it may be better to tackle the causes. I can tell you that in my teaching experiences (at high school level), it’s almost more work to be absent than present when you take into account lesson plan adjustments, grading, catching up, etc. If it’s ever a close call, I err on the side of going to school. But if conditions or operations at school were worse, decisions might turn out differently.
LikeLike
I agree, which is why discussion is an essential element in my ‘game.’ And the point of measuring has to be looking for meaning and meaningful ways of improving. Cannot be ‘gotcha.’
LikeLike
John,
I think the problem is that we are stuck in a “measure to manage” paradigm.
The foremost assumption in this model of how the world works is that a small group of powerful people can sit somewhere and decide on the “right” things to measure, and then the rest of us peons are “held accountable” for the targets that are set in these dimensions. This is a dead end.
When we seek a different paradigm, it will not be found by identifying the magical proportion of test scores, graduation rates and whatever else we can figure out how to measure.
A new paradigm will honor the fact that a great many of the things we value most are not easily captured in any sort of standardized set of indicators. A new paradigm will seek to engage the school communities as the real locus of accountability.
We are likely to find that qualities like perseverance, compassion, collaboration, and creativity are not easily measured, but can be demonstrated. We can look to schools like Mission Hill in Boston to see how this is done.
So long as we are stuck in this top down measure to manage mindset we will be chasing the illusive and ultimately non-existent perfect set of measurements. Let’s turn the page.
LikeLike
jOHN, HERE IS AN OPED i PUBISHED IN PROJO IN 2004:
Fall 2004
[HED:Standardized test are too narrow and unreliable to be the sole measure in determining a youngster’s future.]
Ronald A. Wolk is the founding editor of Education Week and Teacher Magazine. He is now retired and living in Rhode Island.
Multiple Measures
One of the worst things about President Bush’s education law, No Child Left Behind, is that it will intensify America’s obsession with standardized tests. The law requires states to conduct annual testing in grades 3-8—in addition to existing testing in other grades.
The central purpose of schooling is academic learning, and the testing teachers do in class is important in assessing students’ progress. By contrast, the unwarranted emphasis we put on costly and time-consuming standardized testing distorts the process of teaching and learning in schools. Moreover, standardized tests aren’t very good at measuring how much kids really learn. Even worse, the tests don’t measure the values and attitudes that parents repeatedly say they want their children to get from school: diligence, honesty, tolerance, fairness, and compassion. Parents want their kids to be well rounded, to develop the skills they need to continue learning on their own; to become good citizens, productive workers, and fulfilled human beings. That’s what all of us should want.
Despite warnings from testing experts and educators that important decisions should not be based on a single measure, 18 states now require high school students to pass an exit exam to graduate and a half dozen more plan to do so. Graduating from high school is far too important to be decided on the basis of a standardized test score. Without a diploma, young people are much more likely to be underemployed or unemployed.
When I suggested to a friend who has worked in education at both the state and district level that policymakers and educators should develop an assessment system of multiple measures to evaluate not only academic skills but other important attributes, he responded that it would be too complicated and subjective to be practical. “What would you include in the measures?” he challenged.
It’s not really that difficult.
Imagine an assessment system that requires a student to earn 80 points out of a possible 135 to graduate. A student earning 100 to 115 would graduate with honors, and those with more than 115 would receive high honors (and perhaps a state scholarship to college).
Students could earn the points as follows:
* 40 points for passing the mandated state or district exit test.
* 0-20 points for the grade average of all courses. An A average earns 20 points; a B earns 15; a C earns 10; a D earns 5.
* 0-25 points for participation in class and personal work (such as exhibitions, writing, performances). In written evaluations, two teachers would each rate the student on a scale of 0 to 25 points, and the student would be awarded the average.
* 10 points if a student has fewer than 5 unexcused absences.
* 0-25 points for participating and excelling in extracurricular activities. Students receive 15 points for participating in two or more activities, and they can earn up to 5 additional points on the recommendation of the activity’s advisor or coach, and another 5 points for an award received in the activity, like an athletic letter or a writing prize.
* 0-15 points for volunteer work in the community. The number of points earned would be based on the recommendation of the adult supervising the activity.
Students cannot earn enough points to graduate just by passing the mandated exit test; they must show achievement on other important measures. And students who do not pass the exit test can still earn a diploma, but only if they excel on all of the other measures and earn 80 of the remaining 95 points. If a student who failed the exit test earned the maximum for extracurricular activities, volunteer work, and attendance, he or she would still need to earn 30 points (for academic achievement) in either grades or teacher evaluations. The numbers and the categories could undoubtedly be modified.
Skeptics will complain that some of the evaluations are subjective, but that is as it should be. Few of us take standardized tests after we leave school. Instead, we are judged by others on what we produce and how we behave, and assessment of our performance in real life is usually more subjective than objective.
In the present system where test scores are the only measure, educators have little or no say. But teachers, advisors, and coaches are evaluating students continuously, and their assessments should count for something in such an important decision as whether a student graduates or gets promoted.
The reason most often given for high stakes is to motivate students. Anyone who watches kids perform in the student orchestra, or debate, or athletics knows that students are motivated to work hard when they care. They work hard because they are engaged and have something of value to show for their effort. Academic and non-academic achievement are both important in helping students build confidence and become competent and responsible young adults. It’s time we stopped judging students on a single—and badly flawed—measure of only academic achievement.
LikeLike
No surprise that you are and were ahead of the curve. What was the response? Did any educators take this and attempt to run with it?
LikeLike
I got half a dozen queries from around the country, two from school boards the others from supes. Not aware of any action. I didn’t intend to suggest that I was ahead of you, but rather to suggest “refinements” for the game.
LikeLike
I know you did not suggest it. I am comfortable acknowledging it! You have been a role model in lots of ways, my friend.
LikeLike
With this system, up to 40 points are awarded for extracurricular activities and volunteering. Middle and upper-middle class students would benefit most from such a system. As a mom with multiple children on the special needs continuum, even though we are middle class, my kids wouldn’t fare well. We spend after school hours going to speech therapy, mental health counseling, psychiatric appointments to check medication doses, and tutoring just so my kids can remain proficient in a subject they are struggling with. This doesn’t leave time for volunteering or fun extracurriculars after school. Inner city kids without volunteer opportunities, affordable extracurriculars or adult supervision after school would be out of luck. Multiple measures that look at the work students produce in school – including portfolios of work – sound fine. But grade inflation is rampant; social promotion too. Some kids don’t participate in class because they are shy. Creating a multiple measures game that is fair isn’t easy! High school exit exams, where students are expected to demonstrate basic knowledge, perhaps even 8th grade knowledge, at least seem to show that a student leaving the system has grasped the topics we as a society agree our citizens should know.
LikeLike
I agree entirely that “The central purpose of schooling is academic learning, and the testing teachers do in class is important in assessing students’ progress.” In Finland almost all testing is teacher made. They give almost no standardized tests and their students do very well. The main reason, in my opinion, is that they avoid superficial teaching and learning. In the USA the emphasis is on the Scantron God. These tests are academic junk food , for the most part if they do not include essays and open ended portions like AP tests. It is too easy to guess and cheat on multiple choice tests (for students, teachers AND administrators) and test like this encourage superficial learning. And as you mention the tests do nothing to support values and real interest in culture, academics and learning.
But I do want to say something in favor of high school exit exams. I know the CAHSEE well as I have tutored students who have failed it for years after school as a volunteer or in my own CAHSEE class. These tests do NOT require a specific knowledge of any specific book or curriculum but are a basic measure of literacy and numeracy. One five paragraph essay is required (though a student can pass with a partially complete 4 paragraph essay). Student can pass marginally at 350 or achieve “proficient” at grade level which is 380.. There is no time limit to take the test and students can take it multiple times. For ESL/ELD students the time element is the killer on tests like AP tests where they are required to answer 80 questions in 55 minutes or so. That is not a problem for the CAHSEE. And students are HIGHLY motivated to pass these tests. What the CAHSEE has done is make it harder for “D-Dare Devils” to graduate. Many students simply have no ambition except to roll up a lot of D- grades. However, the CAHSEE wakes many up to the fact they can’t be such goof offs anymore. The CAHSEE is a fair test. English learners many not score proficient at grade level but they can pass with a marginal pass. I see many English learners get perfect scores (100) on Reading Comprehension and Literary analysis but fail by a few points to be proficient due to grammar and spelling problems. Administering standardized tests to students that do not count on their transcripts or towards their graduation is a complete waste of time and money in my view. Tests should help us find out where a student’s weak points are and students who do well should be rewarded. Students who finish with all their credits but who fail to pass the CAHSEE can take remedial classes at the Adult School. They have five years to pass the test and have it still count for their original HS diploma. Every year we have students who need to pass one CAHSEE or the other and pass them a few months or years after their class’s graduation. That is more than fair. Students who don’t have both their CAHSEE’s at the graduation date can participate in graduation and get a certificate of completion. With a little effort they can get their diploma. Once again if this test were as hard as the new GED or timed like an AP test I would be totally against it. But because students can take it multiple times, in the summer and even after the graduation date I think it is fair.
LikeLike
elusive!
LikeLike
John,
Anybody who uses the word paradigm should be banned from the game.
LikeLike
Only if they shout it… 🙂
LikeLike
Well said sir , the dialog opportunity is a wonderful reflection .
What is preventing that invitation to join you around your kitchen table ? Is it ego , “I” handy caps ?
You are doing great work keep it coming .
LikeLike
John, I think that if this post gets any traction, it will be because you used humor; also because your example named four people who actually could actually play your game and produce a meaningful outcome. In fact, if I were inclined to look beyond the words on the page I might even suggest that you effectively invited those four in way to which they might actually respond.
As you said, it’s no surprise that Ron Wolk anticipated the idea more than a decade ago. NCLB created the need and Ron was (and still is) one of the guys who leaps in to meet those sorts of needs. That you have now advanced an enhanced version of Ron’s idea speaks to the resilience of the idea as well as the stubbornness of the situation it addresses.
For America to get a handle on education has proven to be a very long process. For sure it goes back to “A Nation at Risk,” and probably beyond that. Since “A Nation at Risk” we have developed a host of excellent ideas about teaching and learning. Those ideas would fill a whole lot of index cards, and those cards could be boiled down to measures of effectiveness. What we haven’t done is persuade the competing parties to work together to reach consensus.
It’s hard to know when you’re approaching the resolution of a very long process, but your idea makes me thing that we may be doing just that. I think that over the last quarter century we have a developed a solid understanding of what works in education. What we have not done is agree, or even to agree to disagree on the purposes of education. It’s no wonder that we can’t agree on how to measure effectiveness in reaching the goals we haven’t agreed on.
I think that if competing interests really sat down to play your multiple measures game, the focus would shift to goals very quickly. I also think that as long as the participants were willing to move forward on the basis of agreeing to disagree, they’d come up with a good set of measures.
Note that I am not suggesting that we can aspire to a set of national goals and measures. The Core-40 experience makes that clear, I think. If we expect states to support the effort, they have to be free to adopt goals and measures from among those reached by consensus. I think that if you could really get your four leaders to play the game, and if they succeeded, the game could be expanded to provide a basis for consensus wherever consensus is required.
Today’s New York Times printed a story–first page, above the fold–announcing that Koch Industries and the Center for American Progress were joining forces to try to reform our country’s justice system. The description of the initial meeting of the participants sounds to me a bit like your multiple measures game without the index cards. I see a lot of parallels between justice reform and education reform: values, socioeconomic factors, unions, demagogues, budgets; also initial distrust but also a shared awareness that our society has a huge stake in the reform effort.
In my 16 years as exec of Pi Lambda Theta, I saw a lot of conferences and meetings among like-minded people, but nothing like what the New York Times described today. Any chance that you and Ron and others of similar standing could get the ball rolling on a similar collaboration in education?
LikeLike
Thanks for your thoughtful reply. I wonder if the prison initiative is happening because enough people have realized that we couldn’t possibly be ‘doing prison’ worse. IE, we’ve hit bottom. In my experience, that’s generally when people change–when they have to admit defeat.
Have we hit bottom in public education? Could be. We realize that the for-profit and the non-profit charter sectors are filled with money-grubbing operators. We know that far too many charter schools are failing to educate their students. Are enough people fed up with that?
We test and test and test, and I see signs that a critical mass of disgruntled parents and local educators may be developing.
More people understand that, to close the achievement gap, we must attend to the opportunity gap, the expectations gap and the leadership gap.
But where’s the leadership? Where’s the honesty? Who’s willing to play “Multiple Measures” as the ice-breaker to get us to focus on what we value, what we want for our children? And who’s willing to call out the (many) people who make a good living promoting and accepting mediocrity?
Two steps are essential: identifying those who benefit from mediocrity AND coming together to play “Multiple Measures.”
LikeLike
Sadly, I understand what you mean. Your point about “identifying those who benefit from mediocrity…” is fundamental, as in “know your adversary,” and must be addressed. In my most cynical moments, I start a lost of those, but it always becomes clear that the list is based on my prejudice. If there is indeed a lobby for mediocracy, I think the game is over and we’ve lost. But what if it turned out that our impasse reflects competing strongly held values but not any lobby for mediocracy? Always the optimist…
LikeLike
loved the post and enjoyed the humor and pathos of the game.
Just finished reading Kissinger’s World Order book and it strikes me that education in the US is a bit like foreign policy differences between European, US, Chinese, Russian and Islamic views – each convinced of the rightness of their views, hence the complexities of finding solutions that are equally acceptable to the parties. Partly the issue devolves around what ends we seek as a consequence of our values – no different in the education realm. And just as unlikely as any treaty to last for very long.
Equal access to education is not the same as equal opportunity to succeed. Making the U.S. Competitive in math and science with other countries based on test scores is yet another goal, but not the same as the first. And certainly not a workable solution in a country convinced that every child should be a lawyer or physician, and should get grade to grade promotions irrespective of ability. Judging faculty prowess based on the success of students is yet another. Seeking education that values diversity equally with the core of classics that form the western canon is yet another. Giving local school boards control over content vs statewide or national standards is another irreconcilable value..etc.
Hence I love the game, knowing full well that there actually are no mutually acceptable solutions. And that means we can keep playing the game without any winners – certainly not the students!
LikeLike
Why no traction in US for the evolved OFSTED audits of schools in the UK? Assign whatever weights one wants, and argue about the weights, or don’t agree on a weighted sum of the many measures. But, there is no cheap substitute for multiple experts conducting professional examinations. OFSTED.GOV.UK
LikeLike
While I like the idea of multiple measures, I do have a concern about the idea of points for extracurricular activities and volunteer work. Of course, we want students to participate in these things, but many children in poverty go immediately to work after school and on weekends. Often, they don’t have transportation to get to places to do volunteer work or for team practices and games. Also, in many schools, extracurricular activities require a fee or money for a uniform or other costs. In urban and high poverty schools, opportunities for volunteering and extracurriculars would have to be built in to the school day or provided after school with transportation.
LikeLike
Fair points, but that’s why each community should play “Multiple Measures” and thrash through what it values most, and then figure out how to measure performance. I’d invoke Ronald Reagan here, “Trust but verify.” He was talking about the Soviet Union, of course, but the concept applies to education. Right now, mistrust rules, with devastating consequences.
LikeLike
” While I like the idea of multiple measures, I do have a concern about the idea of points for extracurricular activities and volunteer work”. You are exactly right that students in disadvantaged homes have problems to get their required community service. Some have to work. And by the way, work for these immigrant youth does not mean the minimum wage. There is no minimum wage or overtime for farm labor. There is no minimum wage or overtime for local restaurants or doughnut shops owned by immigrant small business owners. Kids work off the books. Many kids work 60 hours a week during the summer for $6 an hour (cash) and 40 hours a week (all day Saturday and all day Sunday and then every evening 4-5 hours. Under these circumstances it is very difficult to get students to participate in sports and extra curricular activities or even showing up on time for school. Others have family obligations baby sitting, cleaning and cooking for siblings or their own children. I know one 19 year old woman (and yes she is married) who already has four children. She graduated from high school but just barely. A perky 15 year old who always got A’s and B’s by her senior year she was a D student (mostly due to absenteeism). It was sad also to see her age so quickly and have deep rings under her eyes. Any chance for college for her was foreclosed by her early marriage. In her case she at least has a husband who is employed (a mechanic). There is a huge population of immigrants (legal; illegal; formerly legal; “officially legal” -using a valid SSN of a cousin). And this is the reality they face. Those without valid SSN cannot apply for FAFSA or join the military. For many school is not the highest priority compared to survival. Some come to school primarily for the free breakfast, lunch and dinners (yes we serve dinner between 5-7). If we didn’t many of our athletes would not have supper.
Some do not have papers, some do not (yet) speak the language well and many have little access to transportation. And many of my students (I teach the English learners) haven’t pass their basic Math or English proficiencies yet. It is more urgent for these students to get tutoring to pass these tests than to do community service. Our solution is to for these students to have their community service required waived pending their passing the CAHSEE (high school exit exams). I give extra credit for those who give blood and who are able to do community service but I tell them the best community service they can do now is learn English and get their diploma.
LikeLike
I have written this before, and probably will again, it would be nice to have actual educator involved in the discussion. It might be noticed that in setting up your game at your table you included four individuals with none, or limited, teaching experience. The two men with none, and two women, while once teachers, have been out of the classroom and involved in union issues for about twenty years.
Even in your suggestion about community games of Multiple Measures you don’t specifically identify teachers as participants; and, few of your potential measures to be discussed seem to relate directly to creativity, problem solving, and learning. All of those are things that teachers find important and want their students to succeed in.
As I have written before, there is a lot of research that shows how the brain works little of which has found its way into the classroom; and the testing milieu foisted on educators today continues to push in the wrong direction, away from learning and into rote memorization.
There is also ample evidence that the testing regime in place for better than the last fifteen years is not working. If it were our students graduating today would be “college ready” which the pro-testers keep saying they aren’t; and schools around the country would be more equal, which hasn’t happened either.
The discussion, and “game” focus, really has to be about learning, and how best to bring that to our children.
LikeLike
Frank, you belong at the table in your community, but someone has to get the ball rolling. I put those four in my imaginary game because they wield a lot of power and because it seems to me that they mostly talk at each other.
If communities went through this process, great things might happen. So, please organize a game. (you don’t even have to buy the “Multiple Measures” game from me, but don’t tell anyone else I said that because I’m still counting on make a pile!)
LikeLike
I like the idea of giving students who do these things some real rewards. Right now it is good to be on Academic Decathlon or Mock Trial but only real benefit for most is that it looks good on a college application. For AD if students to very very well they can get some college credit.
We also give special seals to those students who show bilingual competency and literacy.
These ideas you mentioned are great. I also wish I could award students with great attendance. One of the number one reasons for a student failing a class is simply due to absenteeism. They come late to class and miss work or quizzes and then fail to make them up. They might do C or B work occasionally but when you add up the Zeros (not D’s) they end up with F.
I have very mixed feelings about community service requirements as I have mentioned above. Poor and working class students find those hoops impossible. As it is school quickly become secondary after age 16. Surviving economically is the highest goal.
I like these:
0-25 points for participation in class and personal work (such as exhibitions, writing, performances). In written evaluations, two teachers would each rate the student on a scale of 0 to 25 points, and the student would be awarded the average.
* 10 points if a student has fewer than 5 unexcused absences.
LikeLike
John and others, here’s a blog published last week in Ed Week i which Deborah Meier and i discuss the same issue.
http://blogs.edweek.org/edweek/Bridging-Differences/2015/02/what_assessment_system_would_s.html
The Center for School Change published a report in 2000 that described a variety of measures that schools can be used.
Click to access What-Should-We-Do-Report.pdf
Yes, some district & charters have used a number of the measures we discussed.
LikeLike
The Horace Mann League has published a report that uses multiple measures to evaluate how American schools perform compared to nine other nations.
“School Performance in Context: Indicators of School Inputs and Outputs in Nine Similar Nations” at
http://www.hmleague.org/wp-content/uploads/2015/01/School-Performance-in-Context-full.pdf .
LikeLike
Great piece, John.
Might I suggest that you change the attendance metric to “chronic absence.”
Here’s why:
– Chronic absence gets beyond daily attendance averages to see how many students are missing so much school they are headed off track academically
– Chronic absence gets beyond truancy to look at days lost to illness, family obligations and suspensions–often excused but still adding up to academic trouble.
Read more here
http://www.attendanceworks.org/multiple-measures-include-chronic-absence/
LikeLike
Done. Thanks for the good suggestion.
LikeLike