Return to the Rossi Award page.
(November 7, 2008)
I am honored to receive this award named after Peter Rossi, a man
who had such an enormous impact on our profession.
Thank you, Doug, for your generous remarks, but I want you and the audience to know that I donít take them
personally. I accept this award for my work, but also for that of the many other people who, through a
combination of faith, fight, and funds, made a revolution in policy research by showing the feasibility and
value of social experiments.
Photo by Rich Schmitt
When I started down this path over 30 years ago, we were a lonely band of zealots. Random assignment was
anything but chic. You didnít get tenure doing random assignment studies, and the hot shots in my field ó
economics ó favored fancy modeling and complex econometrics and statistics.
Our faith was in a vision of how to improve policy and government and, at the risk of sounding corny, a
way to leave the world a better place. This faith rested on the belief that social experiments could produce
more credible evidence on causality and that, while you could not assure that high-quality evidence would be
used, it had the potential to advance several desirable outcomes: improving peopleís lives, increasing public
support for social programs, and getting a higher return on scarce public investments.
But, while we had to have faith to stick with it, we were never blind true believers. We knew that public
policy decisions were driven not just by facts but by values and that random assignment was no panacea.
It could not answer all questions or resolve trade-offs and choices. It could not and should not arbitrate
Yet our hope was that random assignment could produce more definite, less ambiguous evidence that would
more effectively isolate research from political pressure and help elevate these debates and focus them on
the real choices. In this sense, the revolution went beyond a research method to a vision of how to make
government more rational and effective.
So my first obligation today is to acknowledge three groups of people: first and foremost, my teammates
at MDRC, and there were an army of them, who sustained and inspired me; second, the small but crucial group
of academics (including Rob Hollister, a previous winner of this award) who in defending this methodology gave
it a legitimacy that we, the practitioners of the craft, could not; and third my colleagues and sometimes
competitors, primarily at Mathematica Policy Research and Abt Associates, but now at an increasing number of
universities and other research firms, who have been part of this movement. Over the years, our collective
drive and high standards, reinforced as we met at annual APPAM gatherings, created this revolution.
My second point is to remind us that this was a fight. To many of you in the next generation, the value
of random assignment may seem so obvious as to be uncontroversial. But showing that this method was
feasible, useful, and ethical took a relentless campaign. At the beginning, we were told that it was
fine in the laboratory, but that in the real world you could not get program operators to agree. It would
be akin to asking a doctor to deny patients a known cure.
Throughout these 30 plus years, and despite success after success in implementing random assignment
without legal challenge, launching such studies has almost always been a fight. There is, even now,
enormous pressure to use a weaker, less intrusive design. I have searing memories of people hurling
epithets drawn from skeletons in the medical research closet. Two stand out: of Barbara Goldman of MDRC
returning shaken from Santa Clara County, where she had been called a Nazi, and of testifying before a
committee of the Florida legislature and being told by a dentist turned legislator that I was using
practices akin to those in the infamous Tuskegee experiment (although that wasnít even random assignment).
My third point is the importance of funds, and thus funders. It takes money to realize many dreams.
My work and this revolution would have been stillborn without the support and active partnership of
key people at foundations (particularly, in the critical early days, the Ford Foundation), and in
several federal agencies, notable the U.S. Department of Labor and the Office of the Assistant
Secretary for Planning and Evaluation and the Administration for Children and Families in the Department
of Health and Human Services. To test a new program as an experiment is to make an ex ante admission that
you donít know whether it will work. That is the very reason for the experiment, or in fact for any
evaluation. However, it takes brave funders to sell a grant as a test, rather than as an obvious good
idea that they have confidence will work.
But faith, fight, and funders would not have done it without the cooperation of people in
participating states, cities, and community organizations. They are the true heroes of this story,
since they took the greatest political risks by letting us test their convictions and put them at
risk of unfavorable publicity. Our success is due to their willingness to redefine Justice Brandeisís
famous description of states as laboratories for policy experiments, and also to the hundreds of
thousands of people in these studies who gave their time to answer our questions and allowed us to
collect their data.
In concluding, I want to share three final thoughts. The first is that, while I obviously believe
in the value of social experiments, I also am convinced that they are far from the only source of insight
and evidence. If we are to advance policy and practice, we will need to draw on the insights of
practitioners, managers (the M in APPAM), and those researchers who use different methods to diagnose
the problems and to understand why people behave as they do and how social programs work in practice.
Experiments are not an alternative to these sources of insight and innovation, but a way to confirm or
disprove the expert judgment they suggest. In my experience, the strongest programs and evidence emerge
when these different types of experts work together, using the strengths of their different fields.
The second relates to Peter Rossi and his oft quoted Iron Law of Evaluation: ďThe expected value of
any net impact assessment of any large scale social program is zero.Ē In my experience, fortunately, this
has not been true. Over and over again, social experiments did not confirm the null hypothesis but, instead,
found impacts of clear statistical and policy significance. They showed that many programs worked, although
they very rarely worked miracles. I donít have time today to speculate on why that might be the case.
The third comment is more personal. I accept this award for myself, for many of you, and fittingly in
this historic week also for the person who had the greatest impact on my work, my father. Born over 100
years ago on the Lower East Side of Manhattan, he was a fighter and idealist. He might have been happy
to hear about a methodological triumph, but he would have been proud if I could have told him that we
had been wise enough to harness it to do some good for others. I hope we have delivered on both fronts.
In accepting this award, I turn back to you in the APPAM community and ask you to continue to have
the faith and fight the fight.
Back to top