- About NCEE
- CIEB Supported Research
- Older Reports
- Tucker’s Blogs
Cross-posted at Education Week
On October 21, Education Week published a response from James Harvey and Charles Fowler to a blog I wrote about their report, The Iceberg Effect, in which I held up that report as an example of many reports that attempt to discredit international comparisons of student achievement that show the United States performing poorly relative to other nations. Failing that, these reports then attempt to show that if our students do perform poorly, our schools are not responsible. It’s the fault, they say, of factors that have nothing to do with the actions of American educators or our schools. If that is true, of course, one can only conclude that there are no changes our schools and educators could make that would improve outcomes for our students and so they don’t have to change what they are doing or how they do it. It is that formulation of the problem and that conclusion to which I object.
It stretches credulity to say that our schools have no responsibility for the poor performance of our students. If our schools have no responsibility for the academic performance of our students, what, exactly, are they responsible for? To get to that conclusion, the authors have to destroy the credibility of the measures on which U.S. performance looks bad. Then, just to be sure, the critic must show that, if the reader still harbors some doubt about the superlative performance of our students, the blame for low performance should not fall on the schools.
So Harvey and Fowler supply a list of American scholars who have attacked the international comparative assessments of student performance on methodological grounds. Harvey and Fowler know, of course, that I can easily come up with a list of other equally reputable scholars who are strong supporters of those comparative assessments. It would be astonishing if there were no controversy about this research, given the consequential nature of the findings for the countries that participate. It is important to point out, in this context, that all the periodic international comparative studies of student achievement that have been done have largely confirmed each other’s findings. There are none that would place the United States among the top countries in student achievement and equity.
The most devastating of these studies was the most recent PIAAC study from the OECD, which found that the millennials in the American workforce placed nearly last when compared to the millennials in the workforces of all the other countries studied in reading, mathematics and problem solving. Harvey and Fowler dismiss this study as so methodologically flawed that it was not even worth mentioning in their report. Their critique rests on alleged low response rates and the fact that the responses they got included immigrant workers in the United States but excluded immigrants in most of the countries being compared to the U.S. But, if you look a little closer you will see that the OECD had set a rather unrealistic response rate target of 70 percent. When they predictably did not get it in most of the countries they surveyed, they subjected the data they got to a battery of statistical tests to show that the samples were not biased. The authors of this critique simply ignored that fact. Given the data sets available, it is not surprising that the immigrants were not fully accounted for, but, if they had been, the results would have been changed for other countries by no more than 2.8 percent, which would not have changed the U.S.’s ranking. So, yes, there were methodological problems, but they were corrected where that was possible and where they were not, the remaining difference was so small as not to matter. This is not a trivial point, because the argument that Harvey and Fowler make falls apart when you realize that, over the last 40 years, the United States has gone from having the best educated workforce in the industrialized world to having one of the worst.
Basically, though, Harvey and Fowler, argue that our schools are not responsible for whatever deficiencies their graduates exhibit because of our high rates of child poverty, the levels of violence we see in this country and the lack of support for families with young children. These are serious issues, and they are issues we have to address, but we cannot use them as excuses for failing to change our schools in the ways that the countries outperforming us have changed theirs.
Here’s why. Several of the countries now greatly outperforming us were economic basket cases 40 or 50 years ago, with levels of poverty vastly exceeding anything this country has ever seen. Some of them had levels of violence that we have never seen. They went to work on them and they overcame them. But they also greatly changed their schools. They thought that school performance had to change greatly while at the same time understanding that they also had to change non-school policies and practices if the schools were ever to live up to their expectations. That is just what we have to do, too.
I can understand why our schools and many policy analysts and academics feel besieged these days. I, too, think it is unfair for so many outside the schools to lay all the problems of the schools at the feet of professional educators. And I, too, am outraged at those who think that tough-minded test-based accountability is the route to redemption for our schools.
Many of the people associated with The Iceberg Effect are friends, people whose values and work I admire. Harvey and Fowler say they are very comfortable with our organization’s education reform proposals. So why go out of my way to take issue with their report when there are so many truly wrongheaded reports and proposals abroad in the land?
I decided to write about The Iceberg Effect because our education debate seems to be polarized between those who would hold our teachers, school administrators and unions responsible for the poor performance of our students and the teachers, school administrators and unions who would hold everyone but themselves responsible.
If it is the professional educators—former Secretary of Education William Bennet used the derisive term “the Blob”—then the thing to do is to circumvent them with market mechanisms, blow them up with “disruptive change” and get rid of the worst of them with tough-minded accountability systems. If it is the parents and the society at large who are to blame, then the schools do not have to change. Everyone but the schools needs to change.
What the experience of the countries with the most successful education systems shows is that the problem is not the professional educators or their opponents. It is the design of our education system, invented a century ago for a society and industrial economy that has long since died away. Our professional educators will have to redesign it. It is they who have the professional knowledge needed to redesign it and it is they who will need to make the new design work. But they will be in no mood to do that as long as they are told by people who think of themselves as defending them that our education system does not need to be redesigned.
Cross-posted at Education Week
I’ve just read the new announcement from the Ford Foundation describing the direction the foundation will take under its new President, Darren Walker. It is a deeply thoughtful and quite radical statement of priorities. It tells us not only what the foundation will do, but what it will no longer do and why. Most important from my point of view, it reverses trends in foundation management that are decades old, trends that have been very damaging to the not-for-profit community on which the foundations depend to reach their own goals.
The heart of the matter is a dry-as-dust topic: institutional overhead. For those who need a brush-up on this arcane subject, overhead is what not-for-profits charge for the costs they bear that are not associated with particular foundation-supported projects, but are associated instead with the cost of things that all or many projects share and which, if not paid for, will make it impossible for the organization to function. A few examples will suffice. Typically, the foundation will pay directly for the cost of the space the project itself occupies, but the cost of the hallways, lobby, general management offices and so on are shared among all the projects on some reasonable basis. The same thing might be true for the costs associated with copying, IT support and office supplies. What can reasonably be assigned directly to the project is so assigned; the rest is put in the pool of overhead or indirect costs and shared out. There are many different ways to determine what should be in the overhead pool and what should be assigned directly to the project that are regarded as legitimate by the accounting profession.
When I was just starting out in the not-for-profit world, foundations typically assumed that it was their obligation to pay their fair share of indirect or overhead costs. But that changed over time. First, they chafed at the idea of paying the full overhead costs of universities that applied for research project grants, often charged at rates of 70 percent of the direct costs or even more. They could see why they should pay for their share of the common space for a research facility, but they could not see why they needed to pay for a share of the costs of the gymnasiums or the general libraries, for example.
Many foundations then rethought the nature of their obligation. When I was in government, our grant and contract officers distinguished sharply between the purpose of a grant and the purpose of a contract. They told those of us in charge of research operations that we should use contracts when our goal was to carry out a government purpose. Interestingly, at that time, the government took the position that it should pay its share of all the indirect or overhead costs associated with the proportion of direct costs that we were providing.
That was in an era in which most private foundations, just like government grant makers, saw themselves as providing grants to highly capable people who had their own agendas and their own plans. Even though they were providing aid to people and institutions to support the implementation of their grantees’ plans, and not their own plans, they were still prepared to pay their fair share of overhead costs.
But then it all fell apart. A new wave of private foundations were established by living entrepreneurs who tasked their staffs with coming up with tightly focused agendas, plans for achieving those agendas and timelines and metrics for accomplishing those plans. The locus for agenda-making shifted from the grantees to the foundation. Grantees became agents of the foundations and their agendas. As the transformation I just described took place, the foundations—both old and new—began to look for the best deal they could get from not-for-profits that had become, in effect, private contractors. They no longer took responsibility for the institutional health of the not-for-profits they funded.
The foundations began to put caps on the amount of overhead they would pay, conscious when they did so that the caps were so low that they would not pay for shared costs that had to be incurred if the project they were funding had any chance of succeeding. In effect, every grant they made to a not-for-profit that received all of its income from grants would produce a deficit for the organization receiving the grants. This led to what I can only call a corrupt relationship between the not-for-profit staffs and the foundation staffs in which both worked together to get around the trustee-imposed ceiling by re-categorizing what should have been overhead expenses as direct expenses. But there were limits to the degree to which this was possible, and so many not-for-profits were forced into a hand-to-mouth existence, worrying from month to month whether they would make payroll. Whether they did or not, many were forced by this regime to abandon their own agendas to become contractors to many foundations to pursue the foundations’ increasingly narrow agendas on the foundations’ terms.
Enter the Ford Foundation. To my utter delight, Darren Walker, Ford’s new president, acknowledges in his open letter that the polices of his own foundation—and by implication all other foundations with similar policies—made a big mistake when they decided, in effect, that they would henceforth refuse to pay the full overhead costs associated with the grants they made to not-for-profits. And then he goes much further. He goes on to announce that Ford will henceforth take a direct interest in the health of the not-for-profits they support, as institutions. There are no details, but I must say I was thrilled at this news. Not because I expect that we will be one of those institutions—I imagine the chances of that happening are small—but because this is enlightened policy at its best.
No foundation should feel itself obligated to support any institution that is not a strong contributor. Opportunities abound for cozy relationships between foundation program officers and not-for-profit staff based on close personal relationships or past performance not reflected in current capacity. Foundation boards and CEOs need to be sensitive to that possibility. But foundations will be hard pressed to get the kinds of wins they want if the only institutions out there are job shops interested only in doing what it takes to keep their staff employed. Just as the best not-for-profits cannot survive without foundations willing to take responsibility for their health, the foundations cannot do good work without a supply of not-for-profits with imagination, great leadership and the capacity to chart new waters and implement ambitious plans.
It is really wonderful to see the Ford Foundation taking this kind of leadership in addressing the key issues of institutional capacity in the not-for-profit community that some of us have worried about for so long. I can only hope that others will follow their lead.
Cross-posted at Education Week
Last week’s news regarding disappointing results on the most recent administration of the National Assessment of Education Progress (NAEP) certainly made some waves. Scores decreased for both assessed grade levels in math, decreased for 8th graders in reading, and were flat in 4th grade reading. This marks the first time in 25 years that math scores have declined.
The decline has been blamed on immigration, the family economic stress brought on by the Great Recession, implementation of the Common Core State Standards, failure to implement the Common Core State Standards, tough-minded teacher accountability and the Supermoon. We will never know whether it is due to any of these things, a combination of these things or none of these things, because we have no evidence and never will to adjudicate such claims.
But the most important thing to know about the fall in NAEP scores is that it is the wrong thing to focus on. What you should be focusing on is the lack of change in NAEP scores over time. Take a look at the table below of historical NAEP scores at the high school level since the survey was started four decades ago. We chose the historical scores because the prompts have been the same over the whole period. You can be certain that you are comparing apples to apples. We want you to focus on the high school level because what really matters is the skills that our students leave high school with as they go on to further education or enter the work force.
My colleague David R. Mandel points out that average reading scores at the high school level have been flat since 1971. The proportion of students at or above the “proficient” mark was 39 percent in 1971 and 39 percent in 2012, the last time NAEP reported on the historical series. There was a bump up in these averages in 1992, but the air went out of the bump after that. If you look inside these numbers, you will see that the White/Black gap went from 53 points in 1971 to 20 in 1990, staying fairly constant since then.
The average historical NAEP high school scores for mathematics follow much the same pattern: 304 points in 1973 (the first year NAEP assessed math at the high school level) and 306 in 2012 (the last time historical NAEP was given at the high school level). The last statistically significant change in these scores occurred in 1986. The story for differences in performance between White and Black students is very like the story for reading. The gap declined from 40 points in 1973 down to 21 points in 1990, and has been stable since a rise to 26 in 1992. Roughly the same trajectory has been followed by the White/Hispanic gap.
The average reading scores are still not up to the level they attained before NCLB was passed. The average mathematics scores are about the same. NCLB has contributed not at all to closing the gaps in performance between White and minority students. The data simply do not support the claim that No Child Left Behind and its descendants improve outcomes for our students.
Some of you will be quick to point out that, while the NAEP data are disappointing at the high school level, more progress was made at the 4th and 8th grade levels. As my colleague Brendan Williams-Kief points out, however, even if we stayed on the same heading and speed that we were on during the period from 2003 to 2013 at those levels, before this year’s dip downward, we would have to wait until the year 2105 for 80 percent of our 4th graders to be at or above proficient in mathematics, the year 2147 for 8th grade mathematics, the year 2191 for 4th grade reading, and the year 2173 for 8th grade reading.
The average student in many of the top-performing countries graduates two to three years ahead of the average high school graduate in the United States. The distance between what our high school graduates know and can do and what their high school graduates know and can do has been steadily widening in recent years.
The problem with the NAEP scores is not that they just slipped a bit. It is that those scores are pretty much the same at the high school level as they were 40 years ago. During those 40 years, one nation after another has surpassed our performance. That is the disaster on which we should be focusing. What it says is that there is something deeply wrong with our whole system of education and it has got to get fixed.
Cross-posted at Education Week
A new report from the Council of the Great City Schools has done what seemingly nothing or no one has yet been able to do: Convince the current administration that the rampant over-testing in U.S. schools is proving harmful for the quality of education that our students receive.
The report found that students take, on average, more than 112 standardized tests between pre-K and grade 12, with the average student taking about eight standardized tests per year. Some are intended to “fulfill federal requirements under No Child Left Behind, NCLB waivers, or Race to the Top (RTT), while many others originate at the state and local levels. Others were optional.”
Now the administration is signaling that they see the error of their and their predecessors’ ways. Calling for a two percent cap on the amount of classroom time that is spent on testing, and a host of other proposals, the administration’s mea culpa is an unexpected demonstration of what can occur when the facts are laid bare for all to see. How much is actually done to reverse the over-testing trend will be decided by the actions of incoming acting Secretary of Education John King.
The tone of flexibility in the Department’s announcement is new and welcome, as is its recognition that the Department may share some culpability in the national revolt against testing. Its call for fewer and higher quality assessments is on target, as is its willingness to help the states come up with more sensible approaches.
What I don’t see in the administration’s proposals is understanding that the vast proliferation of indiscriminate testing with cheap, low quality tests is the direct result of federal education policies beginning with No Child Left Behind and continuing with Race to the Top and the current waiver regime. I offer you one phrase in the Department’s announcement in evidence of this proposition: “The Department will work with states that wish to amend their ESEA flexibility waiver plans to reduce testing…while still maintaining teacher and leader evaluation and support systems that include growth in student learning.”
But it is precisely the federal government’s insistence on requiring testing regimes that facilitate teacher and leader evaluations that include student growth metrics that caused all this over-testing in the first place.
Outstanding principals I’ve talked with tell me that when tough-minded, test-based accountability came into vogue, they created or found good interventions that came with their own assessments, each keyed to the intervention they were using. They had always done that. But their district superintendents, also fearful for their jobs under the new regime, mandated other interventions, with their own tests. Then the state piled on with their own mandated programs and tests, all driven by the fear of leaders, at each level, that if student performance did not improve at the required rate, their own jobs were on the line. Few of these interventions were aligned with the new standards or with each other. But time was of the essence. Better a non-aligned instructional program than none at all. Better a cheap test of basic skills they could afford than a much more expensive one they could not afford.
What sent the numbers right over the cliff was pacing. School administrators, focused on having their students score well on the basic skills tests used by the state accountability systems, pushed schools enrolling large numbers of disadvantaged students to figure out where the students needed to be at set intervals during the year. This determined the pace of instruction. It also made it much easier for administrators to get control over the instruction. All that remained was to administer a test at each of those intervals—say every month or couple of months—to see whether the teachers were keeping pace with the scripted curriculum and the students were making enough progress to do well at the end of the semester or year.
Pacing was the drum. Drill and practice was the melody. Each pacing test was a mini-version of the final test to be given by the state. Instantly, for every federally-required test, four to six others just like it were created. But these tests, because there were so many of them, had to be very cheap. In the minds of some people, this was formative evaluation, but, in my mind, its effect was to regiment instruction in a way that the best advocates of formative evaluation never had in mind.
The situation I just described must have seemed a horror to many teachers, especially teachers of disadvantaged students. Whatever discretion they had disappeared. Their lives became a litany of pacing guides, scripted curriculum, drill and practice and a flood of test score data to analyze. In states with letter grade systems of accountability, school administrators quickly figured out how to game the system, concentrating their best teachers on students in the grades with mandated assessments, and, within those grades, on the students just below the thresholds of performance, which, if exceeded, would bump the school’s overall grade up. They gave much less attention to the students who were far behind or who were doing well but could have done much better.
The vast proliferation of testing and the widespread use of cheap tests that fail to measure high standards did not happen because state officials did not know what a good test is or because local officials spontaneously concluded that it would be a good idea to greatly expand the use of pacing guides, drill and practice regimens and the tests that come with them. State and district officials used lousy tests and greatly increased their use because they thought they were the only tools available to respond to the federal government’s requirements for tough test-based accountability. They were simply responding to the incentives they faced. They still face those incentives, so there is no reason to expect them to behave any differently. Giving them lessons on what good tests look like won’t change that. Giving them money to review their testing programs won’t change that.
The only thing that will change that would be to change the accountability systems to do less testing with much higher quality tests and to change the accountability regime to put less pressure on the schools and teachers and, instead, create incentives for students to take tough courses and work hard in school. This is exactly what the top performing countries do. We are, as usual, embracing a strategy that has been considered and rejected by the countries with the most successful school systems and rejecting the testing and accountability strategies they have used with great success. If you would like to see a proposal for an accountability strategy based on their policies on testing and accountability, look here.
The situation for teachers is even more poignant. They have turned against the Common Core not because of the standards themselves, which they like, but because of the way they have been implemented, and, in particular, because of the way they have been used to create teacher evaluation systems. Using a logic I have yet to get my head around, the advocates of teacher evaluation appear to have convinced themselves that teacher and leader evaluation is by itself a highly promising school reform.
The key for great school leaders isn’t formal evaluation and it isn’t firing people. Only Donald Trump, evidently, fired his way to the top. The key is running a great school that great people want to work in, and then spending a lot of time identifying, recruiting and supporting those great people. Principals who work this way often let their staff know that they expect them to work hard. Those who do not want to work so hard go elsewhere. But these principals do not depend on test-based accountability systems to identify the slackers nor do they depend on test-based accountability systems to identify the teachers they want to hire or to develop them once they are hired.. Why should they? They are in classrooms all the time, talking and observing, coaching and supporting.
The data reported by the Council of the Great City Schools reveal a calamity. The cause is our national accountability system. The flexibility offered by the Department of Education is welcome and refreshing, but it is not the answer. The answer will have to wait for the day when the federal government no longer insists that the states and schools use test-based accountability and value-added strategies to assess individual teachers with consequences for individual teachers. John King did not create this system. Perhaps he can help this country change it. We’ll see.
Date: November 20, 2015
Date: November 13, 2015
Date: November 12, 2015
Date: October 28, 2015
Date: October 23, 2015