Elsewhere in this newsletter, we summarize the paper prepared by the OECD for the recent International Teachers Summit in the Netherlands and the remarks made by Andreas Schleicher in his webinar on the subject.  These documents are well worth reading, as is Vivien Stewart’s account of the event.  Here, I will attempt to share some of the dynamics of the summit.

I did not attend the summit, and so have assembled this account on the basis of conversations with several people who were there.  My purpose is to describe some of the differences in views among the participants, because they are consequential, and reveal much about the direction education policy is likely to take in the coming years.

This was the third in the series of summits, the first two of which were held in New York City at the invitation of United States Secretary of Education Arne Duncan, invitations extended to the ministers of education and education labor leaders from the top-performing and most rapidly improving countries.  The United States acted as host country for those meetings and Education International (EI) and the OECD were the principal co-sponsors.  The aim was to provide a venue in which the top officials involved in making policy for teachers and teaching in their countries could, aided by analyses provided by the OECD and EI, compare notes on strategy and implementation, and by so doing, further improve their own education systems.  Nothing quite like this had ever happened before.

The first summit was focused on attracting and recruiting high quality secondary school candidates into the profession.  It covered initial teacher education, strengthening professional practice and retention.  There was broad agreement that no nation could have a high quality education system without high quality teachers.  One could feel a palpable sense of excitement among the participants as they reinforced each other’s conviction that a policy focus on teacher quality could yield great dividends and that the nations around the table could learn a lot from each other.  It ended with a call for a second meeting, one that would go deeper on teacher preparation, teacher supply and demand and school leadership.  Subsequent meetings, the planners thought, might similarly focus on other key aspects of policy for teachers.

The second meeting reached the objectives its planners had for it in the realm of leadership, though it came up a little short on the subject of supply and demand.  But the big difference was a difference of tone.  Key differences in policy direction among the participants emerged, differences grounded in different interpretations of the nature of the challenges facing the industrialized nations’ education systems, and the appropriate responses.  The differences in tone became obvious both in exchanges between the participants at the table and later, when the observers had a chance to ask questions of those participants.  There was, in particular, a certain chill in the exchange between U.S. Secretary of Education Arne Duncan and Sing Kong Lee, the Director of the Singapore’s National Institute of Education, in their exchange on the subject of teacher evaluation and the role of teacher evaluation in the design of accountability systems.  Secretary Duncan appeared to be pressing for some support for the proposition that teacher evaluation—in particular teacher evaluation tied to measured student performance—was an important key to teacher quality.  Sing Kong Lee acknowledged that teacher evaluation was important, but expressed some reservations about the American approach.  There were echoes of this difference at other points during the discussion at the table and again, in somewhat more strident tones, when the observers joined in the discussion.  Though many in the room nodded their heads when Sing Kong Lee spoke on this topic, it was clear that Duncan was not alone in his view that countries interested in improving student outcomes needed strong accountability systems, and that teacher evaluation systems tied to student performance should be part of those systems, but it was just as clear that the labor leaders, teachers in the audience and many ministers were very wary of such systems.

Toward the end of the second summit, the ministers and labor leaders gathered for separate lunches.  Both gatherings acknowledged that the issue of teacher evaluation and appraisal had become the “elephant in the room.”  To the extent that teacher evaluation is tied to promotions, retention, incentives, rewards and so on, such discussions can easily lead to confrontations with the teachers unions.  But it was not just fear of confrontations between governments and unions that was at play here.  Many of the ministers had considered and rejected the idea of basing policy in any important way on tough accountability systems focused on teacher evaluation because they did not think such management strategies would enable them to recruit and retain the kind of high quality professionals they wanted.

Thus, this issue appeared to engage issues of policy, management and strategy central to the work of everyone.  Andreas Schleicher, realizing that the great promise of the summits could be squandered if they did not deal with this issue, pressed those present to make the “elephant in the room” the focus of the next summit.  Rather than trying to push it into a corner, he wanted to deal with it head on.  EI agreed.

Some of the experts and observers in the room argued that teacher evaluation should not be the central topic of the third summit, that it was but one component among many in a high-performing system.  But Schleicher and others agreed that this set of issues was so central that it needed to be dealt with head on and the decision was made to focus the third summit on teacher evaluation and appraisal.

That decision would put great pressure on the OECD to come up with a paper setting the stage for the meeting that all the attendees would regard as a fair point of departure for the discussion.  The planners agreed on the following lens for that paper:

• How should teacher evaluation and appraisal be defined and who should define it?
• What processes and techniques should be used?
• What can research tell as about the impact of teacher evaluation and appraisal?

And thus the stage was set for the third summit.

I was not there, and could not in any case get inside the heads of those who were, but, at this distance am very much inclined to agree with what I take to be Schleicher’s strategy.  Shoving this issue under the rug would have doomed the summits.  Ministers would have drifted away if the discussions were inhibited by very important issues that could not be discussed.  The alternative was to try to frame the issues in such a way that they could be discussed.  This was the path that was chosen.  It was broadly agreed that teacher evaluation and appraisal is very important and that it could be effective only in systems also designed to:

• Make teaching an attractive profession,
• Provide very high-quality initial teacher education,
• Create a school management system in which teachers could act as autonomous professionals within a collaborative culture, and
• Engage teachers in developing the evaluation system.

And that was frame with which OECD and EI opened the third summit.

This is a very sensible approach.  It could potentially provide a roadmap leading to sound policy that would also provide an opportunity for all parties to claim victory, but it would have been too much to expect that it would relieve all the tensions with which the second summit ended.

In the eyes of several observers, no one at the table at the third summit was advocating that teacher evaluation and appraisal be used to weed out bad teachers.  And everyone agreed that teachers both needed and wanted feedback.  But, with that off the table, there was still tension between those who are most comfortable with the use of evaluation for professional growth and development, on the one hand, and those who see it as a vital tool in the design and implementation of tough-minded accountability systems on the other.  And, in the middle, were those who were naturally inclined to the position apparently so well articulated by Andreas Schleicher at the meeting, namely that teacher evaluation is best thought of as an important component of a much larger system built around a conception of teachers as highly capable professionals, not as cogs in a Tayloristic management design.

That vision assumes that the criteria against which teachers are being judged is not limited to student performance on basic skills in a narrow range of subjects but on their ability to help students succeed against the full range of outcomes now widely referred to as 21st century skills, many of which are difficult if not impossible to measure.  In Tayloristic systems, everyone assumes that management will assess the workers in any way they see fit, usually according to fairly simplistic criteria; in professional environments, the direction of accountability is at least as much to one’s colleagues as to one’s superiors in the organizational structure.  So who is to devise the criteria for judging teachers and who is to decide whether an individual teacher meets them?  In blue collar environments, all workers are regarded as equal, if not interchangeable.  But, in a professional environment, the professionals acquire increasing responsibility, authority and compensation as they demonstrate increasing competence and skill.  Perhaps, as nations move toward conceptions of teachers and teaching grounded in the idea of teacher as professional, the idea of teacher evaluation and appraisal should be inextricably connected to the development of formalized career ladders for teachers.

The third summit did indeed address these and other issues.  This made for some tough conversations.  It became very clear that it was going to be hard to resolve these issues without some real trust among the parties, both at this table, and, by implication, within the countries represented.

Nonetheless, the people I talked with about the summit came away encouraged.  The honesty of the conversation, the fact that what had at the preceding meeting been the “elephant in the room” had now been addressed and that there was substantial consensus on many points was a relief to many who had feared going into the meeting that it might end badly.

That it did not is no doubt in part the result of the good will of those who came.

But new cracks emerged.  Among the rules set by the conference organizers is one that says that a country cannot be represented at all unless it is represented by the top education official (usually the minister of education) and the top teachers union official.  But, especially for the Asian nations, there is a strict limit to the number of out-of-country trips officials can make, often no more than two a year.  If a minister more senior than the education minister calls a meeting on the date of the summit, the education minister must cancel the trip to the summit.  Under the current rules, this means that the country is not formally represented and for that reason, a number of jurisdictions that had been invited to the third summit attended in a participating observer status.

The rule could, of course, be abandoned.  But that could easily lead to the summit not being a summit of top officials with policy-making authority, but rather a meeting of functionaries.  No one wants that.

There is another problem.  It is important to the host country to be able to invite observers, people—mostly educators—who are interested in the proceedings and want to express their views on the issues being discussed by the delegates.  But this desire for what has become something of a public fishbowl can inhibit the desire of the organizers of the summits to have a frank discussion among the delegates.  The frankness of the discussion is one of the big attractions of the meetings for the delegates.  The openness of the meetings is a big draw for the host countries.  This potential conflict of goals did not loom large when the summits were first conceived, but, now that the conversation has begun to tread on sensitive issues, it has become clear that some way must be found to resolve the tension between the desire for openness and the need for some measure of privacy.

Lastly, as in so many other international organizations, there are tensions with respect to which nations are invited to sit around the table.  The original conception was to include both top performers (on the PISA rankings) and the countries whose education systems were improving the fastest.  But, if Asian top performers drop out because education ministers are not able to attend, the summit could get to be a meeting dominated by countries that are not among the top performers, and, if that happens, the top performers who remain may decide not to come, and then the summit ceases to be a summit.

These are tough challenges, but they are neither unprecedented among such international meetings nor are they, in principle, insurmountable.  The three meetings that have taken place thus far have served as a unique venue for the people on whose shoulders rest the fundamental redesign of the world’s leading education systems to exchange information, share views and challenge each other’s conception of the right policies and strategies.  That is a very worthwhile function.  I very much hope the organizers are successful as they seek a path through this thicket.

From 2006 to 2011 the International Association for the Evaluation of Educational Achievement (IEA), the organization that produces the TIMMS and PIRLS international assessments, worked with teacher education programs in seventeen countries to administer a survey to determine the depth of mathematics knowledge and skills of future teachers in those countries.  The study also assessed the quality assurance measures each nation has in place for recruiting and training teachers.  The seventeen countries that participated in this Teacher Education and Development Study in Mathematics (TEDS-M) were Botswana, Canada (four provinces), Chile, Georgia, Germany, Malaysia, Norway, Oman (lower-secondary teacher education only), the Philippines, Poland, the Russian Federation, Singapore, Spain (primary teacher education only), Switzerland (German-speaking cantons), Taiwan, Thailand, and the United States (public schools only).

The survey included a test of mathematics content knowledge of teachers in training in the last year of their primary- and lower-secondary teacher education programs.  The subdomains used to develop the test were derived from the subdomains used in the assessment frameworks for IEA’s Trends in Mathematics and Science Study (TIMSS) given every 4 years to students at grades 4 and 8 in more than 60 countries.  A different test was administered to students training to be primary teachers from the test given to students training to be lower-secondary teachers in order to reflect the level of mathematics they will eventually teach.

The charts below show the scaled mean score for teachers from each country, for primary teachers in Chart 1 and lower-secondary teachers in Chart 2.  Canada is not included in either chart as the number of responses from that country did not meet the sample size requirements for TEDS-M.

While not all countries represented in these charts participated in TIMSS 2011, it is clear that top TIMMS performers, such as Singapore and Taiwan, also did well on this measure of future teachers’ math content knowledge at both primary and secondary levels.

Along with the assessment gauging future teachers’ mathematics content knowledge, TEDS-M conducted a survey of teacher training programs in the countries studied to see what quality assurance procedures are in place for recruitment and selection of future teachers.  This study looked at three criteria for quality assurance in the recruitment and selection of future teachers: 1) The extent to which states control enrollment into teacher training programs, 2) The attractiveness and status of primary and secondary teaching as a profession, and 3) High selection requirements for entry into teacher training programs.  Quality assurance procedures from each country were ranked Strong, Moderately Strong or Limited.

The chart below demonstrates how each country scored for each category, according to the criteria developed by the TEDS-M study.  Among the countries that responded to the survey, only two countries were rated strong in all three categories: Taiwan and Singapore, both of which came out at the top of the league tables in the most recent administration of TIMSS Mathematics in 4th and 8th grade.  The United States, meanwhile, rated low on all three measures of quality assurance for recruitment and selection of teachers.

The results of the TEDS-M survey clearly show a large gap in future teachers’ mathematical content knowledge between countries that typically top the league tables in student performance on international tests, like Taiwan and Singapore, and low performing countries.  Top performing countries also have rigorous quality assurance measures for recruiting and training new teachers.  The findings of the TEDS-M survey also suggest that the diversity in teacher recruitment and training procedures represents a policy continuum, which can provide countries working to improve teacher quality with examples of systems that are working hard to improve the mathematics knowledge and skills of their teaching force.  The TEDS-M survey considered other factors in teacher training programs as well, including future teachers’ beliefs about mathematics learning and self-reported past performance in mathematics.  We encourage you to read the full report here.

Education Equity

The issue of educational equity is one that is important to policymakers in all top-performing nations, but the situation in China is unique.  Shanghai has one of the best education systems in the world, but it is an anomaly in a country that is also characterized by a sprawling rural populace without many of the benefits of Shanghai’s system.  As recently as a few years ago in China, not all children had access to nine years of compulsory education, though that changed at the end of 2011.  China’s government has been working to redress this gap, and an article at China.org quotes Chinese Education Minister Yuan Guiren as saying that “[China has] made a lot of progress in improving fairness in education in recent years … My dream is to ensure that we can … provide education for all people without discrimination and cultivate every person in this nation to become a talent.”  A great deal of their policy and financial focus has been directed towards poor, minority, female and rural students over the past five years.

Despite Shanghai’s great strengths in education, policymakers and educators are concerned there, too, with a different equity issue.  Unlike in the rest of China, where female students may have less access to education than male students, in Shanghai, female students are outperforming male students fairly significantly.  In 2008, girls made up more than 60 percent of the top scorers on the gaokao, China’s university entrance examination, up from about 34 percent in 1999.  As a response to this dramatic shift in performance, The Japan Times reports that some Shanghai schools have created classes for boys only, hoping learning in a single-gender environment will help to boost male students’ confidence and improve their performance.

In China’s Special Administrative Region of Hong Kong, there is concern over the equity of the education provided to students from ethnic minorities.  In that region, almost all schools are required to teach all of their classes in Cantonese; this requirement has been in place since 1997.  The New York Times reports that this policy has been detrimental to students who do not speak Cantonese fluently.  As a response, the government has opened “designated schools” in which classes are taught in English and the student body is 95 percent ethnic minority.  However, many feel that the students attending these schools are at a disadvantage due to their separation (both physically and linguistically) from mainstream society.

Education Funding

Following the release of the Review of Funding for Schooling (better known as the “Gonski Report”) in Australia last year, both the federal government and the states have been working to reach agreement on what school funding will look like going forward.  The report proposed a uniform system of funding schools across Australia, with a base funding amount augmented by a school-specific “loading” to address economic, cognitive and physical disadvantages among the student body.  However, the premier of the state of Victoria has rejected this plan, preferring instead to propose his own system.  The Age has the full story.  Other states, too, have rejected the Gonski proposals.  In Queensland, the Education Minister has announced that they will be developing their own funding plan, while Tasmania has emerged as the first state with a Labor government (the same party that is in power in the federal government) to reject the government’s plans.

Nearby in New Zealand, a recently released study from the New Zealand Council for Educational Research found that, according to a survey of the country’s principals, secondary schools are often struggling financially.  Principals reported budgetary deficits, with a majority stating that their finances were worse in 2012 than they were the previous year.  However, the Ministry of Education rejected the report’s findings, with a staffer contending, “schools are adequately funded to deliver the curriculum so that all students are able to learn and achieve.”  Read more at the Otago Times.

China, meanwhile, has been working to increase funding for schooling as part of their overarching strategy to improve the system and create both greater equity and improved student performance.  An article on Xinhuanet reports that China has been increasing education spending since 2009, with an investment totaling nearly US$5 billion over the past four years.  Four percent of GDP is about the same, proportionally, as what the OECD countries spend on average on education each year; top performing systems such as Australia, Finland and the Netherlands also spend about 4% of their GDP on education. 4-Traders reports that the government plans to increase education spending by 9.3% in 2013 and to focus on educating rural students.

Student Pressure

Most of the top-performing East Asian education systems are known for the the extensive hours students spend studying outside of class, often to prepare for university entrance exams.  The Japan Times reports that students in that country are attending cram schools, or juku, earlier than ever before.  Whereas in the past students began attending juku in their teenage years, now it is becoming increasingly common among elementary and even preschool students.  However, there does seem to be a degree of ambivalence on the part of parents: while many feel that they should not have to pay for private tutoring in addition to regular schooling, they often turn to juku in order to ensure that their children are not falling behind their peers.

In China, by contrast, aware that immense pressure on students is often not conducive to a student’s health, many provinces are making strides in changing the culture of “cram.”  Beijing, in particular, is leading the pack in developing policies focused on reducing student stress in an education system where tests are a central element of schooling.   The Global Times reports that after March 19, primary schools in that city will face limitations on testing and homework, and secondary schools will be prohibited from ranking students based on exam scores.  However, the article also reports that parents are not necessarily on board with these changes.  One parent pointed out that as long as the gaokao, China’s university entrance exam, dominates a student’s academic prospects, the system is unlikely to change. The Shanghai Daily has also recently covered the tension between the government efforts to relax education in the primary grades and parents’ concerns about their children’s futures, while Xinhuanet reports that China will be launching a national campaign to ease stress and move towards more comprehensive evaluations of student performance.

Teacher Quality

Both Australia and the Netherlands have produced new policy plans for improving the quality of their teaching forces in the past month.  In the Netherlands, the Ministry of Education, Culture and Science released Working in Education 2012, a policy document that calls for turning schools into professional organizations where teachers would have access to attractive career paths.  The government’s recommendations for improving teacher quality, to be introduced by 2016, include developing a competency document for each teacher that describes their skills and the activities designed to maintain and improve them, implementing a peer review system for teachers, and introducing performance-related pay pilots.

In Australia, both the state and federal governments are concerned with the issue of teacher quality.  In New South Wales, the Education Minister has announced that there will be a new minimum entry standard for teacher education programs.  The Australian reports that teacher candidates would need scores of at least 80 percent in three subjects on the high school leaving exams, including in English.  Another component of the new quality measures is an introduction of a literacy and numeracy test that teacher candidates must pass while they are completing their degree.  The federal government has also announced plans to require literacy and numeracy tests and an assessment interview for students entering teaching programs.  The Education Minister of New South Wales has stated that his state’s plan does not conflict with the federal plan, but would hold teacher candidates to a higher standard than the federal plan.

Early Childhood Education

In Singapore and Shanghai, the government and parents are increasingly focused on early childhood education (ECE).  Singapore plans to spend more than US$2.4 billion on preschool education in the next five years, which doubles their current investment, writes Channel News Asia.  In Shanghai, parents are so eager to enroll their students in early childhood programs that some have begun signing their children up for these programs on the day they are born.  According to the Global Times, however, this demand for preschool has created a boom in the private ECE sector.  This has led to concerns about the quality of the teaching staff and the programs offered, and calls for improved government oversight.

In order to successfully evaluate teachers, education systems first must work with key stakeholders to define what it means to be a good teacher and then develop clear standards for the profession, according to a new background report released by the OECD in advance of the 2013 International Summit on the Teaching Profession.  The report, Teachers for the 21st Century: Using Evaluation to Improve Teaching, provides an analysis of how the countries studied evaluate their teachers by identifying the elements of teacher performance that are most frequently appraised, the tools used in teacher evaluation and the ways in which evaluation results inform teaching and learning.

During a webinar which provided an overview of the report’s findings, Andreas Schleicher, Deputy Director for Education and Skills and Special Advisor on Education Policy to the OECD’s Secretary-General, emphasized that teacher evaluation by itself will not significantly affect student learning, but should be considered as part of a larger set of strategies: teaching must be an attractive career choice, high quality teacher education programs must be available to future teachers, teachers should be granted professional autonomy once they enter the classroom, effective in-service professional development opportunities must be provided and teachers must be active participants in the development of any teacher evaluation system.

Why Evaluate Teachers?
According to the report, teacher evaluations are mainly conducted for two reasons  — to improve teaching and learning and to provide accountability.  Formative evaluations are used to provide teachers with meaningful feedback that can inform profession development.  Summative evaluations can be used as the basis of accountability systems focused on individual teachers and are usually linked to some type of consequence for teachers such as career decisions or salary changes.

While many countries use evaluations for both summative and formative purposes, the report authors point out that the approach used for each should be quite different.  For example, if the goal of the evaluation is to improve teaching practices, then self-evaluations makes sense because teachers are more likely to admit their faults with the expectation that providing this information will lead to effective decisions about their developmental needs and future training opportunities.  However when the purpose is accountability and teachers face potential consequences concerning their career or salaries, self-evaluations do not work.  Summative evaluations need to have a strong external component, such as an accredited external evaluator, and a more formal process to ensure fairness. And, formative evaluations need to be more context-based, taking into account the unique circumstances surrounding the teacher’s history and the school’s setting.

What Elements Are Evaluated?  
During the webinar, Schleicher commented that education systems cannot improve what they cannot define.  Therefore, he said, standards that define what teachers need to know and be able to do are essential to developing effective evaluations systems.  The report emphasizes the importance of involving teachers in developing standards for the profession.  The process used to develop national teacher standards in Australia included a consultation phase that involved all key education stakeholders including teachers, teacher associations, teacher educators, employers, unions and regulatory authorities.  Similarly, in New Zealand, the professional body for teachers led the process of defining standards for the profession with the extensive involvement of teachers, employers and teacher unions.

The report found that the elements of teacher performance that are most frequently evaluated are related to planning and preparation, instruction, the classroom environment and professional responsibilities. Teacher evaluations could also take into account working in teams and managing and sharing leadership responsibilities.  In New Zealand, for example, teaching standards call for appraising professional relationships and values and responsiveness to diverse linguistic and cultural backgrounds.

Evaluation Methods
The report found that the most common methods used to evaluate teachers include classroom observations, teacher portfolios, self-evaluations and performance goals set by the individual teacher in agreement with school management.  Almost all countries use classroom observations to some degree.  As the figure below shows, an average of over 70 percent of high school teachers reported that classroom observations were considered to be of “high or moderate importance” in the teacher evaluations or the feedback they received.  However in using classroom observations as part of teacher evaluation, some experts advise avoiding announced classroom observations, because they do not provide an authentic experience of a teacher’s day-to-day practice.

In some countries, teachers must take tests to assess their general knowledge, but only two of the countries studied in the report, Mexico and Chile, use teacher tests to determine career advancement or dismissal.  A few countries also use surveys of students and parents as one element of gauging teacher competence.

Speaking to a point that is very controversial, the report makes it quite clear that it is challenging to identify the specific contribution that a given teacher makes to a student’s performance.  Student learning is largely influenced by student’s innate abilities, motivations and behaviors and the support students receive from their family, peer group and school.  And students are influenced not only by their current teacher, but also by their former teachers.  The report explains that while value-added models can control for a student’s previous results and have the potential to identify an individual teacher’s contribution to student performance, there is wide consensus in literature that these models should be used only in addition to other evaluation measures.  The report also contends that using student results as an evaluation instrument is likely to be more relevant for whole-school evaluations than for individual teacher evaluations.

Delaware is featured in the report for their work on incorporating student outcomes into teacher evaluations.  The state’s system calls for teachers to use three measures of student progress including performance on state tests, test results on an instrument other than the test used for state accountability and goals for student progress developed by the teacher.  During the 2011-2012 school year, Delaware engaged hundreds of teachers in developing a wide-ranging library of resources that supports implementation of the new policy.

While there is no consensus on the right types of evaluation methods to use to evaluate teachers, the report makes it clear that using several methods is essential to drawing a comprehensive picture of teachers’ abilities.  The Measures of Effective Teaching (MET) study, funded by the Bill and Melinda Gates Foundation, is one of the most sophisticated analyses to-date on how evaluation methods can be used to identify the skills that make teachers effective.  The findings stress that assigning equal weights to multiple measures creates a more accurate assessment of teacher effectiveness than other models in which one measure is given a greater weight over others.

Who Conducts Teacher Evaluations?
While this varies across countries, the most common bodies that conduct teacher evaluations include inspectorates, professional teacher organizations, unions, school leaders and peer teachers.  The report recognizes the importance of using multiple evaluators to assess teacher performance to provide different perspectives.  For example, while external, highly trained evaluators assess teacher performance as accurately as school heads or principals, school leaders have the benefit of being more aware of variables in the particular school context that may affect a teacher’s performance.  On the other hand, some researchers have found that while principals may be able to successfully identify the high- and low-performers, they are unable to distinguish between teachers in the middle of the performance distribution.  Regardless of who is conducting the evaluation, the report notes that, “the effectiveness of appraisals crucially depends on whether evaluators have the knowledge and skills to evaluate teachers reliably in relation to established criteria,” so it is very important that all evaluators receive proper training.

How Are Evaluation Results Used?  
The results from teacher evaluation systems are used in a variety of ways including informing teacher practice; designing professional development opportunities that address teacher shortcomings; establishing rewards and consequences based on evaluation results; and developing lines of communication so the information gathered can inform education policy.

Results from a 2008 teacher survey found that over 40 percent of teachers reported that they did not receive suggestions for improving their practice after an evaluation and 44 percent agreed that teacher evaluations were conducted merely to fulfill an administrative requirement.  During the webinar, Schleicher said that it is very important for teachers to see teacher evaluations as a basis for professional support and career development.

The report also found that of the countries studied, very few use teacher evaluations to reward high-performing teachers with salary increases.  In the chart above, of the countries surveyed only Chile, Korea and Mexico have these types of policies in place.  When countries do use teacher evaluation results to reward teachers, few provide teachers with career advancement opportunities.  Because the organizational structure of schools in many OECD countries is typically flat, with few opportunities for teachers to be promoted or to gain increased responsibilities, the report recommends that education systems should look to high-performers such as Singapore for guidance in using teacher evaluation for career advancement.  This city-state has established a robust appraisal system that is linked to defined career ladders.  Singapore has created career structures at all school levels providing a teacher with the opportunity to advance to master teacher status or move into administration or research and policy.  And as Singaporean teachers move up the career ladder, they are rewarded with higher compensation levels.

The report, Teachers for the 21st Century, is largely based on two prior OECD reports: the Review on Evaluation and Assessment Frameworks for Improving School Outcomes, a 2009 study that involved 24 countries and looked at the various components of evaluation and assessment strategies that countries use, and the latest edition of the Teaching and Learning International Survey (TALIS), published in 2008. To access the new report visit:
http://www.oecd.org/site/eduistp13/TS2013%20Background%20Report.pdf.

In their September 2012 Global Employment Outlook, the International Labour Organization (ILO) drew particular attention to the plight of the young worker worldwide.  They project that the global youth unemployment rate (youth being defined as between the ages of 15 and 24) will climb from 12.7 percent in 2012 to 12.9 percent by 2017.  This is in contrast to the overall unemployment rate, which is expected to remain steady worldwide at 6 percent between 2012 and 2017.  The projected rates of youth unemployment vary, of course, by region.  In East Asia, the youth unemployment rate is projected to increase to 10.4 percent by 2017, up from 9.5 percent, while in the developed economies and the European Union, the rate is actually projected to decline from 17.5 percent in 2012 to 15.6 percent in 2017.  However, the latter figure is not actually cause for celebration – the report notes this is “principally because discouraged young people are withdrawing from the labor market and not because of stronger hiring activity among youngsters.”

We turn to additional ILO data to see what the picture looks like in some of the countries with top-performing education systems, to see if the strength of the primary and secondary systems mitigates to some degree the proportion of young people who are struggling to find work (Figure 1).  The results are somewhat surprising.  Finland, widely acknowledged as having one of the best primary and secondary education systems in the world, also has the highest unemployment rate for people aged 15 to 19 years, and one of the highest unemployment rates for people aged 20 to 24 according to the ILO data.  Singapore and the Netherlands, which have strongly integrated vocational and technical pathways available to students before the age of 18, on the other hand (and unsurprisingly), have quite low youth unemployment rates.

Figure 1

But before jumping to conclusions, it is important to dig deeper into how countries define youth unemployment, because this in and of itself can impact how well a country appears to be doing in terms of moving young people into the workforce.  For the chart above, the ILO definition of “unemployed” included people who were not in paid employment, were available for employment, and were seeking employment.  The ILO points out that these measures are difficult to compare across countries because education systems vary widely, and in some countries a young person may be considered “employed,” for example, if they are engaging in a vocational training program part-time.  In another country, the labor force may be considered as including only the youth who have dropped out of secondary school or who have earned a secondary degree.  This may result in inflated rates of “unemployment” in some countries, for example, Nordic countries, that have more modular vocational and post-secondary education programs and other strong supports for young people, resulting in young people pursuing a combination of part-time training, employment, or other activities such as international travel before settling into a career.

Fortunately, there is another international measure that allows us to compare the proportion of young people who are struggling to enter the workforce or the education sector.  That is the percent of youth not in employment, education or training, often abbreviated as NEET.  The OECD provides data on the percent of NEET youth in most of its member countries; below, we have again shown the data for the top performers (Figure 2).  The chart provides information for three different categories of young people: youth who are unemployed (that is, looking for work), and not in education or training; youth who are inactive (that is, not looking for work), and not in education or training; and the NEET rate, which includes youth who are either unemployed or inactive, and not in education or training.  The NEET rate is represented by the total length of the bar on the chart, as it is a combination of the two other measures.

Figure 2

The Netherlands, which has one of the lowest rates of youth unemployment by ILO measures, also has a very low NEET rate.  Notably, just 1.5 percent of youth in the Netherlands who are not in education or training and are actively seeking work are unable to find jobs.  This is just over 25 percent of the overall OECD rate of 5.8 percent, and significantly smaller than the EU27 (European Union) rate of 6.6 percent.  Denmark and Finland, two Nordic countries which, by overall youth unemployment measures, do not look particularly good, also have very low NEET rates.  These low rates are likely due to the fact that these countries, and particularly the Netherlands and Denmark, have very strong school-to-work pipelines, with multiple pathways for all types of students.  Students in these countries have access to various workplace learning experiences and apprenticeships, as well as a close relationship between industry and these training programs.  On the other end of the spectrum, the United States, New Zealand and the United Kingdom all have high NEET rates in addition to their high youth unemployment rates, suggesting that job training programs or pathways into the workforce in these countries are lacking.

One concern, however, is the possibility of a growing connection between youth unemployment rates and youth NEET rates.  The ILO points out in their Global Employment Outlook that as new economic sectors grow and old sectors decline, people who were either employed in or being trained for jobs in the old sectors will face the loss of these jobs with a sense of discouragement, meaning that NEET rates will rise following the rise in unemployment rates.  This is why it is so important to have education connected to current workplace skill requirements, and particularly, to ensure that vocational and technical education programs are linked closely to industry, so that youth are being prepared for the jobs of the future.

Earlier this month, the Center for International Education Benchmarking introduced a new feature on its web site.  The Study Guide is intended to provide readers with a weekly summary of headlines from the top-performing education systems in the world.  This month’s International Reads highlights some of the most important current issues in the news.

Teachers.  In Japan, changes to teachers’ retirement packages have caused many teachers to think about retiring early before the changes go into effect.  Japan education officials are struggling to figure out a work-around to ensure that students are not left without a teacher for the remainder of the school year, according to The Mainichi.  A Valentine’s Day strike was held by educators in Victoria, Australia in protest of the pay package offered by the state—a modest 2.5 percent a year plus performance-based pay.  The Australian Education Union is demanding a 12 percent raise over three years with no performance-based pay.  But on the issue of improving the quality of the pool entering teaching, the teacher’s union and the government are on the same side—recently, Education Union federal president Angelo Gavrielatos issued a statement supporting the government’s initiative to raise the quality of students entering the teaching profession.  In Ontario, newly elected Education Minister Liz Sandals is facing teacher dissatisfaction.  Last fall, legislation was passed banning teacher strikes in the province.  In January, the new law was used to impose a contract on public secondary school teachers.  In response, teacher unions have asked their members to refrain from supervising extracurricular activities, which they see as outside their regular duties.  Sandals said her first order of business is to ensure that new teacher contracts are the result of negotiation, not legislation.

Early Childhood Education.  In Japan, government officials are considering offering free pre-school to children ages 3-5, in an effort to ease the financial burden on families, according to Inside Japan.  This proposal comes at the same time that President Obama called for free pre-school for all 4-year-olds at or below 200 percent of the poverty line in the United States during his State of the Union address.  Parents in Hong Kong are focused on another concern — that kindergartens are emphasizing grades and tests too much.  In response, they are leading a movement to shift to kindergartens that emphasize learning through play, according to the South China Morning Post.  And in New Zealand, the Pasifika Education Plan 2013-17 aims to lift Pasifika participation in early childhood education from its current rate of 86.8 percent to 98 percent by 2016, according to Radio New Zealand International.

Post-secondary Education.  A February 3rd editorial in the Japan Times calls for major changes in the country’s university entrance exam system, arguing that the current assessments measure knowledge acquired rather than deeper comprehension, aptitude and potential.  Meanwhile, researchers at the Hong Kong Institute of Education (HKIE) say educational inequality is getting worse, despite the increased number of publicly funded university places.  A recent HKIE study found that students from wealthy families are nearly four times more likely to enroll in a university than those living in poverty.  That’s a much wider gap than 20 years ago.  The Netherlands is looking to the liberal arts model to solve some of their higher education challenges related to a lack of differentiation and excellence.  Inside Higher Ed reports liberal art schools (known there as university colleges), “have had an outsized impact on Dutch higher education policies and practices, inspiring the growing movement toward selective admissions and the development of ‘excellence’ programs within a famously egalitarian higher education system.”

Choice and Charters.  The Education Amendment Bill, introduced last year in New Zealand, would create legal recognition of charter schools there.  However, the Treasury has found evidence that school systems using strongly competitive elements do not produce systematically better student outcomes and other critics are arguing that charter schools will take public money but be free from government scrutiny.  While charter schools are not prevalent in Canada, school competition does exist in the form of four separate publicly funded systems catering to the English and French non-religious and Catholic constituencies of Ontario.  With birth rates in that province on the decline, schools are struggling to keep enrollment levels high. But the schools don’t just compete for students in name only: recently, schools have taken to touting their extracurricular programs in advertisements in local media and attending other schools’ open houses in order to gain an edge.  For more on this story, see the article in The Globe and Mail.

International Benchmarking.  And finally, Shanghai is looking forward to the December publication of the 2012 PISA results to show the world that, once again, they are on top of the international education league tables.  According to Shanghai education officials interviewed by the Sydney Morning Herald, “tests recently conducted for the next PISA report…will show Shanghai students have further improved their results and consolidated their lead in the world.”  In the same article, Deputy Director of the OECD Education Division Andreas Schleicher says, “Maybe it’s time to change some of our stereotypes.  What you see today in the school system in Shanghai is what you are going to see in the labour market tomorrow.”  Learn more about the results from the most recent TIMMS and PIRLS international assessments by clicking here.

Check back to our web site on a weekly basis for more education news from the top-performing education systems in the world.

Backtracking from a proposal to do away with the General Certificate of Secondary Education (GCSE), the set of examinations that English secondary students take from age 16 on, Education Minister Michael Gove announced February 7 that the government would not establish the EBAC or English Baccalaureate, a plan that had been introduced by him in September.  Under pressure from Ofqual, the body that regulates school qualifications in England, along with teachers unions and school administrators, Gove dropped the plan to allow only one organization to produce exams for each subject instead of allowing many to do so, as has long been the practice.  But he will still push for some of the other elements originally found in his EBAC proposal.  He wants to get rid of coursework (work done on assignments given by teachers and graded by them) and modules (the ability for students to work on small chunks of curriculum and then be tested on those topics).  He wants to give exams at the end of two-years of study in the core subjects instead of at the end of one year, add extension papers in mathematics and science for “the brightest students”, and, most significantly, change the definition of school success when reported to the public through league tables.  The Minister has proposed to get rid of the current league tables where schools are evaluated based on the percentage of students with 5 or more GCSEs scoring at a C or above.  His plan calls for schools to be evaluated on two measures: the percentage of students passing English and mathematics and a report on the progress that students make from year to year in eight GCSE subjects.

In addition to the teachers unions, Ofqual and the school administrators association, the Education Select Committee in Parliament also weighed in against Gove’s proposal.  The Conservative PM that heads the committee argued that if the country were to institute the qualifications system Gove envisioned, which would have given students who did not complete their EBAC a “statement of achievement” rather than a qualification, those students would be given “a badge of failure” hurting less able students rather than helping them.

For more information on EBacc, visit the Department for Education website.  Gove’s latest proposal, which includes a new national curriculum emphasizing the traditional subjects taught in a more structured way, would go into effect in schools in 2015 with the first examinations to be given in 2017.

I just finished reading a recent book from Routledge, Teacher Education Around the World, edited by Linda Darling-Hammond and Ann Lieberman.  It is a very rewarding read, full of new information and fresh, insightful analysis.  The editors asked an impressive team of researchers to do chapters on Finland, Singapore, the Netherlands, the UK, Hong Kong and Canada.  Darling-Hammond did the chapter on the United States and the two editors pulled the threads together in the last chapter.

Two things jump out at this reviewer.  The first has to do with the scope of the changes taking place around the world as nation after nation concludes that better teachers are the key to their goals for their students.  The other has to do with the nature of the battle for the souls of policymakers over the best strategy to do that and the way policymakers are responding to the contending forces.  I’ll tackle the latter first and then return to the former.

As Darling-Hammond and Lieberman see it, the policy battle taking place around the globe is being fought between those who believe that the effort to professionalize education has failed and those who believe that that effort has only just begun, that student performance will improve radically only if teaching is converted from a blue-collar occupation into a true profession.  More accurately, perhaps, this battle pits those who believe that teaching can and must have all the attributes of a true profession against those who think it neither can nor should do so.

In a way, this is a battle about who is entitled to wear the badge of the reformer.  Through one lens, reform is converting teaching from an occupation requiring little technical knowledge and expertise into a true profession requiring a good deal of both, and, on the other reform is battling an entrenched education bureaucracy with all the tools that market forces and the entrepreneurial spirit can bring to the revitalization of moribund industries.

To some extent, of course, this battle involves the facts.  The first question here is whether those who want to turn teaching into a true profession are making any progress.  Darling-Hammond, Lieberman and their coauthors paint a detailed picture of changes taking place along a broad front within the professional education community over the last 20 years or so, all designed to raise the quality of the pool of young people from whom new teachers are selected, improve their mastery of the subjects they will teach, help them better understand the way young people grow and develop, learn their craft, and practice that craft under the supervision of first rate teachers until they either become first rate teachers themselves or leave teaching.

All of this makes sense, of course, only if one believes that there is a substantial body of professional knowledge and practice that must be mastered if a raw recruit is to become a good teacher, above and beyond the knowledge of the subject one is going to teach.  If you believe that, then reformers are the people who put in place all the elements needed to turn teaching into a true profession.

One of the most important hallmarks of a true profession is the presence of sound professional standards, so I was particularly interested in the way the authors show how new standards—for entering teacher education institutions, for accrediting teacher education institutions, for licensing teachers, for determining who advances up the newly created career ladders and for awarding certification to advanced professionals—are providing powerful drivers for the whole new system of development of high quality teachers in a growing number of countries.  They point out that, whereas teachers used to advance through their own education and training and sometimes through their subsequent careers on the basis of courses taken, they now advance on the basis of careful, sophisticated assessments of their actual performance, such as those recently developed at Stanford University.

The book shows how the most advanced countries have worked hard to identify and use the best research on the factors that make for great teachers, much of it done in the United States, but also to provide teachers with important research skills, enabling them to constantly improve their own practice in a disciplined way.  They show how a new breed of school is developing around the world that serves as an analogue to the teaching hospital in medicine.  The best of these institutions is clearly changing the university at least as much as the school, resulting in a constant dialogue between clinical faculty and research faculty in which both together create the curriculum for the education and training of new teachers and find a way to blend theory and practice in a way the makes the former come alive and that provides insight into the latter in a much more powerful program of instruction than was previously available at these institutions.

The picture one gets from this book of the broad upgrading of the selection, preparation and support of new teachers is nothing if not varied.  The details of how these countries are going about this transformation are very different, and there is no doubt that that fact will enable us all to learn a lot from the variation.  But the themes are clear.  Teaching is increasingly viewed as a profession like other professions.  That means that:

• there is an important body of professional knowledge and practice to be acquired over a period of years,
• so much discretion is required in its application that the trained professional must be trusted to apply that knowledge with wide discretion in the workplace,
• the work needs to be regulated, but the regulations need to be based on professional standards and those standards must come from the profession itself, and
• the advancement of practice will come only with more and better research, the results of which are incorporated into the training of the professionals and the support provided to the professionals as they constantly seek to improve their practice.

From the point of view of the authors of this volume, that is what it means to be a professional, and turning teachers into true professionals is the only way to create mass education systems capable of educating virtually all students to global standards.

And then there is the other camp.  They see all this as a thinly veiled attempt by a failed bureaucratic establishment to hang on to the old ways.  If teacher educators knew how to or even wanted to improve their appalling performance, they would have done it years ago.  No self-respecting high school student who could get into a first-rate university would choose to go to a school of education, which will let anyone in and provides a program with standards so low that no one ever fails.  This camp is very fond of pointing to actual examples of very highly qualified research scientists willing to become high school teachers in their retirement, but who cannot do so because they do not wish to take the intellectually vacuous courses and mindless tests required by the teacher training institutions and the state to become a teacher.

To the people in this camp, it is obvious that there is no craft of teaching that rises to the level of serious intellectual activity.  What is needed are young people and older people who can demonstrate that they know the subject they are expected to teach and the rest will take care of itself.  The way to get the teachers we need is to break the hammer lock of the establishment on teacher training, and open the training of teachers to anyone or any institution prepared to let the market decide whether their product is worth hiring.  The market, in other words, can bring in strong competition for the established institutions and do what markets do best: drive costs down and quality up.  The people in this camp celebrate Teach for America and its relatives in several other countries, because they have succeeded in bringing some of America’s most capable young people into teaching—if only for a couple of years and in very few classrooms—by requiring only a few weeks of teacher training.  All over the world, the people who hold this view are championing policies that allow many kinds of institutions to train teachers, and reduce the training that new recruits get in the craft of teaching and in the research on student learning to a minimum.  It is, I think, not unreasonable to conclude that the people in this camp do not believe that there is, properly speaking, a profession of teaching, but rather that teaching is an occupation or a calling, but not a profession.

What is particularly interesting about this clash as portrayed in this book is the way this conflict is playing out country by country.  The authors present both Singapore and Finland as wholly in the first camp, with policies that are internally consistent, all of which reflect a commitment to the idea that teaching is and ought to be a profession, for which people are selected as professionals, trained as professionals, supported as professionals and managed as professionals.

But the authors show that, after that, the picture on the ground is much more mixed.  If one end of the dimension line is represented by Finland and Singapore, the other is represented by the United States and the UK.  In between, they show us countries in which both sides of the conflict have won their policy battles.  In those countries, we see a real effort to put in place policy measures intended to build a true profession of teaching right alongside others that make it possible for individuals to minimize or even eliminate the training required to become a licensed teacher, the standards for which are being raised in other statutes on the books of the same country.

One gets the sense that the world is in a race.  On one side are those hoping to strengthen the profession of teaching and, on the other, are those who are seeking to blow up the very institutional structure the former are trying to build.  If those who are trying to professionalize teaching succeed fast enough, they will invalidate the case being made by those who are trying to blow up the establishment.  Because education is an inherently conservative enterprise, they may get the time they need. But, if they take too long to reach their objective, or their methods are sufficiently weakened by the other side along the way, they will lose and those who believe that market forces are all, or almost all, of what is needed may prevail.  And then it will be most interesting to see which countries are most successful in educating their children.

By Marc Tucker

My apologies to Maimonides.  But I would not blame you if you were perplexed about the recent dust-up after the latest PIRLS and TIMSS data came out.  Some of the best-known names in education research worldwide came out with guns blazing, mostly at one-another, in a rapid-fire exchange about what the numbers meant.  I thought some of you might welcome a guide to the shooters and the shots, and a bit of commentary on the profound meaning of it all.

Tom Loveless, the head of the Brown Center at the Brookings Institution jumped on the data to say that they called for a “rethinking of the Finnish miracle success story….If Finland were a state taking the 8^th grade NAEP [the sample survey used in the United States to monitor the progress of American students over time], it would probably score in the middle of the pack.”  Jack Buckley, Commissioner of the National Center for Educational Statistics in the U.S. Department of Education confessed that, “I’ve always been a little puzzled” by the high level of attention paid to Finland.  Well, so much for Finland!

Martin Carnoy and Richard Rothstein wrote an analysis of the data claiming to show that while reading achievement of American students on PISA was growing between 2000 and 2009, it was falling by an even larger amount in Finland.  Similarly, they said, in math, US students from the lowest social class were also gaining substantially, while scores of comparable Finnish students declined.  “This is surprising,” they said, “because the proportion of disadvantaged students in Finland also fell…” And they go on to say that, by their analysis, the achievement gap between the most and the least advantaged students in the United States is actually smaller than in “similar postindustrial countries, and often only slightly larger than gaps in top-scoring nations.”

Ha!  That means that the withering criticism showered on American schools for their poor performance was totally undeserved.  The problem, if there is a problem, lies not in the schools, poor Horatio, which have been doing a much better job than anyone has given them credit for, but in the enormous disparities in family income that have opened up in American society.  And Finland, according to this analysis, hardly deserves its status as the model that the United States should be adopting.

Not so fast, say Paul Peterson, Eric Hanushek and Ludger Woessmann.  Peterson is at Harvard, Hanushek at Stanford and Woessmann at the University of Munich.  The data, they say, don’t show that at all.  What they actually show is that, even if such corrections are made, American students at the top do not perform anywhere near as well as the students in the top performing countries, or, at least, not such a high proportion of them do.  Things are just as bad as they always said they were, and the need to turn up the heat on the schools to perform up to international standards is as great as ever.

But wait a minute, says Andreas Schleicher.  The Carnoy-Rothstein analysis depends, he said, on a challenge to the methods used by OECD-PISA to do its survey research, and that challenge, says Schleicher, just won’t hold up in court.  To which Carnoy and Rothstein said in reply to the reply, Oh yes it will.

So what is going on here?  Why are all these people so exercised about this data?  What are their agendas anyway?  Who is right and who is wrong?  Why does it matter?  And what does it mean?

I know that research is supposed to go where the evidence leads it and the researcher is only there to record the ineluctable result, without fear or favor.  But the reality is that researchers have values to support and reputations to protect, and their conclusions are more often than not influenced by both their values and the reputations they have established as a result of the policy positions they have taken.  So, perhaps it would help to sketch in the positions taken on the relevant issues by the people I have named.

It should surprise no one that spokespeople for the Brookings Institution and the United States National Center for Educational Statistics should be waiting to pounce on Finland and on the people who have used the Finns’ standing in the international league tables to make a case for using the educational strategies the Finns have embraced.  Both Brookings and a series of U.S. Department of Education research executives, some of whom have gone to Brookings when they left the Department of Education, have been deeply skeptical of international education benchmarking and ardent advocates of what they have described as the “gold standard” of education research, meaning the use of experimental research techniques as the only legitimate way to attribute cause in social research.  It is obviously impossible to randomly assign national “treatments” to national populations in the arena of education, so, from their point of view, all statements that this or that set of policies “causes” these or those national outcomes in the arena of education policy are necessarily suspect.

Brookings and the Peterson, Hanushek, Woessmann team are both strong supporters of charters and the introduction of market forces generally as school reform strategies.  Brookings, as well other Washington-based think tanks, are eager to deflate the recent enthusiasm for international education benchmarking in part because they fear that the close examination of the strategies used by the top-performing countries will show little evidence that charters or market strategies in general are effective strategies for raising student achievement at a national scale.

Peterson, Hanushek and Woessmann each have their own views on what is most important in education reform, but all are advocates of charters and reform agendas based on market forces, and all appear to believe that it will take fear of foreign competitors to put this reform agenda over the top in the United States.  They have also done research that they say supports their claim that market strategies do work in the top-performing countries.  Implicitly, then, they believe, unlike their Brookings colleagues, that it is possible to do rigorous research using comparative data gleaned from these international surveys that attributes cause and from which, therefore, it is possible to draw policy conclusions.  This team of researchers has consistently advanced the view, like my own organization, that economic ruin will be the fate of any nation that fails to hold its own in international education competition, though their prescriptions as to the most effective policy agenda are different from our own, based on the study of pretty much the same data.

But Carnoy and Rothstein come from a very different place.  They believe that the relatively poor performance of American students on the international surveys of student achievement is a function of the large and increasing disparity in incomes among Americans, in absolute terms and in relation to other countries.  They are outraged that organizations like my own and researchers like Peterson, Hanushek and Woessmann hold the schools accountable for poor student performance, when they think the fault lies not in the schools and teachers, but rather in a society that tolerates gross and increasing disparities in income among Americans.  They would have us focus on promoting policies that would result in a fairer distribution of income in the United States.

Which puts them in direct conflict not just with Peterson, Hanushek and Woessmann, but also with Andreas Schleicher, the driver of the whole PISA system at the OECD.  Schleicher’s primary framework for the analysis of the PISA data displays the country data on two axes, one for student achievement on the subjects assessed by PISA and the other for equity, the pattern of the distribution of results from the poorest to the best performers within countries.  Countries with short tails in that distribution are described as having high equity; those with long tails are described as having low equity.  Schleicher points out that the United States just barely escapes being among those countries in the worst quartile on both measures.  Another table in Schleicher’s slide deck shows that, when socio-economic status is held constant, the schools of some nations do a much better job than others of reducing achievement disparities among students.  Carnoy and Rothstein would take American teachers off the hook, saying that the performance of poor and minority students is actually improving, the gap is not so large as was thought, and the performance of poor and minority students in the top performing countries is actually declining.  To the extent there is a problem, it is a problem caused by socio-economic status of the students, not the teachers’ performance.  Schleicher would say, no, that is not so.  Even when we look at students from comparable socio-economic backgrounds, American schools do less to close the gap with the students from more favored backgrounds than schools in most other countries.  They cannot both be right.

So it is no wonder that Carnoy and Rothstein go after Schleicher and his data and methods with hammer and tongs.

So who is right and who is wrong here?  All of the people I have named are competent researchers from well-regarded institutions.  Just as each of these people have their own values and established positions on the relevant policy issues, the same is true of me and the organization with which I am associated.  Our analysis of the dynamics of the global economy strongly suggests that high wage countries like the United States will find it increasingly difficult to maintain their standard of living unless they figure out how to provide a kind and quality of education to virtually all their children that they formally thought appropriate only for a few.  And we also believe that the most likely source of good ideas for strategies that will enable them to do that is the countries that have already done it.  We think that whether the source of poor performance is mainly growing disparities of income or relatively poor performance of the education system, the dynamics of the global economy are unforgiving and countries like the United States do not have the option of saying that the educators can do nothing, that the only thing that will save us is income redistribution.  We do not think that the only way to learn what strategies are likely to work is research methods derived from the experimental sciences.  Indeed, we think that the record clearly shows that American business recovered from a devastating assault from Japanese firms in part by inventing and using the very method—industrial benchmarking—that we and others are now using in the field of education.

To me, the most important conclusion to be drawn from the debate whose contours I have just rather roughly outlined is that now, for the first time in the United States, the international surveys of student achievement really matter.  That is a big, big change.  It was not the case before that advocates of the most hotly debated education reforms in the United States felt that they needed to take the data from these surveys seriously, to defend their positions or to advance them.  Clearly, they do now.

The second point is that the data from the international surveys is being used to make points not about peripheral issues, but central issues.  It really matters whether the cause of the United States’ relatively low standing in the international league tables is income disparities among the students’ families or poor education in the schools.  It really matters whether or not countries like Finland have important lessons for the rest of the world.  It matters whether the survey methods being used by the organizations that design and administer them bear up to scientific scrutiny or not.  And, lastly, it also matters whether the methods used by those who do research comparing the effects of different policies and practices on student achievement in multiple countries have enough scientific merit to justify their use by policy makers to make national policy. These are consequential questions.  This is the first time that we have seen a sustained debate by some of America’s leading scholars on these matters.  It is not likely to be the last, and that appears to herald an era in which, for the first time in the United States, international surveys of student achievement are likely to take a prominent place in the public debate about education policy.
You may be wondering where I come out on the welter of claims and counterclaims I described above.  Now that I have laid my analytical framework on the table along with those of the other analysts, you are in a position to apply the same dose of skepticism to my conclusions as I urged you to apply to the others.   My take on the data we now have in hand is more or less as follows.

First, the usual note of caution.  One snapshot does not a movie make.  We should not declare a trend before we have more than one data point.  So we might want to see whether the changes in rankings suggested by the recent PIRLS and TIMSS data hold up over time.

Second, as many have pointed out, TIMSS and PIRLS put the accent on measuring how students do on what amounts to a consensus curriculum.  Did they learn what international experts think they should have been taught in the subjects they assess?  PISA measures the capacity of students to apply what they have learned in the classroom to proxies for real-world problems of the sort they might actually encounter outside the classroom.  I have a strong preference for the latter goal over the first, which mainly comes from an experience I had years ago, when Archie Lapointe, the director at that time of the Young Adult Literacy Survey, told me the following.  The survey asked the young people surveyed to add a column of figures and take a percentage of the result. Almost all could do it.  It also asked the same respondents to take a restaurant check, add up the items, get a total and calculate a tip.  Very few could do it.  Like Alfred North Whitehead, I have very little use for what he called “inert knowledge.”

Third, we need to keep in mind that the fine-grained distinctions in the rankings, for most countries that are near one another, are not statistically significant.  What we should really be paying attention to is the groupings of countries in the rankings, when countries are grouped in such a way that the measured differences among the groups are statistically significant.  If you look at it from this perspective, what we see is the United States still has a long way to go before the vast majority of its students score in the front ranks of performance at many grade or age levels in many subjects, which is how I would define top performers.

Fourth, I think it is pretty clear from the OECD data that smaller proportions of American students score in the higher deciles of performance on the PISA tests, and more in the lower deciles than is the case for students from the top-performing countries.  If that is true, then it cannot also be true that the United States would do as well as the top-performing countries if only the poor, Black and Hispanic students were taken out of the rankings, as many American teachers and some policymakers maintain.  It is also clear from the OECD-PISA analysis, as I pointed out above, that, when the data are corrected for students’ socio-economic status, American schools are less effective than the schools of most of the countries measured at closing the gap between these students and students with higher socio-economic status.

This, of course, is not where Carnoy and Rothstein come out, but I think Andreas Schleicher won the battle between him, on the one hand, and Carnoy and Rothstein on the other.  But don’t take my word for it.  Read the claims and arguments made by both sides carefully.  There is a lot at stake in this conflict.

So, what then are we to make of the fact that, if Massachusetts, North Carolina and Florida were countries, they would have done very well indeed in the most recently released rankings?

The case of Florida, I think, is pretty straightforward.  The Florida Center for Reading Research, administered by Florida State University, is one of the nation’s leading centers for reading research.  Its methods are widely admired throughout the United States.  The state of Florida has managed to leverage this research program and its key figures to produce widespread implementation throughout the state of the methods advocated by the Center.  We can see the results in the PIRLS fourth grade reading results.  The question, of course, is what effect, if any, this will have on student performance in the upper grades as the students who have benefitted from these programs mature through the years.  That story has yet to be told.

In North Carolina, we are looking at a program of education reform that began with Governor Terry Sanford, whose first term as governor began in 1961.  Sanford’s unrelenting emphasis on improving education in the state laid the base for Governor James B. Hunt, Jr., who served as governor from 1977 to 1985 and again from 1993 to 2001, making him the longest serving governor in the state’s history.  Through that whole period, he never lost his focus on education as the key to the state’s economic growth, and, during that period, North Carolina showed more progress on student achievement as measured by the National Assessment of Education Progress than any other state in the United States.  Hunt’s agenda for education reform was profoundly affected by what he was learning about the strategies adopted by the top-performing countries in the world.  Like them, he focused on teacher quality, high quality instructional systems and early childhood education.  North Carolina was among the very first states in the United States to send delegations of key state policy-makers abroad to study the top performers.

Massachusetts is a similar story.  In this case the first phase of the reforms were driven by the business community, organized by Jack Rennie, a very successful businessman who worked hard to organize that community, and Paul Reville a public policy analyst.  They played the key role in pushing the Massachusetts Education Reform Act of 1993 through the legislature.  The Act provided hundreds of millions in new funding for the schools in exchange for explicit performance standards for students, set to international benchmarks and carefully drawn curriculum frameworks, also set to international benchmarks; a new comprehensive assessment system set to the standards and curriculum frameworks; much tougher standards for getting to be a teacher, intended to greatly ratchet up teachers’ command of the subjects they intended to teach, and a system to disclose student performance, school by school, with results reported by student subgroups, so that poor performance by these subgroups would not be hidden in the average scores for the school.  Right after the Act was passed, David Driscoll, until then the Deputy Commissioner of Education, was made Commissioner and remained in that position for ten years.  Under Driscoll’s leadership, Massachusetts, despite a great deal of pressure to do so, never backed off of its decision to set and to maintain internationally benchmarked standards, for both student performance and teacher certification.  After Driscoll left, the new governor created a new position in state government, to provide leadership to all the parts of government concerned primarily with education at all levels.  He filled that position with Paul Reville.  Between them, Driscoll and Reville provided the same kind of strength and continuity of leadership that Governor Hunt provided in North Carolina, and for a very similar agenda, an agenda that is in many respects consistent with our own analysis of the strategies used by the top performing nations to get to the top of the league tables.

You may or may not agree with my analysis of the kerfuffle over the release of the TIMSS and PIRLS results.  You may or may not agree with my explanation for the rise of Florida, Massachusetts and North Carolina on the PIRLS and TIMSS league tables.  But, in any case, I urge you to look at the contending papers, and come to your own conclusions.  All of us could benefit greatly from a long, loud, contentious effort to define what it means to be educated, and to better understand why some nations are more successful than others at educating the vast majority of their young people to whatever standard they choose.

* This is a second version of the original post for this month.  We misstated the conclusions presented by Martin Carnoy and Richard Rothstein in the report described in this newsletter.  We believe we have stated those conclusions accurately here, and apologize to the authors for the error.

For the record, however, the version of the Carnoy-Rothstein conclusions that we based our first statement on was itself based on the version of the report that Carnoy and Rothstein originally released, which claimed that their re-estimate of United States PISA scores would result in the United States ranking 4^th among OECD countries in reading, and 10^th in math, a major revision upwards of the US PISA rankings.  In their most recent version of their report, released last week, Rothstein and Carnoy revised these numbers downward somewhat to 6^th in reading and 13^th in math, but, as the post points out, even these numbers are contested.

In December, the results of the 2011 administration of the Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS) were published in three separate reports, each examining international performance in reading (at the fourth grade level), math (at the fourth and eighth grade levels) and science (at the fourth and eighth grade levels).  These assessments provide a picture of international student performance in the years before a student reaches the age of 15, which is the age at which students take the OECD’s Programme of International Student Assessment (PISA).  However, there are some central differences between the TIMSS/PIRLS and PISA assessments.  Michael Martin, the Co-Executive Director of the TIMSS and PIRLS Study Center at Boston College, notes that while PISA is intended to measure a student’s general skills in the arenas of reading, math and science, TIMSS and PIRLS are more focused on content mastery.  Additionally, Jack Buckley, the commissioner of the National Center for Education Statistics, has pointed out that the countries participating in both assessments do vary – the TIMSS and PIRLS groups are smaller and represent a mixture of countries at different levels of economic development as compared to the participants in PISA.

Because of the differences between the assessments, the countries that are in the top ten or fifteen of the TIMSS and PIRLS rankings are somewhat different than the top performers on the last incarnation of PISA in 2009.  While league tables of the top countries based on their average scores always garner the most press when the results of international assessments are released, we decided to take a more in-depth look at what level of proficiency students in the top fifteen countries are actually reaching in these subjects.

The IEA has established four “international benchmarks” on their score scale for these assessments.  While the score scale for both PIRLS and TIMSS runs from 0-1000, the vast majority of scores fall between 300 and 700.  The IEA has identified a score of 400 as the “low” international benchmark, indicating that students at this score point have been educated to a “basic” level.  Beyond that, there is a score of 475, or “intermediate;” a score of 550, or “high,” and a score of 625, or “advanced.”  Below, we have plotted the percent of students at each benchmark in the top fifteen countries on the 2011 administration of PIRLS and TIMSS.  This is useful when thinking about the top performers, because it shows, in a clearer way perhaps than the average scale score, what students in each country are really able to do.

In the fourth grade PIRLS reading assessment, a student who reaches the “low” international benchmark is able to “locate and retrieve an explicitly stated detail” in a literary text, and “locate and reproduce explicitly stated information … at the beginning of the text” in an informational text.  By contrast, at the “advanced” international benchmark, students are able to “integrate ideas and evidence across a text,” and “distinguish and interpret complex information from different parts of a text,” among other skills.

The chart above, like the others to follow, is organized from top to bottom in the order of average scale score.  However, the average scale score does not always correlate to the highest percentage of students reaching the “advanced” benchmark in each country.  In this case, it does not, though Hong Kong does have the highest proportion of students meeting either the “high” benchmark or “advanced” benchmark – 67 percent – while in the United States, just 56 percent of students meet those levels.  The tail of students either meeting the “low” benchmark or not meeting a benchmark is also significantly smaller in the top three countries – Hong Kong, the Russian Federation, and Finland (7, 8 and 8 percent, respectively), than in the majority of the other countries.  This more specific data on student performance is useful in terms of thinking about a country’s overall performance, because it gives a clearer sense, potentially, of the equity of the school system, and the ability of the system to educate all students – or any students – to high levels.  It also demonstrates that there are clear differences in student performance between the top handful of countries and the rest of the countries rounding out the top ten or fifteen.

For fourth grade math, in order to reach the “low” benchmark, a student must be able to demonstrate “basic mathematical knowledge,” such as adding and subtracting integers and being able to recognize familiar shapes.  At the “advanced” benchmark, a student must have an understanding of how to apply their knowledge, for example, by solving word problems with multiple steps, and they must show some understanding of more difficult concepts like fractions and decimals.

In the case of TIMSS fourth grade math, the percent of students reaching the “advanced” benchmark does correlate to the country’s average scale score, at least for the top six performers.  This chart indicates very clearly how well the East Asian countries do compared to the rest of the world in instilling advanced-level math skills in their students, even at an early age, with about a third of students or more reaching the “advanced” benchmark in Singapore, Korea, Hong Kong, Taiwan and Japan, and an overwhelming majority reaching either the “advanced” or “high” benchmarks in all cases.  These countries also have the smallest proportions of students who failed to meet the most basic level.  By contrast, starting with Northern Ireland, which is in sixth place in the overall league table in this subject, the other countries have higher proportions of students failing to reach at least the “intermediate” benchmark, and generally much lower proportions of students reaching the “advanced” benchmark.

In eighth grade math, students at the “low” benchmark “have some knowledge of whole numbers and decimals, operations, and basic graphs.”  At the “advanced” level, students are able to demonstrate many mathematical skills, such as solving linear equations, reasoning with geometric figures, and expressing generalizations algebraically.

The pattern in proficiency seen in the TIMSS fourth grade math results is continued in the TIMSS eighth grade math results.  Andreas Schleicher from the OECD and US Education Secretary Arne Duncan have commented on the drop in math and science skills from fourth grade to eighth grade in the United States, and the data bears this out.  In fourth grade, 47 percent of American students met either the “high” or “advanced” benchmarks; in eighth grade, just 30 percent of students did.  Furthermore, twice as many American students – 8 percent – failed to meet any benchmarks in eighth grade than in fourth grade.  In Singapore, however, the number of students meeting the “advanced” or “high” benchmark holds steady at 78 percent in both grades, and the other East Asian countries also do not lose any substantial ground.  Taiwan increases the number of students at the “advanced” level from 30 percent in fourth grade to about half (49 percent) in eighth grade.

In fourth grade science, students at the “low” benchmark “show some elementary knowledge of life, physical and earth sciences,” and “demonstrate knowledge of some simple facts … interpret simple diagrams, complete simple tables, and provide short written responses to questions requiring factual information.”  At the “advanced” benchmark, students can “apply knowledge and understanding of scientific processes … and show some knowledge of the process of scientific inquiry.”  Additionally, “they have a beginning ability to interpret results in the context of a simple experiment, reason and draw conclusions from descriptions and diagrams, and evaluate and support an argument.”

On the TIMSS fourth grade science assessment, the East Asian countries do not dominate in terms of student proficiency at the “advanced” benchmark as completely as they do in math, although perennial top performers South Korea and Singapore still top the list in this measure.  Fewer students overall, across the board, seem to have reached the “advanced” benchmark in science as compared to reading and math.  The United States seems to have a particular problem in this subject, with 19 percent of students either failing to meet any benchmark or only meeting the “low” benchmark.

At the eighth grade level in science, students meeting the “low” benchmark are expected to “recognize some basic facts from the life and physical sciences,” and can display this knowledge by “interpret[ing] simple diagrams, complet[ing] simple tables, and apply[ing] basic knowledge.  Students at the “advanced” level can “communicate an understanding of complex and abstract concepts in biology, chemistry, physics and earth sciences.”  They also “understand basic features of scientific investigation … [and] combine information from several sources to solve problems and draw conclusions, and … provide written explanations to communicate scientific knowledge.”

Like in fourth grade science, overall, there seem to be fewer students who reach the “advanced” benchmark across the board.  The United States sees a 5 percent decline in the number of students reaching the “advanced” benchmark from fourth to eighth grade, and a four percent decline in students reaching the “high” benchmark.  This is compounded by a large jump in the percent of students who either do not meet any benchmarks (7 percent compared to 4 percent) or meet only the “low” benchmark (20 percent compared to 15 percent) – more than a quarter of all US students, in fact.

A separate, but equally interesting, set of data from the 2011 PIRLS results is the level of proficiency of students in two types of reading – literary and informational – as compared to a country’s overall score.  Debates over the value of each type of reading as emphasized in a curriculum have been raging for some time now, and while the PIRLS data does not solve this debate, it does provide interesting new fodder to the discussion.

The chart above depicts the overall average reading score on PIRLS, which is administered to fourth grade students, for the top fifteen systems on that assessment, as well as the average score on the literary reading tasks and on the informational reading tasks.  The top performing countries (Hong Kong, the Russian Federation, Finland and Singapore) all have average informational reading scores that are higher than or equal to their overall reading score, with literary reading scores somewhat lower than or equal to both the overall score and the informational score.  By contrast, the United States, Northern Ireland, Denmark, Ireland, Canada and England all display the opposite trend – literary reading scores that are higher, often statistically significant, than either their informational reading scores or their overall scores.  There is also, in the case of the United States, Ireland and Northern Ireland, a statistical significance in the difference between the lower informational reading score and the overall score.

This suggests that informational reading may, in fact, help aid a student’s overall reading skills, at least as measured by the PIRLS assessment.  It is notable that several East Asian countries, including Singapore, Hong Kong, and Taiwan, all of which traditionally do very well in the math and science assessments, also have students who perform better on informational reading tasks than on literary reading tasks.  In the case of Hong Kong and Singapore, this results in a very high overall score.  In Taiwan, the informational reading score is extremely high compared to the literary reading score, and actually fairly comparable to Singapore’s informational reading score.  However, in this case the literary reading score of Taiwan’s students brings the overall score down, suggesting a need for balance.  In terms of balance, Finland seems to have gotten this just right; the informational, literary and overall scores are indistinguishable from one another, and are all very high.