In December, the results of the 2011 administration of the Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS) were published in three separate reports, each examining international performance in reading (at the fourth grade level), math (at the fourth and eighth grade levels) and science (at the fourth and eighth grade levels).  These assessments provide a picture of international student performance in the years before a student reaches the age of 15, which is the age at which students take the OECD’s Programme of International Student Assessment (PISA).  However, there are some central differences between the TIMSS/PIRLS and PISA assessments.  Michael Martin, the Co-Executive Director of the TIMSS and PIRLS Study Center at Boston College, notes that while PISA is intended to measure a student’s general skills in the arenas of reading, math and science, TIMSS and PIRLS are more focused on content mastery.  Additionally, Jack Buckley, the commissioner of the National Center for Education Statistics, has pointed out that the countries participating in both assessments do vary – the TIMSS and PIRLS groups are smaller and represent a mixture of countries at different levels of economic development as compared to the participants in PISA.

Because of the differences between the assessments, the countries that are in the top ten or fifteen of the TIMSS and PIRLS rankings are somewhat different than the top performers on the last incarnation of PISA in 2009.  While league tables of the top countries based on their average scores always garner the most press when the results of international assessments are released, we decided to take a more in-depth look at what level of proficiency students in the top fifteen countries are actually reaching in these subjects.

The IEA has established four “international benchmarks” on their score scale for these assessments.  While the score scale for both PIRLS and TIMSS runs from 0-1000, the vast majority of scores fall between 300 and 700.  The IEA has identified a score of 400 as the “low” international benchmark, indicating that students at this score point have been educated to a “basic” level.  Beyond that, there is a score of 475, or “intermediate;” a score of 550, or “high,” and a score of 625, or “advanced.”  Below, we have plotted the percent of students at each benchmark in the top fifteen countries on the 2011 administration of PIRLS and TIMSS.  This is useful when thinking about the top performers, because it shows, in a clearer way perhaps than the average scale score, what students in each country are really able to do.

In the fourth grade PIRLS reading assessment, a student who reaches the “low” international benchmark is able to “locate and retrieve an explicitly stated detail” in a literary text, and “locate and reproduce explicitly stated information … at the beginning of the text” in an informational text.  By contrast, at the “advanced” international benchmark, students are able to “integrate ideas and evidence across a text,” and “distinguish and interpret complex information from different parts of a text,” among other skills.

The chart above, like the others to follow, is organized from top to bottom in the order of average scale score.  However, the average scale score does not always correlate to the highest percentage of students reaching the “advanced” benchmark in each country.  In this case, it does not, though Hong Kong does have the highest proportion of students meeting either the “high” benchmark or “advanced” benchmark – 67 percent – while in the United States, just 56 percent of students meet those levels.  The tail of students either meeting the “low” benchmark or not meeting a benchmark is also significantly smaller in the top three countries – Hong Kong, the Russian Federation, and Finland (7, 8 and 8 percent, respectively), than in the majority of the other countries.  This more specific data on student performance is useful in terms of thinking about a country’s overall performance, because it gives a clearer sense, potentially, of the equity of the school system, and the ability of the system to educate all students – or any students – to high levels.  It also demonstrates that there are clear differences in student performance between the top handful of countries and the rest of the countries rounding out the top ten or fifteen.

For fourth grade math, in order to reach the “low” benchmark, a student must be able to demonstrate “basic mathematical knowledge,” such as adding and subtracting integers and being able to recognize familiar shapes.  At the “advanced” benchmark, a student must have an understanding of how to apply their knowledge, for example, by solving word problems with multiple steps, and they must show some understanding of more difficult concepts like fractions and decimals.

In the case of TIMSS fourth grade math, the percent of students reaching the “advanced” benchmark does correlate to the country’s average scale score, at least for the top six performers.  This chart indicates very clearly how well the East Asian countries do compared to the rest of the world in instilling advanced-level math skills in their students, even at an early age, with about a third of students or more reaching the “advanced” benchmark in Singapore, Korea, Hong Kong, Taiwan and Japan, and an overwhelming majority reaching either the “advanced” or “high” benchmarks in all cases.  These countries also have the smallest proportions of students who failed to meet the most basic level.  By contrast, starting with Northern Ireland, which is in sixth place in the overall league table in this subject, the other countries have higher proportions of students failing to reach at least the “intermediate” benchmark, and generally much lower proportions of students reaching the “advanced” benchmark.

In eighth grade math, students at the “low” benchmark “have some knowledge of whole numbers and decimals, operations, and basic graphs.”  At the “advanced” level, students are able to demonstrate many mathematical skills, such as solving linear equations, reasoning with geometric figures, and expressing generalizations algebraically.

The pattern in proficiency seen in the TIMSS fourth grade math results is continued in the TIMSS eighth grade math results.  Andreas Schleicher from the OECD and US Education Secretary Arne Duncan have commented on the drop in math and science skills from fourth grade to eighth grade in the United States, and the data bears this out.  In fourth grade, 47 percent of American students met either the “high” or “advanced” benchmarks; in eighth grade, just 30 percent of students did.  Furthermore, twice as many American students – 8 percent – failed to meet any benchmarks in eighth grade than in fourth grade.  In Singapore, however, the number of students meeting the “advanced” or “high” benchmark holds steady at 78 percent in both grades, and the other East Asian countries also do not lose any substantial ground.  Taiwan increases the number of students at the “advanced” level from 30 percent in fourth grade to about half (49 percent) in eighth grade.

In fourth grade science, students at the “low” benchmark “show some elementary knowledge of life, physical and earth sciences,” and “demonstrate knowledge of some simple facts … interpret simple diagrams, complete simple tables, and provide short written responses to questions requiring factual information.”  At the “advanced” benchmark, students can “apply knowledge and understanding of scientific processes … and show some knowledge of the process of scientific inquiry.”  Additionally, “they have a beginning ability to interpret results in the context of a simple experiment, reason and draw conclusions from descriptions and diagrams, and evaluate and support an argument.”

On the TIMSS fourth grade science assessment, the East Asian countries do not dominate in terms of student proficiency at the “advanced” benchmark as completely as they do in math, although perennial top performers South Korea and Singapore still top the list in this measure.  Fewer students overall, across the board, seem to have reached the “advanced” benchmark in science as compared to reading and math.  The United States seems to have a particular problem in this subject, with 19 percent of students either failing to meet any benchmark or only meeting the “low” benchmark.

At the eighth grade level in science, students meeting the “low” benchmark are expected to “recognize some basic facts from the life and physical sciences,” and can display this knowledge by “interpret[ing] simple diagrams, complet[ing] simple tables, and apply[ing] basic knowledge.  Students at the “advanced” level can “communicate an understanding of complex and abstract concepts in biology, chemistry, physics and earth sciences.”  They also “understand basic features of scientific investigation … [and] combine information from several sources to solve problems and draw conclusions, and … provide written explanations to communicate scientific knowledge.”

Like in fourth grade science, overall, there seem to be fewer students who reach the “advanced” benchmark across the board.  The United States sees a 5 percent decline in the number of students reaching the “advanced” benchmark from fourth to eighth grade, and a four percent decline in students reaching the “high” benchmark.  This is compounded by a large jump in the percent of students who either do not meet any benchmarks (7 percent compared to 4 percent) or meet only the “low” benchmark (20 percent compared to 15 percent) – more than a quarter of all US students, in fact.

A separate, but equally interesting, set of data from the 2011 PIRLS results is the level of proficiency of students in two types of reading – literary and informational – as compared to a country’s overall score.  Debates over the value of each type of reading as emphasized in a curriculum have been raging for some time now, and while the PIRLS data does not solve this debate, it does provide interesting new fodder to the discussion.

The chart above depicts the overall average reading score on PIRLS, which is administered to fourth grade students, for the top fifteen systems on that assessment, as well as the average score on the literary reading tasks and on the informational reading tasks.  The top performing countries (Hong Kong, the Russian Federation, Finland and Singapore) all have average informational reading scores that are higher than or equal to their overall reading score, with literary reading scores somewhat lower than or equal to both the overall score and the informational score.  By contrast, the United States, Northern Ireland, Denmark, Ireland, Canada and England all display the opposite trend – literary reading scores that are higher, often statistically significant, than either their informational reading scores or their overall scores.  There is also, in the case of the United States, Ireland and Northern Ireland, a statistical significance in the difference between the lower informational reading score and the overall score.

This suggests that informational reading may, in fact, help aid a student’s overall reading skills, at least as measured by the PIRLS assessment.  It is notable that several East Asian countries, including Singapore, Hong Kong, and Taiwan, all of which traditionally do very well in the math and science assessments, also have students who perform better on informational reading tasks than on literary reading tasks.  In the case of Hong Kong and Singapore, this results in a very high overall score.  In Taiwan, the informational reading score is extremely high compared to the literary reading score, and actually fairly comparable to Singapore’s informational reading score.  However, in this case the literary reading score of Taiwan’s students brings the overall score down, suggesting a need for balance.  In terms of balance, Finland seems to have gotten this just right; the informational, literary and overall scores are indistinguishable from one another, and are all very high.

Learning Beyond Fifteen: Ten Years After PISA finds that the largest gains in reading achievement typically occur during compulsory education. But the study shows that many adolescents who leave high school with relatively low reading comprehension greatly improve their reading ability and some of these students can close the gap altogether.

The study examines reading gains of Canadian youth between the ages of 15 and 24.  The report uses data that Canada collected from the PISA assessment of 15-year- olds (PISA-15) in 2000, the first time the test was administered.  The test was given to 30,000 students across all ten Canadian provinces.  In 2009, when the original PISA-15 students were 24 years old, a subset of the cohort took the PISA exam again (PISA-24).

The report also takes into account information from Canada’s Youth in Transition Survey (YITS), developed to analyze education and labor-market outcomes of Canadian youth.  The survey collected data from the PISA-15 respondents and their parents at the time they took their initial PISA test at age 15.  Every two years thereafter, the survey also collected information from students about their education and employment experiences, their life choices, and their attitudes.  In 2009, when the surveyed students were 24 years old, a subset of the cohort completed a PISA reassessment.

Overall, there was a clear improvement in reading performance among all the students that were reassessed for the Learning Beyond Fifteen study, with the average reading score increasing 57 points from 2000 to 2009, which is equivalent to the difference in average proficiency scores for 15-year-olds in Canada and their counterparts in countries like Croatia, Israel, and Austria.  To put it in another context, the learning gains were equal to roughly one school year in Canada.  In 2000, 21.4 percent of Canadian 15-year-olds scored below proficiency Level 3 on the PISA scale, which indicates an ability to locate multiple pieces of information, make links between different parts of text, and relate it to familiar everyday knowledge.  By 2009, 7 percent of the test takers still scored below proficiency Level 3.

OECD (2012). Learning Beyond Fifteen: Ten Years After PISA

Examining the reading proficiency gains by demographic characteristics, Learning Beyond Fifteen finds that by age 24, women continue to outperform men by a slightly smaller margin than they did when they were 15 years old.  In almost all participating OECD countries, girls outperform boys in reading by a large margin.  The participating Canadian women achieved an average increase of 50 points in reading performance during the nine-year period, while males achieved an average increase of 63 points.  Similar to many other subgroups, the poorer-performing groups acquired reading skills at a quicker pace than the better-performing groups, but were still not able to close the achievement gap entirely.

This was the case when examining reading gains by family socio-economic background.  The evidence suggests that students from disadvantaged homes continue to have lower scores at age 24 than their more advantaged peers (by a difference of 50 percentage points).  A similar pattern was found for Canadian French speakers versus English speakers and rural students versus urban students, with French speakers and rural students narrowing but not completely closing the achievement gap by the time they were age 24.

What was quite interesting in this report is that twenty-four-year-olds with an immigrant background fully bridged the gap in reading performance.  These students scored an average of 524 points in PISA-15 while those born in Canada averaged 545 points.  In PISA-24, all students, regardless of their birthplace, averaged around 600 points.  Students born outside of Canada improved 77 score points from the time they were 15 years old until they were 24 years old, an improvement equivalent to more than one proficiency level on the PISA reading scale.  The authors point out that these results demonstrate that the appropriate integration policies can help to reduce, if not eliminate, differences in student performance, at least for immigrants in Canada.

OECD (2012). Learning Beyond Fifteen: Ten Years After PISA

The report finds, not surprisingly, that participation in some form of post-secondary education is consistently and substantially related to growth in reading skills between ages 15 and 24.  For example, students with a university degree at age 24 had an average score of 652 points in PISA-24, while those students with only a high school education at age 24 had an average score of 564 points, almost 100 points lower than college graduates.  When those same college graduates took the test at age 15, they had averaged 596 points on PISA, a score still substantially above the scores attained nine years later by those students that had not obtained any formal education beyond high school.

Students who spent four or more years in a higher education institute between ages 15 and 24 but did not complete a degree, still showed improvements in skills that were similar to or greater than (70 points or more) those skills observed among young people who did complete a college degree.

On the other hand, students that were poor performers at age 15 and entered the labor market soon after compulsory education tended to continue to be poor-performers at age 24 with their general rate of improvement being relatively modest.

This report also took a look at the factors that relate to improvements in reading skills when students were 15 to age 24.  From childhood to age 15, the report study finds that the strongest influences on reading proficiency are parents and the home environment along with the quality of teachers and the classroom-learning environment.  When students are older, the degree of control they feel over their life is strongly related to improvements in reading.   Independence and the capacity to make individual life choices is generally related to larger improvements in reading performance, particularly if it is coupled with participation in post-secondary education.

Young people who had the advantage of a supportive learning environment up until age 15 showed relatively slower learning improvements as they made their transition to independence.  On the other hand, those students that did not succeed in their school, made greater improvements if they experienced a life change, for example changing the status of their relationship (from single to married) or moving out of their parents’ home.

Other Reports of Note
Education Week Quality Counts 2012. “Canada Musters Resources to Serve Diverse Student Needs.”
This article, part of Education Week’s special 2012 Quality Counts edition focused on “The Global Challenge for Education,” examines Canada’s commitment to equality in its public schools, and particularly the provinces’ ability to provide a high quality education for their most at-risk students by managing school funding at the provincial, rather than the local, level.  Although Canada has a higher immigrant population and a higher proportion of students living in poverty than many other OECD countries, they have been able to integrate these students into mainstream classrooms while still giving them targeted support both in the classroom and out, with some districts even providing subsidized health services like vision and hearing screenings.  In addition to the article, Education Week has made a video, produced for the Quality Counts release event, as well as audio from the live event.

OECD. (May 2012). “Does performance-based pay improve teaching?”
Performance-based pay for teachers is a hot topic in many countries.  So this month’s PISA in Focus will be of interest to many of our readers.  The authors explain that,  “A look at the overall picture tells us that there is no relationship between average student performance in a country and the use of performance-based pay schemes.”  But in countries with comparatively low teacher salaries (less than 15 percent above GDP per capita), student performance tends to be better when performance-based pay systems are in place, while in countries where teachers are relatively well paid (more than 15 percent above average GDP per capita), the opposite is true.  So for countries that do not have the resources to pay all of their teachers well, it is worth having a look at the experience of those countries that have introduced performance-based pay schemes.  This finding, of course, is consistent with our own finding in Surpassing Shanghai that relatively poor countries just starting out on the economic development curve that cannot afford to pay their teachers developed world salaries will tend to use Tayloristic management schemes because their teachers will not have the professional skills required to succeed in a professional work environment.  Conversely, the same Tayloristic management methods won’t work when a country is employing highly educated and trained teachers.  Put another way, blue-collar work organization is appropriate for relatively low-skilled teachers and for use in the early stages of economic development, but professional norms of work organization are needed as a country moves up the economic development ladder and begins to employ highly educated and trained teachers.  Only the latter are likely to produce world-class high quality, high equity education systems.

International Association for the Evaluation of Educational Attainment. (2012). “Policy, Practice and Readiness to Teach Primary and Secondary Mathematics in 17 Countries: Findings from the IEA Teacher Education and Development Study in Mathematics (TEDS-M).”
Using findings from the 2008 Teacher Education and Development Study in Mathematics (TEDS-M), this report examines country-level policies related to the preparation of future mathematics teachers, how these policies impact the participating countries’ teacher education programs and instructional practices, and the implications of these polices and practices for student learning. The International Association for the Evaluation of Educational Attainment (IEA) published initial results from the TEDS-M study in 2009. The participating countries include Botswana, Canada (four provinces), Chile, Taiwan, Georgia, Germany, Malaysia, Norway, Oman (lower-secondary teacher education only), the Philippines, Poland, the Russian Federation, Singapore, Spain (primary teacher education only), Switzerland (German-speaking cantons), Thailand, and the United States.  According to Policy, Practice and Readiness to Teach Primary and Secondary Mathematics in 17 Countries, the countries that best prepare math teachers have implemented a number of common practices including rigorous math instruction for all high school students, including potential teachers; teacher-preparation programs that are highly selective and demanding; and an attractive profession with excellent pay, benefits and job security.  According to this study, Taiwan and Singapore top the list of the countries that do the best job of preparing math teachers and Russia also scored highly.  Poland, Switzerland and Germany did well but this is partially explained by their reliance on specialist teachers in the lower-grades.  The United States generally finished below this group but above other countries that scored below the international average.

OECD Education Working Papers. (May 2012). “School Funding Formulas: Review of Main Characteristics and Impacts.”
This working paper provides a literature review on school funding formulas across OECD countries.  It examines what kinds of school formula funding schemes exist and how they are used, in particular, for promoting the needs of socially disadvantaged pupils and how school formula funding systems perform according to equity and efficiency standards.  The paper discusses the difficulties of striking the right balance in school funding formulas between more or less weight given to local differences.  For example, when funding formulas give more consideration to the local costs of education and other local specificities, this can lead to more convoluted and obscure formula designs.  The authors also focus on the challenges of measuring how much it costs to educate students with a given background to a pre-defined standard and ensuring school autonomy while making sure funding is spent on what it was intended for.

(Source: OECD. (2011). Education at a Glance: 2011, OECD. (2010). PISA 2009 Results: Overcoming Social Background, Vol. II.)

One challenge shared by many countries as they work toward improving student performance is insuring that new immigrants are provided the kind and quality of education needed for success.  Often, these students, who may speak different languages at school and at home, and who may come from cultures very different from that of their adopted country, are invoked as a reason for low scores on international exams.  Some top performers like Finland and the Netherlands have in recent years begun to integrate large immigrant populations into their school systems, and are working to establish effective policies to ensure that these students are able to achieve at the same high levels as native-born students.  Other top performers, however, have a long history of educating immigrant students to high levels.  Australia, New Zealand and Canada fit into this category.  All three countries have student bodies in which nearly a quarter of students are immigrants, defined by the OECD as either first- or second-generation transplants (Figure 1).  And notwithstanding the high proportion of immigrant students in these countries, they consistently top the international league tables of student performance.  In some cases, notably in Australia, immigrant students in fact do better than native students on international assessments, and in Canada, there is only a small gap between immigrant and native students’ performance (Figure 2).  Furthermore, because Canada instituted a longitudinal study of students who first took the PISA exam at age 15 in 2000, we are able to see that immigrant students’ performance in that country has improved over time.

(Source: OECD. (2011). Education at a Glance: 2011.)

In their annual report, Education at a Glance, the OECD takes immigrant students’ socioeconomic background into account when assessing the gaps between immigrant students and native-born students in order to determine to what degree socioeconomic status may impact performance.  They found that when controlling for socioeconomic status, the gap between native and immigrant students decreased in almost every case.  On average across OECD countries, the performance gap in reading decreased from 44 points to 27 points.  In the United States, while native students outperform immigrants by more than 20 points before accounting for socioeconomic status, immigrant students actually outperform native students by about 10 points once their socioeconomic background is accounted for.  Other countries, particularly Canada, Australia and New Zealand, do not have a large gap in the performance of immigrant students before and after taking into account their socioeconomic status, suggesting that they are more successful than other countries including the United States in integrating the vast majority of their immigrant students into their education systems.

The successful integration of immigrants into a country’s school system can be seen clearly in the case of Canada, which has been able to produce high achievement among its immigrant students, as demonstrated by a very close relationship between immigrant students’ average PISA scores and the national average (Figure 3).  Australia and New Zealand have also achieved some measure of success here, while Finland and the Netherlands have large gaps between the national average and the average of their immigrant students.  In the report Pathways to Success: How Knowledge and Skills at Age 15 Shape Future Lives in Canada, OECD observes that new immigrants tend to perform less well than their Canadian counterparts.  However, immigrants who have been in Canada for five or more years nearly match native students’ performance.  They also note that more immigrant students go on to university (61%) than do native-born students (43%).

(Source: OECD. (2011). Education at a Glance: 2011; OECD. (2010). PISA Ranking by Mean Score for Reading, Mathematics and Science.)

In a recent look at the results from a Canadian longitudinal study of students who took the PISA exam in 2000, the OECD found that by the age of 24, immigrant students were able to erase the gap that had been present between immigrant and native students when they took their PISA exam at the age of 15, with the average PISA score of immigrant students at the second administration of the test (at age 24) being two points higher (601) than that of Canadian-born students (Figure 4).  This is an increase of 77 points over the average score of immigrant students at the age of 15, which represents more than one proficiency level on the PISA scale.  Canadian-born students, by contrast, gained just 54 points between the ages of 15 and 24 – still a large improvement, but not as large as the improvement made by students who had immigrated to Canada.  It is important to note that immigrants are an exception.  Other groups of students who were low performers on the 2000 administration of PISA (among them students from disadvantaged socioeconomic backgrounds and Francophone students) continued to be low performers in the second administration of the exam, and the performance gap between these groups and the overall Canadian average was relatively unchanged ten years later.  The authors of this report argue that the more equitable outcomes at the age of 24 are a result of both Canadian immigration and education policies, and that Canada provides effective education pathways to immigrant students, particularly beyond the level of compulsory education.  The authors suggest that Canada’s policies can be useful to other countries experiencing high rates of immigration, and note that skills gaps can be addressed in the years between lower secondary school and the time at which students enter the workforce.

While Learning Beyond Fifteen does not go into real detail about the policies Canada employs to produce success among its immigrant students, the authors of the 2010 OECD publication Strong Performers and Successful Reformers in Education: Lessons from PISA for the United States, have identified several strategies.  The first is Canada’s immigration policies.  Canada has encouraged, through public policy, the highly educated to move to Canada to fill specific economic and labor force needs.  Because preference is given to these highly educated immigrants, they are not seen as a major threat to native-born Canadians and are therefore accepted more readily into Canadian society.  Their children, too, have access to better resources. Immigrant children in Canada, unlike in most OECD countries, generally have access to resources that are equal to or greater than what their native-born counterparts are provided including high quality school buildings, low teacher-student ratios, healthy classroom climates and positive teacher morale.  Immigrant students are typically mainstreamed into classes taught in English or French, to quickly integrate them into the system.  At the same time, these students are provided with additional language classes both in and out of the classroom to allow them to keep up with their peers.

(Source: OECD. (2012). Learning Beyond Fifteen - 10 Years After PISA.)