School quality: can we measure this with standardized tests?

Martijn Meeter and Melanie Ehren

The OECD’s PISA study has altered how countries think about education. For some, there was cause to celebrate, for others to mourn and fret. Germany famously experienced a “PISA shock” when in 2000, the scores indicated the country was doing less well than its neighboring countries. At that time, the Netherlands was one of the countries that compared favorably to Germany. However, in the past two decades, scores have seen a slow but steady decline, combined with mounting concerns over increasing inequality. While performance differences between students are not very large compared to other OECD countries, they are determined by social background to a larger extent than elsewhere...

The Netherlands: increasing inequality

The OECD’s PISA study has altered how countries think about education. For some, there was cause to celebrate, for others to mourn and fret. Germany famously experienced a “PISA shock” when in 2000, the scores indicated the country was doing less well than its neighboring countries. At that time, the Netherlands was one of the countries that compared favorably to Germany. However, in the past two decades, scores have seen a slow but steady decline, combined with mounting concerns over increasing inequality. While performance differences between students are not very large compared to other OECD countries, they are determined by social background to a larger extent than elsewhere. 

Disagreement about causes of inequality

There is considerable disagreement on the reasons for the decline and on what to do about it. Some, including the Dutch Inspectorate of Education argue for more time and emphasis on basic skills, reading, writing and arithmetic, rather than time for other goals such as healthy behaviors or life skills. They point to evidence from standardized tests showing that a large number of students exit school without mastering a basic level in mathematics, or not able to write coherent texts. An underlying cause for low skills in mathematics might be teachers who are not competent in the subject themselves. For writing, the Inspectorate points to the lack of deep feedback that students receive on their writing products, with teachers only commenting on basic errors of spelling and layout. 

For reading, the problem is one of motivation and students not reading enough books outside of school.

Others however do not necessarily agree that declining test scores are problematic and instead argue that education is becoming too much about testing and valuing test scores. The association of secondary school boards has claimed, for example, has claimed that emphasis on basic skills hampers students’ social and mental development. 

Should we use standardized tests to measure school quality?

These viewpoints reflect a long-standing debate about whether school quality should be measured by (only) standardized test scores, and whether these tests are the best way to understand student performance. Elsewhere we have argued for multiple measures, but here we address the second question and discuss it for primary education.

Can standardized tests of students’ basic skills be used as measures of school quality?

For primary education, there are three simple criteria to assess an assessment : does the assessment predict success in secondary education? And does it reflect variables that the school has a major influence on (as opposed to, e.g., a child’s genetic makeup or social background)? And is it difficult to game? Standardized tests of arithmetic skills and of reading comprehension meet all three: They reflect the residue of years of in-school training on these skills, because of this they are difficult to game (except through outright fraud), and they are highly predictive of later success in education.

In fact, “basic skills” is something of a misnomer, given what is needed for successful reading comprehension. Successfully answering questions about the meaning of a text not only requires good text decoding skills, but also good knowledge of vocabulary and understanding of the world, and an ability to make inferences. In fact, reading comprehension is more of a capstone skill than a basic skill (the same holds for advanced tests of arithmetic). 

Should we extend standardized tests to measure other outcomes?

If standardized tests of arithmetic and reading comprehension are indeed valid ways to measure school quality, are there other measures that might be added? We argue that for now it is best to focus on academic subjects only, as many of the available alternative measures of non-academic skills or attitudes tend to be unreliable. For example, the social-emotional skills measures championed by the OECD tend to actually measure personality traits; and schools have little influence over these. Possible additions to the palette could be student satisfaction, because of the intrinsic value of a child’s happiness during school hours, and civics education – although the predictive value of the latter for later civic attitudes still needs to be established. And rather than overburdening schools and students with a battery of tests, its best to focus on the basics that every students needs to do well in school and life in general.

MEET THE AUTHOR

prof.dr. Martijn Meeter

Full Professor, Faculty of Behavioural and Movement Sciences, Educational and Family Studies

LEARN! - Learning sciences

prof.dr. Melanie Ehren

Professor in Educational Governance

Director of Research Institute LEARN!

Vrije Universiteit Amsterdam

Honorary Professor
University College London