Pages

Tuesday, 3 December 2013

10 Things You Should Know About PISA

`

1. PISA isn't precise but isn't useless either. No methodology designed to compare countries with completely different cultures and education systems will ever be perfect. The OECD acknowledge that their rankings aren't exact and it makes more sense to look at buckets of countries who have similar scores:






It may be that they're underplaying the statistical issues with the tests (see Professor David Spiegelhalter's concerns here). But that doesn't mean that PISA is worthless. In fact it's far more reliable (along with the other global tests - TIMSS and PIRLS) than any other method of comparison. While we shouldn't rely on the exact rankings we can look at broad trends - especially over time.


2. PISA tests something quite specific. The questions PISA uses are applied - i.e. are focused on using literacy and numeracy in "everyday" situations. Some education systems focus their curriculum more on this type of learning than others. By contrast TIMSS tests whether pupils have mastered specific knowledge and skills which, again, some systems focus on more than others. The UK and US do much better in TIMSS than PISA - and always have.

So with those caveats...

3. The UK is very very average. In Maths and Reading there's no statistically significant difference between the UK and the OECD average. Even on each individual question type (subscales) in Maths the UK is bang on average. Only in Science is the UK slightly above average - as it was last time. There is also no real change from the previous round of PISA. The UK's science score is exactly the same as last time; while Maths and Reading have seen a fractional but not significant improvement. As the graph below shows the UK is one of fairly large group of countries that have seen no meaningful change over the past ten years. All those policies; all those rows and  - at least on what PISA measures - no change.




4. The Far East is dominant. Far Eastern countries have always done well in PISA but they are moving away from everyone else. The top seven jurisdictions in Maths are all Far Eastern (though four of the seven are cities or city-states). Shanghai's 15 year olds are now a full three years ahead of the OECD average - and thus the UK - in Maths (40 points translates roughly into one year of learning). Also look, in the table below, at the number of "top performers" in Shanghai compared to the OECD average:


It's worth noting that there are some question marks about Shanghai's performance. For instance they exclude the children of migrant workers. And of course Shanghai is not China. If you plucked London out of the UK it would almost certainly do better than the country average.


5. Scandinavia is in decline. One of the big stories from this PISA dataset is Finland's significant drop - especially in Maths. They've actually been declining over the past few iterations but the drop this time is much bigger. In fact they're one of only four countries where Maths scores are falling at an accelerating pace (see chart below on the left). But something broader is happening in Scandinavia. If you look at the chart on the right you can see Denmark, Sweden and Iceland have been falling steadily over the past ten years too. Norway has remained static over the same period. Given these countries have fairly different policy environments there may be demographic factors at play here - for instance increasing immigration could be making these countries less socially homogenous.



6. The emerging economies are on the rise. The charts below show the countries that have consistently improved in reading since 2000. While there's a mix of different countries; what stands out is the number of "emerging economies" in this group like Brazil; Indonesia; Mexico and Turkey. These countries are starting from a low base and still do considerably worse than the UK and other developed nations but their rapid improvement is encouraging from the perspective of global prosperity. Often the improvement in these countries is a result of getting more children into school and keeping them their longer. Between 2000 and 2011 the number of children not in school, globally, fell by almost half. We can also see big improvements in many of the former USSR/Warsaw Pact countries like Estonia, Poland, Russia and Hungary. The former two are now among the best performers in Europe.



7. National scores hide huge regional variations. Some countries run the tests in such a way as to allow sub-regions to be given separate scores. The differences are often startling. For instance the Trento region in the north of Italy would be in the top ten globally for Maths if it was a country but pupils in Calabria in the south are around two and half years behind. Likewise in Australia pupils in the Capital Territory (i.e. Canberra) are almost two years ahead of those in the Northern Territory (where many indigenous Australians live). And Flemish Belgium does miles better than French Belgium. In the next PISA we'll be able to see England by region and we can expect to see London and the South-East outperforming other areas. We can already see Wales significantly underperforming the rest of the United Kingdom - the gap's got fractionally larger since last time.

8. There are some policies that many of the "rising" countries seem to share. One of the trickiest things about PISA is making causal links between specific policies and changes in countries' scores. It's very hard to not just cherry-pick examples that support one's existing views. There do seem to be, though, some strong themes around the most successful and most improved countries. One is selection - Germany and Poland are both reducing selection in their systems and have seen improvements and a reduction in the impact of socio-economic status on performance. Singapore is really the only high-performing country to have any selection in their system. A focus on the status of teaching does also seem to be important. This has always been true in the Far East but many of the most improved countries like Estonia, Mexico and Israel have been toughening entry criteria to the profession; raising teacher pay and improving access to professional development. Most successful countries also seem to give a reasonable amount of autonomy to schools. And most have "system stability" - i.e. they have planned reforms backed by much of the system taking place over an extended period of time; rather than constant, uncoordinated, changes.

9. High expectations are absolutely key. The OECD argue that the single biggest reason why the Far East does so well is that they do not have the fixation with innate ability that many Western countries have (yes I'm looking at you Boris). As they put it:

"The PISA 2012 assessment dispels the widespread notion that mathematics achievement is mainly
a product of innate ability rather than hard work. On average across all countries, 32% of 15-year-olds do not reach the baseline Level 2 on the PISA mathematics scale (24% across OECD countries), meaning that those students can perform –at best – routine mathematical procedures following direct instructions. But in Japan and Korea, fewer than 10% of students – and in Shanghai-China, fewer than 4% of students – do not reach this level of proficiency. In these education systems, high expectations for all students are not a mantra but a reality; students who start to fall behind are identified quickly, their problems are promptly and accurately diagnosed, and the appropriate course of action for improvement is quickly taken."

 

10. There's loads more interesting stuff. The above points are all taken from Volume 1 of the PISA report. There are another five volumes that will need to be trawled for further insights. Volume 2 is particularly important because it looks at the impact of socio-economic status (SES) on performance. Again SES seems to explain an average amount of the UK's variation but in the most successful countries it plays much less of a role.


Here's the link to the full report: bit.ly/PISA2012

13 comments:

  1. Sam

    A very interesting and useful summary, I think.

    You said in section 8 that...

    "It's very hard to not just cherry-pick examples that support one's existing views."

    Do you think you managed to do this in section 8 ?


    Are the characteristics you identify (which seem to be almost an frameowrk for Teach first) seen only in countries which are improving most?

    ReplyDelete
  2. Really interesting question is eg why Far East does better in maths? Could it be nothing to do with schools and teaching at all but the pressure from parents prepared to cram kids through 13 hour days in a culture where obedience is paramount in a subject where practice and practice really makes most difference? Why do we say this is a good thing? I accept that getting to a functional level of maths to do everyday things is important and the fact that most people don't seem to have the basic maths and science to shoot down much of the stuff claimed with statistics (including PISA) by politicians is an issue but would I have put my own kids through that? No. Do I think it would make much difference to their effectiveness in the jobs they do now? I doubt it. There are other things like getting them to be good networkers that have been far more important to them and their quality of life and we give it virtually no weight at all. This is why PISA is dangerous. It leads us along a path that could be just as harmful as it is beneficial because we don't really understand the consequences of those numbers. What if it took drugging all the kids to get a higher score? Would we do it? Exam results are not unimportant but they have become the apotheosis of learning to the point where they are counter-productive.

    ReplyDelete
    Replies
    1. I would love, love, love for there to be a hypothetical society which did not have the "cramming for tests" mentality you describe as common in the Far East, but did have the "math achievement is a function of hard work more than innate ability" mentality, also common in the Far East. I'd be very curious to see how they did.

      Delete
  3. Does language affect the outcome? I read (but cannot now link to) a piece that said Swedish speaking Finns do slightly worse than Finnish speakers despite being richer. Second, everyone's favourite data miner Malcolm Gladwell says several E Asian languages are more logical in their numbers. So you learn to count earlier and/or learn to manipulate numbers more easily or earlier.

    ReplyDelete
  4. "Cramming" is clearly not the most important factor because
    1) The various Far East countries have very different levels of cramming.
    2) The main difference between the highest performing countries and, e.g. the U.K, is the small proportion of students which are left behind with little maths or reading ability. If we had more cramming in the UK, you would expect that to help the best but put off more educationally disadvantaged children.

    Don't only accept the headline (mean) numbers from PISA. Look at the distribution of scores and be astonished at the performance of these countries across the board, from most able to least able. In many of these countries (Japan is the clear exception), this is only the first or second generation with widespread access to education. No parent, no child, and no teacher will be willing to believe that poor performance reflects background when most people have just come off the fields / sweatshops .

    ReplyDelete
  5. Interesting analysis, especially when it comes to your later points. While I find the PISA report interesting reading, I caution against putting too much faith into these results. Results which don't equalize for the inherent difference in the data are at best questionable and at worst useless, even if you state you don't think so in your point 1.

    China for instance does not allow testing in certain areas to be included in the country's overall score, which would otherwise be considerably lower (as your point 7 shows for Italy).

    German kids at the point of testing have two years less schooling than their Scottish counterparts (thus PISA pins year 9 students against year 11 students). Korean students spend double the hours of average European students on learning in class which is why the analysis says they're up to three years ahead. They have simply spent three years or more learning the same stuff our kids are supposed to master in far less time.

    I'd love to see the data compiled for countries with the same education system and finding out why they perform as they do. In that sense the Scandinavian decline you mention in point 5 is very much asking us to take a much closer look for the reasons behind it.

    What also interests me is what particular countries have done to improve their performance between PISA 4 and 5, such as Poland and Germany. But contrary to Gove's statement to parliament, Germany has not reformed its education system in the last years. Instead, the improvement of its scores is due to increased performance by the least able students. This is most likely due to the fact that they have stopped tinkering with the education system after almost 20 years of constant upheaval.

    Education policy is not in the federal remit, instead, each Bundesstaat decides its own curriculum. In general, most have opted to use the East German 12 year system over the West German 13 year system and the West German streaming system over the East German comprehensive one.

    Yes, as you mention in point 8, some German selection has been reduced, but the high achievers are still sorted out after four years of primary school and sent to the Gymnasium. The ones left behind are sorted again into those expected to sit GCSEs and those who aren't.

    It's the latter who in about a third of the federal states are now left in the same class with their brighter counterparts and maybe that's where the higher expectations you discuss in point 9 come into play. I'd have to look at the individual regional results to know if it made a difference or not. I do think it's too early to tell as some of those selection changes have been made as late as 2010.

    ReplyDelete
  6. It's a small point, but I don't think your suggested reason for the decline in Scandinavian test scores can be correct. Iceland's somewhat more diverse than it was 20 years ago, but even compared to the rest of Scandinavia it's monolithically white and monolithically Icelandic. If it were down to decreasing homogeneity, you'd expect it to be Norway dropping and Iceland holding steady.

    It seems more likely to me that their are different causative factors in different parts of Norden.

    ReplyDelete
  7. "If you plucked London out of the UK it would almost certainly do better than the country average"

    Why would it?

    ReplyDelete
  8. Some interesting observations, although I think it is clear that higher PISA scores result from better focussed teaching for poorer children, and that "Free" Schools in Sweden have been a disaster, probably because of the way they have led to educational segregation and fragmentation.

    One of the countries that has failed to hit the PISA headlines is Chile; Chile has had a Gove-ist fragmented and unequal education system with "deregulated" ITE, overtesting and PRP, for a very long time, probably longer than anywhere else on the planet. It languishes near the bottom on the table, despite being one of the wealthiest countries in Latin America.

    This suggests that factors such as equality, quality of training for teachers and the level of fragmentation in a system have a direct influence. Badly trained teachers teaching in a system which is fragmented and in which schools and teachers are competing rather than collaborating, is a recipe for educational failure. Unfortunately the UK is in the early stages of setting up an identitical system.

    ReplyDelete
  9. German improvement is mostly down to revised expectations. The country was genuinely shocked by early PISA results and a nationwide realisation that something had to be done led to many disparate groups coming together in the States to demand better performance. Would that UK parents, unions, local authorities and politicians could achieve this sort of consensus instead of working against each other. The record of the current government is far from encouraging either - undermining unions, teachers, local authorities, local politicians. Expect further decline.

    ReplyDelete
  10. I would be careful about use of the phrase "high expectations" here. Putting in a lot of effort to ensure that you help the poorest performing students is doubtless a good thing (when compared to ignoring them and assuming they won't do well)."High expectations" is a phrase typically used by the right in the context of their belief that the liberal elite have let down working class kids by being nice to them and teaching them rap rather than Shakespeare and doing touchy feely things to do with "independent thinking" rather than rigorously caning them every time they get a maths question wrong.

    It's also important to think about causation in the context of analysing this data (which I think you do a splendid job of). With the increased autonomy given to schools in better performing countries, for example, it might well be the case that where schools are performing well, they are allowed autonomy, but where schools as a whole are seen as failing, there is pressure to exert more central control/apply fixed standards.

    ReplyDelete
  11. According to the Israeli press children who were tested in hebrew had a math results of 489 while children who were tested in Arabic scored 388 . That means that the gap between Arabs and Jews is 1 SD (100 point).
    Consider that Ashkenazim/Mizrahim ratio among Hebrew testers is 50/50 , and that Ashkenazim typically score 0.75 SD (75 point) then Mizrahim (e.g on the Israeli SAT) , then Ashkenazim score in math is about : 489 + 75 * 0.5 ~ 521 .

    ReplyDelete
  12. Thanks for sharing information about 10 Things You Should Know About PISA.
    Stromanbieter Vergleich

    ReplyDelete