Poor Wales.

When the PISA results came out on Tuesday, they got a bit of a hammering.

Its reading scores were particularly poor, with the mean (466) sitting 10 points below the OECD average (476) and almost 30 points below England (496) and Scotland (493).

But, having dug into it a bit more, things might not quite be as bad as it first seems.

PISA tests in Welsh

Something that sets Wales apart from the rest of the UK is that a sizeable minority of 15-year-olds take the PISA test in a language other than English. Table 1 – produced using the PISA database – puts this figure at 17% (although the code “Welsh-M,R, English-S” is somewhat ambiguous).

From the database, one can also see that the decision of whether to do the test in English or Welsh is usually a school level decision (all children in each school usually did the test in either English or Welsh).

One interesting point about those that did the PISA test in Welsh is that they come from more socio-economically advantaged families. This is illustrated using parental education in Table 2, but holds for other measures of socio-economic status as well.

Now, one thing we know about children from higher socio-economic backgrounds is that they tend to do better at school, right? Meaning PISA reading scores should be higher amongst those that did the test in Welsh.

Wrong!

In fact, as Table 3 illustrates, the average PISA reading scores amongst those taking the Welsh version of the PISA reading test was inexplicably low. It was around 60 points lower than their peers that did the test in English – the equivalent of two years of schooling. Remember, this is despite the Welsh-taking group coming from MORE socioeconomically advantaged backgrounds.

So, something doesn’t smell quite right to me. (And I have probed this issue before.)

What impact has this had upon PISA reading scores in Wales

Whether there is some translation issue here, or something else, I’m not entirely sure. But something fishy is clearly going on.

What I have therefore done is set the PISA reading scores for the 17% of children who did the test in Welsh to missing, and predicted (imputed) what one would expect this group to score given their socio-economic background.

This suggests that reading scores in Wales are about 11 points too low, increasing from 466 to 477. This swing of 11 points that puts Wales in line with the OECD average (476), though admittedly still some way behind the rest of the UK.

Now, this is a bit of a rough and ready approach. And it could (and should) be refined with more time and better data (and I will happily offer my services to the Welsh Government for free if they are willing to provide me with the data I need).

But my hunch is that this problem leading to an 11-point downward bias in PISA reading scores in Wales is probably in the right ballpark.

Looking into the future, the OECD should also consider what exactly has happened here, and what can be done to put it right next time around.