This is one of several blogposts in Datalab’s ‘Who’s Left 2018’ series of posts. The full series can be found here.

As well as analysing the picture of moves by secondary school-age pupils in England we have tested what the effect would be of the policy solution that we propose – reweighting league tables in proportion to the amount of time that pupils have spent on-roll at particular establishments.

In short, rather than all pupils who finish secondary schooling at a particular establishment counting equally, as happens currently in school league tables, reweighting involves weighting some pupils more heavily than others.

For instance, the results of a pupil who had been on-roll for each of the 15 terms that make up secondary education receive a greater weight than those of a pupil who had only spent six terms at the school – and pupils would still count to some extent even where they were no longer on the roll of a particular school. Full details of how our reweighting approach works can be found here.

As well as doing this at school-level, it’s possible to do it at multi-academy trust (MAT)-level, following the methodology that the Department for Education uses in producing MAT league tables.

In January we noted that the MAT league tables showed considerable variation in the Progress 8 scores achieved by the trusts that featured – but cautioned that they should not necessarily be taken at face value.

That’s because, as simple aggregations of school-level data, they repeat one of the weaknesses of the DfE’s school-level league tables – that, broadly speaking, only pupils who remain on-roll in January of Year 11 count in the figures.

So what is the impact of reweighting MAT league tables?

Flattening out

The chart below shows the impact of reweighting for the 62 trusts that featured in the MAT league tables in 2017, with trusts ordered by their MAT league table Progress 8 (P8) score

The overall impact of reweighting is to flatten out the range of MAT P8 scores to some degree.

There would not be huge changes in the rank ordering of trusts, but, in general, the gap in P8 scores between trusts with the very highest scores and those towards the middle of the pack would decrease.

In total, 15 trusts would see a positive impact, three would see no change, and the remaining 44 would see a negative impact. In many cases, the impact is very small, but in individual cases they are as large as +0.22 and -0.23.

A simple average of these trust-level figures shows a mean impact of -0.04 on Progress 8 scores.

Leavers

There are likely to be a number of factors that explain the impact observed.

Firstly, several of these trusts are London-based, and inner London is the region where the greatest impact of reweighting results is seen.

Secondly, the schools that feature in these trusts are predominantly sponsored academies. As the table below shows, looking across all schools in our Who’s Left analysis, as a group, sponsored academies would see a negative impact of a move to a reweighted approach.

The size of the reweighting impact is not directly related to the number of pupils who leave.

The chart below shows the impact of reweighting versus the number of pupils who finish secondary school at the schools counting in the MAT league tables. (It’s worth saying that this doesn’t include pupils who leave to another school that is part of the same MAT but which doesn’t count in the MAT league tables, on account of having joined the MAT recently.)

And there’s a similar picture when just the number of pupils who leave these MAT schools and leave the state system entirely is considered.

This shows that it is not simply the number of pupils who leave that determines the reweighting impact – it is down to a combination of the number of pupils who leave, at what point those pupils leave, and what those pupils go on to attain (and what their prior attainment was), as well as the number and characteristics of pupils who have joined schools within the trust.

To be clear, this is not saying that trusts where there is a large impact of reweighting are engaging in off-rolling. As we have stated elsewhere, from data alone we do not think it is possible to reach that conclusion.

But, if as seems likely, MAT league tables are going to assume ever-greater importance, as a field we should ask whether the comparison they make is a meaningful one, and whether we think they send the right messages in terms of the importance of inclusion, and of establishments admitting pupils in need of a school place.

An alternative method of reweighting

In this first set of reweighted figures, pupils are included even if they are not recorded as having taken any GCSEs or equivalent qualifications. If a pupil is not recorded as having taken any exams, they are given an A8 score of 0. Pupils are also included if they took some exams but their results did not count towards any establishment.

In some cases, this will include pupils who have left the system entirely, for example through emigration, rather than just those who remain in the system but did not take any exams.

To show the effect of taking out of the analysis pupils who are not recorded as having taken any GCSEs or equivalent qualifications, or, if they did, whose results did not count towards any establishment, we have produced a second set of reweighted figures that excludes these pupils.

As the chart below shows, in most cases this reweighting (labelled as reweighted P8 scores, version two) gives a value somewhere between the published P8 value for the trust and the value given in our first version of reweighting.

If a reweighting approach were to be adopted, there would need to be an extension of a mechanism currently in place, to allow schools to request amendments to their results in cases where a child had left England permanently, or died, at any point between Year 7 and Year 11.

In the absence of this, the true impact of reweighting, ignoring pupils who were legitimately no longer part of the cohort, is likely to fall somewhere between those resulting from the two reweighting approaches described above.

Full data, giving the impact of both reweighting calculations, is given in the table below, with details of the methodology used available here. MATs that wish to gain a better understanding of their own data are encouraged to contact the team.

Reweighting impact by multi-academy trust, 2017 cohort

Now read the next post in the Who’s Left 2018 series.

Want to stay up-to-date with the latest research from FFT Education Datalab? Sign up to Datalab’s mailing list to get notifications about new blogposts, or to receive the team’s half-termly newsletter.