Provisional Progress 8 scores for 2019 will be published tomorrow.

The first set of calculations shared with schools for checking were missing a chunk of BTEC results. These have since been found and P8 scores recalculated.

In truth, the missing data won’t have had much impact on schools’ P8 scores unless they had lots of pupils who didn’t have at least three other subjects that could be counted in the open slots of P8.

But based on our previous work, we do know that pupils’ results in BTECs tend to be higher than in their English and maths GCSEs. There are also some GCSEs which are scored less generously than English and maths, particularly modern foreign languages (MFL).

What if there was a way of adjusting the point scores awarded to different qualifications to make them more like GCSE English and maths?

Why do we need to change the scoring of some qualifications?

Let’s go back to 2018 Key Stage 4 data.

The chart below plots each school in England’s average Progress 8 score in English and maths against their Progress 8 score for the “open” slots.^[1]

Highlighted in the chart above are 61 schools where the open score is at least 1 point higher than the average of English and maths. In other words, pupils achieve one grade higher (on average) in each of the three open slot subjects than in English and maths.

To what extent is this caused by entering more generously scored qualifications?

Rescoring qualifications

Progress 8 guidance lists the point scores that are associated with achieving each of the different grades in the different types of qualification eligible for inclusion in the measure.

At present, a grade 4 in a GCSE yields 4 points. This is the case in all subjects. But if MFL is scored less generously, perhaps more points should be awarded.

We’ve previously calculated what we called a qualification-neutral Progress 8 measure. This uses a method that rescales the point scores for each grade in each qualification in line with the English and maths scores of the pupils who took that qualification – so that fewer points are awarded in qualifications that appear to be graded more favourably. The methodology is set out more fully here. We’ve updated it using the 2018 data.

By way of example, the chart below shows the rescaled point scores awarded to grade 4 GCSE in different subjects. In some subjects, such as art and design, a grade 4 receives less than 4 points. In MFL and computing, it receives more than 5.

In the most generously scored major qualification, BTEC PE, the rescaling process reduces the points awarded considerably – shown in the table below.

The impact of rescoring on school P8 scores

The chart below shows the same thing as the first chart in this post, but using these rescaled scores that we’ve calculated. There’s a closer correlation between P8 scores for the open slots and English and maths results after rescoring: compared to the published data, the correlation improves from 0.71 to 0.83.

On the whole, the open slot scores of the 61 schools seen above have reduced, and they have been brought closer to the pack. For the most part, their open slot scores are still on the high side. So not only were their pupils more likely to enter more generously-scored qualifications, they were also more likely to do well in them.

Impact on overall Progress 8 scores

As was the case in our earlier blogpost, this ultimately makes very little difference to schools’ P8 scores.

The correlation between published P8 and qualification-neutral P8 scores is staggeringly high: 0.99.

The school most affected by the change would see it’s Progress 8 score fall by 0.4 points, dropping from the 100^th to the 96^th percentile in the process. In total, the P8 scores of 23 schools would fall by 0.25 points or more. Just one school would improve by a similar margin.

But for the most part, rescoring qualifications makes very little difference. For 2,679 schools out of 3,146 the published P8 and qualification-neutral P8 scores change by no more than 0.1 (either up or down) ^[2].

Is it worth doing?

Adjusting the scores for different qualifications adds extra complexity. Given how little the scores change, it might not be worth the hassle.

However, doing something like this would remove any incentives for schools to offer particular qualifications just to rack up points for the sake of school performance tables.

Now read the other post in this pair of blogposts, which looks at how things would change if P8 took account of context.

Want to stay up-to-date with the latest research from FFT Education Datalab? Sign up to Datalab’s mailing list to get notifications about new blogposts, or to receive the team’s half-termly newsletter.