A total of 1,670 schools have now participated in the FFT Key Stage 4 Early Results Service, which we offered for the first time this year. More than half of state-funded mainstream schools took part.

Earlier this year we looked at the impact of changes to GCSE examinations in 2017 on Attainment 8 and Progress 8 Pupil Premium gaps. We concluded that the introduction of new examinations (a 9-1 scale in English/mathematics, different points scores for other GCSEs) had led to very little change on the overall Progress 8 gap. However, then looking at individual elements, we found:

• for English, the gap had widened substantially;
• for mathematics, the gap had widened;
• for EBacc subjects, the gap had narrowed;
• for other subjects, the gap had narrowed slightly.

We concluded that the narrowing of gaps in EBacc and other elements was probably due to an increase in entry that was masking the impact of the changes in point scores. Subsequent analysis confirms that this was correct:

In 2018, reformed GCSEs cover most large-entry subjects. What impact will this have had on Pupil Premium gaps?

Dave Thomson’s analysis (based on a slightly earlier KS4 Early Results Service dataset of 1,548 schools) showed that the schools formed a representative sample and that it was likely that they would reflect national trends. Dave’s blogpost showed that the gap in both Attainment 8 and Progress 8 had widened slightly from 2017 to 2018.

The KS4 Early Results Service dataset includes scores for Attainment 8, English and mathematics. While we cannot, therefore, look at EBacc and other elements separately, we can calculate a combined score for these two elements by subtracting the English and mathematics scores from the overall total for each pupil.

The chart below shows the change in Pupil Premium attainment gaps from 2015 to 2018 (Attainment 8 scores for 2015 have been retrospectively calculated).^[1] Figures for all years are for schools in the KS4 Early Results Service dataset – therefore providing a fair comparison.

We can see that, apart from mathematics, gaps have widened from 2017 to 2018:

• The gap for English has continued to widen. This is a surprise given that English was measured on the 9-1 scale in both 2017 and 2018.
• The gap for EBacc and other subjects combined has widened, and by nearly as much as it has narrowed from 2016 to 2017. Given recent trends it is likely that the gap in entries will have, at the very least, remained constant (and will probably have decreased) – so it looks like we are seeing a widening in the gap as a consequence of the change to reformed GCSEs.

Dave Thomson’s analysis looked at the change in scores from 2017 to 2018 when pupils with the same Key Stage 2 attainment were compared. It showed that, for pupils with average or below average KS2 scores, outcomes in 2018 were slightly higher. For pupils attaining the old Level 5, outcomes in 2018 were slightly lower.

Given that overall KS2 attainment for Pupil Premium pupils is lower than for other pupils, we might expect their 2018 outcomes to be higher and for the gaps to have closed. So why does the opposite appear to be happening?

The chart below shows the difference between 2017 and 2018 Attainment 8 scores for each KS2 fine grade, with Pupil Premium (PP) and other pupils shown separately.

While the overall trends for both groups of pupils are the same (higher points scores for lower KS2 attainers), the line for Pupil Premium pupils is quite consistently below that for other pupils. If we look at the difference between the two lines this becomes even clearer:

It’s clear that – even when we compare pupils with the same KS2 scores – attainment for Pupil Premium pupils is consistently lower in 2018.

What impact has this had on Progress 8 gaps? The chart below shows the gap in value-added scores between Pupil Premium pupils and other pupils from 2015 to 2018 for schools in the KS4 Early Results Service dataset (again Progress 8 scores for 2015 have been retrospectively calculated).

Having reduced each year from 2015 to 2017, the Progress 8 Pupil Premium gap increased by 0.12 from 2017 to 2018. This represents a difference of 1.2 points in overall Attainment 8 i.e. equivalent to just over one grade lower in one subject. This might seem relatively small, but in the context of Progress 8 scores it represents a substantial change in the overall Pupil Premium gap.

Gaps also increased for individual components within Attainment 8 – particularly for EBacc/other subjects and for English. Why has this happened?

It is surprising to see a widening of the gap in English and mathematics from 2017 to 2018 given that the examinations have remained the same. Perhaps, as they have started to become used to a new examinations, schools have found it harder to increase attainment for Pupil Premium pupils.

Assuming that gaps in entry for EBacc/other subjects have, at the very least, remained the same in 2018, then the widening of the gap suggests that, relative to other pupils, Pupil Premium pupils have found the new reformed GCSEs to be more difficult.

When national data is available we will be able to undertake a more detailed analysis to see if these trends and conclusions hold firm.

Finally, this analysis confirms conclusions we formed previously – that “any comparison of changes in Progress 8 gaps over time will need to be done with considerable caution”.

The changes to examinations mean that looking at trends in attainment gaps is, as we have seen from this analysis, problematic even at national level.

It is, for a variety of reasons, much more problematic at school level where a better approach would be to analyse the attainment and progress of Pupil Premium pupils relative to other pupils nationally within each year and then to aggregate data over three years. We are currently working on just such a report.

Want to stay up-to-date with the latest research from FFT Education Datalab? Follow us on Twitter to get all of our research as it comes out.