One criticism of the current Progress 8 measure is that the points awarded to some qualifications are more (or less) favourable than to others.
There is a risk that any inequality in the way different qualifications are valued incentivises schools to pursue those treated more favourably at the expense of working to improve grades awarded in those treated less favourably.
In a previous blogpost, we looked at the impact of a single qualification, the European Computer Driving Licence (ECDL) on schools’ Progress 8 scores. We showed that for some schools, entering all (or almost all) pupils for ECDL had a large, positive effect compared to an alternative version of Progress 8 in which we rescaled the points awarded to ECDL grades.
But could schools’ other qualification choices have an impact on Progress 8 scores? For instance, it is well-established that modern foreign languages (MFL) are graded more severely than other GCSEs. So schools that tend to enter pupils for MFL at the expense of other English Baccalaureate (EBacc) options may be at a disadvantage under Progress 8.
It might seem sensible then to try to calculate a “qualification neutral” version of Progress 8 in which value added is first calculated for every subject. The average VA score for every subject would be zero, and so any apparent variation in the severity of grading would not have an impact on overall qualification neutral Progress 8 scores.
But would it work?
Methodology
The current Progress 8 calculation works broadly as shown below. An Attainment 8 score is firstly calculated for each pupil, and then a value added calculation is performed on these A8 scores – comparing the A8 scores to estimated values based on a pupil’s Key Stage 2 results.
Current Progress 8 calculation
The qualification neutral version works differently. The coverage of pupils is identical. However, rather than identifying pupils’ best results based on points scores achieved, we firstly calculate value added in every subject within the scope of performance tables. We then find the result with the best VA score for each of the eight slots of Progress 8.
Qualification neutral Progress 8 calculation
A full methodology can be found here [PDF].
Impact on school scores
For the vast majority of schools, the qualification neutral Progress 8 score is not radically different to the existing P8 score. The correlation between the two measures is 0.97.
Progress 8 vs qualification neutral Progress 8
There are large changes for some though. One school, a grammar, changes from +0.16 under the existing method to +0.85 under the qualification neutral method. In fact, the P8 gap between selective and non-selective schools would widen under the qualification neutral method.
By contrast, one sponsored academy already provisionally below the proposed floor with a P8 score of -0.93 would see its score fall to -1.59 under the qualification neutral method. At this school 90% of pupils were entered for ECDL.
In total, 266 out of 292 schools provisionally below the floor standard under the existing method would remain so under the qualification neutral method, while 26 schools provisionally below it would not be if the qualification neutral method was adopted. However, 51 schools would drop below the floor standard.
National scores
Under the qualification neutral method, the national average P8 score is not zero, it is +0.02. To explain why this is the case, we first show the national average P8 score for each area in the chart below.
Average value added score per slot, qualification neutral Progress 8
Firstly, the national average score for English is +0.25. This is because for most pupils, there is a choice of more than one grade. In many cases, a grade in language (or literature) will be higher than the other.
In maths, the national average score is -0.02. Among pupils entered for maths, the national average score is zero. However, when the small percentage (2%) of pupils who were not entered is included (all of whom have by definition negative P8 scores), the average P8 score for maths falls slightly below zero.
This – unfilled slots – also explains why the average score for the EBacc slots is negative. However, the impact of unfilled slots is even greater than it is for maths alone: the average score is -0.24.
Finally, the average score for the open slots is positive at +0.16. Most pupils will have more than three subjects available to fill these slots. As subjects with higher VA scores are preferred by definition, the average rises above zero.
Should we go for this approach?
So given it neutralises issues with inter-subject comparability, should we adopt qualification neutral Progress 8? Er, no. There are a number of serious problems.
The biggest is that the average score for each value of KS2 mean fine grade is not zero. As the chart below shows, the average P8 score increases with prior attainment from -1.0 for pupils with a mean KS2 fine grade of 1.5 to 0.2 for pupils with a mean fine grade of 5.7.
Mean qualification neutral Progress 8 score by KS2 fine grade
This is clearly unacceptable and is caused by a) penalising pupils with unfilled slots and b) pupils with higher prior attainment tending to be able to discard a greater number of lower-scoring results.
This method also does nothing to improve the non-constant variance in P8 scores with respect to prior attainment outlined here, or reduce the impact on the overall school average of pupils with anomalously large negative scores.
Final thoughts
The qualification neutral method is attractive in that it would remove any incentive to pursue an easier qualification option that is in the best interests of the school, but not of the pupils. The disadvantage of the method, even if the bias with respect to prior attainment could be ironed out, will be the additional complexity. As value added measures go, the current Progress 8 measure is as simple as you can get (beyond the intricacies of calculating Attainment 8).
The optimal approach still seems to be achieving better between-subject comparability in grades (or points) awarded.
The introduction of grades 9-1 was an opportunity to address the disparity in grades awarded in MFL but Ofqual has given precedence to maintaining comparability with A*-G grades.
Different points could nonetheless be awarded in different subjects and we will return to this in the new year. But could the education sector live with, for example, a grade C in French being valued more highly than a grade C in history?
Want to stay up-to-date with the latest research from Education Datalab? Follow Education Datalab on Twitter to get all of Datalab’s research as it comes out.
Mr Thomson,
Enjoyed reading this. I work in a grammar school that makes every student take a language at GCSE.
Even if the subject neutral approach were not adopted, better information on calculating P8 subject by subject, rather than element by element, would be very useful for schools in measuring the progress of students and the effectiveness of subject teaching. Currently we only have A8 estimates by element (bucket) and this is not good enough to assess the impact of subject staff and subject leadership. Schools would also then have better data to discuss with Ofsted and would be ina better position to set targets. Of course, none of this will really be possible until the current turbulence with changes to the assessment system settles down.