Progress 8 is the new measure by which secondary schools will be judged. It works by comparing each child’s achievement in eight subjects at GCSE with the average GCSE results for other children who got the same results in exams taken at age 11. The Department for Education designed it to incentivise schools to provide a broad academic curriculum and focus on helping all pupils make good progress. By comparing pupils with similar levels of academic attainment, it is supposed to provide a fair test of a school’s performance and trigger intervention where an individual school falls below the ‘floor standard’.

In 2015 the average Progress 8 score (using FFT’s Progress 8 estimate) across the 13 fully-selective local authorities was about zero, meaning their secondary schools were no better or worse than in non-selective areas. But while the grammar schools in selective areas had a Progress 8 score of +0.34 (i.e. each pupil scored about a third of a grade higher across eight GCSEs), the secondary moderns had a negative Progress 8 score of -0.11. Progress 8 measures therefore appear to show that selection has no effect on overall attainment, but does boost the attainment of grammar pupils at the expense of their neighbours attending secondary moderns.

But there’s reason to be sceptical. Progress 8 almost certainly overstates grammar school performance while understating secondary modern performance. This is because a Key Stage 2 test score is quite a noisy measure of a child’s educational attainment at age 11. It is not very highly correlated with 11-plus results, for example. While a KS2 score of 5.2 (indicated by the green line in the chart below) indicates the point where the majority attend a grammar, there are many with lower KS2 scores who pass the 11-plus and vice versa.
gram_p8_1

Why does this matter? Well, a student who scores as low as 4.5 on their KS2 and yet has managed to pass the 11-plus is probably more able than students scoring 4.5 who do not pass. They may have just had a bad day when they took their KS2 exams. We would therefore expect them to achieve much better GCSEs, regardless of school attended. Conversely, the student who scores as high as 5.5 on their KS2 and yet fails their 11-plus is probably less able than other students scoring 5.5. Failing the 11-plus suggests they may have been lucky with their KS2 exams: perhaps the questions they wanted all came up. This is the noise – things other than ability – picked up by the exam.

The chart below shows how Progress 8 scores relate to KS2 scores for both grammars and secondary moderns. Those who have very low KS2 scores (on the left of the graph) and yet pass the 11-plus (the pink grammar line) achieve incredibly high Progress 8 scores simply because their KS2 score almost certainly understates their academic ability at age 11. Some of the progress made during primary school is then unfairly attributed to their grammar school. By contrast, those who have very high KS2 scores (on the right of the graph) and yet have failed the 11-plus (the blue secondary modern line) have very low Progress 8 scores. Again, this is because they are likely to be less capable, on average, than others with the same KS2 score. Some of the progress at their secondary modern is therefore unfairly attributed to their primary school.

gram_p8_2

What can we do to fairly judge the performance of grammars and secondary moderns? The chart below shows the relationship between KS2 score and Progress 8 in fully selective areas. Progress 8 scores above zero indicate that pupils are outperforming their peers, and vice versa. The grey line on the chart below shows that in selective areas pupils with low prior attainment tend to achieve below what is expected, while high KS2 achievers tend to outperform their peers in the rest of the country. One option for judging performance is to judge schools in these areas collectively, recognising the sorting of pupils that is taking place. However this does not allow a judgment of whether individual schools have fallen below the floor standard and would therefore undermine the point of the Progress 8 measure.

gram_p8_3

Second, we could recalculate Progress 8 for grammar schools and secondary moderns by excluding students who pass (or are close to passing) the 11-plus. But KS2 is such a poor predictor of passing the 11-plus that this would mean excluding half the children in these areas from our estimates. This would probably make school’s Progress 8 scores very unstable over time. It would also undermine the government’s aim for Progress 8 to incentivise equal focus on all children.

A third, more sensible approach, would be to change how we calculate average progress. In selective areas, instead of comparing pupil’s progress to that of all other pupils with similar prior attainment, we should instead compare pupil’s progress with other pupils with similar prior attainment in selective schools (or non-selective schools). That would achieve all the main aims of the Progress 8 measure, including assessing whether a particular school meets the floor standard. But it would still not allow us to compare grammars and secondary moderns because, for any given level of KS2, we would always expect a pupil who passes the 11-plus to make more progress than a pupil who did not.