This blogpost supports a new report available for secondary schools in FFT Aspire, but may be of interest to a wider audience.
Let’s imagine you’re a member of the senior leadership team of a school and you’re reviewing your school’s 2017 performance data. You decide to examine how pupils from each of your eight main primary feeder schools (those from which at least 10 pupils were recruited) performed using Progress 8.
To do this, you produce a boxplot of P8 scores for each of the eight schools (labelled A to H) such as that shown below, including a final box for pupils recruited from other schools, labelled O.
Your eye is immediately drawn to School H. On average, the pupils from this school went on to achieve a Progress 8 score of -1.9.
Assuming a notional 9-1 grading system for all qualifications, this equates to almost two grades per subject lower performance than that of pupils with similar prior attainment nationally. Just four pupils achieved a P8 score above 0.
Your curiosity is piqued.
How to interpret a box plot
The charts used here are box plots, a type of chart which most people are likely to recognise – but the details of which we may not all remember.
The line across the middle of the box is the median – the value which exactly half of results fall below, and the other half fall above. The box itself shows the range between the lower quartile – the value below which exactly a quarter of results fall – and the upper quartile – the value above which exactly a quarter of results fall. This definition means that exactly 50% of results are within the range of values covered by the box.
The difference between the lower quartile and the upper quartile is the inter-quartile range. The whiskers stretching out from the box reach to the maximum and minimum values, excluding outliers, which are shown as dots. Outliers are defined as results more than 1.5 times the inter-quartile range away from either the lower or upper quartile.
What were pupils’ KS2 results like?
Apart from a couple of outliers, pupils from School H achieved very high KS2 results, as the below chart shows.
In total 18 of the 23 pupils from School H achieved an average fine grade of 5.5 or higher in reading and maths.
Was the 2017 Progress 8 score a one-off?
You decide to examine data from 2015 and 2016 as well. As Progress 8 is affected by pupil demographics such as disadvantage, you decide you’ll use a contextual value added (CVA) version of Progress 8, which we’ll call Attainment 8 CVA. (This is available to FFT Aspire users).
The next three charts show the outcome of this, for 2015, 2016 and 2017.
So, no, the 2017 data was not a one-off. Pupils from School H have consistently achieved results two grades lower per subject at Key Stage 4 on average than pupils with similar prior attainment nationally.
This is a real example. The secondary school is in an Ofsted category, criticised in its latest inspection for the progress of its most able pupils.
School H is rated outstanding.
We need to ask difficult questions about the Key Stage 2 results of School H.
Was it really the case that pupils performed at such a level at School H that they simply could not maintain their progress at secondary school?
Could they have received additional assistance when taking their KS2 tests?
Does test security need to be improved?
What can be done?
FFT Aspire users
If you are a secondary school using FFT Aspire, log in to the site to access your report – see the ‘Downloads’ section.
Not an FFT Aspire user?
We have produced a report for FFT Aspire users to enable them to see a breakdown of Progress 8 by primary school (see the sidebar for details of how to access this).
Are there any primary schools from which pupils perform particularly, and consistently, poorly? Or even consistently well, perhaps indicating some under-performance at Key Stage 2? How does your P8 score look with these schools removed?
Looking at Attainment 8 CVA for the last three years and taking account of secondary school attended, we found around 30 feeder primary schools with a score of -1 or lower, indicating that pupils achieved at least one grade below expectation per subject on average.
There were none with a score of +1 or higher.
The Department for Education plans to cap the P8 scores of pupils with anomalously large scores. Ofsted’s Inspection Data Summary Report (IDSR) shows the effect of removing outliers on schools’ value added scores [PDF, p.5].
We would also recommend to both DfE and Ofsted that they also consider making adjustments for pupils who attended primary schools with anomalously high KS2 results.
Better still, don’t rely on a single metric to decide which (if any) schools are performing poorly.
Want to stay up-to-date with the latest research from Education Datalab? Sign up to our mailing list to get notifications about new blogposts, or to receive our half-termly newsletter.
 As secondary schools vary in Attainment 8 CVA scores, with some performing poorly and some very well, pupils’ CVA scores are centred around the secondary school average to enable a fairer comparison of primary schools.