//Difficult questions about some schools’ Key Stage 2 results

Difficult questions about some schools’ Key Stage 2 results

By |2018-09-27T17:16:32+00:0017th April 2018|Exams and assessment, Reports, School accountability|

This blogpost supports a new report available for secondary schools in FFT Aspire, but may be of interest to a wider audience.

Download the report for your school now – log in to FFT Aspire, go to the downloads menu in the top right-hand corner (↓) and choose ‘FFT special reports’.

Let’s imagine you’re a member of the senior leadership team of a school and you’re reviewing your school’s 2017 performance data. You decide to examine how pupils from each of your eight main primary feeder schools (those from which at least 10 pupils were recruited) performed using Progress 8.

To do this, you produce a boxplot of P8 scores for each of the eight schools (labelled A to H) such as that shown below, including a final box for pupils recruited from other schools, labelled O.

Your eye is immediately drawn to School H. On average, the pupils from this school went on to achieve a Progress 8 score of -1.9.

Assuming a notional 9-1 grading system for all qualifications, this equates to almost two grades per subject lower performance than that of pupils with similar prior attainment nationally. Just four pupils achieved a P8 score above 0.

Your curiosity is piqued.

How to interpret a box plot

The charts used here are box plots, a type of chart which most people are likely to recognise – but the details of which we may not all remember.

The line across the middle of the box is the median – the value which exactly half of results fall below, and the other half fall above. The box itself shows the range between the lower quartile – the value below which exactly a quarter of results fall – and the upper quartile – the value above which exactly a quarter of results fall. This definition means that exactly 50% of results are within the range of values covered by the box.

The difference between the lower quartile and the upper quartile is the inter-quartile range. The whiskers stretching out from the box reach to the maximum and minimum values, excluding outliers, which are shown as dots. Outliers are defined as results more than 1.5 times the inter-quartile range away from either the lower or upper quartile.

What were pupils’ KS2 results like?

Apart from a couple of outliers, pupils from School H achieved very high KS2 results, as the below chart shows.

In total 18 of the 23 pupils from School H achieved an average fine grade of 5.5 or higher in reading and maths.

Was the 2017 Progress 8 score a one-off?

You decide to examine data from 2015 and 2016 as well. As Progress 8 is affected by pupil demographics such as disadvantage, you decide you’ll use a contextual value added (CVA) version of Progress 8, which we’ll call Attainment 8 CVA. (This is available to FFT Aspire users).

The next three charts show the outcome of this, for 2015, 2016 and 2017.

So, no, the 2017 data was not a one-off. Pupils from School H have consistently achieved results two grades lower per subject at Key Stage 4 on average than pupils with similar prior attainment nationally.

Difficult questions

This is a real example. The secondary school is in an Ofsted category, criticised in its latest inspection for the progress of its most able pupils.

School H is rated outstanding.

We need to ask difficult questions about the Key Stage 2 results of School H.

Was it really the case that pupils performed at such a level at School H that they simply could not maintain their progress at secondary school?

Could they have received additional assistance when taking their KS2 tests?

Does test security need to be improved?

What can be done?

FFT Aspire users

If you are a secondary school using FFT Aspire, log in to the site, go to the downloads menu in the top right-hand corner (↓) and choose ‘FFT special reports’.

Not an FFT Aspire user?

Learn more here.

We have produced a report for FFT Aspire users to enable them to see a breakdown of Progress 8 by primary school (see the sidebar for details of how to access this).

Are there any primary schools from which pupils perform particularly, and consistently, poorly? Or even consistently well, perhaps indicating some under-performance at Key Stage 2? How does your P8 score look with these schools removed?

Looking at Attainment 8 CVA for the last three years and taking account of secondary school attended[1], we found around 30 feeder primary schools with a score of -1 or lower, indicating that pupils achieved at least one grade below expectation per subject on average.

There were none with a score of +1 or higher.

The Department for Education plans to cap the P8 scores of pupils with anomalously large scores. Ofsted’s Inspection Data Summary Report (IDSR) shows the effect of removing outliers on schools’ value added scores  [PDF, p.5].

We would also recommend to both DfE and Ofsted that they also consider making adjustments for pupils who attended primary schools with anomalously high KS2 results.

Better still, don’t rely on a single metric to decide which (if any) schools are performing poorly.

Now explore your data by feeder primary school in the new report available in FFT Aspire. To download a copy, log in to FFT Aspire, go to the downloads menu in the top right-hand corner (↓) and choose ‘FFT special reports’.

Want to stay up-to-date with the latest research from Education Datalab? Sign up to our mailing list to get notifications about new blogposts, or to receive our half-termly newsletter.

Notes

1. As secondary schools vary in Attainment 8 CVA scores, with some performing poorly and some very well, pupils’ CVA scores are centred around the secondary school average to enable a fairer comparison of primary schools.

Categories

Tags

About the Author:

Dave Thomson is chief statistician at FFT with over fifteen years’ experience working with educational attainment data to raise attainment in local government, higher education and the commercial sector. His current research interests include linking education and workplace datasets to improve estimates of adult attainment and study the impact of education on employment and benefits outcomes.

18 Comments

  1. Deb 17 April, 2018 at 3:00 pm - Reply

    Interesting article, brilliant that we can look at the report in FFT Aspire too.

  2. Nick 18 April, 2018 at 11:36 am - Reply

    As the data manager at a school with one of ‘the 30’ feeder primary schools mentioned, this report is so helpful to me.
    Every time we have an OFSTED data meeting I bring up this issue. Every time I get the ‘all secondary schools say this’ response and it then takes me several hours to ‘persuade’ them that this is a real issue and that our overall progress figure (now P8 but previously 3 and 4 levels progress) would be so much better. It obviously impacts hugely on our P8 figure for ‘High PA’ pupils as all of these pupils are being judged as level 5. Ofsted come into the school with the preconceived idea that the High ability students are performing very badly.

  3. Steffan Woodhead 18 April, 2018 at 11:46 am - Reply

    Is this report available in Wales? I can’t seem to find it.

    • Dave Thomson 19 April, 2018 at 9:58 am - Reply

      Hi Steffan. No, it’s currently only available for England (as Progress 8 doesn’t apply in Wales).

  4. Stephan 19 April, 2018 at 9:30 am - Reply

    It’s interesting that the immediate reaction to high end of KS2 test scores is that the primary school is somehow fiddling results. I would ask how the test scores correlate with the end of KS2 teacher assessment, what the secondary schools baseline assessment on entry for these pupils looks like etc before deciding that these results can be deemed inaccurate (with the possible consequence that these pupils are not given the opportunities they need to fulfil their potential).

    • Clive 20 April, 2018 at 10:50 am - Reply

      I have been beating this drum for a while
      The issue has always been the difference between the KS2 score and their CAT score.
      Plus when we internally test the skills of the pupils with our own version of a SAT paper we notice that the same pupils underperform against the expected result you would have by looking at the score they achieved at KS2.
      But always came up against a brick wall when you tried to discuss this with OfSted or County.

  5. Chris 19 April, 2018 at 5:53 pm - Reply

    Is this a pdf report that is already generated ie is available in the ‘downloads’ from self evaluation > see all downloads area? That one is black and white only; I am just wondering if I am meant to be able to generate this myself somewhere within aspire.

    • Dave Thomson 20 April, 2018 at 3:27 pm - Reply

      Hi Chris. Yes you go to “see all” in the downloads section. On the reports for schools the funnel plots are in grey, so if you’ve managed to find a 3 page pdf with the first page titled “Progress 8 by Feeder School (2017)” then you’ve found it.

  6. SID 24 April, 2018 at 2:43 pm - Reply

    Hello,
    Can you expand on this:

    Looking at Attainment 8 CVA for the last three years and taking account of secondary school attended[1], we found around 30 feeder primary schools with a score of -1 or lower, indicating that pupils achieved at least one grade below expectation per subject on average.

    Is it a three year average? Also is it the average of the FFT Progress 8 CVA on the bottom of the download sheet?

  7. James 28 April, 2018 at 3:13 pm - Reply

    I may be stupid but where is this report?

    • Philip Nye 30 April, 2018 at 8:26 am - Reply

      Hi James, it should be in the downloads section of Aspire, but I’ve found what I believe is your school, from your email address, and if I’m correct it isn’t showing for you because of a change of LAEstab upon academisation. Could you drop me an email to philip.nye@fft.org.uk, please, with your LAEstab to check I’m looking at the right school, and I’ll get the report added. Thanks, Philip.

  8. Anna Perry 14 May, 2018 at 9:46 am - Reply

    Interesting. Clearly KS2 results are not always reliable (open to manipulation, though we may only hypothesize what form this manipulation takes), so why are Ofsted so hung up on the Progress 8 measurement?

  9. Emma 17 May, 2018 at 9:44 am - Reply

    Hi

    I don’t have the report in my downloads and would be really interested to see this please I think this might be down our school having two accounts with two DFE numbers?

    • Philip Nye 17 May, 2018 at 10:04 am - Reply

      Hi Emma, sorry about this – that will indeed be the reason. I’m getting the report added, though it might be tomorrow before it’s live, if I could ask you to check back towards the end of tomorrow.

  10. Alex 5 November, 2018 at 10:35 am - Reply

    will this be updated for 2018? And if so, when?

    • Philip Nye 20 November, 2018 at 10:56 am - Reply

      Hi Alex, we haven’t finalised which reports we’ll be producing based on 2018 data yet – but this one has been very popular, so I think would be a strong contender.

  11. Steve Winyard 20 November, 2018 at 9:18 am - Reply

    Where can we find a list of the 30 schools?

    • Philip Nye 20 November, 2018 at 10:54 am - Reply

      We haven’t published this list of schools, I’m afraid, Steve.

Leave A Comment