Congratulations! You’ve passed your 11-plus. How confident do you feel that you would have passed if you had sat a slightly different test on a different day of the week?

In our recent analysis of the Kent 11-plus we expressed concern that the issue of who passes is incredibly sensitive to slight variation in candidate performance because of the design of the test. Short papers, of 25-30 minutes, coupled with the need to meet a minimum mark on each of the three papers, means that 8% of those who pass are just one mark away from failing.

This sensitivity to chance events – a small error on one question, being distracted by a noise outside, feeling slightly under the weather – is important because it can mean the test violates the principle that it sorts the most academically capable into grammar schools and the rest into secondary moderns[1].

Reliability

We can get some insight into how reliable 11-plus tests are by looking at data from two grammar schools in North Yorkshire, where pupils are required to sit the test in two parts, a week apart.

In the case of these two schools – Ermysted’s and Ripon – a child’s best verbal and non-verbal reasoning scores are used to allocate places[2]. But we can use the data to explore test reliability – with conclusions for the rest of the country where the 11-plus is taken in a single sitting.

The chart below shows the pupil performance on two 50-minute verbal reasoning tests taken one week apart at Ermysted’s (we don’t want to combine the data from the two schools because they have different pass marks reflecting local pupil demographics).

Fifty minutes is nearly twice as long as the Kent reasoning paper, and with 80 questions the overall test reliability should be quite high.

Nevertheless, the correlation between the scores on the two North Yorkshire verbal reasoning tests is only 0.89.

If we look at the 107 highest scores in each of the two tests, 85 children pupils appear in both lists. But there are 44 pupils who appear in only one of the lists. (This figure – 107 – was chosen as it is the number deemed to be suitable for grammar school in the data that we have.)

The 40-minute non-verbal reasoning test has a similar correlation between sittings, of 0.85. A total of 86 pupils appear at the top of the list in both tests, with 42 children appearing at the top in one but not the other.

What conclusions can we draw?

Even the highest quality tests will result in pupils getting slightly different results from one test to the next.

These North Yorkshire schools are trying their best to reduce this element of chance by allowing children to take their best scores over two weekends.

This comes at the cost of increased time and stress involved in the selection process, so it is understandable that other areas choose not to do this. However, the consequence is that one-sitting selection processes will be highly sensitive to chance variation in pupil results.

Sometimes less academically capable students will pass the 11-plus and more academic capable students will fail. Society needs to decide how much of this misallocation it can tolerate.

This analysis is based on data made publicly available by North Yorkshire County Council as a result of a Freedom of Information Act request by Joanne Bartley of Comprehensive Future. We would like to thank North Yorkshire County Council for facilitating the release of this data in the interest of openness and transparency. Education Datalab is an independent research organisation that has not received any funding to write this research.

Want to stay up-to-date with the latest research from Education Datalab? Sign up to our mailing list to get notifications about new blogposts, or to receive our half-termly newsletter.

Notes

1. There are processes designed to correct for issues such as this – headteacher panels and an appeals process in the case of the Kent 11-plus. In the case of the headteacher panels, however, we concluded that they may not be correcting for issues such as disadvantaged children being under-represented among those passing the 11-plus as much as we might expect.
2. You can read the details of the two schools’ admissions processes here.