One criticism often made about Ofsted is the consistency and reliability of its inspections. I am currently collaborating with Sam Sims and Christian Bokhove on this very issue and the interest shown in our work has been quite overwhelming.

In my view, this is perhaps the most important part of the Education Select Committee’s ongoing inquiry into Ofsted. It is great to see that they will be taking evidence my colleague Sam Sims tomorrow on this very matter.

But why don’t we know more about this issue? While Sam, Christian and I have tried to push forward research in this area, the evidence base remains far too thin. In fact, there are whole areas – such as inspections of Early Years settings (where judgements are often based upon the views of a single inspector on a single day) – where there is virtually no evidence in the public domain.

Here are my own personal reflections on why this is the case.

The unique position of Ofsted to investigate inspection reliability

In many ways, Ofsted is in the best position to conduct such research. It has the resources, access to inspectors and a great research team.

Yet, throughout its 30-year history, Ofsted has published little on this matter. There was a nice study back in 2017 where it looked into the consistency of its ungraded inspections and some interesting research into pockets of specific inspection tasks in preparation for the introduction of the Education Inspection Framework. But, really, not a lot else.

Why?

One reason is likely because some give Ofsted a hammering whenever such studies show anything less than 100% reliability. For instance, in its study into the reliability of ungraded inspections, Ofsted found 92% agreement across judgements made by different inspectors. But some still were critical, pointing to the 8% that were not.

This reflects an unreasonable expectation amongst some for inspections to be completely reliable. But anything involving human judgement – such as inspections – will never be so. (VAR, anyone!?).

Rather, as Sam, Christian and I argued in our submission to the Education Select Committee, inspections need to be reliable enough for the purpose they are used. Whether they are, or not, is exactly why we need evidence on this matter.

Ofsted, then, becomes stuck between a rock and a hard place.

If it conducts research and reports its inspections to have a high level of reliability, it does not garner much external credibility. Cynics will argue that Ofsted has “marked its own homework” and say “well, of course Ofsted would say that”.

On the other hand, were Ofsted to conduct such a study and find inspections to have low levels of reliability, they would find themselves in a very awkward position. Would any governmental organisation be willing to come out and say “look, what we do isn’t particularly reliable”?

So, for Ofsted, such research becomes a no-win situation. They’re damned if they do, and damned if they don’t.

This then leads to a vacuum in the evidence base.

The challenges faced by researchers

In such situations, it is the role for independent researchers to step in. They should be the “honest brokers”, providing robust, high-quality evidence in an area where there would otherwise be none.

So why don’t they?

The simple answer is that many would LOVE to. But they lack the necessary data.

Sam, Christian and I have had to be really creative in our work – scraping information from inspection reports and collaborating with journalists on information they have obtained via Freedom of Information requests. But this is a time-consuming, sub-optimal process, with the data obtained patchier than it should be. It is, however, the best that any non-partisan researcher currently has.

This is an area where Ofsted must do better.

The paper we published on Friday into whether schools with “inside knowledge” get better inspection outcomes is a prime example. Rather than trying to facilitate such research, when Schools Week requested the data, Ofsted were not exactly helpful.

Rather than release the data when is was requested back in October 2022, Schools Week had to wait for the Information Commissioner’s Office to get involved before Ofsted were made to release it. This is despite Ofsted having published the names of its inspectors since 2015. Why not just add the details of the schools where these individuals work to the list of names it already published (for many inspectors, this information can already be found via a quick Google search)?

Then, even when the information was released, Ofsted only provided a list of organisation names, some of which were ridiculously vague (e.g. “Barnet”). Why not provide the URNs of the schools, next to these names?

And, ultimately, we would be able to further drill down into the results of our paper were Ofsted to make additional information available. This would result in a better collective understanding of an important issue surrounding inspection consistency and reliability. But no such offer has yet been received.

Conclusions

Together, the above means the evidence available regarding inspection reliability and consistency is far too thin.

Ofsted, who are in pole position to generate such evidence, struggle to do so. At least in a way that garners the credibility and acceptance it needs from across the education sector.

Yet independent researchers don’t have access to the data they need to play the role of the “honest broker”.

This is something I hope the Education Select Committee inquiry will look to change, ensuring that future debates about inspection consistency and reliability can be properly evidence based.