One criticism often made about Ofsted is the consistency and reliability of its inspections. I am currently collaborating with Sam Sims and Christian Bokhove on this very issue and the interest shown in our work has been quite overwhelming.
In my view, this is perhaps the most important part of the Education Select Committee’s ongoing inquiry into Ofsted. It is great to see that they will be taking evidence my colleague Sam Sims tomorrow on this very matter.
But why don’t we know more about this issue? While Sam, Christian and I have tried to push forward research in this area, the evidence base remains far too thin. In fact, there are whole areas – such as inspections of Early Years settings (where judgements are often based upon the views of a single inspector on a single day) – where there is virtually no evidence in the public domain.
Here are my own personal reflections on why this is the case.
The unique position of Ofsted to investigate inspection reliability
In many ways, Ofsted is in the best position to conduct such research. It has the resources, access to inspectors and a great research team.
Yet, throughout its 30-year history, Ofsted has published little on this matter. There was a nice study back in 2017 where it looked into the consistency of its ungraded inspections and some interesting research into pockets of specific inspection tasks in preparation for the introduction of the Education Inspection Framework. But, really, not a lot else.
Why?
One reason is likely because some give Ofsted a hammering whenever such studies show anything less than 100% reliability. For instance, in its study into the reliability of ungraded inspections, Ofsted found 92% agreement across judgements made by different inspectors. But some still were critical, pointing to the 8% that were not.
This reflects an unreasonable expectation amongst some for inspections to be completely reliable. But anything involving human judgement – such as inspections – will never be so. (VAR, anyone!?).
Rather, as Sam, Christian and I argued in our submission to the Education Select Committee, inspections need to be reliable enough for the purpose they are used. Whether they are, or not, is exactly why we need evidence on this matter.
Ofsted, then, becomes stuck between a rock and a hard place.
If it conducts research and reports its inspections to have a high level of reliability, it does not garner much external credibility. Cynics will argue that Ofsted has “marked its own homework” and say “well, of course Ofsted would say that”.
On the other hand, were Ofsted to conduct such a study and find inspections to have low levels of reliability, they would find themselves in a very awkward position. Would any governmental organisation be willing to come out and say “look, what we do isn’t particularly reliable”?
So, for Ofsted, such research becomes a no-win situation. They’re damned if they do, and damned if they don’t.
This then leads to a vacuum in the evidence base.
The challenges faced by researchers
In such situations, it is the role for independent researchers to step in. They should be the “honest brokers”, providing robust, high-quality evidence in an area where there would otherwise be none.
So why don’t they?
The simple answer is that many would LOVE to. But they lack the necessary data.
Sam, Christian and I have had to be really creative in our work – scraping information from inspection reports and collaborating with journalists on information they have obtained via Freedom of Information requests. But this is a time-consuming, sub-optimal process, with the data obtained patchier than it should be. It is, however, the best that any non-partisan researcher currently has.
This is an area where Ofsted must do better.
The paper we published on Friday into whether schools with “inside knowledge” get better inspection outcomes is a prime example. Rather than trying to facilitate such research, when Schools Week requested the data, Ofsted were not exactly helpful.
Rather than release the data when is was requested back in October 2022, Schools Week had to wait for the Information Commissioner’s Office to get involved before Ofsted were made to release it. This is despite Ofsted having published the names of its inspectors since 2015. Why not just add the details of the schools where these individuals work to the list of names it already published (for many inspectors, this information can already be found via a quick Google search)?
Then, even when the information was released, Ofsted only provided a list of organisation names, some of which were ridiculously vague (e.g. “Barnet”). Why not provide the URNs of the schools, next to these names?
And, ultimately, we would be able to further drill down into the results of our paper were Ofsted to make additional information available. This would result in a better collective understanding of an important issue surrounding inspection consistency and reliability. But no such offer has yet been received.
Conclusions
Together, the above means the evidence available regarding inspection reliability and consistency is far too thin.
Ofsted, who are in pole position to generate such evidence, struggle to do so. At least in a way that garners the credibility and acceptance it needs from across the education sector.
Yet independent researchers don’t have access to the data they need to play the role of the “honest broker”.
This is something I hope the Education Select Committee inquiry will look to change, ensuring that future debates about inspection consistency and reliability can be properly evidence based.
My experience of Ofsted is rather old but…… As a Maths teacher I once had an inspector from ILEA come into my classroom – he made an assumption that I could not explain why you cannot have the square root of -1, rather than that my knowledge of the student was thet he would find it difficult. The difference between that and Ofsted inspection was that, at the end of the lesson, I found the ILEA inspector, told him off and received an apology. Ofsted inspectors disappeared and teachers would be unable to challenge. I don’t know if that is still the case.
Having worked with the inspectorate as a national Senior Policy Manager with a brief for the quality of education and training for the Learning and Skills Council I understand the inspection process in some depth. The change occurred with the arrival of Ofsted as we know it now. The inspection methodology before the Adult Learning Inspectorate (ALI) was merged with Ofsted was different. Inspection teams were required to reach consensus with all doubts and variances discussed until a whole inspection team agreed. The inspection team also included the nominee for this purpose. Judgements were sound and the inspected organisation very much part of the process and involved not subject to it. Ofsted changed this so that the lead inspector was solely responsible for the judgements and the draft text of the report etc. This was a step back in the quality of the process and in increase in subjectivity plus a more ‘judged and damned’ aspect to inspections that learning providers can feel excluded from.
Tha nks! Interesting to know some of the historical context surrounding this!
Given my past as a teacher of A Level sociology in a sixth Form College and then as a published researcher into teacher education n a UK University Department, it would be perverse for me not to support the research you are planning. Good luck with your work! I hop be able to read the final report once it has been published
Tha ks!!!