Moderation and Verification of Centre Assessment Judgements – Ofqual Consultation

..Sharing some thoughts and ideas on the implications of the consultation paper 

We all want reliable and consistent standards that can be maintained. AOs all work to that end. But we are about to see a change that could mean an extra big shake up of the sector as Ofqual proposes to change some of the rules. The interesting thing in this scenario is the rules are being flexed to accommodate the uniqueness of vocational and technical qualifications delivery.

Amid a policy regime driven by Government where assessment is becoming more centralised, more closely controlled than ever, take T levels and Apprenticeship End Point Assessment. It does seem Ofqual is perhaps better informed and more on-side in recognising that not all assessment can be delivered this way, nor easily moderated by the AO, if we want to keep regulated qualifications for certain sectors and industries. Here we need to mention fork-lift truck driving, first aid at work, and also that ability to bring employees much more quickly onto the construction/work site. 

My wish for the proposals is that it doesn’t turn some sector provision away from being regulated. That qualification standards are lost, as the provision returns to become part of open market provision. That would be a real race to the bottom.

A little bit crazy

The high-level summary of the proposals is about a tightening of the terminology, having a written down centre assurance strategy and most controversially a 3-visit a year regime, 1 to be unannounced. The craziness and wasteful idea of an unannounced visit, and how 3-visits will build costs and resource burden takes away from some otherwise sensible things that AOs tend to have in place or do anyway, that support the consultation proposals. 

When is moderation, moderation?

I’m wondering what Ofqual expect to see for moderation where the assessment is MCQs, will the review of trend data be enough? The practice of ‘moderation of marks’, essentially looking at trend data, it’s not just about MCQ tests, it has been around a long time in some disciplines. 

Then there is the serious question of how exactly does Ofqual see moderation of performance based, practical, skills, or oral assessments work? The drama exam, carrying out CPR, the receptive exam for British Sign Language, also the coach on the river bank being assessed, are good examples of this. We know sampling these kinds of assessments is much more resource intensive. Local IQA (internal quality assurance) is an effective way to make sure that assessor judgements are reliable and consistent in these circumstances. So will there always be a case for continuing to invest in IQAs at centres; or will centres feel inclined to leave all the moderation burden to the AO? 

Quality Assurance Activity – the verification model

At the Ofqual consultation event I attended all the issues were of course hotly discussed and there seemed to be some listening particularly to the idea that the 3-visit regime should be 3 quality assurance interventions. This would allow a more proportionate and balanced approach to individual centre risks.  

Amongst those quality assurance interventions could be remote moderation; online checking-in on centre activity, including on centre induction, training and standardisation events; remote reviews of centre documentation, policy and process; facetime interviews with assessors, trainers, and/or Learners; mandatory participation in AO training and standardisation; and more. This would allow a more flexible approach, depending on the risk associated with the centre, and allow AOs to apply different treatments to different needs, to ensure that quality control at centre level is happening.

Verification plus, plus

The idea shared in the consultation that newly-approved, a new assessment team, a new qualification type, or other scenarios might dictate a greater level of verification scrutiny goes hand-in-hand with most AOs current strategy. This ramping up of the control measures to ensure standards are in place and maintained makes sense. 

Scale of the burden

On the face of it, it could seem the burden is just about scaling up resources for a 3-visit regime. However, the recording systems – for itemising or holding the assessment evidence; the sampling plan, records and decisions applied; retaining the assessment evidence in a way that allows it to be easily retrievable; the planning and transparency of quality assurance activity, its recording and ability to follow it through are sizeable challenges too.

The availability of systems to do all of this, and for it to be auditable, for many AOs may require some investment, save for the precarious use of spreadsheets and groaning file drives. For AOs that have centres that operate as franchises, allow satellites, work outside national boundaries or in a second language the ability to capture this additional complexity in the system will be moreover challenging. 

At least a two-dimensional approach to quality models

The centre assurance strategy that’s proposed is often already there at many AOs, but possibly not fully documented in places. Areas such as sampling strategy, approach to standardisation (personalised, rather than routine), and the frequency and participation in standardisation events are sometimes more bendable to make them more effective and timely; will this be allowed in this new world? Documenting this brings a more detailed level of policy and process to be written and followed. Staff and EQAs will need to be more on board than ever with the requirements. For compliance this is good news, but the reality is this could be a lot of work and training-in needed.

For some AOs there may be a quality assurance strategy per qualification, or suite of qualifications. This may already be articulated in the assessment strategy. Going forward, each revised or new qualification development, an additional step, to evaluate the most appropriate moderation or verification model to apply to its delivery will be sensible. 

Balancing the qualification quality assurance requirement and the centre situation will probably need a bit more triangulation. In this situation a more sophisticated approach to risk assessment and management will be beneficial. This is perhaps where Ofqual can help more, to provide some models of risk management, and the ingredients of quality assurance models they have seen work well. Ofqual have the chance to contribute to sharing good practice. 

A strong case for e-certificates

Reading the consultation it feels like Ofqual expects the frequency of needing to correct results and revoke certificates will increase. Currently revoking certificates is quite limited to certain situations. And we know obtaining certificates back that have been issued is a hard exercise. To make it more possible e-certificates may be a good way forward, despite is not being an entirely fool proof system.

What would be helpful is a decision model to help inform the re-issue/revoke decision, not just guidance from Ofqual. For me, timescales need to be closely considered, there is a point after which revoking a certificate will have no positive impact, unless it’s an area where license to practice is in place. An e-certificate system that provides on-going verification checks for prospective employers and other regulators would provide value here. 

Appeals

As for the impact on appeals, it has to be they will grow? There is an odd idea in the consultation that no-one connected with the AO can be the decision maker for the appeal. My experience is appeals tend to be pretty clear cut, there is little wriggle room for judgement/decision taking, its more black and white decisions. I’m not sure what’s being suggested here, whether it’s okay for the investigation to be done by an employee/contractor, but the decision by some separate entity?

More assessment and quality assurance data, more record keeping, more tracking of data, more auditable data

Everything in the consultation points to more of everything data wise. I can’t think that I have seen an IT system, or a set of integrated systems, that will capture all that’s needed, or get near to it. The comforting thing is we will all be working with imperfect systems! That aside, the proposed implementation date is unrealistic if Ofqual want a better prepared sector going forward. 

The cost projections for this step-change from Ofqual look light too. Evaluating what qualifications best fit which model; upscaling the recruitment and training activities to support these changes; the strategizing; the policy and procedures writing; the logistics and planning; the training in; as well as planning for, managing and monitoring the data. That’s a lot of effort.

Heather Venis

Awarding First

E: heather@awardingfirst.co,uk T: 01588 650 152, or 0789 479 6262

Awarding First supports EPAOs and Awarding Organisations with professional knowledge,  skills, and guidance on regulatory requirements, compliance, also EPA, qualifications and quality assurance arrangements and developments. Heather has a breadth and depth of experience in the sector and works with a range of AOs.