Validity agenda – what it can mean
Most often when I’m considering validity it's about whether the assessments are testing what they were intended to test. However, it straddles a couple of things firstly, it is whether assessments can and do test those things intended, and secondly whether the qualification then meets its intended purpose in the eyes of its users — learners, employers and delivery centres. So is this two separate things?
When it comes to a validity strategy my thoughts are this is about making sure that users are satisfied that the qualification is meeting the need, wherever in its lifecycle. This involves taking and listening to feedback all the way through, from the initial qualification design through to certificate fulfilment 1, 3, or 5 years on. Feedback might be structured — regular surveys, focus groups with stakeholders, or some other research activity, or combinations of. Unstructured feedback might be heard via complaints, the appeals system, from social media, a centre visit, etc. Capturing this unstructured, reactive feedback needs to be managed so that important learning isn't lost.
So for me validity strategies are about having ears and engaging with users. Then doing something with that feedback if it's valid, that might involve adjusting unit or syllabus components, amending assessment methods, changing the level of the qualification or components, etc.
Then there is validity associated directly with making sure assessments test what they are intended to test which involves a whole different set of strategies. It can include making sure the most important parts of the syllabus or individual learning outcomes are tested. This involves assessment planning, making sure that the assessment method can and does effectively assess those items. Are mark schemes weighted appropriately to bring out the items? Does the question setter, marker, or workplace assessor have the right skills and competences to design an effective assessment? Is the evaluation of assessment outcomes monitored sufficiently to identify whether assessment measures are effective?
I'm not an academic. But some of these issues do present some academic challenge for us.
They also pose real life, operational challenges. I keep learning, each time I look at this area. How about you? What do you think you should do to satisfy the validity agenda? I always welcome your feedback.
Heather Venis
Principal, Awarding First
Watch for my next blog post on 17 June