Abdi Ltd writes: We are exploring delivering one-day programmes on Learning Assessments, so that participants really understand what indicators to look for when assessing learning and how and where to start.
In the most recent of our quarterly sessions to support the team’s continuing professional development, we reviewed and discussed the quality of Level 2 assessment that we see when working on evaluations and reviewing submissions. Level 2 is the measurement of knowledge, skills and attitudes. Where an investment intends to improve knowledge, build skills or help to change attitudes in order to influence behaviours, or whether these are formal education and training programmes or the provision of information and/or quick checks on whether someone can do what they are expected to do, they must be accurate, fair and objectively measureable. The assessment needs to provide assurance that before someone starts implementing a task, adhering to a new policy, changing their behaviour in accordance with a new expected standard, they know what to do and how to do it to the expected standard and under conditions they will experience when they are using these knowledge or skills in their work. Obvious really – but in practice very frequently poorly evaluated. The first mistake is to assess something that is neither essential nor relevant to the change in performance or behaviour that the investment aims to deliver. We looked as many examples of assessment tools the team (our associates and staff) had shared for the CPD session.
For example one was a multiple choice questionnaire that apart from its very dubious right/wrong answers to questions was in fact assessing information the participants on a training programme had been told during the course. But these participants were on the course to improve their customer service skills and there was no assessment of these skills and behaviours! We then looked at an example of the observation of participants undertaking a simulated training session where the observer was asked to provide comments. No descriptors to provide objective indicators of what the observer was supposed to look for and no scoring requirement, just a subjective narrative. We also looked at a much higher quality tool which had clear unambiguous descriptors and a simple scoring requirement. Finally, we compared participants self-rated scores on competence and confidence and compared the results to the results collected from an observation tool where there were very well developed descriptors, rating scales and standardisation of results had taken place. Curiously, the participants had all rated their skills at the end of the programme at a very high level with the exception of fewer than 10% who were rather more modest in their self-rating. The observation results (where there was consistency of results across all trainers/observers and participants) showed a very different picture with over 30% scoring below acceptable ratings, 10% achieving the acceptable standard and the balance rated as showing progress towards acceptable. Self-rating is perhaps not that reliable?
What is the worst offence: training programmes for purposes of regulatory compliance that either only record whether someone did the course or at least sat through a face to face or online programme. Regulators who are prepared to accept ‘presence’ as valid information about whether someone knows what they are supposed to do and can demonstrate behaviours in accordance with the standard have only themselves to blame when non-compliance leads to the kinds of serious failures in behaviour and performance we have seen frequently in sectors as diverse as banking and healthcare.
______________________________________