What visuals do you use in your certification test design conversation?
Do you wish you had a visual to help you and your clients through your initial test design conversations? For many of us buyers and suppliers of training & certification the conversation initially revolves around low stakes versus high stakes testing. You may branch off into the levels of testing and discuss your strategy for beginner, intermediate and advanced levels which will surely lead to how the test will be implemented. We can rely on familiar testing options which we’ve seen others implement or have implemented in the past but we may arrive at a different outcome if we look at the range of solutions available as we have this conversation.
In my research I haven’t seen a visual that brings together the range of options in certification tests in a holistic manner. I’ve seen graphics which provide portions of the conversation but nothing that includes the stakes, level, type, format and implementation options that need to be covered. (Keep in mind this is just about the testing components not the entire certification strategy.)
The Spectrum of Testing graphic helps my clients view testing along a continuum, a spectrum where certain selections are best for some things and not feasible for others. We’re able to see the range of solutions and make adjustments to bring the desired result into clearer focus.
Work the spectrum conversation from the bottom up.
Low Stakes – High Stakes – The stakes refer not to the test itself but to the outcome, the consequences of the outcome of the test. This discussion branches off into revealing your initial thoughts around basic, intermediate and advanced offerings of the certification and looking at cognitive and psychomotor (physically demonstrating performance) requirements. All are foundation elements and help segue to the next level, knowledge.
Level of Knowledge – Many initially skip level of knowledge in the test conversation. The sooner this conversation is held and documented the better. Clearly setting the standard for the level of knowledge required for your certification has so many implications for the certification’s success. It lays the groundwork for the item types, format and implementation to follow.
The level of cognitive knowledge required has been categorized into six major areas and is most often referred to as Bloom’s Taxonomy. The taxonomy levels include: knowledge, comprehension, application, analysis, synthesis and evaluation. It’s easy to see where a basic level of knowledge requires less rigorous test items than certifying that someone has achieved mastery and is capable of evaluating a subject. So why discuss this now? It’s critical to the success of the certification to have the target level of knowledge clearly identified in the test plan because of the next layers in the spectrum. When this isn’t done we see tests that don’t meet the stakeholder’s nor the participant’s standards and the certification lacks credibility. Everyone involved from management to development needs to share the same understanding about the target level of the exam.
Item (Question) Types – The types of questions that can be asked on your test are usually left up to the discretion of your subject matter experts or your test developers. The types range from true-false, matching, fill-in-the blank, drag and drop, multiple choice to essay or an actual performance of the task required. Having the discussion with the key stakeholders and providing the implications of format as it relates to the other areas is critical to setting the test’s foundation. For example, having true/false items for a high stakes evaluation level exam gives the candidate a fifty-fifty chance at getting it right, not what you want on a certification where people’s safety is at stake. But if you want to be able to have a high stakes test and you want the candidates to be able to apply their skills then scenarios with a multiple choice item type may be an option.
Test Format – The test format should be dictated by the previous levels in the spectrum and the audience demographics. Test formats commonly range from paper-based, scanner, web-based, computer-based, and adaptive, to simulation, lab, board/expert review, provide service, or actually provide test items.
Test Implementation – Implementation revolves around live in-person rollouts to remote rollouts. This spectrum level is a bit different in that the areas are not as exclusive and the issues of proctoring, security and advanced options like biometrics can be discussed which may change you to alter a decision from a previous level.
Including each area of the spectrum; stakes, level, type, format and implementation options allows for a more complete test design consultation.
What visuals do you use in your certification test design conversation?
