Last week we began a blog series based on the first stage in the Assessment Life Cycle, Defining Target and Creating Specifications. This is our fourth post in this series and will focus on Blueprinting: Calculating Survey-based Competency Weights and Making Holistic Blueprinting Decisions. CLICK HERE to read our most recent post on this topic, Determining the Weighting Formula.

In addition to providing a means of weighting individual competencies, determining the weighting formula allows higher-level weights to be (indirectly) calculated such as those for Competency Areas. Though we generally use a more direct process for the final determination of weights, this calculation based on the survey and weighting formula provides an additional source of information for panelists to consider when setting final weights. The process involves first summing the weights for all competencies to define a kind of exam total weight. This process is then repeated for each Competency Area, thus defining a total weight per Area. Dividing the weight per Area by the exam total weight provides a proportion for each and may be interpreted as the expected weighting of each Area on the exam as a whole. The same process can be repeated for General Competencies or any level that is an aggregate of individual competencies. This aggregated information is used in the next task, making holistic blueprint decisions.

On the survey, panelists are asked directly how much weight they would give to each major ‘facet’ of the blueprint such as Competency Area, Cognitive Level and Task Type. In one version of this process, panelists make these initial judgments independently. Their judgments are then tabulated and areas of significant discrepancy among panelists are flagged for discussion. Those Competencies whose weightings derived from the survey differed significantly from the panelists judgments are also flagged. These discussions take place before the final round where panelists are invited, but not required to revisit their initial judgments.

As a side note, in our experience, it is this opportunity to share different perspectives on professional practice that directly leads to satisfaction with the blueprinting process. The blueprint is also better based on the fact that through the discussion process, panelists often gain a ‘bigger picture’ of the profession and are therefore able to better represent the profession as a whole through their ratings.

At the end of the session, panelist will have completed the final round of judgments and therefore have provided all the data required to create draft blueprints.