Standard Setting is most often a judgmental process. It typically involves a panel of subject matter experts and stakeholders who must estimate the difficulty of each question for so-called minimally-competent, or borderline candidates. Those judgments are then aggregated in order to arrive at a passing score across sets of questions. Standard setting methods differ in the way that those judgments are made by panelists, and also in the way that question-level judgments are aggregated to create a passing score. We will talk about one of the most common methods of standard setting, the Modified Angoff method and save the Bookmark method for another time.

The central task of the Modified Angoff method is for panelists to estimate the percentage of minimally competent candidates who would answer each item correctly. The instructions to panelists would be to examine the question carefully, both in terms of the ‘structure’ of the question, and of the difficulty of the competency being tested. This information is used to make a judgment regarding the expected performance on that question of the minimally competent candidate. Two types of judgments are common, either the probability that any single candidate would answer the question correctly, or the number out of 100 minimally competent candidates that would answer the question correctly.

These judgments are summed for each panelist to create a recommendation for a passing score. This makes sense, because the item-level judgments are themselves pass-level judgments per question. For example, if a panelist provided judgments of 0.7, or 70% for each and every question, the passing standard would logically be 70% for the entire exam.

Typically these judgments are made over multiple rounds after which the judgments become increasingly refined. Between rounds, various types of information can be provided to panelists regarding the reasonableness of their judgments. A common type of information is impact data, or the number and percentages of candidates who would passed based upon the average or median passing score recommendation by panelists. Other types of feedback include the difficulty of each question for the candidates or the consistency of each panelist’s judgments.

As shown in the figure below, the changes in item-level ratings impact the overall recommended passing score. The passing score recommendation made in the final round is the one that is ultimately adopted.

For more detailed information on standard setting procedures and methods see the book, Setting Performance Standards: Concepts, Methods, and Perspectives, edited by Gregory Cizek and Robert Sternberg.

Learn more about this method and all of the psychometric services offered by the Yardstick team.

The Assessment Life Cycle is a conceptual framework for the psychometric processes required in a certification/licensure context. Essentially the Assessment Life Cycle is a way of organizing the processes involved in creating sound assessments into a series of easy-to-understand stages and it looks like this:

This series begins with an overview of the Assessment Life Cycle and why it is important to organizations that administer examinations.

Our goal is to demystify the processes that are unfortunately too often shrouded by the cloaks of psychometric mumbo jumbo. We hope that you find these white papers valuable and look forward to feedback!

Click HERE to learn more about the Yardstick White Paper Library.

1. The Assessment Life Cycle
This paper is an overview of the components of the Assessment Life Cycle. Here we talk about organizing the processess involved in creating sound assessments into a series of easy-to-understand stages.

2. Stage 1: Defining Targets and Creating Specifications
This first stage in the cycle is where any assessment program starts. Here test developers decide what they’d like their assessment to be able to conclude about test takers, and finish by creating a crystal clear blueprint on the number and types of test questions that need to be written.

3. Stage 2: Item Development
Stage 2 of the Assessment Life Cycle is where the test questions are created to populate your high-stakes certification/licensure item bank.

4. Stage 3: Assembling Exam Form(s)
Stage 3 discusses how to construct one or more exam forms from the quality items that have been authored and reviewed in the item bank.

5. Stage 4: Test Administration
Stage 4 discusses the sound administration of the exam form(s). All the work that has gone into building the blueprint, creating the question content, establishing the item bank, and assembling the test forms is about to go to the next level.

6. Stage 5: Item & Test Analysis
Stage 5 of the Assessment Life Cycle involves analyzing the data that has been collected from examination administration to psychometrically evaluate the quality of the items and exam forms.

7. Stage 6: Standard Setting
Stage 6 is all about the business of setting passing scores, (AKA ‘cut scores’), for examinations. The cut score(s) serves to classify candidates into categories; those who score above the cut score are judged to possess the minimum level of proficiency required for inclusion into the category, while those who score below the cut score are deemed not to possess the minimum level of proficiency required, and are classified accordingly.

Now you can have access to expert psychometric white papers, written by Yardstick’s psychometricians Greg Sadesky and Greg Pope.

Click HERE to learn more about the Yardstick White Paper Library.

Let’s review the anatomy of a multiple-choice question and some basic guidelines for the development of each.

The Stem

The stem, or “lead-in” of a question, is the main textual body that sets up the choice of options to be considered as answers. The stem should pose a clear question so that the candidate can form an answer without even looking at the options.

They should avoid biases, stereotyping, or unnecessary details. They may reference gender, race, ethnicity, culture, religion, lifestyle, age, or disability, but they should only include details that are relevant and necessary to the process of determining the answer.

Distractors (aka options)

These are incorrect answer options for test-takers to choose from. Distractors should fall into the same category as the answer. They should be consistent, logically compatible, and of similar length to the answer. They should also appear to be plausible and reasonable, not obviously wrong or outlandish. Well crafted distractors should be statements that are accurate but do not quite fully meet the requirements to be correct, so they look correct to candidates that do not know the material well enough.

It is important to be careful that distractors don’t provide grammatical cues such as subject-verb disagreement (between the stem and the distractor), inconsistent tenses, or references that are inappropriately singular or plural.

Avoid creating double options; meaning that one of the distractors could also be the correct response. When more than one option has some element of truth or accuracy but the correct response is the best, the stem should ask the student to select the best answer rather than the correct answer.

Distractors should not be opposites or absolutes (e.g., “always”, “never”) as test-wise candidates easily eliminate these options.

When possible, distractors should be listed in a sequential order by set type such as alphabetical, numerical, length, or logical sets.

Avoid “All of the above”, “None of the above”, “Some of the above” and “Fill in the blank”.

The Key (correct answer)

The key is the answer option that is correct.

Design the correct answer so that:

1. Across all questions the correct response is not consistently longer nor shorter than the distractors
2. Not full of technology, terminology or jargon unless the stem and the distractors contain the same language
3. Similar in nature to at least one of the distractors (if possible)

It can sometimes be the case that item writers tend to position the key in the middle of the options list (often “B” or “C”), and that test-wise candidates look for the answer in these same positions. Be sure to vary your placement of the correct answer.

The next step is to test the quality of the questions and edit as necessary.

Learn more…

Item analyses take many forms, but generally an item analysis is a report that provides statistical information on how well a question performs in line with the content of the question. Sometimes the reports use graphs to show the performance of questions, in addition to the statistics that provide numerical meat. The statistics and graphs are used in conjunction with the question content to determine the quality of the question (and potentially the quality of the instruction). An example of the item analysis that we have designed here at Yardstick, and use successfully with our clients, is provided below:

The interpretation of the statistics (i.e., what they mean in practical terms) is where people tend to glaze over, lapsing into a zombie-like fog. This particular item analysis is based on something called “Classical Test Theory” (CTT), a very commonly used model for evaluating the psychometric performance of items. Another model is Item Response Theory (IRT), used in applications like Computerized Adaptive Testing (CAT)…but here comes the fog so we should digress. The item analysis is composed of several sections; on the left are summary statistics:

• P-value: The proportion of candidates that got the item correct. In the example above a p-value of 0.80 means that 80% of candidates who took the item got it right (this item is fairly easy!)

• RPB: The point-biserial correlation between the test score and the item score. This is a measure of something called “discrimination” or how well the item discriminates between candidates who know their stuff and those who don’t know their stuff. We want to see a high positive correlation between the item score (e.g., for multiple choice questions 1/0, right/wrong) and the test score because this means that candidates who achieved high-test scores (e.g., 90%+) got the item right more often than candidates who achieved low-test scores (e.g., less than 30%)…which is what we would expect. In the example above the RPB is only 0.15, pretty darn low, which indicates poor item discrimination

• CRPB: The corrected point-biserial correlation between the test score and the item score. This is similar to the RPB but is “corrected” to remove the question score from the total score to avoid artificially inflating the correlation, which is why the CRPB is lower than the RPB. In the example above we see that the CRPB is -0.09…yikes! A negative discrimination means that we actually have smart candidates (who get high test scores) getting the question wrong more often than the not-so-smart candidates (who get low test scores)

Lets not get too bogged down by the RPB, CRPB mumbo jumbo, the rest of the item analysis report provides information to help make these concepts become clearer.

Moving to the right on the report is a table providing a wealth of detail on how candidates responded to the question.

The “Low”, “Med”, and “High” refer to groups of candidates in terms of their total test scores with “Low” being the group of candidates with the lowest test scores, “Med” being in the middle, and “High” being the candidates with the highest test scores. What we expect is that the “Low” candidates select the correct answer significantly less frequently than the “Med” who select the correct answer significantly less frequently than the “High”. Remember the description of discrimination? We wouldn’t expect to have candidates who get really low test scores getting the question right almost as often (or heaven forbid more often) than candidates with moderate or high test scores. What we see in the example above is just that problematic pattern; the low, med, and high candidates are all quite close in terms of the proportions selecting the correct answer, hence low discrimination.

The table also shows the “Overall” percentage of candidates selecting each alternative as well as a “Count” of the number of candidates who selected each alternative.

Below the table we have some auto-generated messages to point us in the right direction in terms of problem areas for the question.

A negative discrimination statistic is a classic sign that we might have a mis-keyed question so the message gives us that hint. We also are told that a couple of the alternatives are not drawing any candidates.

The fancy graph (sometimes referred to as a non-parametric item response model item characteristic curve…or just “the fancy graph”) on the item analysis shows the performance of the question options color-coded to the alternatives. It is a great way to get a quick visual interpretation of the item and is based on pioneering work done by Dr. Jim Ramsay, Professor Emeritus at McGill University.

The Y-axis is the proportion of candidates selecting the alternative and the X-axis is the exam score (low exam score on the left of the graph moving to higher exam scores on the right of the graph). The graph displays the information we see in the table but in more visual detail (not just the “Low”, “Mid”, “High” groups). We see that for the correct answer “B” candidates with low-test scores start off by selecting this choice more than any of the distractors, and then the green line levels off flat. This is typical of what we would see in a non-discriminating item. What we would like to see is a graph something more like the below:

This item is looking good, nice steep curve for the right answer, high discrimination statistics, the p-value is a bit low (39% of candidates got the question right) but not extreme, overall this item looks pretty nice (insert wolf whistle here)!

The example questions used in this blog article were referenced from the great resource: Diamond, J. J., & Evans, W. J. . An investigation of the cognitive correlates of test-wiseness. Journal of Educational Measurement, 9, 145-150 in the context of test-wiseness.

We are passionate about item analysis and all things testing. Visit the Meet the Experts section of our website to learn more about why we love psychometrics!

DOWNLOAD this article.

In addition to sponsoring the event and highlighting the features of our product and services, Yardstick psychometrician, Dr. Greg Sadesky, will be presenting on two topics focused on high stakes testing and assessment.

Meet Dr. Sadesky

Monday, February 4 @ 10:00 am
The Life or Death of an Item through the Eyes of a Reviewer

Participants in this workshop will have the opportunity to join in determining if items are appropriate to use on an operational assessment. Participants will receive training on basic definitions and will discuss some considerations for determining what requirements might be needed for a given purpose. For example, the difficulty threshold is typically higher on a certification exam than on an educational exam. Participants will have the opportunity to ask any questions that might help them understand the information presented in this part of the session. In order for all participants to use the same rules to determine the acceptability of an item for the purposes of this experience, the presenters will facilitate a discussion about the requirements for educational assessments prior to the mock item review. This open discussion will also allow participants to give some consideration as to similar and different requirements based on the type of exam. The facilitators will use items from a middle school math assessment to allow participants to practice their understanding of the definitions. Presenters will provide items to the participants and have them determine whether or not the item would be acceptable for operational use. Given the high-stakes nature of many exam programs, this knowledge will allow participants to be better prepared to create exam forms appropriate for the full exam population.

Tuesday, February 5 @ 3:45 pm
Automated Test Assembly (ATA) Cooking Show: The Recipe Guide for Exam Optimization

Optimizing an exam is like creating a delicious meal. There are ingredients required (items and their associated metadata) to bring together in certain amounts and configurations (exam blueprints) in order to create the most tasty exam possible. Along the way, we will need to make adjustments here or there, flavoring to taste, to avoid the creation of bias (balancing of keys, balancing of gender in item content, balancing word counts, passages, case scenarios, and so on). We then need to have stakeholders taste the dish (review/validate the exam form) to make sure we are delivering something tasty and valid. Finally we need to create a shopping list of new item development targets to plan for our next meal. This session will demonstrate the fundamentals of ATA in building an exam form from start to finish using an accessible and fun cooking show model. Attendees will learn how to use ATA to create targets to optimize an exam form and build new forms in an efficient way. Attendees will have the opportunity to participate in discussion of real world ATA issues and how to solve them.

To learn more about this event, visit the ATP website!

Item analysis, a key component in the Assessment Life Cycle, is a check-up of a program’s item bank and is done to ensure that the item bank is aligned to what the examination is measuring.

As part of our ongoing effort to offer psychometric products to customers in a comprehensive and demystifying way, Yardstick is now offering an Item Analysis and Realignment package, wherein we perform an analysis of items and deliver a scorecard of how well the items are performing. The scorecard will tell you how many items you may still need to write. It will also make recommendations on how you may improve and align items.

How it Works

1. Export current examination items to an excel or CSV file (we also work with bubble sheets).
2. Remove/redact actual question content (if necessary).
3. Provide the items to Yardstick’s psychometricians.
4. Receive an item analysis report in 5-7 business days.
5. Review the report and align your items, with or without the help of psychometricians.

DOWNLOAD A BROCHURE & LEARN MORE about this exciting psychometric product!

Natasha will work out of Yardstick’s Toronto office, connecting with clients and industry stakeholders and offering Yardstick services to certification and licensure bodies in all parts of Canada and the United States.

Learn more about Natasha and our entire Psychometrics team here

This exciting event, known as Psychometrics 101: Defensibility Bootcamp, was an enormous success, providing attendees with an overview of all psychometric processes involved in a high stakes certification/licensure assessment context represented by the Assessment Life Cycle. Seminar attendees were exposed to an understanding of each of the six stages of the Assessment Life Cycle through instruction and practical, applied, hands-on group work.

Stages of the Assessment Life Cycle

Stage 1: Defining purpose, targets, and creating specifications
Stage 2: Filling your item bank (item development)
Stage 3: Assembling your test (test development)
Stage 4: Test administration (secure testing and standards)
Stage 5: Item and test analysis (testing the quality of your items)
Stage 6: Standard setting (establishing cut scores)

This event was taught by Yardstick psychometricians Greg Sadesky, Ph.D., Greg Pope, MSc., and Michaela Geddes, Ph.D., who interacted and exchanged ideas with representatives from Canadian and US regulators, association representatives, test developers, educators, testing program executives, and members of credentialing boards.

Thank you to all who participated in the Yardstick psychometric seminar. We look forward to hosting another event in London, UK in late January and in Ottawa, Ontario in February!

About Our Experts

Greg Sadesky, Ph.D.
As VP Psychometric Services and the big man on campus for our Professional Testing Services, Greg applies his expertise to ensure that your tests are fair, sound, and interpretable. He works with our partners to demystify the process at any stage of the assessment cycle.

Greg Pope, MSc.
As VP Examination Services Greg is obsessed with ensuring that our high stakes customers get the services they need for their examination program, on time, and of the highest quality. Greg Pope and Greg Sadesky are joined at the hip when it comes to psychometrics, providing high stakes customers with soup to nuts psychometric services framed within the Assessment Life Cycle.

Michaela Geddes, Ph.D.
With a Ph.D. from the University of British Columbia, specializing in Measurement, Evaluation, and Research Methodology, Michaela works with Greg Sadesky, PhD. and Greg Pope, MSc. to provide the full gamut of psychometric services in the Assessment Life Cycle to high stakes testing clients in any sector. She comes to Yardstick with over a decade of psychometric experience working for well-respected organizations such as CTB/McGraw-Hill, the Certified General Accountants of Canada, and the government of New Zealand.

Join us for our next Boot Camp! Download the registration form.

In business since 2006, the Yardstick Edmonton office has been a constant environment of growth. We are a dynamic, entrepreneurial group that loves to work-hard, play-hard and empower our people to achieve their potential. We’re actively involved in the local community, and sponsor events with YEGrb, the Edmonton Ruby Meetup, and StartupEdmonton.

Ruby and JS Developer

Contract Position
Edmonton, Alberta
(Start date: ASAP)

Are you a developer passionate about building powerful experiences and platforms? Do you enjoy working with a great team to solve interesting challenges and deliver pioneering products? Then you need to be part of the development team at Yardstick.

Yardstick is actively seeking a full-time developer with experience in Ruby on Rails, JavaScript, and HTML5 to join our team. Experience building scalable architectures in MySQL and MongoDB is also a plus.

DOWNLOAD THE RUBY/JS DEVELOPER JOB DESCRIPTION

For more information on this opportunity or to apply, email us at: GreatJobs@GetYardstick.com