Addressing Bias in Certification Programs: A Q&A With Clarence Chaffee

The concept of fairness is referenced throughout the “Standards for the Accreditation of Certification Programs” published by the National Commission for Certifying Agencies (NCCA). For example, certification programs must have a publicly available nondiscrimination and fairness policy (Standard 6), and policies related to appeals and reconsideration of adverse decisions must be fair (Standards 6 and 7). Additionally, there should be fair opportunities for a broad range of subject matter expert (SME) participation (Standard 13) and programs should ensure the fairness of the exam for all populations (Standard 19).

When talking about fairness, the topics of diversity, equity and bias naturally follow. The 2021 I.C.E. Exchange program, as well as other recent conference programs, included several sessions on these topics, but understanding how to critically evaluate existing certification programs — and successfully implement any necessary changes — can be challenging.

Clarence “Buck” Chaffee is the founder and president of Caviart Group, a testing and certification consulting company based in Virginia. Chaffee has provided counsel to numerous certification organizations and spoken extensively on addressing bias in certification programs, most recently giving a presentation at the American Board of Medical Specialties conference called “Multidimensional Approaches to Addressing Bias in Certification Programs." The following Q&A explores the topic of his presentation further.

What is Caviart Group’s experience in addressing bias in certification programs?

My career in the testing industry started over 40 years ago, with responsibilities for certification and licensure programs in the U.S. and abroad. I founded Caviart Group 16 years ago, and we specialize in developing new certification programs and advancing existing programs, particularly in areas like judgment, ethical decision-making and other skills that are harder to assess with a multiple-choice test.

Addressing bias applies to every type of assessment. In performance testing, when the candidate is in-person, it’s obvious where bias comes in; but bias also exists in all types of computerized testing.

What types of bias do you commonly see across credentialing programs?

We define bias as anything that affects judgment of the candidate that is not pertinent to the knowledge, skill or ability we are trying to measure. Bias can be positive or negative — it goes both ways. Bias can occur at any stage of the program, from exam design and construction to content, administration and scoring.

Starting with the job task analysis: How do you define the profession? Who do you survey? If you only survey association members, they might not represent the full spectrum of the profession and you already have a biased view.

There can be bias related to age, appearance, background, employer, language or skin tone; but even if there is diversity in appearance, there still could be bias if everyone has the same educational background or works in the same part of the country.

I’ve seen instances on an item-writing committee where the whole group agreed, but one person said, “That’s not true where I come from.” If that person wasn’t in the room, the item would have been biased against candidates from that location.

The SMEs may be considered experts, but that doesn’t mean they know everything that’s going on in the field, across the country or in the world. It’s important to define expertise in a broad, inclusive manner, rather than an overly simplified way. This type of broad, inclusive thinking must permeate the whole process.

What are some of the approaches you recommend to address bias in certification programs?

It starts with diversity in recruiting for the job task analysis and your SMEs — diversity of thought, diversity of practice and other dimensions of diversity. One major source of bias I’ve seen over the years is only surveying senior practitioners, who may not be in touch with what it’s like to be an entry-level professional today. It’s the subtle aspects of bias that are most dangerous because they can slip through easily without anyone realizing it.

How can a certification organization check if there is bias in its program?

Start by looking at the results of the assessment. To do that, you need to collect demographic information. If certain populations are performing differently than other groups, you need to explore why.

It’s also important to get feedback from candidates, to survey them after the exam and ask if the assessment reflects their practice. If some candidates are saying no, why is that? There’s the reality of fairness and there’s also the perception — candidates and the industry should perceive the exam as rigorous and a fair reflection of the profession.

To really ensure fairness, organizations must delve into the real issues that affect real people, with real outcomes — so that everyone has the same opportunity to succeed.