Aligning Credentialing Assessment Practices With Diversity, Equity and Inclusion

Over the past few years, I’ve had the pleasure of engaging with the topic of diversity, equity and inclusion (DEI) and learning about the ideas, experiences and efforts associated with this topic from a wide variety of organizations and individuals. In 2022, I was a member of the I.C.E. DEI Task Force and presented on the topic of DEI at the I.C.E. Exchange. Three of the key takeaways from those experiences were:

1. There is a widespread interest — and often passion — for DEI.
2. Many are unsure, unaware or hesitant about how to implement DEI into their work in a meaningful way.
3. Sometimes DEI is discussed in a narrow way when it is instead far-reaching and universally applicable.

It is not hard to understand why individuals and organizations sometimes feel analysis paralysis or trepidation when they consider how to enhance and promote DEI within their practices, policies and procedures. Acknowledging and addressing current and historical inequities within our organizations and society at large can be challenging and sometimes emotionally taxing. Implementing DEI practices can be seen by some as punitive or reactionary. And, of course, DEI has such far-reaching applications that it can be difficult to know where to start.

In this article, I will not seek to educate on specific recommendations for credentialing assessments. There is already excellent guidance available from I.C.E. Exchange presenters and other credentialing and testing channels on addressing bias in exam content, selection of diverse panels of subject matter experts (SMEs) and so forth. Also, there is a risk with any “how-to” or procedural and that is the risk of applying advice in a way that assumes it is universally applicable. While there are perhaps more similarities than differences among credentialing programs within the I.C.E. community, each program is unique and may have its own challenges or requirements.

In my experience, rather than a dearth of practical guidance, the barrier to implementing lasting DEI changes often comes down to not fully recognizing DEI as relating to our inherent responsibilities to an ever-changing environment of credentialing candidates, certificants and other stakeholders. In this article, I will do my best to re-frame DEI as an opportunity to improve the consistency, interpretability and fairness of assessment processes.

Connecting Validity, Reliability and Fairness With DEI

One question I am often asked as a psychometrician is how to incorporate DEI into the design, development and evaluation of credentialing assessments. To answer that question, I must always start with a discussion of validity, reliability and fairness. If you are a psychometrician or work with one, there is no doubt you’ve heard these terms, which are further described in the Standards for Educational and Psychological Testing.

• Validity is the accuracy and interpretability of assessment outcomes. In credentialing, the main focus is content validity, the degree to which assessment content reflects the real-world competencies required and situations experienced by the target test-taking population.

• Reliability is the consistency of assessment outcomes. It can be measured in several ways according to the type of consistency desired (e.g., among exam items, among raters, among repeat administrations, among forms). Consistency is the minimum requirement to have any chance at interpretability.
• Fairness is the equitability of the assessment process and outcomes. The most recent (2014) edition of the standards introduced fairness alongside reliability and validity, in part to reflect the importance of fairness throughout the testing process, rather than just the assessment instrument itself.

Astute readers will already see this framework of validity, reliability and fairness espoused by psychometricians and other measurement professionals can serve to advance much of the same objectives of DEI for credentialing assessments. A closer look at each aspect of DEI and how it relates to validity, reliability and fairness can also show us how it can break down:

• Diversity is the inherent variability among persons and their perspectives. It is easy to think of test takers as a highly uniform group if they have similar jobs or education, but this doesn’t tell the whole story. Even within a specific test-taking population, individuals will differ in innumerable ways. Achieving consistency in assessment outcomes starts with an acknowledgment that individuals differ. In fact, this is the very reason we assess individuals: We cannot simply assume that someone has the requisite competencies.
  
  The other trap is to try to define diversity in a checklist format. For example, stratifying the selection of panelists according to United States regions sounds good on the surface, but it can lead to overrepresentation. One could successfully recruit three individuals from the “Midwest,” but they could all be from the same state or city. This also assumes that individuals within the “Midwest” share the same characteristics. This is a relatively innocuous example, but one can see how this could be even more problematic with highly charged personal characteristics.

• Equity is the fairness and justice in organizational processes and outcomes. Consistency is the backbone of fairness and justice. If we do not give every credentialing candidate a consistent testing experience, then we cannot view a pass/fail decision as a meaningful interpretation. This is an area that already receives significant attention in credentialing, and rightly so. Mitigation of bias in exam content, evaluation of testing bias (e.g., differential item functioning [DIF]) and test adaptation (e.g., translation, localization and accommodations) are all topics addressed by the National Commission for Certifying Agencies (NCCA) Standards with the objective of equitability and fairness.
  
  The one notion I’d like to add here is that there is no finish line for this important work and improvements can come from unexpected places. For example, my assessment development colleagues, who are highly proficient and diligent reviewers of bias in exam content, have noticed that I sometimes detect ambiguous language, idiomatic expressions and other readability issues more readily because I am a non-native English speaker. The reason I say this is unexpected is that, despite receiving 100% of my education in English and in the United States, my experience learning English as my third language (and also supporting family members and friends in doing so) has given me some additional perspective on identifying and addressing language and wording that can be difficult or misleading.

• Inclusion is the process of extending the opportunity for meaningful contribution and leveraging individual talent. Inclusion should be thought of as the implementation of equitable processes, rather than an open gate for any and all. The idea here should be that all qualified individuals have equal access to the opportunity to demonstrate their competencies.
  
  For example, there is often a tendency for credentialing bodies to return to SMEs who’ve had prior involvement. Of course, exam development requires training before contributors are fully effective and these individuals have already shown that they can contribute effectively. But without outreach to recruit new perspectives, a panel entirely or predominantly consisting of veteran SMEs can lead to stagnation of content development (e.g., narrow scope of exam content, outdated content), a situation where a few SMEs have undue authority or sense of ownership over the process, or other issues related to representativeness of panelists. Targeted and effortful recruitment practices can help bridge the gap, especially when in consideration of traditionally underrepresented demographics (e.g., physicians with a doctor of osteopathic medicine [DO] degree, women in STEM professions or senior-level roles, individuals with non-European ancestry).

In 2018, Deloitte published a diversity and inclusion maturity model and it casts DEI as a change management process. DEI often starts as a response to compliance requirements and is championed by a select few. With purposeful and concerted efforts and thoughtful consideration, DEI can progress toward integration into policies and procedures, then organizational strategy and, ultimately, organizational culture.

One of my colleagues often says, “Everyone is on a different step of their personal DEI journey.” I would further add that practical guidance and advice are best employed with a mindset that recognizes that we as individuals and organizations differ, but we also change over time. I invite you to take that next step, wherever you are on your journey.