Leadership Insight

I.C.E. Exchange Session Preview — From Pilot to Practice: Implementing AI for Test Form Assembly

As AI technology develops, there is a critical partnership between technology vendors, psychometric vendors and credentialing organizations when it comes to piloting and evaluating AI-driven assessment tools.

In the upcoming I.C.E. Exchange session “From Pilot to Practice: Implementing AI for Test Form Assembly”, Rory McCorkle, PhD, MBA, CAE, ICE-CCP, PMP, SPHR — along with Anna C. Rubin, ICE-CCP, and Susan Davis-Becker, PhD — will provide best practices and key challenges of conducting pilot studies of AI-assessment technology.

Read McCorkle’s responses below on what session attendees can take away from the session.

I.C.E.: What inspired you to explore this topic, and why is it relevant now?

Rory McCorkle (RM): Many sessions at credentialing conferences have focused on using AI for automated item generation. However, far fewer have focused on how AI enables credentialing organizations to create significant efficiencies in other areas of their test development processes. This discussion on automated test assembly shows one example of how AI can significantly decrease psychometric labor — up to 95% — and in a way that has been supported by external psychometric validation.

I.C.E.: What are one or two key takeaways you hope attendees leave with?

RM: We hope that attendees walk away with a framework of how these AI-enabled innovations can be created and validated.

We also hope that they take away:

  1. The importance of cohesive partnerships between credentialing bodies and industry vendors.
  2. The critical need for pilots to gather data for third-party validation of the efforts.
  3. The importance of gathering quantitative evidence to support these evolutions, both for defensibility and accreditation.

I.C.E.: What’s a common misconception about this topic that you hope to challenge?

RM: The primary misconception we hope to challenge is the potential cost of these innovations. By using an open-source AI model, we can enable these innovations at significantly lower cost than the major commercial AI models — e.g., ChatGPT, Anthropic — and also better protect our organizational data by privately hosting these models.

I.C.E.: How do you hope your session influences future conversations or practices in the field?

RM: We hope that this session will generate ideas among attendees on other labor-intensive test development tasks. These types of projects can reduce staff time and help organizations pivot their team's time to more activities that create greater value for them and their stakeholders.

Interested in the session? Register for I.C.E. Exchange today and make sure to mark your calendars for 2:30 p.m. ET on Wednesday, Nov. 19.