When Nightmare Becomes Reality: A Testing Breach Case Study
In certification and licensure today, loss of a test form is a testing program’s worst nightmare. A compromised form impacts score validity, disrupts test delivery timelines and costs thousands of dollars to replace. Unfortunately, in February 2024, ASIS International (ASIS) discovered that the nightmare had become reality when it received notice that an entire live form of their Physical Security Professional (PSP®) certification exam was being sold online as a test preparation resource. This article will share the immediate impact of the content leak; the steps ASIS took to address the incident and lessons learned during the investigation and remediation process.
A Growing Concern
Every testing program with a desirable credential should expect bad actors to attempt to gain unauthorized access to your secure exam content and exploit it for their own profit. The individuals and businesses involved in harvesting and distribution of live test items are growing increasingly sophisticated and persistent and are being fueled by new technology developments such as generative AI and AI agents. For valuable testing credentials, the issue often is not a question of “if” content will be compromised, but a question of “when” and “how much” compromise will occur. Understanding the market for test preparation services and routinely conducting online monitoring can provide useful and actionable exam security data to mitigate the risk of compromise.
Taking Quick Action After a Leak
For ASIS, notice of its content leak came from a trusted source who discovered live PSP® exam content online during a routine search. Discovery of live content for sale required urgent action by the organization. ASIS immediately acquired the full content being sold and confirmed an entire PSP® form was compromised. Since a back-up form was not readily available, ASIS made the difficult decision to temporarily suspend administration of the exam.
While suspending the exam resulted in disruption to candidates and a fiscal impact to ASIS, it appropriately ensured that no other candidates would be tested using the compromised form. It also provided ASIS time to implement its corrective action procedures. The procedures provided valuable guidance to ASIS in responding to the 2024 incident in alignment with the ANAB ISO/IEC 17024:2012 accreditation process.
A large content breach is highly disruptive for an organization. For ASIS, that included the need to coordinate with the vendor on next steps, notify candidates and reschedule exams, quickly develop new content and publish new forms and respond to questions from key stakeholders. The organization’s official corrective action procedures provided the basic framework for ASIS to gather information, guide development and publication of new forms, and document the timeline and resolution of the problem. ASIS also used the corrective action procedures to help draft communication plans for addressing questions with candidates and other stakeholders and to guide additional remediation activities.
Improving Upon Current Procedures
While the corrective action documentation was helpful, ASIS identified several opportunities for improvement in the materials.
For example, ASIS soon discovered that the corrective action procedures were not entirely sufficient to address the investigation process. ASIS turned to experienced exam security experts for guidance on next steps, including development of an investigation plan, review of operational processes, data analyses, interviews and reporting. Working collaboratively with a vendor, ASIS sought to identify the source of the content compromise, the likely date of the compromise and whether other scores were impacted by the breach. In addition, the investigation vendor provided guidance concerning potential security gaps for follow-up consideration by ASIS.
The investigation was fruitful, and ASIS was able to identify critical information regarding the source and timing of the content compromise, as well as the scope of exposure. This was critical information for ASIS in finalizing its remediation actions and empowered ASIS to take several steps to further secure the exam and address score validity. Collaborating with experienced exam security professionals to guide this stage of the response gave ASIS confidence in the defensibility and reasonableness of the steps it took to remediate the issue.
Protecting the Test Against Future Leaks
Based on the information from the investigation and incident response process, ASIS staff, and the Professional Certification Board, subsequently identified and prioritized a series of enhancements to its foundational exam security practices. These practices include development of a form assembly and item usage policy, a test security strategic plan, a defined incident response plan, updated web monitoring practices, enhanced forensics capabilities and more defined candidate disciplinary guidelines. Development of these capabilities will better position ASIS to respond to future incidents promptly and effectively.
ASIS plans to include new contractual incentives and training requirements related to proctor performance for in-person and remote administration into its agreement with its testing vendor. For remote proctoring, this will include a better understanding of how the delivery vendor blocks unauthorized programs from being used by test takers and whether the vendor captures key candidate data during the test event. These responsibilities can make an investigation more successful and allow for more timely resolution of issues. As always, taking time to build a positive relationship with key vendor(s) remains important, because any content exposure will require a testing program to work through the incident in close communication with its vendor(s).
The PSP® exam resumed on June 1, 2024 — two and a half months after the suspension was implemented. While no significant form breach is easy to navigate, the February 2024 incident experienced by ASIS was more manageable due to the corrective action materials prepared for ANSI National Accreditation Board (ANAB) accreditation and the assistance of experienced and trusted exam security professionals.
Today, the incident has caused the PSP® to be supported by a strengthened security program. Building and implementing a comprehensive security plan, however, is an iterative and on-going process. ASIS continues to incorporate new policies and procedures arising out of the learnings from this incident that took place more than a year ago.
Does Your Organization Have a Security Program?
The nightmare of losing a test form remains a concern for every credentialing program. Efforts to steal exam content and undermine assessments are an ongoing and growing problem for testing programs. If your organization does not have a comprehensive exam security program in place, we encourage you to learn from this case study and work with experts to develop a robust program that addresses risks at each stage of the testing lifecycle. After all, if your credential is desirable, it is not a question of “if” you will experience an exam security incident, but a question of “when” and whether you are prepared.
Did you enjoy this article? I.C.E. provides education, networking and other resources for individuals who work in and serve the credentialing industry. Learn about the benefits of joining I.C.E. today. And if you enjoyed, share this article with a friend or on your social media page.