Responding to Exam Security Breaches

By Sarah Toton, PhD, Caveon Test Security, and Mark C. Franco, ICE-CCP, Whiteford, Taylor & Preston, LLP

Disclaimer: This article is for general information purposes only and should not be construed as legal advice.

Uh-oh. You’ve received information that your program’s test content and/or test security protocols have been breached. What should you do? Take a breath, don’t panic and read on. Below is a condensed guide to help you address a test security breach methodically, rather than reacting as the situation evolves. Preparing ahead of time is better than thinking through test security for your program during a breach, but this article is designed to help when that is simply not the case.

Identify Resources

To successfully navigate a breach, you should seek sources of guidance. First, you will need to identify all of your organization’s applicable policies and procedures related to a test security breach as these will dictate how you act in any specific situation. Next, you will need to reach out to experts that can help you deal with different aspects of the breach. These may include in-house or outside legal counsel, your current testing vendor or a test-security vendor. Then, set aside time for addressing and navigating the breach. Often, we all have full plates before a breach occurs and very little bandwidth for dealing with a crisis situation. However, a breach is the top priority and you will likely need as many resources as you can muster.

Assess the Current Situation

What do you know? How do you know it? This will help you 1) design a plan for dealing with the breach and 2) inform the people who will be involved in executing the plan. Channel your inner detective and try to answer the questions “Who?”, “What?”, “When?” and “Where?” Start with the information you already have or could find out internally and work from there. This is not the time to contact individuals allegedly involved in the breach.

For example, let’s say you receive a tip that a group of candidates is sharing content and the tipster sends you a PDF of the alleged test content:

Who? Try to determine the tipster’s identity and which candidates are implicated in the sharing of this content. Sometimes the document itself allows the author of the document to be identified. For example, the document properties of the PDF may contain the name of the author, or the order of the test items in the document may be unique to the test for a particular candidate.

What? Assess the content for legitimacy and accuracy. Is it actually your test content? How accurately has the content been captured? How much of your content is in the document? Is there an answer key? What organizational policies and procedures need to be applied or considered?

When? When was the content disclosed? Document properties and the tipster may provide helpful information.

Where? Are the candidates implicated in the sharing of this content linked to a particular test site, training center or company?

Some of these questions may be unanswerable until additional information is obtained.

Collect Additional Information

Gather all of the information you have based on your initial answers to the questions above. If the breach may involve a particular test site, look into that site’s history and characteristics. If there are specific candidates implicated, take stock of all the information that your program collects about candidates and their test session. Do you take video recordings of candidates as they test? Proctor logs? Do you have information about the candidate’s employer, training center or previous exam attempts? You don’t know what you don’t measure, so start with what you have available.

If you choose to partner with your testing vendor or a test-security vendor, they may be able to help by conducting investigations, data forensics or monitoring the web for disclosed content.

If the potential breach involves sharing test content on the internet, consider searching the web for disclosed content. Beware though, entering your actual item text into search engines can actually cause a breach by disclosing your item content. If you try this yourself, it is best to search general terms like “answer key for x exam” and use more than one search engine.

Data forensics is the statistical analysis of test data to identify unusual patterns that may be associated with cheating or test fraud. The results obtained from such analyses may not directly prove that cheating or a breach of test security protocols has occurred, but could provide the information needed to demonstrate a reasonable basis for taking certain types of action, such as invalidation of test scores.

Assess Your Policies

Do you have a candidate agreement, handbook or internal policy documents? If you use a third-party test administrator or delivery vendor, do they have exam-related policies? Generally, these materials will dictate what remedial measures are available following an exam security breach.

If you encounter circumstances that are not addressed by policies, then you should proceed carefully in considering any potential measures. You will need to balance the importance of maintaining the integrity of the exam against an obligation to ensure every candidate is treated with fundamental fairness.

If consistent with any applicable policies or prior decisions, a program can consider invalidation of test results. Otherwise, you may have to consider other courses of action. You will typically only want to take actions consistent with your organization’s policies and if you are unable to determine what those actions may be, seek legal advice before taking any action.

Take Action

Once you’ve discovered a breach, you need to address it. Your responsibilities as a testing program include ensuring the validity of test scores and protecting the integrity of the testing program. The specific actions you take will depend on the nature of the breach and your organization’s policies. If you discover a small group of candidates shared answers, score invalidation may be appropriate, as well as other punitive measures such as denial of future eligibility. If you discover that administrators at a specific test site have not been following testing protocols, you may decide to replace testing staff or shut down testing at the site. If you discover a large braindump of your test content on the web, you could deploy an emergency form, stop testing until replacement forms are ready and/or use data forensics to identify invalid scores. The goal is not just to take action, but to take justified action that is appropriate for the situation. Inappropriate action will exacerbate an already bad situation.

It is important to remember that whatever action you take remains consistent with your organizational policies and procedures. Depending on the particular circumstances, your policies and procedures may not provide enough guidance. For example, what if you don’t become aware of a violation of exam security until many months after the exam occurred and results were provided. Do your exam-related policies still apply or could there be an applicable code of conduct that would be more relevant? These are not questions that you will want to determine on your own and it would be good to consult with experts that have appropriate legal, exam, and industry or profession-related perspectives.

Communicating About Actions

Legal support is essential when communicating your actions to various stakeholders following a breach. What and how you communicate will depend on the particular circumstances of the exam security breach. It should be made clear why a specific action was taken by the organization and what opportunity, if any, is available to retest. In certain situations where revocation of a credential is also a possibility, you will need to provide an affected individual an opportunity to refute the findings and to appeal any decisions. You will also want to make sure your communications do not create any additional problems for the organization and it is recommended to have your legal support perform a thorough review on any candidate-facing communications regarding the breach.

Move Forward

If you experience a serious breach for which you were not prepared, you should proceed carefully and make sure you act in accordance with your own policies, procedures and applicable laws. Even though a breach may be stressful and uninvited, it presents an opportunity to learn from the experience to improve your program’s security.

In many situations, a serious breach has ended up being a huge catalyst for positive change and stronger programs have emerged with better policies, stakeholder buy-in and methods for monitoring test security. Do the best you can within your existing policies, leverage your resources and bring in experts to help. Then, make a plan to prevent it from happening again, detect it if it does and handle it in a fair and consistent way.

If you are fortunate enough to be reading this article and not dealing with a potential breach, stay tuned for another article on designing test security measures for your program, where we will cover topics such as candidate agreements that allow you to take action, auditing test deliveries for potential test security violations, conducting data forensics to proactively monitor your program’s test security, developing emergency forms and creating an incident response plan before a breach occurs.