Taking Test Security to the Next Level Through Design

Introduction

In 2007, Gene Radwin of EMC was wrestling with a rather stark and disheartening situation. High-quality copies of items from his exams with correct answer keys were being distributed on the internet, and test scores were skyrocketing. He devised a rather unique solution. He would publish easy, non-scored, mis-keyed items (i.e., Trojan Horse items) with the exam. Would it be reasonable to think that individuals of very low ability with high scores on the exam might answer the Trojan Horse items with the published key instead of the correct answer? It worked! He reliably detected cheaters using this approach. It was rather ingenious, but Gene Radwin’s real contribution to test security is the idea that you can design security elements into your exams. You can read his description of the method here.

The premise of this article is that by building and designing security measures and countermeasures into your program, you can take exam security to the next level. It is absolutely critical that security components included within your program collect data so that you can 1) assess whether the security measures work; 2) determine the nature and scope of attacks against the security of your program; and 3) devise appropriate responses for security breaches. Because cheaters and thieves keep devising clever ways to circumvent security, the same test security measures you have relied upon in the past will not be effective in the future. You must be equally clever in devising stronger and more effective security measures.

The two prior articles in this series introduced and emphasized that the test security process can be described in four phases: protect, detect, respond and improve. Those articles discussed data forensics, web monitoring, test site monitoring, and use of proctors to detect and prevent security breaches. These time-honored techniques should be used by all testing programs.

Most test security threats fall into two classes: theft and cheating. Hence, enhanced security measures that you implement within your program should be designed to counter these two threat classes. The sections below provide examples of enhanced security techniques that can take exam security to the next level. These techniques require enhanced technologies and data collection capabilities that have been specifically designed to prevent and detect test security breaches.

Theft Prevention – Item Banks

Most testing programs use item banking systems to create items and publish exams. An insider or hacker who gains access to the item banking system can easily steal exam content, which could require suspension of all testing while the items are replaced and the tests are republished. Because the potential losses are so great, unauthorized access to the item banks must be detected and prevented, if at all possible. Thus, standard information security protocols should be strengthened.

The following techniques can be used to prevent item bank theft:

• Manage and restrict access to item banks.
• Digitally shred files and physically shred hard copies of items that are no longer needed.
• Isolate item bank access from the internet.
• Implement role-based access rights so that content writers cannot view published exams.

Theft Detection – Item Banks

Item banking software should record and maintain audit logs of systems activities and access. The software should also implement item version control and record every change that is made to an item and by whom. These records can be invaluable when an internal leak is suspected. For example, if every single version of the item is maintained, leaked items may be compared with item bank versions to determine the window of time and persons who had access to the leaked versions of the items.

The following techniques can be used to detect and identify item bank thieves:

• Create and audit item bank access logs.
• Maintain item version control logs.

Theft Prevention – Test Administration

Item harvesting is prevented when test takers are only shown a portion of the exam content. The computer adaptive test (CAT) stops administering items as quickly as possible and attempts to present items at or near an individual’s ability level. Because of this, it is very difficult for test takers to harvest the CAT item bank. Another way to administer a subset of the item pool is by using linear-on-the-fly tests (LOFT). David Foster introduced the discrete option multiple choice (DOMC) item type as a way to limit exposure of item content by presenting only a subset of an item’s answer choices to the test taker. Cheaters who use disclosed items that were administered using either of these techniques gain little advantage, because the content overlap is small between any two test administrations.

Another way that items have been harvested is through hacking electronic item banks that have been downloaded to servers within test centers. Unfortunately, a few corrupt test center employees have facilitated recording and theft of item banks.

The following techniques can be used to prevent theft of items during the testing session:

• Use a secure internet server to render and display the items, preventing hackers from accessing downloaded item banks.
• Use algorithms that administer a subset of the item pool (e.g., CAT and LOFT).
• Use item types designed to specifically prevent theft (e.g., DOMC).
• Use camera and cellphone detectors in a proctored environment.

Theft Detection – Test Administration

If the testing session when items were harvested may be identified, investigators can quickly determine who, where, when and how the items were stolen. Therefore, there is great interest among testing programs for determining the testing session when the items were harvested. The general technique is to digitally “watermark” presented items. The digital watermark allows unique identification of the testing session when the recovered items were presented. Examples of this are 1) record the order in which the items were administered; 2) record the exact subset of items that were administered; and 3) record the order in which answer options were presented. While these techniques may not work for all testing situations, items may be digitally watermarked using other techniques for tracking the source of the item theft.

The following techniques can be used to detect test pirates:

• Digitally watermark the items.
• Digitally watermark the test forms (e.g., by keeping track of displayed item subsets).
• Use forensic techniques to identify potential acts of theft, that determine where, when and by whom watermarked items and forms were stolen.

Cheating Prevention – Test Administration

Cheating can be defined as an attempt to gain an unfair advantage by improperly raising a test score. A quick internet search will reveal a variety of cheating methods. These can be grouped into using pre-knowledge of exam content, eliciting content during the exam, using unauthorized assistance by an insider or other individual, collaborating or colluding during the exam, and using a surrogate or proxy test taker. Theft and disclosure of exam content on the internet is especially troubling because it facilitates widespread cheating through pre-knowledge and decimates the investment in the item pool. Advantage gained through pre-knowledge of exam content can be negated by devaluing the disclosed content. This can be done by quickly determining the content that has been stolen and then revising the exam in such a way that cheaters (i.e., “brain dump users”) believe the disclosed content is still valid when, in fact, it is not.

The following techniques can be used to prevent brain dump users and proxy test takers from gaining an advantage:

• Republish the exam using item variants that look like original items, but are different.
• Implement and use biometric authentication to detect proxy test takers.
• Use multiple forms to thwart use of pre-knowledge.

Cheating Detection – Test Administration

All cheating detection methods are designed and implemented to detect specific kinds of cheating. Hence, greater intelligence concerning potential cheating attacks is extremely helpful. For example, a cheating method that has been observed in schools is that of educators changing answers on student answer sheets. This method may not be as easily used when tests are given by computer. As more testing is done on computers, testing programs should take advantage of computational power to detect cheating.

The following techniques can be used to detect cheating:

• Scan data using standard forensics, such as analysis of gain scores and similar responses.
• Use forensic techniques that rely upon errors in brain dump answer keys to detect users.
• Implement embedded verification questions into the exam.
• Implement Trojan Horse questions to detect use of stolen answer keys.

Summary

Because cheaters are devising more ingenious methods to cheat and the test administration environment is changing, testing program managers need to design and implement creative test security measures and countermeasures. Test security basics are essential, but can only do so much to prevent theft and cheating. The methods presented here allow you to select and design security measures that take test security to the next level and provide greater peace of mind and protection from test fraud.

This article is the third in a series focusing on strengthening exam security by using data and technology. See the first two here and here.