One Size Does Not Fit All: Making Test Security Configurable and Scalable

By Nathan Thompson, Ph.D., and Chad Kinart, MS, MBA 

Developing an organization’s test security plan involves taking into consideration many factors around test development, publishing and delivery. With software-as-a-service (SaaS) becoming more dominant, much of this cycle is now becoming cloud-based. While there are risks involved with using cloud-based platforms, they provide increased flexibility and control for organizations, as well as additional advantages (e.g., immediate republishing). This article will discuss how we can define and leverage these advantages as we rethink test security within the globalized economy in which we live and work.

Flexibility within cloud-based platforms can be especially useful to assessment organizations. Different programs and tests within an organization might vary widely in their test development, psychometric and business needs. The cliché of “one size does not fit all” is true with test security. The issue of scalability and flexibility of test security can be solved by a single configurable cloud-based platform or, in some cases, a few intimately linked platforms. The end goal is to have an assessment platform that can be easily configured to meet the various needs of the organization while maximizing the scalability of security. Running the business of assessment on a number of different platforms can be cost inefficient and increase the likelihood of errors, among other risks.

This article will begin by providing a model to evaluate security risks and identify relevant tactics to minimize those risks. Next, we will discuss the process of determining your organization’s functionality needs when it comes to systems for developing and delivering assessments, as well as the surrounding operational processes. The goal is to first think programmatically and strategically about your security needs and then about how you might identify and configure an assessment platform that best fits those needs and ensure the integrity of your tests and the resulting scores.
 

A Risk-Based Model

Before you implement scalable security, you must fully define your business requirements. This will allow you to determine your business needs and to assign risk to those needs, which will help you determine the appropriate solutions. Follow these steps:
 

1. Define your tests/programs.
2. Define your relevant threats to validity and security.
3. Define your evaluation metrics.
4. Rate the threats using your metrics.
5. Define options to minimize the threats.
    

This process can be driven by a table (see Table 1 and 2), as it allows you to better organize thoughts while providing documentation for validity and defensibility. Because there are three axes (tests/programs, threats and ratings/options) and tables are typically only two-dimensional, we recommend breaking out each test/program as its own table.

Step two is to identify a list of security risks and validity threats that are relevant to your organization. There are many possible security risks and validity threats that might be relevant to your organization, but here are a few examples.
 

1. Using test content pre-knowledge
2. Using a proxy tester
3. Using unauthorized test aids including internet
4. Receiving outside help while taking the test (earpiece, cellphone)
5. Receiving help from a proctor
6. Collusion of examinees
7. Memorizing test content and providing it to a brain dump site
    

Next, map the list of threats to each test or program. Some threats might be completely irrelevant for certain programs. Bear in mind that, as the stakes of the test increase, so do the number and severity of the threats.

A deeper evaluation of relevant threats is then necessary so they can be fully understood and minimized. A convenient first step might be to rate each threat with a Likert-type scale of concern (see Tables 1 and 2, which used a scale of 1-5). Specific outcomes should be noted, such as a proxy tester leading to a completely invalidated score. Any cost or investment concerns should be noted; theft of the item bank for a brain dump website is certainly an expensive outcome. Finally, it is important to determine methods of minimizing the threats that are most salient, their cost, potential side effects and other concerns.
 

Table 1: International Certification

Table 2: Practice Test

Implementation

In the brief example above, one method to solve these risks would be to utilize a minimum of two different platforms in order to account for different levels of security. For example, the certification could be published via a computer-based testing (CBT) vendor and the practice exam with an internet-based testing (IBT) vendor. Additionally, each of these might have its own registration and scheduling system, and the organization might additionally have a certification management system. This already brings the number of platforms to five, with only two straightforward programs, and we have not yet touched on other topics like job analysis surveys, psychometric analysis or data forensics. The downside to this approach is inefficiency. There is a good chance the multiple systems in this scenario will not talk with each other very well. In addition, some information will be redundantly stored within each of the systems. This situation can make it easier for errors to occur while making it more difficult to update information stored in systems that do not communicate well with each other.

Another method would be to look for a system that has flexibility built in to handle different levels of security needs. Having one configurable system allows the following:
 

• Preserves data integrity with less movement between systems
• Facilitates audits by having information in one place
• Houses documentation for validity evidence in one place
• Encourages collaboration and discourages silos
• Provides cost efficiency
• Offers a multitude of options for delivery without importing/exporting to multiple outside systems
   

Having all of your data in one place is great, but if a system can’t handle the different security and business needs of an organization, this option won’t necessarily help you in the long run. In order for a single solution to be scalable in terms of security, the following items need to be configurable:
 

Item banking

• Role-based security (i.e., administrators, item writers, reviewers, test schedulers, read-only)
• Process control for writing items
• Content-based security in item bank or not (i.e., by bank, by hierarchy)
• Internal vs. external item writer (i.e., within system or import items)
   

Publishing/Delivery

• Multiple delivery methods (i.e., Linear, LOFT, CAT)
• Supports job analysis surveys
• Quick republishing
• Delivery details: time limits, sections, allow review per section or test, requires answer, item randomization, option randomization
• Control of retakes
• Registration windows
• Delivery windows
• Lockdown browser
• IP location limitations
• Virtual proctoring
• Video recording
• Brick and mortar test centers
• Examinee log in:
  • Group log in, individual pass codes, group proctor codes, individual examinee proctor code, information confirmation
• Immediate results vs. delayed results
   

Data forensics and reporting

• Collusion indices
• Evaluate proctors/locations
• Parameter drift or other psychometric analysis
• Complete event tracking during delivery, with an audit report detailing everything the candidate did
   

Having the flexibility of the above functionalities in one configurable program allows an organization to easily satisfy its unique business requirements efficiently while minimizing the risks of dealing with multiple programs that have limited ability to talk with each other. In addition, the same system can be integrated with other aspects, such as certification management.
 

Conclusion

Cloud-based software is not going away anytime soon. Therefore, as organizations look at their business and security needs related to their portfolio of programs, efficiency and scalability need to be part of the equation. After an organization has identified all of the risks associated with its programs/tests, it becomes much easier to determine how much potential scalability the organization will need in the foreseeable future in regard to security. By minimizing the number of platforms being used and scaling up a smaller number of intimately connected systems, an organization can better preserve the integrity of its data while facilitating audits and best practices. As always, the ultimate goal is to support the validity of the score interpretations.

It is important to always remember that, when security increases, convenience decreases. This is why striving for a scalable security solution allows an organization to efficiently meet its complex business needs without redundancy and careless risk.