Evolving a Continuing Certification Program

By Mary Post, MBA, CAE, and Ann Harman, PhD

How MOCA Minute Came About 

The American Board of Anesthesiology (ABA) is one of 24 medical specialty certifying boards that collectively form the American Board of Medical Specialties (ABMS). The 24 Member Boards that comprise the ABMS have evolved programs for continuing certification since the introduction of recertification in the 1970s. Over the years, ABMS created Maintenance of Certification (MOC) and established standards for Member Boards to follow. The ABMS Member Board community has since come under increasing pressure from diplomates and professional medical societies to make continuing certification programs more relevant, less burdensome and less costly for diplomates. A frequent source of dissatisfaction has been with the use of high-stakes examinations that take place once every 10 years. Many diplomates believe these exams, due to their infrequency, are a poor indicator of whether they’re keeping their knowledge current. Additionally, standardized tests only encourage diplomates to cram in preparation for the exam; they retain the information long enough to pass the test, but then quickly forget it once the test is over. 

Being responsive to our diplomates’ feedback is important; however, we believe that significant changes to any Member Board’s MOC program must be done deliberatively and as part of a coherent framework that ensures continued alignment with the purpose, meaning or value of our certificates. The ABA’s transition from a periodic high-stakes examination to the Maintenance of Certification in Anesthesiology™ (MOCA) Minute longitudinal assessment was a result of a carefully considered and clearly articulated MOCA^® program redesign, called MOCA 2.0^®. 

The MOCA 2.0 redesign process began by ensuring that the board understood why it wanted to redesign our MOC process and by articulating a clear set of core program design principles. These principles guided our redesign efforts and kept us grounded in choosing to replace some traditional components of the program with new processes.

What Came Before MOCA Minute?

The ABMS conceptualized the MOC concept in 1999 and mandated in 2000 that each Member Board implement specialty-specific MOC programs. As a Member Board of the ABMS, the ABA was charged with implementing MOC activities to ensure certified anesthesiologists maintained an up-to-date fund of knowledge and demonstrated commitment to quality clinical outcomes and patient safety.

We started issuing time-limed certificates in 2000 and officially launched our MOCA^® program in 2004. Our initial MOCA program was based on a 10-year cycle. It included an assessment of:

• Professionalism and professional standing (medical licensure)
• Lifelong learning and self-assessment
• An examination of knowledge, judgement and skills every 10 years
• Periodic evaluations of practice performance and improvement

This four-part framework guides all ABMS MOC programs and allows diplomates to demonstrate proficiency in six general physician competencies.

ABA diplomates certified in 2000 or after hold a time-limited certificate and are enrolled in MOCA after initial board certification. Participation in MOCA by non-time limited diplomates, those certified before 2000, is voluntary and encouraged.

Over the 16 years that MOC has been a part of the board certification process, there has been a growing body of data suggesting that MOC programs are beneficial. At the same time, there is evidence that these programs have weaknesses and many diplomates have reservations about the process. They particularly dislike the periodic high-stakes standardized tests that were a component of all MOC programs, questioning the benefit and relevance of such a time-consuming and costly requirement. They argued that the exam only encouraged “just-in-time learning for just-in-time forgetting.”

Adding Value with MOCA 2.0

After several years of gathering feedback on MOCA, the ABA began redesigning the program in 2011. This involved careful consideration and recommitment to the program’s purpose; what we intended for a diplomate’s participation in MOCA to signify to patients, the public and the profession; and what value the program should provide to diplomates, patients, the public and the profession. The goal was to create an updated MOCA program that had high relevance to a diplomate’s practice, a high impact on quality of patient care and the least burden possible to our diplomates.

As we began the redesign process, the board reaffirmed the purpose of MOC programs is to create dependable mechanisms by which diplomates, colleagues and patients can be assured they are staying current in their field and striving to improve the quality of care throughout their career. While MOC is an extension of initial certification, it has a different purpose than initial certification, which is to certify, at the end of residency training, that an anesthesiologist is ready for unsupervised practice in the field. 

We believe that through our MOCA program we should not primarily serve as gate-keepers into the profession, as we do with initial certification and as we did with recertification, which preceded MOC. Rather, we should serve as a valued and trusted colleague to diplomates as we work together to support their commitment to lifelong learning and improving patient outcomes.

The vision for MOCA 2.0 that emerged over many conversations by the board focused on a program that could help our diplomates access online, self-directed learning that is relevant to their individual professional development and learning goals, and anchored to performance assessment and improvement. Based on this vision, the board established eight design principles to guide the MOCA 2.0 redesign work:

• Promote and support continuous lifelong learning
• Focus on content relevant to diplomates’ practices
• Develop a professionally and publicly credible process
• Incorporate ongoing continuous assessment
• Encourage group discussion
• Incorporate principles of adult learning theory
• Facilitate Quality Improvement and Patient Safety
• Integrate Parts I, II, III and IV of the ABMS MOC framework

The core of the redesigned MOCA 2.0 program is a new longitudinal assessment, called the MOCA Minute^®, that replaces the once-every-10-years examination to assess whether diplomates are keeping their specialty-specific medical knowledge current. In 2013, the ABA began developing the MOCA Minute, which was designed to be central to and supportive of the other components of our MOCA program.

In 2014, the ABA initiated a small-scale pilot test of a longitudinal assessment model that, we hoped, could replace our previous periodic secure assessment. Analyses of the early pilot test results showed diplomates who actively participated scored higher on the once-every-10-year high-stakes examination than diplomates who did not.¹ As a result, in 2016, the ABA launched a large-scale pilot test of the MOCA Minute^® to begin the transition from our periodic secure examination to a continuous assessment model. This new longitudinal assessment approach to assessing diplomates’ knowledge, judgment and skills has been was very well-received by our diplomates.  

In 2018 the ABMS approved the MOCA Minute as a permanent part of the ABA’s MOCA program. Since the 2016 launch of the MOCA Minute pilot, diplomates of other boards have become increasingly aware of this alternative to high-stakes examination for maintaining certification and pushback from the community of diplomates has continued. With the increasing calls for other boards to adopt longitudinal assessments in their MOC programs, many member boards have begun to rapidly replace their current point-in-time examinations with longitudinal assessments.

MOCA^® Minute: Continuous, Personalized Demonstration of Knowledge and Specific Feedback

The MOCA Minute supports busy physicians by engaging them in a continuous, dynamic assessment of their knowledge about topics that all anesthesiologists should know and be able to recall quickly. It offers diplomates a more relevant and personalized approach to lifelong learning and can inform their continuing medical education (CME) choices. Aggregated data from the MOCA Minute also allows us to identify knowledge gaps across the diplomate corps, which we can help to address by feeding this information back to diplomates and the CME provider community. 

In MOCA 2.0, diplomates fill out a practice profile, which drives customized content for their MOCA Minute questions. They answer 30 MOCA Minute questions per calendar quarter, 120 questions per calendar year. Diplomates have one minute to answer each question. All questions are single-best-answer, multiple choice questions with four response options. They are the same type of question that was formerly on the MOCA Exam and address the same type of “walking around” knowledge.

Once a diplomate answers a MOCA Minute question the countdown timer stops, and they are presented with two additional follow-up questions related to the MOCA Minute question they just answered. The first follow-up question asks how confident they are that their response to the MOCA Minute question is correct. They have three response options to choose from: Very confident, Somewhat confident, and Not at all confident. Diplomates are required to answer this confidence question because it is used by the MOCA Minute question algorithm to determine which questions will be repeated to the diplomate in the future; questions about which the diplomate is confidently wrong are prioritized for repeating in the future to reinforce the diplomate’s learning.

The second follow up question asks the diplomate to tell us how relevant the content of the question is to their practice. This question is optional, but diplomates are encouraged to provide feedback. Their responses to this relevance question are used to help better match specific MOCA Minute questions in the future. 

Each diplomate receives immediate feedback on whether he or she answered the question correctly. Every diplomate is immediately linked to the feedback for the MOCA Minute question he or she just answered, regardless of whether the question was answered correctly. Every feedback page follows the same format and contains the same elements, which are as follows:

• The MOCA Minute question and the four response options
• The correct answer
• The answer the diplomate selected
• The key point of the question
• A critique, which includes 1 to 3 brief paragraphs explaining the key point of the question and why the correct answer is correct, and the other options are incorrect.
• Links to references in the literature that the diplomate can access to learn more about the topic 

Diplomates may access their MOCA Minute question histories in their portal accounts. From the question history page, a diplomate can return to the feedback page for any MOCA Minute question they have previously answered and access the provided references or connect to specific CME products that have been linked by CME providers to specific MOCA Minute questions. 

Evaluating MOCA Minute Performance

The purpose of the MOCA Minute component of MOCA 2.0 is to help anesthesiologists continually self-assess their general and specialty-specific medical knowledge, identify possible knowledge gaps and build a personalized learning plan. The MOCA Minute also gives the ABA a mechanism to make fair and defensible decisions about whether diplomates are maintaining their medical knowledge over time. To provide an appropriate and useful metric to meet both these purposes the ABA is using Measurement Decision Theory (MDT). 

MDT is an alternative to typical scoring models. It estimates the probability the physician is keeping their medical knowledge up-to-date based on their pattern of responses to MOCA Minute questions over time. The MDT probability estimate is continually updated as each new MOCA Minute question is answered. Any changes in the trend of the probability estimate over time can be detected and evaluated against an established performance standard.

The threshold for concern that medical knowledge is not up-to-date is at a probability value or p-value of less than 0.1; that is, there is a less than one in 10 chance the physician is keeping their medical knowledge up-to-date based on their pattern of MOCA Minute question responses. Physicians can see their current MDT p-value and trend line on their web portal account.

Diplomate Engagement and Response 

Eighty-nine percent of diplomates who had previously taken the MOCA exam considered MOCA Minute as a better model, based on responses to a 2017 survey designed to gauge perceptions of MOCA Minute. Other survey results from the approximately 4,000 respondents indicate that MOCA minute addressed some of the intended goals (see Table 1). For example, 77 percent agreed that MOCA Minute works very well or somewhat well as a self-assessment tool to help identify knowledge gaps

Table 1. Survey results

Response scale: * Very / Somewhat / Neutral / Not very / Not at all; ** Very / Somewhat / Not at all; *** Very / Somewhat / Not very / Not at all

Conclusion

The integration of the MOCA Minute longitudinal assessment into our MOCA 2.0 program has been very successful. We believe in the context of continuing certification that longitudinal assessment methods are most appropriate because they provide ongoing feedback to our diplomates about whether they are keeping their specialty-specific medical knowledge current. It also provides a rich data collection and feedback looping system that not only allows the diplomate to identify areas of continuing medical education that may be most useful to them, but also to the CME-provider community about where additional CME products and resources may be needed by the anesthesiologist community writ large. 

The change as significant as transitioning from our periodic high-stakes MOCA examination to the MOCA Minute longitudinal assessment was done deliberatively and as part of a coherent framework that ensured continued alignment of our overall MOCA program with the purpose, meaning and value of our certificates. The success of this endeavor was the result of our board clearly understanding why it wanted to redesign our MOC process and by articulating a clear set of core program design principles to guide our redesign efforts throughout the process and keep us grounded about why we were choosing to adopt a newer alternative assessment process. 

We learned several lessons from the MOCA Minute initial and large-scale pilots about how to build a new product on a short development timeline. MOCA Minute evolved, shifting its value proposition and purpose. We did not have an opportunity to fully vet early use cases that had an impact later, causing complications in development. Regular modifications to our technical infrastructure were required to support development and enhancements. New features and interfaces were added quickly to improve functionality, causing the scope of work to expand continually. We operated under a short timeframe with the goal of demonstrating to our diplomates that we heard their feedback and could deliver value in continuing certification program. More time to deeply vet the use of the product with our customers would have allowed us to map a development course for the product through several iterations, saving us time, resources and challenges. 

Aligning our communications and policy decisions within this short timeframe was also a challenge. Our leadership was positioned to present about this product while development and decisions were in progress. We were often pivoting to accommodate strategic discussions and development challenges. Significant effort was required to ensure our communications across the organization remained aligned and focused on key messages under constantly changing conditions. It would have been more prudent to position Board leadership with the global perspective and message and allow time for the operational components to be worked out. 

Despite these challenges, the success of this endeavor has been recognized throughout the ABMS Board community, and today all ABMS Member Boards have or are in the process of implementing some form of longitudinal assessment program or alternative to the burdensome highly secure, point-in-time examination as part of continuing certification. 

Reference:

1. Sun H, Zhou Y, Culley DJ, Lien CA, Harman AE, Warner DO. Association between Participation in an Intensive Longitudinal Assessment Program and Performance on a Cognitive Examination in the Maintenance of Certification in Anesthesiology Program(R). Anesthesiology 2016;125:1046-55.

Additional Resources 

For CEO members, materials from an executive conversation on strategic re-imagining of maintenance of certification can be found below:

Presentation: ICE Executive Conversation on a Strategic Re-imagining of a MoC

Recording: ICE Executive Conversation on a Strategic Re-imagining of a Maintenance of Certification