American Board of Radiology Response to CNN story on Test Recalls

ABR’s EXAM SECURITY

By Gary J. Becker, MD, ABR Executive Director

In the many comments on the CNN story on test recall questions, the same questions are being asked. It is important that we make clear the following points:

The American Board of Radiology (ABR) serves the public.
The ABR plays an important role in protecting patients and the public by ensuring that radiologists are able to safely and accurately interpret and perform imaging studies and image-guided procedures. After a long and arduous path of education and training (see final page of this document), the certifying process demonstrates that doctors are competent to safely and effectively interpret imaging studies and perform image-guided procedures. Once in practice, they must participate in a program of continuous professional development known as Maintenance of Certification (MOC).

ABR is replacing its proven and time-tested exam format, which includes an oral exam, with new, highly-standardized, computer-based exams that will be more objective and better able to assess the abilities of the physician examinees, without measuring extraneous factors not uncommon when candidates take the orals, such as nervousness. As we proceed, we leave behind a proven test of clinical reasoning. Not surprisingly, ABR is determined to safeguard the integrity of its certification decisions by ensuring the security of its new examinations.

Today we live in a world of instant information sharing and are aware that recalled test questions (“recalls”) have been passed around and used not only to guide study, but at least by some to memorize exact questions and answer options, so as to increase chances of passing when these questions were encountered again on future examinations. The latter is a violation of ABR rules and constitutes cheating on the examination. Still, that was the past, and it involved the “written” Qualifying Examinations that are taken in residency, not the ABR’s highest-stakes Certifying Examinations which serve as the final assessment in the certification process, at the completion of residency training.

We cannot change the past. But we CAN prepare well for the future. Since December of 2007, ABR staff, volunteers, and the ABR Board of Trustees have been working tirelessly to produce our computer- based Core and Certifying exams that ABR will begin to use next year. These new tests will assess the doctors’ abilities to actually interpret imaging studies and perform image-guided procedures rather than just memorize facts. In March of 2011, we began to put in place a critical foundation block of this effort: an exam security culture change. In our outreach to residency training program directors, faculty, program coordinators, department chairs, and residents, we are doing everything in our power to ensure that the exams are a fair and accurate reflection of a doctor’s abilities and not just a test of memorization. Today the definition of cheating is both crystal clear to all involved and rigorously enforced. As a very important addition, we are attacking the root cause—the motivation for use of recalls—by increasingly providing useful study and preparation aids, such as study guides and practice exams, and emphasizing the critical importance of candidates’ professionalism and ensuring the public trust.

All examinees must attest.
ABR also requires each prospective examinee to sign a document attesting to his/her understanding of ABR’s Exam Security Policy, and pledging that he/she has not and will not cheat. If we identify someone cheating, we take action that is serious enough to deter others. Recently, a few test-takers caught in the act of cheating have had their exam results invalidated and have been banned from ABR examinations for three to five years. Such adverse actions have a serious damaging impact on a doctor’s career.

ABR’s Exam Security Policy addresses expectations of individual candidates for certification, training programs, and program directors:
ABR’s Exam Security Policy is designed to protect the integrity of all of its certification decisions which are examination-based. Security is necessary to ensure that ABR’s exam results always and only reflect examinee attainment and not unauthorized access to information sources. Unauthorized access may include, but is not limited to: 1) giving or receiving confidential exam information at any time prior to, during, or after the administration of the exam, and/or 2) possession of study materials in any medium during the exam (this provision includes ABR’s Electronic Device Policy).

ABR has always monitored, statistically analyzed, and compared the exam performance of residency programs. We take this very seriously and have invested in tools and resources to ensure that our exams are not being compromised.

Potential consequences to individuals of ABR Exam Security Policy violations include: letter of warning, invalidation of exam results, requirement to retake an examination, barring from the examination for a period of time or until a specified milestone has been met, reporting of the incident by ABR to other parties, permanent barring from certification, and/or prosecution for copyright violation.

Potential consequences to program directors and faculty may include certificate actions based on breach of professionalism. The ABR may withdraw certification on a permanent (revocation) or temporary (suspension) basis.
ABR is also concerned that consequences at the training program level be consistent with ABR policy and its enforcement by the Board. Therefore, we have made a formal written request of the Accreditation Council for Graduate Medical Education (ACGME), which accredits the residency training programs in all specialties, to regard evidence of recall-sharing in a program as evidence of deficiency in teaching professionalism (one of the six core competencies that must be taught in every residency program), potentially warranting a program citation.

Within the past year, ABR has had to remove one candidate from the oral examination process for violation of the Electronic Device Policy. In addition, we have had to bar one candidate from the examination process for three years and another for five years, due to irregular behavior exhibited and documented during an ABR examination. In each instance, the candidate’s exam results were invalidated. In the new exam centers in which future ABR exams will be delivered, security features include biometric identification, security cameras, and proctoring.
Outside of the actual exam setting itself, ABR utilizes web surveillance and statistical methods to detect unauthorized sharing of questions. We have contacted and are currently investigating the programs for which we have evidence of sharing of recalled questions. Residency program and institutional officials have been extremely cooperative and have responded in accordance with ABR demands to remove content, completely dismantle websites, etc. ABR is now prioritizing its focus and use of resources to investigate programs and individuals about whom we have evidence of question-sharing that has occurred after the ABR’s aggressive communication campaign concerning exam security and professionalism began in the spring of 2011. All residency training program directors, program coordinators, and radiology department chairs will be required to sign attestations of understanding, support and enforcement of ABR’s Exam Security Policy.

To be certain about its certification decisions, ABR utilizes best practices for high-stakes testing organizations.  Exam integrity is the bedrock of ABR’s certification process.
We have always made sure the exams are revised regularly. Half of the questions in each Qualifying Examination are new. The other half are previously used questions drawn from a variety of prior exams. Performance data on each question are analyzed for any evidence of the question’s use as a recall. Potentially compromised questions are eliminated from the pool.

As mentioned above, ABR also has been vigilant in safeguarding exam integrity through analysis of test results by program, surveillance of Web activity and postings, and taking action whenever we have any evidence that cheating is occurring or has occurred. Statistical analysis is one important way in which ABR searches for aberrations or exceptional scores that might indicate a concern for possible effects of recall sharing. Thus far, we have not detected significant differences in scores among the programs, nor are we seeing an unexpected number of ill-prepared candidates arriving for their final oral certifying exams, which would be another indicator.

It should also be noted that each resident who takes ABR’s high-stakes Oral Certifying Examination must complete training, then go on to be passed by 10 independent oral examiner-experts in the field in order to become ABR board certified. Candidates for board-certification in radiology are among the highest performers in their medical school classes, as they must be in order to earn a residency training position in a very competitive field. And year after year, new candidates for ABR certification continue to demonstrate their observational and clinical reasoning skills on our Oral Certifying Examination.

Is the memorization of recalled questions a legitimate study activity? Has it been in the past? Will it be in the future?
A broader look at the culture of high-stakes testing in the United States is instructive, in that it helps one understand the use of recalled questions from the point of view of the young person taking one of ABR’s examinations.
Administered in residency, our Qualifying Examinations are not the highest-stakes final ABR Certifying Examination. Still, residents regard them as critical to future success. Consequently, they seek out as much information about them as they can possibly obtain. The backdrop for all this is our American culture, whose norms in the area of education and high-stakes assessment are known to all of us. One need not labor to find many examples in which used exam questions serve as study aids. It is the rare high school student who doesn’t use books of test questions to prepare for the ACTs or SATs, and the same is true for aspiring future lawyers preparing for their LSATs (Law School Admissions Test), accounting students for the CPA examinations, etc. Also, beliefs come into play. Many of ABR’s test- takers believe that the Qualifying Examinations consist of mysterious, random medical facts that no one could possibly learn without studying what’s been on other exams, thus the interest in recalled questions. Although this perception is far from the reality, the belief must be addressed.

The ABR is addressing root causes of the urge to use recalls.
The ABR is taking several steps to better inform candidates about the exam format and content, and to ensure the relevance of the exam material. Almost every question on our new computer-based Core and Certifying Examinations will be case-based and image-rich, and therefore very clinically relevant material that a well-prepared young radiologist should know. To help residents prepare, we have already published a Core Examination study guide on our website, and we are planning to release a practice version of our future Core Examination by the end of 2012 – almost a year before the first residents will actually take it in October 2013. The practice exam will give residents a chance to familiarize themselves with the user interface, question types, viewing tools, and breadth and depth of content. Finally, in the spring of 2013, a full-length Pilot Core Exam will be administered to the first group of residents who are scheduled to take the actual Core Exam in the fall of 2013. Detailed feedback will be provided to the residents to help guide their further study. In the past, the absence of these kinds of tools and opportunities may have played a contributing role in the use of recalls by radiology trainees.

Differences between the exams of the present and those to be implemented next year:
A fundamental difference between the current examination paradigm (computer-based Qualifying Exams and oral Certifying Exam) and the one to be implemented next year (all computer-based) is the nature of the questions themselves. I am focusing on the Qualifying Exams in particular, because residents have used recalls to prepare for these examinations in the past. By comparison, the use of recalls in preparation for the oral Certifying Examinations is useless, even detrimental to test-takers. The Qualifying Examinations test candidates at the level of knowledge and comprehension. At these lower levels of educational objective, achievement is demonstrated mainly by recall of material. Thus, memorization activities seem aptly matched for the task at hand. Memorization is still a very important part of learning. Think of multiplication tables, alphabet, etc. But keep in mind that whether residents memorize such basic material from textbooks or recalled questions, they still don’t become ABR- certified by simply passing the Qualifying Examinations. Indeed, each candidate who passes the Qualifying Examinations must still successfully complete training and pass the highest-stakes Oral Certifying Examination, during which 10 different examiners evaluate not just knowledge, but also application, analysis, management, clinical reasoning, and judgment. The result is that the public is protected because the ABR’s certification process guarantees that every radiologist we certify has met the requisite standard for safe and competent practice.

A few more points on memorization: There is a vast difference between ABR’s current Qualifying Examinations, on which recall of memorized material is largely the way in which one demonstrates the requisite level of achievement for the stage of training, and ABR’s future Core and Certifying Examinations. On the latter, memorization of test questions as “recalls” would undermine the integrity of the exams by reducing questions originally designed to assess clinical reasoning to a simple assessment of knowledge and memorization ability--not at all what we intend to measure on these new examinations. Fortunately, the questions on the Core and Certifying Examinations will be case-based and image-rich, and therefore will not lend themselves to memorization as well as a written text question. Still, our shift to the new exam format makes our need for tightening exam security very apparent. As we move forward with our new examinations, we simply cannot allow the integrity of ABR’s certification decisions to be undermined.

ABR’s certification decisions:
The arduous path to ABR board certification (see final page of this document) cannot be conquered by merely memorizing Qualifying Examination questions. The integrity of ABR’s certification decisions is intact, because each board-certified radiologist has succeeded in all of the following after high school:
1) four years of college study with at least a bachelor’s degree and high grade point average; 2) four years of medical school to earn a medical degree;
3) passing the U.S. Medical Licensing Examination;
4) a year of clinical internship;
5) four years of diagnostic radiology residency, mastering the physics of our very technical field and the medical and procedural skills required to practice radiology;
6) passing the physics and clinical qualifying examinations;
7) obtaining, at the completion of training, a signed attestation of the residency program director as to one’s readiness for the independent practice of radiology; and finally,
8) passing the oral certifying examination by demonstrating extensive professional knowledge, skills, and judgment to 10 independent radiology experts
Finally, once in practice, every diagnostic radiologist certified since May of 2002 must maintain his/her board certification by participating in a program of lifelong learning and continuous professional development (Maintenance of Certification), and by passing an additional secure examination at least every 10 years. Since May of 2002, the ABR no longer issues lifetime certificates of any kind. In radiation oncology, the ABR has issued exclusively time-limited certificates since 1995.

ABR’s decision to aggressively communicate its exam security policy:
In May 2011, our first Core Pilot Examination exposed our future exam questions for the first time. Just two months before this pilot exam, we realized how profoundly our situation was about to change. The ABR, which has always relied on the Oral Certifying Examination as the ultimate test of observational skills, analysis, management, clinical reasoning, and judgment, would be phasing out the oral Certifying Exam and replacing it with our rigorously constructed and highly objective computer-based Core and Certifying Examinations. The questions on the new exams will test these higher-level skills, too, rather than just knowledge and comprehension, but only if they are seen for the first time rather than memorized. This is really a brand new challenge the ABR faces to protect the integrity of its certification decisions. Any use of recalls will necessarily be regarded as a threat to the new certification paradigm, the integrity of ABR’s certification decisions, patient welfare, and the public trust. It is vital that our program directors, program coordinators, residents, and department chairs support and enforce ABR exam security. They have all agreed to do so and to sign the new attestation forms.