Competency-based medical education (CBME) is a promising framework to drive education, training, and lifelong learning as well as improvement in health care, according to presenters of the Lois Margaret Nora Endowed Lecture Plenary at ABMS Conference 2023.
The time is ripe for CBME because many trainees are ill-prepared to provide unsupervised care for patients, which ultimately results in poor patient outcomes, agreed the speakers who provided various proof points. Holly Caretta-Weyer, MD, MHPE, Director of Assessment and Interim Associate Dean of Admissions at Stanford University School of Medicine, noted that between 45 percent and 65 percent of program directors graduate at least one person every year who they wouldn’t trust to care for their family members. Dowin Boatright, MD, MBA, MHS, Vice Chair for Research in the Ronald O. Perelman Department of Emergency Medicine, and Associate Professor of Emergency Medicine and Population Health at New York University Grossman School of Medicine, cited studies demonstrating that the number of avoidable deaths is on the rise, and that every American will experience at least one diagnostic error in their lifetime. “Since 2014, life expectancy in the United States has dropped,” stated Warren Newton, MD, MPH, President and Chief Executive Officer of the American Board of Family Medicine (ABFM). “Our performance with population health outcomes is the worst among developed countries,” he added.
CBME offers a model
“Medical education ought to be a continuum from the day you decide to go to medical school until the day you retire,” Dr. Caretta-Weyer stated.
While many specialties focus on entrustable professional activities (EPAs), which she rebrands as “everyday physician activities,” assessment can’t just be EPAs. It has to be programmatic assessment, she stressed. EPAs are essential activities, but teamwork, communication, and professionalism are also necessary to provide care. Assessing the latter competencies must be done in a way that is meaningful and complimentary to initial and continuing certification, she said. As an example, ad hoc entrustment decisions provide feedback and coaching, tying assessment back to individualized learning. Similarly, feedback is provided as part of the continuing certification process.
Medical educators and specialty boards need to think about how summative entrustment decisions are being made because they are the bridge between graduate medical education and initial certification. These decisions should be made in a robust and rigorous way that is tied to initial certification using a shared mental model, Dr. Caretta-Weyer said. Some of the questions she raised include: How do we assess the competence of trainees or physicians, and either keep them at a certain level or move them to the next level where they need to be? How do we provide frequent judgments of an individual’s readiness to practice or ability to continue to practice? How often should these judgments be made and in what format? How do we assess the things we actually care about? Finally, how do we provide the appropriate feedback to connect the data to individualized learning across the continuum? All these decisions come with risks and assumptions that must be verified with the stakeholders, that is, the learners and patients. “CBME is a patient-focused and learner-centered framework,” she said. “It’s about meeting the patients’ needs and doing right by our learners.”
Regarding programmatic assessment from a board perspective, similar questions must be addressed. How do we design certification processes, particularly continuing certification, to facilitate lifelong learning? How do we get physicians to recognize when the current knowledge they have is insufficient to address the issue in front of them and how do we get them to fill that gap? “CBME gives us a model,” she concluded.
CBME may be vulnerable to bias
Dr. Boatright raised concerns that CBME may be vulnerable to bias. Among the literature identifying bias and discrimination in medical education he cited was an Association of American Medical Colleges study that showed approximately nine percent of underrepresented in medicine (URiM) medical students reported receiving lower assessments based on race and ethnicity. Five percent of Asian students reported the same. In another study, researchers from the University of California, San Francisco Medical Center found that all its URiM medical students received lower assessments, on average, than their white peers. Although these differences were small — often less than one-tenth of a point — they resulted in URiM students receiving half as many honors, on average, as their white counterparts.
Another study evaluated word categories (i.e., standout traits, ability, grindstone habits, and compassion) used to describe medical students based on race and gender in the dean’s letter. While white medical students were significantly more likely to be described with standout traits (e.g., exceptional, best), Black students were more likely to be described as competent. Female students were more likely to be described as caring and compassionate.
CBME can help determine more objectively the skill set of trainees, identifying those who may need additional preparation to be ready for unsupervised practice, Dr. Boatright said. He also acknowledged that milestones and CBME are linked to patient outcomes. One study found that residents who scored lower on professionalism and communication milestones toward the end of residency were significantly more likely to receive patient complaints when they first start practicing. When evaluating assessment of emergency medicine milestones to identify differences in race, ethnicity, and sex, Dr. Boatright and his colleagues found that at the initial point of assessment, all residents were given the same rating. But by the end of the fourth year of training, all non-white residents received lower milestone scores than whites. The same pattern was consistent across all six competencies, he said, which appears to represent a global devaluation of all students of color that is also being driven by gender. “We must acknowledge that CBME has limitations and vulnerabilities that we should consider when refining it,” Dr. Boatright concluded.
Competency-based continuing certification
“Competency-based assessment can help drive change in health and health care, but it is a means, rather than an end,” said Dr. Newton, who elaborated on the future of board certification.
ABFM is working toward competency-based board eligibility and has developed core outcomes, which are similar to the EPAs used in other specialties. As of July 2023, all family medicine residents will be held to these standards, which will be phased in over three years. The discipline is launching a national strategy for faculty development, partnering with the family medicine academic organizations.
At the end of residency, residents become proficient in practice. But certifying boards along with other organizations, are trying to support practicing physicians in attaining mastery in practice, Dr. Newton noted. He cited studies that show cognitive skills need to be kept current, individual self-assessment is insufficient, testing enhances learning and motivation, and goals and consequences are important motivators.
“The last five years for us and many other boards has brought a spirit of continuous assessment and learning,” he said. ABFM has implemented longitudinal assessment and started a national journal club, as well as refreshed knowledge self-assessment and professionalism guidelines. Longitudinal assessment is a game changer because it offers both summative and formative assessment, Dr. Newton stated. All 104,000-plus ABFM diplomates participate in continuing certification.
Member boards traditionally have focused on knowledge and patient care; this focus will continue as we consider how to more effectively close gaps through continuing medical education, he said. But it is the other core competencies where the major focus for competency-based continuing certification should be. Gaps in communication are critical to a host of outcomes, including medication adherence and shared decision-making. It’s a difficult space to assess and the context is critical, Dr. Newton said. Some boards assess communication during oral exams; one board uses patients to assess communication as a component of board eligibility. Continuing medical education providers are beginning to use crowd sourcing assessments via video.
Another area for innovation is systems-based practice – improving performance improvement. While specialties differ, there is agreement that there are too many and often inappropriate quality measures, Dr. Newton noted. Through the Center for Professionalism and Value in Health Care led by Bob Phillips, MD, MSPH, ABFM aims to develop, test, implement, and disseminate measures that matter for primary care. More broadly, performance improvement must address the major problems of the profession — health disparities, burnout, and cost — if the profession is to remain relevant, he added.
Professionalism and the social contract must also be a focus, Dr. Newton said. The social contract affords physicians prestige, better compensation, and more professional autonomy in return for abiding by an ethical code, self-regulating, and addressing the needs of society. An important problem is that many physicians do not understand their obligations under the social contract. In addition, some younger physicians see the language of professionalism as outdated and patriarchal. “We must rekindle the profession’s values and develop language that resonates across physicians, even as we develop data supporting improvement of professionalism,” Dr. Newton concluded.
