Authors: Mark Olaf, DO and Liza Smith, MD, on behalf of the Advising Students Committee in Emergency Medicine (ASC-EM)

The Standard Video Interview (SVI) was developed by the Association of American Medical Colleges (AAMC) in response to the perceived need to provide information which would speak to the professionalism and communication skills of an applicant distinct from the academic metrics already available in the Electronic residency Application Service (ERAS) application.  The SVI was added as a required component of the ERAS application for students applying to Emergency Medicine (EM) in the 2019 Match and will continue to be required for the current 2020 Match cycle.

Over the year since our last SVI update, two papers have been published in Academic Medicine addressing the validity of the SVI as an independent evaluative tool as well as the reactions of residency program leaders to its implementation.  

The first paper, Innovation in Residency Selection by Steven Bird et al., methodically describes the rigorous development and creation process of the SVI for use as an evaluative tool and makes the case that it does appear to represent a unique facet of the ERAS application.  

The authors describe the implementation of the initial institutional review board (IRB)-approved research pilot study which took place from June to December 2016 as well as the operational launch of the SVI for the 2018 ERAS application cycle which collected data from 3,532 emergency medicine applicants.

Analysis of the data from both the pilot and operational launch showed distributions that were nearly normal.  Only small correlations were noted between the SVI score and USMLE Step 1 score, Step 2 CK score, Step 2 CS score, Alpha Omega Alpha Honor Medical Society membership, and Gold humanism Honor Society membership. The pilot study results showed small differences in score means when compared among races, suggesting racial or ethnic influence on SVI scores. Additional training was provided to the operational phase SVI raters, none of whom were the same as the raters for the pilot study. On review of the data from the operational launch, comparisons between the SVI and race/ethnicity in the operational phase yielded no differences, thus effectively eliminating any racial or ethnic influence.  In addition, in the pilot study, score mean differences were medium or large when comparing applicant type (United States MD, Osteopathic, Foreign Medical Graduate and International Medical Graduate) and these differences persisted into the operational phase.

The authors offer that although early measures of validity for the SVI demonstrate promise of its utility as a unique evaluative tool in the application process, their conclusions are not without reservation.

They also bring up the issue of cost and resource utilization required to expand or even maintain this large-scale operational assessment which might necessitate transitioning to computer-based scoring of video interviews.  Computer-scoring is widely adopted in high-stakes writing assessments, but in order to even begin to accept the premise of professionalism and communication competencies being scored by a computer, it will need to be demonstrated that this method can be reliable, fair, and valid.  They also comment on the potential pitfalls related to the variability with which the SVI scores might be incorporated into the residency selection process and caution that these scores should be used in the broad context of holistic application review and not as another filterable data point.  

Program response to the operational phase of the SVI is addressed in the second paper, The AAMC Standardized Video Interview: reactions and Use by Residency Programs During the 2018 Application Cycle by Fiona Gallahue, et al.  An initial study was performed in November 2017 and utilized a program director survey to evaluate reactions to the SVI during the operational phase followed by an additional study done in January 2018 which analyzed each program’s usage of SVI video responses.

Survey results were available from 125 programs while video usage analysis from all 175 program was available. Survey data indicated that program directors were cautious regarding their use of the SVI when evaluating applicants for interviews or ranking.  Most program directors did not use the SVI in the selection process, with the most common reason cited for watching a video was curiosity. Programs were more likely to view videos of applicants with higher USMLE scores and United States MD applicants. More than half of programs indicated that they would be at least somewhat likely to use the SVI scores or videos in the future, demonstrating a willingness to add the SVI into the already complicated process of evaluating residency candidates.  

Among those programs that utilized scores in evaluating candidates for interviews or ranking, mixed reactions and divided opinions were apparent. Slightly more than half (54%) of programs utilized the SVI as part of the applicant selection process, while at the same time 70% of  of programs reported that the SVI was not important in deciding whom to invite for interviews. The most common use of the SVI was as a “tie-breaker” between another applicant with a similar profile. The methods of discerning meaning from the SVI varied as well, with a third of programs watching videos in infer meaning, and fewer programs using the SVI score distribution and percentile rank tables, and performing a direct comparison to other objective measures on the ERAS applications.

On the other hand, 46% of programs didn’t consider the SVI at all, often citing uncertainty about the validity of the SVI as reasons for not using the score.

These mixed reactions prevent us from drawing clear conclusions about adoption of the SVI, however, it may simply be that the SVI will follow the usual pattern of adoption of novel ideas or technology, with relatively few early adopters, followed by a gradual increase.

Uncertainty regarding whether the AAMC will continue to conduct the SVI beyond the 2020 Match cycle remains, as do questions about potential costs to students and/or programs should it be incorporated in the long term.  The AAMC continues to partner closely with representatives from the Society for Academic Emergency Medicine (SAEM), Association of Academic Chairs of Emergency Medicine (AACEM), Clerkship Directors in Emergency Medicine (CDEM), Council of Emergency Medicine Residency Directors (CORD), Emergency Medicine Residency Association (EMRA), American Academy of Emergency Medicine Resident and Student Association (AAEM-RSA), and the AAMC Group on Student Affairs (GSA) to address this issue and others in addition to ongoing research regarding the SVI’s incorporation and utility.  Future research into the SVI is ongoing, including seventeen EM programs having partnered with the AAMC to investigate the relationship between SVI score and performance outcomes during residency. Undoubtedly, the AAMC hopes to demonstrate that the SVI is measuring attributes that translate into an evaluation of professional and interpersonal skills that are evident in residency training.

All that being said, students applying to emergency medicine for the ERAS 2020 Match are expected  to participate in the SVI between early June and July 16, 2019. They should be encouraged to access the free, online preparation materials provided by the AAMC and reassured that expensive and exhaustive commercial preparation has not been shown to offer any benefit over these free resources.  The CORD Advising Students Committee hopes to contribute to future progress on understanding the validity, impact and use of the SVI through ongoing research through the AAMC and anticipated surveys of applicants and advisors regarding preparation for the SVI by students. We’ll keep you updated on any new developments.