In August, 2006, the Emergency Medicine Residents' Association (EMRA) published a survey of boards review courses in emergency medicine. Two principals of Challenger corporation responded via the following letter to the EMRA Newsletter editor. In addition to correcting assorted factual errors, the response letter provides a structured commentary on the research design and methodological shortcomings of this type of survey approach. The letter also covers formal statistical shortcomings of the EMRA survey and points to methods of improving these techniques. Finally, the letter reports the actual effects of Challenger methodology on boards review candidates and invites the EMRA authors and others to consult with Challenger forward-going in the design and use of physician customer survey information.

October 31, 2006

Editor
EMRA Newsletter
Emergency Medicine Residents' Association
ACEP Building
1125 Executive Circle
Irving, TX 75038

Re: Board Review Course Survey Results

Dear Colleague:

Your newsletter published a report on Boards Review Course Survey Results, conducted by Drs. Landman, Devore and Gardner, in the August/September issue of your newsletter. Med-Challenger, a board review course published by Challenger Corporation was among the courses, seminars, and other media included in the report. We are writing to comment on the design, credibility, and conclusions of this document.

Let us mention to begin with that it would be useful when someone carries out this kind of study to make an effort to be current about items such as course titles, corporate contact information and Web pages. While those things appeared to be correct for most, perhaps all, of the other media reviewed, in our case, we noted the following deficiencies:

1. the phone number listed for our company is five years out of date
2. there was no Web link provided, although the other media reviewed had links and ours is easily obtainable by typing our name into any Web search engine
3. the title of the product reviewed is not accurate, and we are not even clear as to which product was reviewed. We shall have more to say on this point below.

Challenger Corporation produces and publishes several programs in the Med-Challenger series. One of these is entitled Med-Challenger EM for Residents. However, we also have several programs for practicing physicians with titles that are approximately in this space. There is good reason to believe that your respondents were actually using a medley of products, as opposed to one product, and using them at different times or under different release dates. The reason we say so is that the respondents quote a variety of wide ranging prices and a variety of venues in which they encountered the "Med-Challenger" course they were using. So, the survey does not actually uncover a real response to the use of the same course, and it appears that many of the respondents were apparently using a course aimed at practicing physicians as opposed to residents in training. In such circumstances, it would not be unusual to find the resident user complaining that the course was too complex or contained too much content.

Secondly, it is a common experience for us to find that hospitals and residencies continue to try to use out of date versions of training courses, including ours, perhaps in the desire to save money. We get a fair number of calls from folks asking us why "the course" doesn't work on their computers at the hospital. Upon investigation, we find out that the version they have is five or ten years old and has not been updated. Why anyone would want to be using a medical training resource that is ten years old is a mystery to us, but it happens all the time! It is not clear from your survey, nor could the authors possibly be in a position to know, whether the actual version of the course being used by any given respondent to the survey was current at the time or an out of date version. Since our courses are peer reviewed and then released as an update on about a two-year cycle, the version/age of the course being used is an additional serious potential confound in the study.

Next, we, meaning the authors of this letter, have many years experience in the design of research studies, including user surveys, and the evaluation of the credibility of studies of one kind or another. We recognize that the EMRA board review report is not intended to be "scientific," but that is not a good enough reason to refrain from pointing out its shortcomings. Here are a few:

1. The theory of heuristics¹ tells us among other things that the use of anecdotal information is a poor barometer for the underlying base rate of any human behavior or condition, whether it is the incidence of a disease, the true risk attached to a range of activities [e.g., the average person thinks plane crashes are more of a risk than driving to the airport], or the effectiveness of any given treatment. In the case of the EMRA report, you are commenting on the effectiveness of educational "treatments," and relying on anecdotal information. Anecdotes tend to stand out because they are salient, not because they are accurate reflections of the underlying specificity or sensitivity of a treatment. They are conspicuous, but a poor measure of reality.
2. Survey researchers well know that one of the shortcomings of most survey techniques is that they are biased in favor of outlier respondents. The average target of a survey is commonly either moderately unhappy or moderately happy; they are not sufficiently engaged by the topic to participate. In contrast, the person who is motivated to respond is either the truly disgruntled user or the one who has missionary zeal for the product consumed. If one just looks at the range of responses made about the reviewed Challenger course [and some of the others as well], you will see mostly outlier remarks—"it was fabulous" or "I hated it." What does that tell us? It doesn't tell us much at all about the average user, and more likely it tells us that at least some of the respondents, as we mentioned before, were using different products!
3. Statistical theory depends for its credibility on a few important factors, among which we might include research design, power, and effect sizes. Once again, while acknowledging that the EMRA report was not intended to be formal, we can still identify some issues in all these regards. First, any effective study design has to avoid as much as possible the deleterious effects of selection bias and the presence of confounds. Selection bias can manifest itself in a lot of ways: an unrepresentative sample [e.g., residents from different years using different versions of different products but all responding to the same survey]; a survey format that screens out or inadvertently leaves out the normal part of the expected "tail" of distribution of respondents; who defects or fails to complete the treatment; and so on. Confounds are those factors that can call into question whether a study is actually evaluating what it claims to be evaluating. For example, when you lump live seminars with computer programs, there are a lot of other factors—cost, compression of time [one week for a seminar versus a distributed learning tenure for a course you can use for months on end], instructional mode, group study versus study on one's own—that can potentially confound a query about what type of study resource is more or less effective. It also is distressing if the category allocations are either overlapping [for example, do the authors realize that people attending the ICEP and Ohio ACEP seminars are simultaneously using the Challenger course in a parallel lab during those courses to round out their study?!] or if the media reviewed are not placed in the correct or in all relevant categories. For example, all of our courses are both available in offline but also in online format, so why are we not included in the Web-delivered category? Several of the very alternative media you are reviewing actually include the Challenger content in their instructional media or our training method at their review courses! When that occurs, you have a research design confound that is highly material to the supposed findings of this survey.
4. Perhaps the most significant deficiency from a purely statistical perspective is the low power of the sample size. The overall survey went to some 3,500 addresses, of which, once the winnowing had occurred, 29 [!] claimed to have used our course, or at least some version of one of our courses. Even if the study design had been robust, selection bias had been overcome, and confounds were adjusted for, the opinions of such a small sample of respondents would not tell us much of anything about the population of residents using our materials in the real world….because the sample is not large enough to do so!!! A statistician would say that the "p value" is so high as to be meaningless. In other words, even with correction of all design errors, these results would as often as not be obtained purely by chance! If you did the survey over again, you could get another chance set of responses that might wildly differ from these. In fact, if you restructured the survey to address the confounds and so forth, you certainly would get a different response.

We did not write this letter to beat up on the survey, its authors, or their motivations. But documents like the EMRA report can create impressions that are not a fair reflection of what they claim to measure. Those impressions can have a significant impact on a small firm such as ours that depends entirely on its reputation for its success.

In fact, Challenger has been in business for 16 years and has a longstanding interest in the effectiveness of its materials for its physician customers. We devote enormous time and money, relative to our size, to collect and evaluate information from our physician clients. More than 200 independent clinical authors, editors, and reviewers are engaged in this process, and that output is monitored by a Physician Advisory Board (PAB) whose views are not controlled by commercial motivations. It also may interest you to know that 50 prominent American residencies, including eighteen emergency medicine residencies, use our content and management information system to train, evaluate and remediate more than 1,000 residents. That is a far more representative sample of the resident user environment than the 29 outliers in the EMRA study, and we keep detailed, current and verifiable information on how they perform, both over time individually, and by comparison with a "blind" pool of peers nationally. We would be happy to invite the authors of your report to consult with our PAB members and our Development Department with respect to how those comparisons are calculated and how our conclusions are arrived at. While specific information on any individual resident or any individual program is of course confidential, the collective performance of our user audience is not.

In the course of nearly 16 years in which our emergency medicine training course has been commercially available in one form or another for emergency medicine physicians and residents, we have had approximately 16,000 individual physician users, of whom perhaps 3,500 have been residents. Of those 16,000 total users, we can verify that about 6,000 have formally used the Challenger courses for boards preparation, although we have good reason to believe that many more have used all or part of the courses informally or to supplement some other media—such as seminars, books, and so forth. Our best estimate is that about 1,500, that is 25% of these formal "students" for the boards, have been residents. While perhaps 15 or so of the physicians in practice have failed the test after using our course, we have NEVER had a resident fail the boards after using our course to prepare for the initial boards exam. That is over the record of 12 years. That is a standard that would be difficult for any organization to match, let alone exceed. It is also the reason that many of the very alternative organizations that you surveyed—such as OHIO ACEP, ICEP, NEMBR—as well as others that you did not survey—e.g., PA ACEP, engage Challenger to provide content, imagery, or computerized content resources in their own review programs!

Physicians, like any other group, learn in different ways. Some prefer seminars, or books, or self-study. Others want a more rigorous method such as our programmed learning approach. Still others want to use the Internet or a computer. A fair number just wing it, believe it or not, with no study at all! Most physicians use some combination of these techniques. It is not easy to tease out one from the other. If a physician reads review guides, goes to a seminar, and then use all or part of a Challenger program to enhance his or her preparation for the boards, what is the true agent of success if a passing grade is forthcoming?

We appreciate the opportunity to present our views to the EMRA Newsletter audience and request that our response be posted as a link to the EMRA report summary online if you are going to maintain it there on an ongoing basis.

Sincerely,

Dan Jones, MD
Founder and Director, Clinical CME
4052-A East Van Buren
Eureka Springs, AR 72632

Robert E. Sweeney, PhD, MS
CEO and President
Challenger Corporation
www.chall.com
5100 Poplar Avenue, Ste. 310
Memphis, Tennessee 38137


¹ Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-1131.