Someone on the Evidence Based Medicine email discussion group asked about how to appraise a “before and after” design. This is effectively the same as using a historical control group. Historical control groups have a bad reputation.
A classic reference
- H Sacks, TC Chalmers, H Jr Smith. Randomized versus historical controls for clinical trials. Am J Med 1982: 72(2); 233-40. Article is behind a pay wall.
compared a series of studies, some randomized (RCTs) and some using a historica control group (HCTs), that compared the same intervention.
We found six therapies for which 50 RCTs and 56 HCTs were reported. Forty-four of 56 HCTs (79 percent) found the therapy better than the control regimen, but only 10 of 50 RCTs (20 percent) agreed.
and concludes that
biases in patient selection may irretrievably weight the outcome of HCTs in favor of new therapies.
Another article
- D. G. Altman, J. M. Bland. Statistics notes. Treatment allocation in controlled trials: why randomise? British Medical Journal 1999: 318(7192); 1209. Available in html format or pdf format.
is also very critical of historical controls
Problems will arise from the mixture of retrospective and prospective studies, and we can never satisfactorily eliminate possible biases due to other factors (apart from treatment) that may have changed over time.
Most experts recommend that you only use a historical control group as a last resort. It’s hard to find anyone to defend historical control groups, but Sir Michael Rawlins, in a web page at Pharmafocus [broken link] argues that there is a time and a place for this type of study.
Historical controls can be very useful, particularly where one is investigating otherwise untreatable conditions where there is a biologically plausible basis for the treatment, and where the outcome untreated is homogenous and either very disabling or fatal. When I was at the CSM, for example, we agreed to licence valganciclovir (Roche’s Valcyte) for CMV retinitis in AIDS patients. That was done entirely on the basis of historical controls.
At the time the FDA refused to accept historical controls and insisted on a placebo-controlled trial, which two years later came out with the same answer. So I think historical control studies are something we will to have to consider again.
The problem is we start collecting historical control trial data too late - we need to start to collect them now. For example, work is now being done on variant CJD. The investigators are collecting very careful historical controls because of the virtual impossibility of randomised controlled trials.
I don’t think that there is a formal appraisal tool for a historical control study. Perhaps you could adapt (very carefully) an appraisal tool for cohort designs. Historical controls can be thought of as an inferior version of a cohort design.
You can find an earlier version of this page on my old website.