An email on the MedStats group outlines a new treatment that is:
- without any significant competing treatments,
- utilized in a heterogenous patient population,
- difficult to study in a randomized trial.
There are a variety of alternatives to a randomized study, but I suspect that this person wants to use a historical control study. It sounds like he wants an informal endorsement from a group of professional statisticians to use a historical control study instead of a randomized study. Professor Mean normally charges serious money for an endorsement, but he is in a generous mood and will offer an endorsement for free.
I wrote a section in my book with the title “Randomization is overrated” and one of the reviewers (bless his soul) gave me a lot of grief over this. I still believe what I wrote. Although I would not agree with the sentiment (to quote from the original email) that randomized trials “are often short-sighted and uninteresting and focused at simple instead of interesting questions.” I would say that randomized trials are often performed in an artificial environment which does not reflect patient care as it is practiced in the real world.
You should recognize, however, that historical control studies are not well accepted in the research community and the effort you expend on this may not pay off in terms of convincing others to adopt this new therapy that you seem to like. The only time that historical controls are found to be acceptable is when:
a) there is no competing therapy available; and\
b) the disease causes close to 100% mortality or morbidity.
Some have added a third condition
c) the disease is well characterized in a highly homogenous population.
This latter condition would exclude any disease defined as a “syndrome” (a disease that is defined by a constellation of symptoms rather than a positive diagnostic test and/or a disease that defined as effects that cannot be explained by any other cause).
If patients for the most part refuse to agree to randomization, then you don’t need anyone’s endorsement for using a non-randomized approach. I make a bad joke about this. Most patients will refuse to be part of randomized trial that involves birth control methods, because they already have strong opinions about the types of birth control that they find acceptable or unacceptable. And they really get upset when they find out that one of the arms in the study is a placebo birth control pill.
If randomization is truly impossible, then I would still encourage you to consider alternatives to a historical control group. There are more rigorous designs that don’t require randomization at the patient level.
If randomization is possible, but it just takes too much time and effort, then you need to make a value judgment. The reversion to a weaker form of evidence has a cost associated with it. Randomization has a cost associated with it. Which cost is greater? In most situations (but not all situations), I believe that the cost of reversion to a weaker form of evidence is greater. In other words, anytime you can randomize, you should randomize.
The question is not, what evidence do you need to justify what you are currently doing, but rather what evidence do you need to convince a skeptical colleague (not a cynical colleague, of course, as cynics can never be persuaded). What you will find is that most of your skeptical colleagues will not change their practice on the basis of a single historical control study. So why are you doing research that is essentially unpersuasive?
You can find an earlier version of this page on my old website.