IRB review of studies with methodological flaws

Steve Simon


Someone (AK) wrote into the IRBForum asking about how the IRB should do when they discover methodological flaws in the proposed research. If everything else is in order (e.g., appropriate consent forms), is the IRB still justified in asking that the flaw be fixed prior to approval of the research.

You don’t have to look too hard for justifying a review of a study’s methodology. Point #18 of the Declaration of Helsinki reads:

Medical research involving human subjects should only be conducted if the importance of the objective outweighs the inherent risks and burdens to the subject. This is especially important when the human subjects are healthy volunteers.

Point #2 of the Nuremberg code reads:

The experiment should be such as to yield fruitful results for the good of society, unprocurable by other methods or means of study, and not random and unnecessary in nature.

When someone volunteers for a medical experiment, they are, at a minimum, sacrificing some level of convenience. They may endure some additional level of pain (such as a needlestick). Sometimes they endure an increase in risk in order to be part of the research.

Why do patients volunteer under these circumstances? Some are in it for the money (or the free medicine and checkups), and others are curious about the scientific process. But a very common reason that people participate in medical experiments is altruism. They want to help out other people who have the same disease that they have.

If you ask someone to be part of a research study, and the study has such a serious flaw that it produces data that are effectively unusable, then you’ve broken an implicit contract with that subject and abused their good will.

That being said, all research studies have flaws. The perfect research study has never been done. Even the gold standard of research, the double-blind randomized clinical trial, has well documented problems.

There will also be some tension between the need for methodological rigor and the need for patient autonomy. For example, a study that requires a patient to sign an informed consent form is flawed because this study will be subject to volunteer bias. So tolerate a few existing methodological flaws because your review process might itself introduce even more serious methodological flaws.

Please get this idea out of your head that your job is to fix flaws in research studies. You can’t do it and you shouldn’t do it. If you desire to only approve research without methodological flaws, you will be forced to reject pretty much any study that comes before your committee.

So the question that IRBs should ask is not “Is this study flawed?” but rather “Does this study have a fatal flaw?

Other people will argue that the question is whether the benefits of the study outweigh the risks and burdens. This might be true in special cases, such as a study that requires patients to undergo a spinal tap or a placebo surgery, but most studies involve risks and burdens that are minimal to the extent that if the study produces anything of reasonable value, it will outweigh the risks and burdens.

I’m sure that people will argue this point, but I believe that the IRB should hold up a study only if it finds a fatal flaw. That doesn’t mean that you can’t offer a suggestion (e.g., we approve this study as it currently stands, but we’d like you to consider using a concealed allocation scheme as part of the research).

So what’s a fatal flaw? There are two good standards. First, is a flaw so serious that the research is unlikely to be published in any peer reviewed journal? Alternately, you could ask whether a flaw is so serious that it would fail to meet the quality screen typically used by the folks who do meta-analysis?

Data that does not get published in the peer-reviewed literature and data that does not get incorporated into a meta-analysis represents data that does not produce fruitful results.

One final comment. IRBs already have the (at least partially deserved) reputations of being nitpickers. Do you want to add an extra dimension of nitpickiness to your review process? Stay away from arguments about methodological rigor unless the issue is drop-dead important.

You can find an earlier version of this page on my old website.