Dear Professor Mean,
I am the new chair of the IRB at a county hospital. Many of the studies we review are pilot studies with small samples. I have been trying to locate criteria for the scientific review of pilot studies, but have not found a consensus in the literature that I have seen. Is a pilot study merely a “dry run” of the procedures that will be used in a later, larger-scale study? Or, is it reasonable for the IRB to demand that the investigator provide specific criteria for determining whether the pilot has been a success? And, should the IRB furthermore demand that specific hypotheses be formulated? My impression is that many investigators declare their studies to be pilots in order to avoid more rigorous scrutiny of their proposals.
Professor Mean was thinking that we need more pilot studies. In particular studies of commercial airline pilots. Pilots who fly regularly to the Caribbean. And it is important for Professor Mean to be on some of those flights while the data is being collected. Could you arrange for a nice hotel suite when I arrive?
Short definition: a pilot study is a study that offers no direct benefits of its own, but only indirect benefits through helping the planning of a larger scale study. Some people might quibble with that definition, but it is a good starting point for discussion. Sometimes people use the term “pilot study” when they should be using the term “exploratory study” instead.
It’s a myth that all research must have a formal hypothesis, and this is especially true of pilot studies. I would not encourage researchers to try to force fit a hypothesis driven research model onto a type of research where it does not fit well.
If I were on an IRB, I would instead demand details about the larger scale study and why that larger scale study could not proceed effectively without information from the pilot study. An economic argument would not fly (we’re running a pilot because we don’t have enough money to run a large study), because no large scale study will occur and no benefit will therefore accrue from the pilot study.
You might want to assess the sincerity of the researchers. Do they (or their colleagues) really plan to do a trial ten times larger and ten times more expensive than the pilot if the pilot is successful? I generally dislike assessing things like sincerity because it presupposes that you are able to look inside the soul of another human being. So I would only apply the sincerity test if there was major evidence that something fishy was going on.
Another important point to remember is that if the goals of a pilot study are qualitative, then it is reasonable to use a qualitative justification of the sample size. A common application of a pilot study is to identify Murphy’s Law while the stakes are still low. If something does go wrong (especially something that would have been easily fixable), find out before you invest fifty thousand dollars. Typically, you would want a sample size large enough to insure that the full spectrum of research subjects are included, because problems are likely to occur at one end of the spectrum or the other but not both. Purposive sampling or quota sampling can be very helpful here.
Suppose the goal of a pilot study is to establish reliability of a measurement system prior to using it in a large scale trial. Reliability is typically measured by some type of correlation, so just show that the confidence interval for this correlation is reasonably precise: something fairly easy to do, even with a small sample size.
Note this was actually a question posted on the IRBForum by DD, but the question was so well written that I quoted it directly.
Closely related web pages:
- Stats: Designing a pilot study
- Stats: Getting IRB approval for your research
- Stats: Design and analysis of pilot studies
- Stats: IRBs and scientific validity
You can find an earlier version of this page on my old website.