There’s a common saying among pediatricians: children are not little adults. You can’t take a drug therapy that works in adults and scale it down to a kid-sized treatment. Children are actively growing, their livers metabolize drugs differently, and they have a stage of life called puberty that most of the rest of us have long forgotten.
Likewise, pilot studies are not little clinical trials. Please do not take a poorly funded clinical trial and try to sneak your inadequate sample size through peer review by calling it a pilot. Pilot studies can serve one or more important purposes. Make sure you use them for what they are designed for rather than pretending they are kid-sized studies.
A pilot study has no value in and of itself. Its value is to serve the needs a full-scale clinical trial that you or one of your colleagues is expecting to run in the future.
Sample size calculation
A pilot study can help provide an estimate of that elusive standard deviation–the one that you need to nail down your sample size justification. Sometimes you can get a standard deviation from a previous study, but this often requires a difficult extrapolation from a different study run at a different part of the country with a different demographic mix of patients. It is so much nicer to get an estimate of variation from the same place where the full-scale clinical trial will be run.
There are other things that influence your sample size justification. How many of your patients will read your consent form and then decide they are better off going to the local witch doctor? How many start the study, but quit before providing any useful information? You should factor in the refusal and drop-out rates in your sample size calculation.
The pilot study can help you estimate how quickly you can recruit patients. Do you get dozens every week, or do you need to wait a full month just to get a couple of volunteers? The pilot study will help you estimate whether the study can finish before the year is over or if it will drag on past your retirement date.
Study logistics
A pilot study can help you test the study logistics. Can you fit in the extra testing and measuring that your trial will need into a clinic where everyone is working at DEFCON 2? Will your research team have the drugs ready to distribute in the right dosages? Will your data get entered properly into your research database?
A pilot study can help you vet your questionnaires. Do your patients understand what you are asking and provide clear and unambiguous responses. You don’t want a series of Likert scale items that look nice but the respondents are circling two numbers, leaving half of the questions blank, or writing detailed notes in the margin.
A pilot study can help you estimate resource requirements. How much time does an interview really take? It’s best to find out early if you have enough storage room for all the extra supplies your clinical trial will need? You should also find out how long it takes to prepare and mail out forty surveys before you face having to mail out four thousand surveys during the full-scale study. That interview that you plan for each patient–does it take five minutes or half an hour?
Unexpected problems
The pilot study can help you discover unexpected problems. What kind of unexpected problems. Well, I don’t know and you don’t know either. But “Murphy’s Law” guarantees that something unexpected will go wrong. Better for it to go wrong during the pilot study.
Also keep in mind that what we ask our patients to endure with a new intervention may be too much to ask. Does your proposed research provoke patients to stage a sit-down strike? For that matter, does it cause a riot among your staff? If an intervention is intractable from either side, better to find out after a ten thousand dollar pilot instead of in the middle of a million dollar clinical trial.
How big should your pilot study be?
Everyone wants to know how big your pilot study needs to be. There is no one-size-fits-all answer. If a primary reason for running is to get a quantitative measure (a standard deviation, a refusal rate, a drop-out rate) then make the pilot big enough to produce a reasonably narrow confidence intervals for that measure.
If, however, the primary goal of your pilot study is qualitative (testing the study logistics, identifying unexpected problems), then you should use a qualitative approach to decide how big your pilot should be. Make it big enough to stress the capacity of your clinic or to give a reasonable opportunity to produce the unexpected problem. Here “big enough” becomes a qualitative assessment.
Running the pilot without a control group
You may consider dispensing with the control group in your pilot. What? No control group? That seems like heresy. You do lose the ability to test your research hypothesis. But your pilot is not a kid-sized version of a bigger study. Admit it. You really didn’t hope to get any power or precision anyway with the small size of your pilot study. So why bother?
Consider dispensing with the control group if you already have a lot of experience providing the intervention associated with the control group. Dispense with the control group if you believe that all of the potential for unexpected trouble is likely to occur only with the intervention associated with the new treatment. In this case it may make more sense to put all your eggs in one basket and double the number of patients in your pilot who are getting the new treatment.
Summary
A pilot study is not a label you slap on when you know the sample size is too small. You run a pilot study to get data you need to justify the sample size in a future large-scale study. Or maybe you run a pilot study to test the logistics of that future large-scale study. Or maybe you run a pilot to identify unexpected problems before you invest money in that future large-scale study. In any case, a pilot study exists not for itself but only to serve the needs of a future research study.