Oops! I found statistical significance in my pilot study

Steve Simon

2006-09-18

Dear Professor Mean, I ran a pilot study to help me estimate the sample size for a larger study. There were 15 patients in each arm of the pilot, and when I analyzed the pilot data for efficacy in the primary outcome, I was shocked to find out that the results were already statistically significant. Do I still need to run the full study?

Before you do anything else, go out and buy a lottery ticket. You obviously have a Midas touch.

The first thing you should do is to go back and look at the original protocol. Did you include a sentence like

“If the results of the pilot study achieve statistical significance, we will..."

I didn’t think so. If you had that level of foresight, you wouldn’t be writing to me. So what you are looking at is a post hoc modification of the original research protocol.

Post hoc modifications of research protocols happen all the time. Stephen Senn has a cute definition:

“Medical statistician: one who will not accept that Columbus discovered America ... because he said he was looking for India in the trial plan."

Still, it is worth noting that post hoc modifications will dilute the credibility of your research finding. Is your research finding so robust that it can withstand such dilution?

If you answer yes, then you have to put yourself conceptually in a time machine and ask the question “If I was smart enough during the writing of the protocol to consider the possibility of stopping the study after the pilot, how would I have written the protocol?” You’d probably include some sort of adjustment to the p-value to reflect the fact that you have potentially multiple peeks at the data. Analyze the data under the new protocol procedures.

If you answer no, then pretend that you never looked at statistical significance in the pilot study. Proceed as originally planned with the full study. The extra work is intended not to offer proof of what you already know, but rather to ensure that your results will be treated seriously when you do publish.

Another possibility is to treat the pilot study as a preliminary finding in need of replication and then replicate the pilot study with an additional 15 patients in each arm.

More information about Early Stopping in Clinical Trials.

Other pages on this website:

Books:

Research articles:

Web pages: