Stopping rules for a Data Safety Monitoring Plan

Steve Simon

2006-07-05

If a research study requires a DSMP (Data Safety Monitoring Plan), that plan should outline conditions that would cause a study to end early. It is difficult to specify what those conditions would be, but it is important to at least think about and comment on each of the major areas listed below.

Not all trials should have explicit rules for stopping early. Early stopping may not be all that critical if you are examining a condition that does not cause serious and irreversible morbidity. For practical reasons you can’t stop a short term study early because it will be over before the statistician has a chance to analyze the data. Likewise, early stopping is problematic when you are looking at long term outcomes. Don’t stop a study early for efficacy reasons if a careful assessment of side effects might be incomplete and that assessment is a critical component of the research.

On the other hand, trials involving critically ill patients should almost always have regular evaluation and careful consideration of early stopping.

You should use the following as a general guide, but don’t be afraid to include whatever considerations for early stopping that your particular research warrants.

Early evidence of efficacy. If you have a two arm study, and evidence emerges during the course of the study that one arm is inferior, it would be wise to end the study and give the superior drug/treatment to all future patients. This must be done carefully in order to preserve the scientific integrity of the study.

A statistician will review the unblinded efficacy data after roughly 1/3 of the target sample size has accumulated and after roughly 2/3 of the target sample size has accumulated. If the primary endpoint is statistically significant after using the O’Brien-Fleming p-value adjustment, then the study will end early.

A good reference for O’Brien-Fleming and related methods is

Not every trial should have a rule for stopping if there is early evidence of efficacy. If this is the case, you should note this and offer a rationale. For example,

There will be no interim analysis of the data to compare the relative efficacy of the treatment group and the control group. We need to get a complete profile of both safety and efficacy of the two groups in order to provide a comprehensive picture of the advantages and disadvantages of the new treatment.

Early evidence of futility. This is closely related to early evidence of efficacy, but needs to be mentioned separately. Sometimes your initial beliefs about the variability of your outcome measure are wildly optimistic. Sometimes, your initial estimate of how much a new therapy can improve things over the current therapy is also hopelessly naive. If so, you will accumulate evidence during the study that makes you wish that you had planned things better. It may come to the point where it is painfully obvious that continuing the study is unlikely to produce a statistically significant finding, and it may make more sense to invest your limited research budget in more promising areas. Stopping a study early for futility is controversial and it requires careful handling to preserve scientific integrity.

Accrual problems. A research study that takes 30 years to finish is probably a study that you should not start, but even if you do start such a study, if you find it is difficult or impossible to complete the study in a timely fashion, then that may provide grounds for stopping the study early.

You may decide not to end a study early even if you have problems recruiting patients. An explicit statement along these lines would still be useful. For example,

Our goal is to recruit [state sample size goal] patients, but if the accrual rate is slower than accepted, we will continue to accrue patients until [specify date]. If the required number are not recruited by then, we will end the study at that time and analyze the data using the same techniques, but will provide an appropriate cautionary statement in the discussion section of any publication.

Sometimes you may wish to scale back on the very complex analyses if a target sample size is not met, as these analyses may not work properly with a smaller sample size.

What you want to avoid is an open ended commitment

We will recruit [state sample size goal] patients for as long as it takes and we will continue until hell freezes over, if needed.

Sometimes accrual problems go hand in hand with early evidence of futility, because the outrageously optimistic assessments of variability and treatment effect go hand in hand with outrageously optimistic assessments of how many patients you will be able to sign up. It is possible to combine the futility and accrual considerations into a single analysis, though this needs to be done prior to data collection, if at all possible.

Early evidence of safety problems. It is impossible to find a valid medical therapy that does not carry some level of risk with it. When you start the study, it is with the belief that the risks are small relative to the benefits. Or the benefits are so large, that even serious side effects are worth the cost. Or you believe that the risks are high, but only because you are dealing with a very ill population of patients. Depending on your initial perspective, your stopping rules might differ. For example, if a therapy is considered to be relatively benign and the condition being treated does not result in serious morbidity or mortality, then a single serious adverse event might be enough to stop the study early. For example,

We will stop the study early if any patient dies, if any patient requires emergency surgery, or if any patient suffers a permanent and irreversible disability, unless it can be shown that this event was unrelated to participation in the clinical trial.

The last provision is important, because you shouldn’t end a drug trial if your patient dies while climbing Mount Everest.

If the risks and benefits are both large, then perhaps you need to do an early assessment of efficacy where the efficacy calculation involves a composite assessment of both the benefits and side effects. If serious side effects are expected because you are dealing with a very ill population, then you need to examine if the side effects are disproportionate in one treatment arm versus another. If possible, state prior to data collection the side effects that will be monitored on a regular basis with the understanding that additional side effects might also be analyzed if evidence warrants it.

We will evaluate the risk of skin rash associated with this topical ointment and will stop the study early if the rate is significantly higher in the treatment group than the placebo group at an unadjusted alpha level of 0.05. We do not anticipate any other common side effects, but if they occur in more than ten patients we will subject those side effects to a similar analysis.

As a general rule, safety endpoints do not require the same adjustments that efficacy endpoints do.

You should also stop a study at least temporarily if an adverse event not mentioned in the consent form occurs. When this happens, you need at a minimum to revise your consent form and get IRB re-approval of the project.

Stopping a study early for safety reasons can harm the scientific validity of the study, but it is generally accepted that the safety of individual research subjects is more important than the scientific validity of the study as a whole.

Further reading: