Testing baseline imbalance in a randomized study

Steve Simon


Randomization will roughly balance out the covariates between the treatment group and the control group because of the law of large numbers. Once in a while, though, an important amount of covariate imbalance will creep into a randomized study. Just as a flip of 100 coins will not always yield exactly 50 heads and 50 tails, a randomized study will not always yield perfect covariate imbalance.

When such an imbalance occurs, it is called a chance bias or accidental bias. It can seriously affect the quality of your analysis.

One reference,

argues that the likelihood of covariate imbalance is small if the total sample size is 200 or more, while another,

argues that a total sample size of 20 provides inadequate protection against even a single covariate imbalance, while a total sample size of 80 protects even against imbalance across multiple covariates.

It is always good to examine the covariates in a randomized study to insure that you have reasonable balance. But most statisticians recommend that you avoid a statistical test of significance for this situation.

One reference,

makes the point that not only should you avoid a hypothesis test, but that you should also avoid visual inspection of differences between groups.

These authors recommend that you select baseline covariates prior to data collection and adjust for them in an analysis of covariance model.

The reason to avoid a test of significance for baseline values in a randomized trial is that such a test addresses only whether there was a flaw in the randomization process. Perhaps the numbers that you thought were random really weren’t so random after all.

A more serious concern is subversion of randomization, where some of the physicians recruiting patients for the study will try to steer certain types of patients towards one treatment or away from the other treatment. They could do this subconciously by applying the inclusion and exclusion criteria more rigidly for certain types of patients if they know that a certain treatment will be assigned. They could also deliberately delay a patient’s entry into a trial for a day or two so that a different patient gets the less preferred treatment.

Physicians who deliberately tries to subvert the randomization process are behaving unethically. They recognize that equipoise is not applicable for this particular patient, so they should offer the patient the preferred treatment outside of the randomized trial.

For this reason, the randomization list should be concealed from the recruiting physicians, so they will not know what treatment is coming up next. Concealing the randomization list is always possible, even when the study is not blinded.

So a test of significance is only appropriate if you think there was a flaw with the randomization process, such as a physician’s attempt to subvert the randomization of the study.

You can find an earlier version of this page on my original website.