Dear Professor Mean, I’m running an experiment where I randomize by alternating between the treatment and the control. I was told this is not the proper way to do this. Why not?
When statisticians talk about randomization, they mean a sequence that as unpredictable as the roll of a die or the flip of a coin. Any systematic and predictable assignment will have problems.
There are two reasons to randomize. First, randomization equalizes the assignment. It prevents the sicker patients, the younger patients, or the poorer patients from gravitating to one group preferentially. Second, randomization creates an unpredictable sequence. An unpredictable sequence prevents conscious or subconscious attempts to manipulate the findings.
Randomization equalizes assignment
I heard a story about some research about water pollution. The scientists had five tanks with differing levels of a pollutant. From a big tank of fish, they took the 20 fish and put them in the first tank. Then they took another 20 fish and put them in the second tank. The next 20 fish went in the third tank and so forth. The scientists counted the number of fish who died after being exposed to varying levels of the water pollution.
The results were quite interesting. The number of fish that died was not related to the pollution levels but instead to the order in which the tanks were filled. The first tank had the most deaths and the last tank had the least deaths. How could this have happened?
Think about the process of catching fish. The first fish you catch are the slow moving ones. They’re kind of sluggish. They may only be a few hours from floating at the top of the tank. The last fish you catch are the vigorous swimmers who always darted out of the way of the net at the last second.
So when the scientists filled the tanks in a systematic order, they put the slow sick fish in the first tank and the fast healthy fish in the last tank. This is a problem. The order of assignment becomes an artificial factor that interferes with your ability to distinguish among levels of water pollution.
Alternating assignments can create the same problem. Consider how cabbages grown in a garden. You will often see a pattern of big cabbage, little cabbage, big cabbage, little cabbage. These plants are competing for resources. One plant gets a tiny head start, and its roots extend over into the other plant’s space, grabbing some extra water and nutrients. This leads to more growth and more grabbing of the other plant’s resources.
This type of pattern can also occur in factories. Consider a mill that cuts small pieces of wire from a large spool of wire. The machine might cut the first piece a bit large because it waited a fraction of a second too long. When this happens, the next piece is likely to be too small. If a piece is cut a bit small, the bit of extra left over might make the next piece too big.
Suppose you are monitoring the amount of time a doctor spends with each patient. If the first patient goes a bit long, the doctor is likely to rush the next patient. A doctor that can wrap up a patient quickly, will then feel more relaxed and take more time with the next patient. So there might be a tendency for visits to go long, short, long, short. It doesn’t always have to happen this way. The days when I visit the doctor it seems like the pattern for the four patients in front of me is long, long, long, long.
It’s also true that not every garden has the big cabbage, small cabbage, big cabbage, small cabbage. Still, as long as there is a possibility for this pattern, you want to avoid assigning your treatments the same way. A random assignment will mix things up so that the first, third, and fifth cabbages (which all happen to be big) aren’t all assigned to the Miracle Gro group and the second, fourth, and sixth cabbages (which all happen to be small) aren’t all assigned to the manure group. A non-random alternation can turn the whole research project into a big pile of manure.
Randomization creates unpredictability
In some research settings, it helps if the people involved don’t know what’s coming next. An element of unpredictability prevents conscious or unconscious biases from affecting the research.
For example, there is often some level of subjectivity in deciding whether a patient is eligible for a research study. If the doctor doing the recruiting knows which treatment is coming up next, he or she might consciously or subconsciously try to manipulate the eligibility requirements. For example, suppose a newly arriving patient might be scheduled to be placed into a treatment group that the doctor views as less favorable for that patient. One way to manipulate the assignment is to delay entry into the study, so that the patient gets enrolled into the desired treatment group.
There are some serious and complex ethical issues involved if a doctor agrees to recruit patients into a research study, but believes that all (or even some) of his/her patients would be better off with one experimental therapy rather than the other one offered. Nevertheless, some doctors still participate in such a study.
You can prevent a doctor from manipulating the study by first blinding the allocation list of subjects. This means that you don’t tell the doctor which treatment the patient will be assigned to until after the eligibility requirements are all verified. You can do this by using a sealed envelop that the doctor opens only after verifying eligibility or by having the doctor call a research coordinator to find out the assignment.
You also need to randomize the allocation list of subjects. Suppose a doctor does not know which assignment is coming up next, but he/she knows that the previous six subjects were assigned as B, A, B, A, B, A. That doctor would have a pretty good idea that upcoming patient would be assigned to B.
Two publications (Colditz et al 1989, Schulz 1996) have quantified significant amounts of bias in the findings of research studies that alternate between assignments rather than randomize.
Another situation where unpredictability is important is when you expose your research subjects to two or more conditions in sequence. If the subjects can guess the next item in the sequence, they many consciously or subconsciously manipulate the results of the experiment. The most obvious example of this is in research that attempts to assess psychic ability, such as the ability to predict which card will come next in a deck of cards. If the cards are not shuffled, then the ability to predict the next card is not as much psychic ability as it is the ability to recognize patterns in the unshuffled card deck. This seems like a trivial example, but there are situations much like the psychic prediction of card sequences where failure to properly randomize led to the subjects being able to guess the correct answer without having any psychic ability.
A recent publication (Marks and Colwell 2000) showed just such a situation. Rupert Sheldrake has claimed that blindfolded subjects can sense when they are being stared at, and that this is evidence “an influence somehow reaches out” from the starer. There is empirical evidence to support such a finding, and this evidence is examined in Marks and Colwell. These authors argue that the order in which the research subjects were stared at followed a pattern that became readily discernable over time. When these researchers substituted a truly random sequence, the subjects could not perform better than expected due to chance.
- How study design affects outcomes in comparisons of therapy. I: Medical. Colditz GA, Miller JN, Mosteller F. Stat Med (1989), 8: 441-454.
- The Psychic Staring Effect. An Artifact of Pseudo Randomization. Marks DF, Colwell J. Skeptical Inquirer (2001), 24(5):41-44,49.
- Randomised trials, human nature, and reporting guidelines. Schulz, KF. Lancet, 348: 596-598 (1996).
You can find an earlier version of this page on my original website.