I get lots of questions about how small a sample size can be before you can’t perform a quantitative analysis and instead are forced to summarize the data in a qualitative fashion. The most recent question involved looking at infants with feeding disorders. There were 29 of these infants, but a subgroup of 5 had disorders so severe that they still required a feeding tube at 3 years of age. The researcher wanted to compare this group of 5 to the remaining 24.
There’s a fuzzy border between the sample sizes that are so small that you describe them qualitatively and the sample sizes that are just large enough to allow a quantitative comparison. It’s based largely on emotional reactions, which is fine. The problem, of course, is that a sample size that one person “feels” is too small might be more than enough for another person.
Although there’s very little quantitative justification for this border, I do talk about sample sizes that are so small that only an “all or nothing” comparison will achieve statistical significance.
By “all or nothing” I mean that the worst case in the first group is still higher than the best case in the second group. A total sample size of 5 and 5 (10 total) represents an “all or nothing” threshold. Since you have exceeded that value in the second group, you probably have sufficient justification for running an analysis. Keep in mind, of course, that it would take a Godzilla sized difference to achieve statistical significance, but that shouldn’t stop you.
If you or your boss are uncomfortable, I certainly would not object if you used a qualitative discussion. There’s no right or wrong answer here.
You can find an earlier version of this page on my original website.
Also see this page.