A colleague of mine is guest editor for journal that is preparing special issue on research methodology. He wanted to know if I was interested in contributing a paer, and of course I said yes. I’m not quite sure what he is looking for, so I mentioned several ideas that might be worth pursuing. We’ll see if he likes any of these ideas, but I thought it wouldn’t hurt to post those ideas here to give you a sense of some of the important issues in research methodology that I might want to write about.
Let me write up a few suggestions of topics and see what you think of them. These are roughly in order of my preference, but I can tackle any of these topics. If reviewing these makes you think of a different topic that might work better than these, just let me know. I can write a good review paper in a variety of areas.
Excessive reliance on p-values in research.
You’re aware of the recent ASA statement on p-values. Nice, but a lot of people are unsure what do in practice because we have all become so reliant on p-values. I could talk about weaning yourself away from p-values (as opposed to dropping them cold turkey). There are several places where p-values are clearly unwanted: testing assumptions, especially normality; testing baseline imbalance in randomized studies; using p-values to build models and identify confounders. If we stopped using p-values in places where there is already strong consensus that they are unwanted, that’s half the battle already. And it comes relatively pain free.
Sample size shortfalls in research.
I can’t tell you the number of times that I’ve had clients come to me asking my permission to stop a study before getting the planned number of subjects. It seems that they had hoped to get a hundred patients within a year, but after two years they have just a couple dozen. This is a bit of irony because if they had exceeded the proposed sample size, they would have to get IRB approval. A change in sample size potentially upsets the cost-benefit balance. But if they fall short of the proposed sample size, they don’t have to get IRB approval, even though this also upsets the cost-benefit balance. The big problem is that no one considered monitoring the accrual rate as part of the ongoing management of the trial. If they had caught the slow accrual problem early, they could make adjustments early enough in the trial (e.g., adding extra centers to a multi-center trial, hiring a clinical research coordinator) that it could have an impact. Instead they don’t do any thing until the trial is almost done. Then it’s too late. There’s a Bayesian model for monitoring accrual that I helped develop and it has nice properties for planning prior to data collection, monitoring during data collection, and post study review.
I could write a review of when and why to use a Bayesian approach. While Bayesian methods can apply anywhere, its greatest strength is with hierarchical models, latent models, and imputation of missing values. I’m not a rabid anyone-who-doesn’t-use-Bayesian-methods-is-an-idiot Bayesian. I use it when it clearly has advantages over more traditional approaches, but I don’t see it being a complete replacement.
The proper use of pilot studies in research
Too often, researchers will slap on the label “pilot study” when they have a study that they know is grossly underpowered as a way to deflect criticism of their poor planning. A pilot study is not a miniature version of a full-scale trial. It exists to provide information. It help you estimate a standard deviation to help with sample size justification. It quantifies resource requirements to help plan a budget. It tests the palatability of a new intervention both with the patients and the health care team. It identifies where Murphy’s Law is likely to strike. I would also like to address the question of how big a pilot study should be, as there is conflicting advice out in the literature.
Quasi-experimental designs
I dislike the term “quasi” because it implies a level of inferiority. You adopt a quasi-experimental approach, in spite of its limitations because it is still superior to an approach like pure randomization. I could review the problems with randomization (heresy, I know, for a statistician to criticize randomization) and then explain some of the more recent approaches to experimental studies where the research team (wisely) rejects randomization because randomization is an inferior approach in a many situations.
Community-based participatory research (CBPR)
There’s a cynical saying out there about us. It goes something like “Researchers are like mosquitoes; they take your blood and then leave.” Many of the problems with research (such as sample size shortfalls and slow patient accrual mentioned above) stem directly from our failure to engage with the communities that we use for research subjects. Using CBPR is very discomforting. It requires you to give up a lot of control, it requires developing listening skills, and it takes a lot of time and energy. But failure to apply CBPR to avoid this discomfort comes at the expense of the quality of your study. One of the most important principles of CBPR is that the community owns the data. I’d like to provide a bit of historical background. Research during the AIDS epidemic was often very confrontational. Advocacy groups made life unpleasant for the research community, but ended up getting many important changes in how we conduct trials today.