I got a question that I had never heard before, and it sort of is a statistics question and sort of isn’t. A researcher was comparing two methods of training residents in a particular surgical procedure and wanted to know how long you should wait between the training and the evaluation of whether that training was effective.
It’s hard to imagine how to turn this into a quantitative question. If there is reliable data on outcomes across a wide range of time intervals, then perhaps you can look to see if there is a threshold. If effectiveness levels off after three months and does not change much after that, it’s hard to justify the extra time, trouble, and expense of a two year study. But if there is already a wide range of time intervals, that probably means that your problem has already been studied to death. So the question of how many months to wait before re-testing is usually a question that you can only answer with qualitative data.
Looking at things more broadly than educational interventions, there is a general rule that long term outcome measures are preferred to short term outcome measures. The amount of weight you lose after a month on a particular diet is not that important because most people will eventually gain back what they lost in the short term. A much better measure of a diet’s success is how much weight people can keep off after a year on the diet.
A study of smoking cessation education materials showed that most of the lessons learned by teenagers were forgotten after four months. The two things that the teens did remember was how expensive a smoking habit really is and how smoking turns your teeth yellow, which isn’t too surprising. The two issues that teenagers worry a lot about are appearance and money. All the other important points about why you shouldn’t smoke weren’t effectively conveyed because the teens only remembered them for a short period of time.
In general, interventions are only successful if they “stick” for the long term. So when you think about when to evaluate the residents, the longer the time frame, the better. But there’s a competing concern. You need to finish your study in a reasonably narrow time frame. It gets a lot harder to evaluate the surgical skill of a resident once they’ve left your program. You may have other constraints as well. Waiting a full year before re-testing makes no sense if you’ve made a commitment to present your results at the summer research conference.
Sometimes there are certain time thresholds that you cross that are meaningful. For most cancers, for example, if you can stay in remission for five years, you are considered to be cured of that cancer.
When your considerations are largely qualitative, the opinions of your peers is important. They’re the ones who are most in tune with the standards of research in your field, and they should be able to tell you how long a time frame you need to make your research persuasive.
You can find an earlier version of this page on my blog.