This is an HTML format version of an email by Ronan Conroy on April 9, 1999 to edstat-l, an Internet list and to sci.stat.edu, a USENET group. This email summarized a presentation he made about how to develop ideas for research. I have made some minor formatting changes (mostly the use of bolding, bulleting, and indenting to highlight the major themes), but all of the credit for writing up this summary belongs to Ronan Conroy. Part of this presentation represents a summary of discussions on edstat-l and sci.stat.edu. Here is the acknowledgement in Dr. Conroy’s original email.
I’d like to thank the many people who took part in the discussion, or who wrote to me privately, and to stress that the quotes in it are often the person who made the point most memorably, rather than the only person who said it.
Many thanks to Chris Zorn, Gabriele Susinno, Giovanni S. Leonardi, the... inimitable dennis... roberts, Joe Ward, Michael Granaas, Roland Andersson, Jay Warner, Alex Heath, The Anthonys, Bob Frick, Jerry Dallal, Tjen-Sien Lim, Tim Cole, John F. Schnell, Robert C. Knodt, Paul Velleman and Joseph L. McCrary. Did I say Herman Rubin? I did now.
This material is reproduced with permission from Dr. Conroy. For what it’s worth, I have included a copy of the original email. It’s pretty clear that there was some formatting in Dr. Conroy’s original document that got lost when it was translated to a text format for email.
Introduction
This paper tackles one of the questions that statisticians dread most: the most basic one of all. How do you start formulating a research project? It began life as a talk at a research seminar in the Rotunda Hospital, Dublin. Trying to write it up, I decided to mail the statistics lists that I subscribe to. This paper has been greatly enriched by the ideas and discussion generated on edstat-l, the statistics teaching list, as well as contributions from subscribers to the stata list and the UK statistics list allstat. Quotes are often attributed to the only person who made the point most memorably, but many of the ideas emerged repeatedly in different postings. I’d like to thank all those who took part in the discussion.
Exploring your environment
The first thing you need to do is identify your resources for research. This is often easier when you first arrive somewhere. After a while you begin seeing an environment as the place where you work or live or eat. You need to see it with a fresh eye to see it as a potential research environment.
Don’t forget that your research environment includes not just your patients and your colleagues, but also includes any source of data, ideas or help that you have access to. Many of my own research projects have taken shape because my office is next door to the psychology department; a casual remark has often triggered a flurry of speculation, articles rooted out, contacts mentioned and so on.
The internet is also a valuable environment. Discussion lists abound,which can provide not just free advice but also an insight into current controversies and new directions in research. Simply subscribing to a list and reading the postings (the word for a person who does this is a lurker) without taking part in the discussions will often give you ideas.
General resources
How much time will you be able to devote to research? To what extent can you integrate it into your daily work?
Will colleagues help? For instance, if you need blood taken outside working hours, will the doctor on call oblige? Will nursing staff collaborate by collecting extra information?
Do you have access to a person, unit or department with a specific research interest? They can often be a useful source of ideas. Never underestimate the value of just going for coffee with someone who does a lot of research, or, better, a research team. The speed with which a bunch of researchers can take a vague idea and shape it into a research design is amazing. Most of these ideas go nowhere, but eavesdropping on the process can help you to do it yourself.
Giovanni Leonardi of the Environmental Epidemiology Unit at London School of Hygiene and Tropical Medicine put it like this: “There are many potential research ideas that never make it to becoming research projects, and the likelihood that a research idea will become a research project is heavily influenced by this idea having being selected and refined in an environment where potential ideas are routinely tested for their viability. Think of this as ‘natural selection’ of research ideas within the research environment."
Do you have access to a statistician, or someone who can advise you on study design and sample size?
What library facilities do you have access to? Skimming journals is a good ideas generator, which I will deal with in more detail later; but access to a good library, including literature searching and reprint ordering facilities, is a must. Add extra points for library staff who are willing to do literature searching with you looking over their shoulder to refine the search.
What computer facilities are available?
- Ideally you should be able to write your own research papers and reports.
- This is helped greatly by being able to create tables and graphs, so if you can’t do this, learn.
- The next real bonus is having a dedicated bibliographic package which you can use to store, organize and annotate your references, and to generate bibliographies for papers -- well worth investing in.
- Finally, have you a statistical package, or access to someone who will do your statistics for you? Ideally, you should be able to examine your data using graphs and tables, rather than handing the whole analysis over to a number cruncher. Professional statisticians are best used to answer complex questions, but you should be able to do the simple statistics yourself, perhaps guided by a statistician. Not being able to do any statistics means that you lost contact with your data at the vital point when it is being investigated for patterns, anticipated and unanticipated.
What are they funding this year? This sounds like a cynical point, but if there are funds available for research in specific areas, make use of them. What charities are there who might be interested in your research area? Talk to colleagues; there is often no single listing of available research sponsors, and you have to rely on the grapevine.
Specific resources
Do you have access to information already collected which could be the basis for a research project? This information could have been collected as routine clinical information. Although you probably cannot do a research project solely on the contents of patients' charts, routinely collected information may allow you to
- Identify patient groups that are interesting to study (and figure out if there are enough of them to be worth studying!)
- Identify controls
- Add already-collected clinical information to the data that you will collect in the course of your project.
Information may also be available as an offshoot of another research project. You may liaise with another research project and
- Study a subgroup of their patients in more detail
- Follow up a previously-studied group
- Add a sub-study of your own to a study that is being planned or, not so easy, ongoing.
It is a good idea to talk to people who are doing research in the setting in which you work. They will be able to spot potential difficulties in proposals, and may also have useful ideas as to what they would do if they had access to your facilities.
Potential projects
Now all you need to do is to get an idea for a project which will be realistic, given the resources available to you. This is often a stumbling block. I had one person come into the office to discuss a research project with me. ‘I have 24 patients with rapid cycling mood disorder’ he said. And stopped, waiting for me to say something. The trouble is that 24 patients with rapid cycling mood disorder is no more a research project than 24 trout in a shoebox. What you need to ask yourself is ‘what do we not know about rapid cycling mood disorder’?
One very important piece of advice that recurred frequently in the edstat-l discussion was the need to develop many ideas simultaneously. Christie Brown, Assistant Professor of Marketing at University of Michigan Business School tells her students to imagine that inside them they have a large basket of research ideas, some better than others:
‘I point students toward Donald Campbell’s work on creativity. Campbell suggests one secret to generating better ideas lies in the QUANTITY of ideas generated. In other words you stand the best chance of pulling an idea from the “high” end of your good-idea basket if you make a lot of draws.' (Campbell, Donald T. “Blind variation and selective retentions in creative thought as in other knowledge processes.” Psychological Review. 1960;67:380-400.)
Don’t focus prematurely on a single idea. Develop a few together. It’s like the process of conception: the chances of a child resulting from a single act of sexual intercourse are small. But the chances of a child not resulting from regular sexual intercourse are likewise small. Carry a notebook and write down every idea that you get, good or bad. You will learn from thinking about why the bad ones are bad as well as from why the good ones are good.
Christie Brown again: ‘Write down everything. Do not self-censor. Keep a log of your baby-ideas in case they end up being worth pursuing. Get in the habit of generating at least one idea based on everything you read in your domain and even out of it.
Bob Frick, a cognitive scientist, actually forces students to develop a number of research ideas as a learning exercise. ‘The assignment was to come up with three “kernels”, and the students had about a month to do it. The notion was that they were supposed to find some original idea they had. It usually ended up being an original observation. Original to them -- it didn’t have to be original to the field of psychology. Their original idea would then be a kernel that could be developed into an experiment. Most people have these, but they don’t pay attention.’
Extending the ideas of others
Much of the discussion on edstat-l centred around where ideas for research projects come from. The sources of ideas divide into two:
- Replicating/extending the work of others
- Doing something original.
I’ll take the easy one first!
Repeating research that has been done by others doesn’t sound like task, but there are several important reasons why it needs to be done, and there are some other benefits too. The reasons why research needs to be replicated include:
Local research is needed to make sure that findings from other countries apply locally. Indeed, basic research is constantly needed to monitor local health needs and to evaluate the services being delivered.
- Is the problem the same here? For instance, research has shown that liaison psychiatry in Ireland sees a very different spectrum of morbidity than its counterpart in the US. Planning services using US models is inappropriate. If you suspect that the service in which you work sees a spectrum of patients or problems that is different to what you would expect, research this. It can lead to a better understanding of the health service needs in your service.
- Do the strategies worked out elsewhere apply? Many treatments and interventions have been researched in settings that may be quite different to your own. You might well ask if they are best for your setting.
All research needs extension to new contexts and development along an obvious line - Clinical trials are often done on homogeneous, idealised patient groups; they need extension to realistic groups such as those with comorbidity, or beyond the age range of the original research. Think of
- Treatment of hypertension in the elderly. This is a classic case where treatment was wrong for many years because it was based on the results of studies of younger patients.
- Use of treatments in neonates that have only been studied in older children or adults. This example has become topical recently, as paediatricians have realised that very young children are often excluded from treatment trials, yet the treatments end up being used to treat them,
- Many treatments are evaluated on patients who have ‘pure’ diagnoses, while they are frequently applied to people who have more than one disorder.
Factors which have been identified in a disease may be present in other similar diseases. Since its role in peptic ulcer disease was uncovered, H pylori has been investigated for many other unsolved crimes.
- Is it responsible for the digestive symptoms some pregnant women experience?
- Is it linked to cancers?
Yes, there is a feeling of a bandwagon rolling along, but someone has to check out these questions.
You may spot an explanation which the original study failed to identify and test. This is, of course, classic ‘stroke-of-genius’ research. Just remember, though, that the explanations that are most often overlooked are the commonest, most familiar things.
You may not believe a piece of research. Not all research is good research. I have, several times, replicated and extended research because I didn’t believe it. Incredible research deserves to be replicated. If you confirm the original findings, you have helped to overcome the resistance that they will find in being accepted. If you fail to confirm the findings, this in itself is interesting. Though try to make sure that the original author isn’t asked to review your paper!
Even doing a straightforward replication of a previous study can be a very worthwhile exercise. As a first project, it means that you already have a ‘canned’ methodology, and you will learn a lot about running, analysing and presenting research, But there are often surprises too.
Chris Zorn of Emory University wrote: ‘As a social scientist responsible for training grad students in statistics, one thing that I’ve always found useful is replications While the main reason I use replications is to teach students statistics and/or software, these exercises often prompt them to extend the work they are replicating. These can range from the simple (e.g. testing for relationships in the data that the original investigators didn’t look for) to the very involved. The result is often interesting, if a bit derivative, research projects, some of which have led to PhD theses, etc.’
Andersson Roland puts it simply: Dig where you stand. That is, make use of all the data that is already at hand and that nobody had time to analyse. Almost always there will be unexpected or unknown patterns in these data that can be detected if you analyse them with an open mind. You do not always need to have an research idea ready when you start. They will come up when you try to formulate an explanation for the patterns that you find in your data.
Alex Heath, an economist from Australia, wrote: A good way to get started thinking about research questions for me is to find things which have been done overseas (usually the US or the UK) and adapt them to Australian data. I find that once you start replicating things you find interesting twists and turns which allow you to say something completely new.
Although I have replicated several studies because I didn’t believe them, this probably isn’t the best spirit in which to replicate. But neither should you simply accept the original research as scripture. Paul Velleman, the person responsible for the DataDesk statistical package and ActivStats, a statistical teaching package, wrote in praise of an attitude of well-informed skepticism: This misses the most important part of the process -- an abiding skepticism. You must know your science before you can be intelligently skeptical about it, but just because you know what is common wisdom doesn’t mean you should believe it. Indeed, if science is to progress, you must maintain a willingness to disbelieve. You don’t do research by replicating previous results but by doubting them.
Dennis Roberts, responding to this, said: a good replication study does not have to be done BECAUSE one doubts them but rather, to bolster the case that the research findings made ...
I think that he and Paul really just differ in emphasis, with Dennis arguing that ‘replication is very valuable ... we don’t do enough of it ... ' while Paul cautions against literal-minded repetition. I think everyone would agree that the scientific idea of replication is doing something more intelligent than just looking for what the other guys already saw.
Paul makes the point, too, that it is hard to sit down and work carefully through a set of data without coming up with at least one pattern that needs further investigation. You may start by replicating a study, but this is almost guaranteed to act as a springboard to innovative questions of your own.
Getting a research idea by reading papers.
You can simply bury yourself in the library with a whole year’s worth of your favourite journal and, starting from the most recent issue, use a series of filters to identify studies that you would be interested in and capable of extending. Even when I’m not in need of a research project, I often graze my way through a small stack of journals, picking up an interesting methodological approach here, or a useful measurement technique there. Many of my more prolific colleagues do this a lot. One, in particular, seems able to rummage out a half-a-dozen relevant journal articles from her shelves on any topic in about five minutes.
If I am looking for a potential project, I look at each article in turn and ask:
- Does the title sound interesting? If so, look at the abstract. There is little point in taking on a research project that doesn’t sound interesting. It will be a lot of work, so it will have to hold your attention.
- Can I extend this study to a relevant patient group? Think about the sorts of patients you have access to--would it be interesting to try this on them?
- Can I extend the methods by assessing something not assessed in the original paper, or increasing quality of assessment? Remember that you are one of your own best research resources. Can you refine the study design by substituting a personal interview or a medical examination for a self-administered questionnaire?
- Read the end of the discussion to see if the authors identify areas for future research. Usually the researchers will make a big play to inflate the importance of their area of interest, and do a paragraph of ‘we-simply-must-know’ stuff. They are usually either working on this themselves or they have applied for a grant already, so if you decide to follow up their suggestion, move fast.
- If you find that an article catches your attention as a possible project, do a literature search to get a feel for the area (and look carefully through the references in the original article, of course; aside from identifying useful articles to read, you will get an idea of the sorts of journals that might take your paper, and you can target it accordingly).
- Contact the authors. Authors love to hear from people who think that what they are researching is interesting. They often provide advice, access to unpublished manuscripts, free consultation time simply because that’s the way research works. Be prepared to do the same in your turn! The easiest way to contact the authors is get an email address. Search the website of their institution if the article doesn’t have an email address, or try using an email search engine on the internet to find them. People reply faster to email, and you can bombard them with small queries which would be irritating in letters. And, of course, keep them in touch with your work. They may be a great help when it comes to analysing and writing up.
- Finally, you should also think about plundering the abstracts from conferences. There may well be something there, and the work will be newer than published articles. If you are actually at a conference, seek out the people who are doing research that would interest you. Over coffee you can usually get ideas for several projects from them. One favourite ploy I use is to ask them how they would do their research if they could start all over again, knowing what they know now.
Getting your own ideas
This is an even harder subject to write about than extending and developing the ideas of others. (Did I say plagiarising? -- Never!). The secret seems to be keeping your eyes and ears open all the time. The observation doesn’t have to be complicated. On the contrary, spotting an obvious question in an everyday event often has greater potential.
Jack Schnell of Department of Economics at the University of Alabama in Huntsville remembers simple advice he got as a student: ‘look out of the window’, meaning ‘pay attention to what is happening out there in the world, look for issues that are ripe for investigation’. And since that time I have tried to do just that. For me, this has been more intellectually sustaining than, say, combing through some literature in the hopes of seeing a useful extension.
A simple observation can spark off a whole train of ideas. Roland Andersson, of the Department of Surgery in Joenkoeping, Sweden, said For me it started like this: I observed that we had had 12 patients with appendicitis during one week. The following weeks we had only one or two. I wondered: ‘Had we had an epidemic of appendicitis?’. I happened to know about Knox space-time analysis and I started off from there and finally have written a thesis about ‘Appendicitis - epidemiology and diagnosis’. Lots of new questions arise and I am now involved in a (as it seems) never ending project about aspects of appendicitis. (And please, don’t worry if you have no idea what Knox space-time analysis is; the important point is that Roland brought together a specialised theoretical framework which he already knew and a common everyday observation. In other words, he applied the theory he knew to the world outside the window.)
But what frame of mind, what view of the world do you need in order to have productive research ideas? A lot of discussion focussed on this question. At one extreme was Robert Hamer, who very much doubted whether you could teach anyone how to look at the world in a questioning manner. I don’t think that this is true, though. We are brought up in a way that does not encourage us to question the explanations we are given for things. Don’t forget that all children are hungry to find things out, to know why things are so. This voice of hunger for knowledge and delight in figuring things out is much smaller and more timid by the time we have grown up, but with patience it can be called back. It takes time to rid ourselves of this learned uncuriuosity.
The trick is doing what children do: asking lots of questions and teasing out the logical consequences of the answers. Paul Velleman again: “Dennis is right that the problem is nudging the mind. We need to start that process in childhood. We must cultivate in our children and our students a broad-based skepticism coupled with a sense that there *is* order in the universe."
These are the sorts of questions that scientists and other children ask.
- How does this work?
- What is the proposed biological explanation for the thing I am looking at?
- How do we know?
- Is there another possible explanation?
One must maintain an active and abiding skepticism about the explanations and models that have been proposed in science. Skepticism, which Paul Velleman identifies as a key attitude, doesn’t involve simple disbelief, but rather being able to entertain a number of different explanations at once.
This struck a chord with Robert Knodt: After being involved with masters and doctoral students for over thirty-years and looking back for an answer to the original post, I find that the statement above applied to over 90% of those I helped... The first person I worked with was bothered by a statement in a 10th grade Biology book which said that trees were pruned in the fall in order to make them fill out areas and become more symmetrical. This still bothered him eight years later. He finally did is work on ‘wound’ hormones in plants.
Says who? Many pieces of medical knowledge are folkloric, and the evidence is slender. In particular
- Clinical signs and diagnostic tests--are they really reliable and valid? The apex beat, for instance, is taught as a religion in some cardiology departments. Research into the repeatability of this clinical sign shows that it is hard for two observers to locate it in the same patient, much less in the same place.
- Factors are supposed to be associated with prognosis or diagnosis: where’s the evidence? Women with small feet, for instance, are supposed to be at higher risk of caesarian section because of the baby’s head being too big for the pelvis. But this assertion rests on a couple of small and doubtful studies. Better research (already done--sorry!) has shown that this is just a medical myth.
- Rituals of patient management (surgical masks, changing wound dressings) may have no evidence to support their usefulness.
I don’t believe that! Always trust your disbelief. Often a trip to the library will put your mind at rest, but think about
- Alternative explanations for things you have been told
- Your feeling that what you see in your own practice is not what you have been told you will see
Why are we doing this? At every point in clinical practice there are decision forks. Some may be invisible (we always do X when Y happens) but these are the most interesting! For example
- What is the most informative sequence of diagnostic tests?
- What are the treatment options at a given moment?
- How do we decide between them?
Why are they both right? Some disagreements in the literature are because no-one has yet spotted the reason why two different sets of investigators should have observed data that were seemingly contradictory.
Can we learn from the abnormal? We learn once from describing the normal--normal course of disease, normal range of variation etc. We learn a second time by examining cases that do not fit the general picture. Rare, pathological conditions can give us an insight into how more subtle, commonplace processes work.
- Congenital homocysteinemia and CHD led to the investigation of homocysteine in CHD in ‘normal’ people
- Research into doctor-patient interactions in hypochondriasis gave leads for investigating health anxiety as a dynamic in these interactions in normal patients.
Final thoughts
I don’t know where ideas come from, but I do know that you get more ideas if you try to remember everything that happens that doesn’t have a good explanation. I carry a little black notebook which can simply be used to note phone numbers and things I have to buy next time I go shopping, but it also means that I have a way of writing down an idea the instant I spot something interesting.
The last thing I want to say is based on my experiences teaching music to adolescents, as much as teaching research methods to medical students. The biggest obstacle you encounter is a feeling that you can’t do this; that you aren’t the sort of person who can sing, or make interesting observations or pose original questions. Just remember: this is what you did as a child, before you were taught any different. So you already know how to do this; just think of yourself as a little rusty.
The copyright for this page belongs to Ronan Conroy.
You can find an earlier version of this page on my original website.