[StATS]: So you want to write a questionnaire (July 12, 2002)
Dear Professor Mean, I need to write a questionnaire for a research study I am conducting. Can you help me write it? – Cautious Carmen
If I wrote the questionnaire, it would include a bad joke on every page.
You need to think about several issues while writing a questionnaire:
- What is the purpose of your questionnaire?
- What level of anonymity can you provide?
- How will you minimize non response?
- Are you asking the right questions?
Also make sure that you do some pilot testing of your questionnaire.
I’ll presume, for the most part, that you are giving a questionnaires to individual patients, but you can also send them out to groups and organizations as well.
What is the purpose of your questionnaire?
You need to identify the purpose of your survey. Are you trying to identify
or some combination of the above.
Who are you sampling from and who do you want to generalize these results to. Will you make any<U+00A0>extrapolations from this survey?
Will you collect the data
- through a web page
- through postal mail
- through FAX
- through email
- through the phone
- through a<U+00A0> face to face interview
Have you considered a focus group as an alternative method for data collection?
What level of anonymity can you provide?
You should provide the greatest degree of anonymity possible and you should inform your patients what level of anonymity you can provide.
For may questionnaires, you will publish only aggregate results; individual responses will not be reported. If you have to link the questionnaire data to a medical record or other source of information, then you need to inform your patients.
Sometimes you need to track the survey in order to find out who to send a follow up reminder notice to. This could be done with the use of a code number on the survey, but if you can, you should assure the patient that this code will not be used beyond the use of reminder notices.
I am starting to write up a web page about privacy concerns in research.
You need to identify any sensitive questions. Some examples include
- genetic information
- information about mental illnesses
- information about sexual attitudes, preferences or practices
- information on the use or abuse of alcohol and other drugs
- information on illegal activities
What is sensitive may depend on what group you are asking. Questions about smoking and alcohol consumption might be more threatening, if your population is a group of pregnant women.
Ask yourself if the disclosure of this information might embarrass or harm the respondent. If it can, then you need to take special precautions.
How will you minimize non response?
Some of the people you send your questionnaire to will not receive it. Some of them will not return the questionnaire. And those who do respond, may not respond to all the questions. All of this can cause a serious bias in your data analysis.
Think first about motivation. Why would anyone take the time to fill out and return your questionnaire? You need to give them some incentive.
- You might include something of value with your survey such as cash or a gift certificate. Your budget probably can’t afford a large incentive, but a large incentive might be considered coercive anyway.
- Sometimes people are motivated by altruism, so you should explain how your questionnaire will help make the world a better place. Make sure that your patients see a link between this questionnaire and something that is important to them.
- Curiosity can also motivate; consider offering a summary of your research findings after the questionnaires have been analyzed.
Also be sure to avoid common demotivators.
- Don’t give your patients an undue work burden with an overly long and complex questionnaire.
- Don’t ask for information that you don’t need or which you already have.
- Don’t make your patients pay for a stamp or a long distance phone call.
- If you are sending a survey to an organization, don’t send it to the wrong department and certainly don’t send it to a general address with the hope that it will find its way to the right person.
If possible, use follow up reminders by phone, email, or postal mail to those who do not respond by the deadline. These reminders can sometimes raise concerns about anonymity, so be careful about this. Use coded numbers on the surveys to track who has responded and let them know that the link between the codes and any personal identifiers will be destroyed.
The best way to minimize the number of non respondents is to make the survey clean, simple, and easy to respond to.
- Most of us have limited attention spans. Be brief.
- Most of us are easily confused and befuddled. Ask one question at a time. If you are using conjunctions (and/or) in a question, try splitting it into two more more simpler questions.
- Most of us have dreadful memories. Minimize the amount of recollection that your patients have to do. Don’t ask for exact numbers when a range will do.
- Most of us do not handle abstractions well. Try to ask questions about tangible items and give examples. Avoid questions about concepts that are not encountered in daily living.
- Most of us are impatient. Ask questions that your patients can answer rapidly and without much mental effort. Avoid questions that involve arithmetic computations, such as adding up several sources of income. Avoid questions that involve ranking or selecting preferences from a long list.
Finally, be sure choose an appropriate language level. For many questionnaires, you should write at a fifth grade reading level.
In spite of all this some people will not respond. If you can get some abbreviated information from them, such as demographics or their reasons for not participating, that might help. It might determine a good statistical adjustment for your data. Even if you can’t adjust for it, this information might help you determine the direction and severity of any bias caused by non response.
Are you asking the right questions?
Use standardized questions and scales whenever you can. These standards were developed and tested over a long period of time, so you know how they will behave. By using standardized questions, you also make it easier for anyone who might incorporate your research into a systematic review or meta-analysis.
A good example of a standardized scale is the Burns Anxiety Inventory. This is a series of 33 questions about anxious feelings, anxious thoughts, and physical symptoms. Here are six of the items:
- Feeling that things around you are strange, unreal or foggy.
- Apprehension or a sense of impending doom.
- Racing thoughts or having your mind jump from one thing to the next.
- Feeling that you’re on the verge of losing control.
- Butterflies or discomfort in the stomach.
- Tight, tense muscles.
By asking a wide range of questions about anxiety, you are helping to get an accurate assessment of anxiety, especially for those patients who might show anxiety in some ways but not in others.
Lack of standards can cause problems. Jadad and Gagliardi (1998) criticize scales used to rate web sites providing health information. There were too many of them, most of them did not present any justification for their
When you are using categories, use the same categories that others use. Birthweights, for example, as classified as low (LBW) if less than 2500 grams, very low (VLBW) if less than 1500 grams, and extremely low (ELBW) if less than 1000 grams.<U+00A0>
Running a pilot study of the questionnaire
Pilot test your questionnaire.
- What are you thinking?
- Remember to read aloud for me.
- Can you tell me more about that?
- Could you describe that for me?
- Remember to tell me what you are doing.
– Dillman, page 143.
Here are some other issues to examine during a pilot.
- Were any items skipped frequently?
- Were any items answered incorrectly or ambiguously?
- Were any items redundant (no variation, or perfect correlation with another item?)
- Should you add extra categories to certain questions?<U+00A0>
Try to estimate the resources you need to conduct this questionnaire.
[This section is not yet available.]
Thank you for filling out this survey. We don’t have enough money to include a pre-printed envelope. Be sure to use extra postage, since the survey weighs more than one ounce. We’re not sure how we will use this data and maybe we’ll disclose this information to other researchers. This survey hasn’t had any pilot testing, so if we goof up badly, you might have to fill out a better one later. No one has used the survey form before, so we’re not sure if we’ll find out anything interesting.
Survey Research Methods Second Edition. Earl Babbie (1990) Belmont, California: Wadsworth Publishing Company.
Examination of a survey methodology. Dillman’s Total Design Method. FE Crosby, MR Ventura, MJ Feldman. Nurs Res 1989: 3856-58.
Mail and Internet Surveys: The Tailored Design Method. Don A. Dillman (2000) Canada: John Wiley & Sons, Inc.
Mail and telephone surveys: the total design method. DA Dillman. New York et al.: Jon Wiley & Sons 1978:
How Surveys Answer A Key Question: Are Consumers Satisfied With Managed Care?. Karen Donelan. Accessed on 2003-10-20. www.managedcaremag.com/archiveMC/9602/MC9602.survey.shtml
Survey Research Methods Second Edition. Floyd J. Jr. Fowler (1993) Newbury Park, CA: Sage Publications, Inc.
Improving Survey Questions: Design and Evaluation. Floyd J. Jr. Fowler (1995) Thousand Oaks, CA: Sage Publications, Inc.
A Brief Guide to Questionnaire Development. Robert Frary. Accessed on 2001-01-04. www.testscoring.vt.edu/fraryquest.html
Survey research. JA Krosnick. Annu Rev Psychol 1999: 50537-67. [Abstract]
Research Resources. Laurier Institute for the Study of Public Opinion and Policy, Laurier Institute for the Study of Public Opinion and Policy. Accessed on 2003-10-20. www.wlu.ca/lispop/lispop.htm
How to Measure Survey Reliability and Validity. Mark S. Litwin (1995) Thousands Oaks, CA: Sage Publications.
How to Conduct your Own Survey. Priscilla Salant, Don A. Dillman (1994) Toronto: John Wiley & Sons, Inc.
Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. Howard Schuman, Stanley Presser (1996) Thousand Oaks, CA: Sage Publications.
Health Measurement Scales A Practical Guide to Their Development and Use. David L. Streiner, Geoffrey R. Norman (1989) New York: Oxford University Press, Inc.
Standard Definitions: Final Dispostions of Case Codes and Outcomes Rates for Surveys. Mischael W. Traugott, Murray Edelman, Warren J. Mitofsky, The American Association for Public Opinion Research. Accessed on 2000-www.aapor.org/default.asp?page=survey_methods/standards_and_best_practices/standard_definitions
The Survey Research Handbook Second Edition Guidelines and Strategies for Conducting a Survey. Alreck PL, Settle, Robert B. (1995) Chicago, IL: Irwin Professional Publishing.
PEDAKSI: methodology for collecting data about survey non-respondents [pdf]. Lynn PJ, Institute for Social & Economic Research. Accessed on 2005-04-06. www.iser.essex.ac.uk/pubs/workpaps/pdf/2002-05.pdf
Separating Refusal Bias and Non-Contact Bias: Evidence from UK National Surveys [pdf]. Lynn PJ, Clarke P, Institute for Social & Economic Research, Working Paper 2001-24 (November 2001). Accessed on 2005-04-06. www.iser.essex.ac.uk/pubs/workpaps/pdf/2001-24.pdf
Brochures about Survey Research. ASA Series: What is a Survey? American Statistical Association Survey Research Methods Section. www.amstat.org/sections/SRMS/whatsurvey.html
Further reading – Empirical evidence of response bias
Effect of UK national guidelines on services to treat patients with acute low back pain: follow up questionnaire survey. A. G. Barnett, M. R. Underwood, M. R. Vickers. British Medical Journal 1999: 318(7188); 919-20. [Medline] [Full text] [PDF]
Mortality and cancer rates in nonrespondents to a prospective study of older women: 5-year follow-up. K. M. Bisgard, A. R. Folsom, C. P. Hong, T. A. Sellers. American Journal of Epidemiology 1994: 139(10); 990-1000. [Medline]
Characteristics of non-responders and the impact of non-response on prevalence estimates of dementia. F. Boersma, J. A. Eefsting, W. van den Brink, W. van Tilburg. International Journal of Epidemiology 1997: 26(5); 1055-62. [Medline]
Non-response bias in a lifestyle survey. A. Hill, J. Roberts, P. Ewings, D. Gunnell. J Public Health Med 1997: 19(2); 203-7.
The Tromso Heart Study: responders and non-responders to a health questionnaire, do they differ? B. K. Jacobsen, D. S. Thelle. Scand J Soc Med 1988: 16(2); 101-4.
Do safety practices differ between responders and non-responders to a safety questionnaire? D. Kendrick, R. Hapgood, P. Marsh. Injury Prevention 2001: 7(2); 100-3. [Medline] [Abstract] [Full text] [PDF]
Nonresponse bias in a national study of dentists’ infection control practices and attitudes related to HIV. G. M. McCarthy, J. K. MacDonald. Community Dent Oral Epidemiol 1997: 25(4); 319-23.
Comparison of early and late respondents to a postal health survey questionnaire. A. Paganini-Hill, G. Hsu, A. Chao, R. K. Ross. Epidemiology 1993: 4(4); 375-9.
Quality of response in different population groups in mail and telephone surveys. J. Siemiatycki, S. Campbell, L. Richardson, D. Aubert. Am J Epidemiol 1984: 120(2); 302-14.
Representativeness and response rates from the Domestic/International Gastroenterology Surveillance Study (DIGEST). J. G. Tijssen. Scand J Gastroenterol Suppl 1999: 23115-9. [Medline]
What are the characteristics of general practitioners who routinely do not return postal questionnaires: a cross sectional study. N. Stocks, D. Gunnell. J Epidemiol Community Health 2000: 54(12); 940-1. [Medline]
Further reading – Ambiguous Questions
Would you say you “had sex” if...? S. A. Sanders, J. M. Reinisch. Jama 1999: 281(3); 275-7.
Collection of Race and Ethnicity Data in Clinical Trials. U.S. Food and Drug Administration. Accessed on 2003-02-25. www.fda.gov/cber/gdlns/racethclin.htm
Further reading – Format of Your Survey
Different response rates in a trial of two envelope styles in mail survey research. D. A. Asch, N. A. Christakis. Epidemiology 1994: 5(3); 364-5. [Medline]
A comparison of responses to mailed questionnaires and telephone interviews in a mixed mode health survey. D. J. Brambilla, S. M. McKinlay. American Journal of Epidemiology 1987: 126(5); 962-71. [Medline]
Increasing response rates to postal questionnaires: systematic review. Phil Edwards, Ian Roberts, Mike Clarke, Carolyn DiGuiseppi, Sarah Pratap, Reinhard Wentz, Irene Kwan. BMJ 2002: 324(7347); 1183-. [Abstract] [Full text] [PDF]
Measuring later health status of high risk infants: randomised comparison of two simple methods of data collection. D. Field, E. S. Draper, M. J. Gompels, C. Green, A. Johnson, D. Shortland, M. Blair, B. Manktelow, C. R. Lamming, C. Law. British Medical Journal 2001: 323(7324); 1276-81.
Increasing response rates for mailed surveys of Medicaid clients and other low-income populations. P. J. Gibson, T. D. Koepsell, P. Diehr, C. Hale. Am J Epidemiol 1999: 149(11); 1057-62.
Do postage-stamps increase response rates to postal surveys? A randomized controlled trial. R. A. Harrison, D. Holt, P. J. Elton. Int J Epidemiol 2002: 31(4); 872-4. [Medline]
A comparison on nonresponse in mail, telephone, and face-to-face surveys. J. J. Hox, D De Leeuw. Quality and Quantity 1994: 28(4); 329-344.
Does length of questionnaire matter? A randomised trial of response rates to a mailed questionnaire. C. Iglesias, D. Torgerson. J Health Serv Res Policy 2000: 5(4); 219-21. [Medline]
Improving the measurement of quality of life in older people: the York SF-12. C.P. Iglesias, Y.F. Birks, D.J. Torgerson. QJM 2001: 94(12); 695-698. [Abstract]
Increasing response rates to postal questionnaires. Cynthia P Iglesias, Yvonne F Birks, David J Torgerson, Paula-J Roberts, Chris Roberts, Bonnie Sibbald. BMJ 2002: 325(7361); 444-. [Full text]
Response rate according to title and length of questionnaire. E. Lund, I. T. Gram. Scand J Soc Med 1998: 26(2); 154-60.
Comparability of telephone and household breast cancer screening surveys with differing response rates. R. M. Mickey, J. K. Worden, P. M. Vacek, J. M. Skelly, M. C. Costanza. Epidemiology 1994: 5(4); 462-5.
Methods for the design and administration of web-based surveys. T. K. Schleyer, J. L. Forrest. J Am Med Inform Assoc 2000: 7(4); 416-25. [Medline] [Abstract] [Full text] [PDF]
Understanding Implementation. The mechanics of polling.. The Statistical Assessment Service. Accessed on 2003-10-20. www.stats.org/record.jsp?type=news&ID=378
Question Time. The Statistical Assessment Service. Accessed on 2003-10-20. www.stats.org/record.jsp?type=news&ID=382
Response to mail surveys: effect of a request to explain refusal to participate. The ARIC Study Investigators. E. Shahar, K. M. Bisgard, A. R. Folsom. Epidemiology 1993: 4(5); 480-2.
A comparison of mail, telephone, and home interview strategies for household health surveys. J. Siemiatycki. Am J Public Health 1979: 69(3); 238-45.
Nonresponse bias and early versus all responders in mail and telephone surveys. J. Siemiatycki, S. Campbell. Am J Epidemiol 1984: 120(2); 291-301.
Improving the response rates to questionnaires. Liam Smeeth, Astrid E Fletcher. BMJ 2002: 324(7347); 1168-1169. [Medline] [Full text] [PDF]
Increasing response rates in telephone surveys: a randomized trial. W. Smith, T. Chey, B. Jalaludin, G. Salkeld, T. Capon. J Public Health Med 1995: 17(1); 33-8.
Using the Visual Analog Scale. Chad Starkey, Pete Koehneke, Daniel Sedory, Paula Turocy. Accessed on 2003-06-23. www.cewl.com/clined/acpm/app_c.html
Is Shorter Always Better? Relative Importance of Questionnaire Length and Cognitive Ease on Response Rates and Data Quality for Two Dietary Questionnaires. Amy F. Subar, Regina G. Ziegler, Frances E. Thompson, Christine Cole Johnson, Joel L. Weissfeld, Douglas Reding, Katherine H. Kavounis, Richard B. Hayes. Am. J. Epidemiol. 2001: 153(4); 404-409.
Comparative Response to a Survey Executed by Post, E-mail, & Web Form. Gi Woong Yun, Craig W. Trumbo. JCMC 2000: 6(1); [Full text]
Refusal and information bias associated with postal questionnaires and face-to-face interviews in very elderly subjects. R. Hebert, G. Bravo, N. Korner-Bitensky, L. Voyer. J Clin Epidemiol 1996: 49(3); 373-81.
Further reading – Fraud
Interviewer Falsification in Survey Research.. Section on Survey Research Methods, American Statistical Association. Accessed on 2003-05-15. www.aapor.org/interviewfalse.pdf
Further reading – Interviewer Effects
Do interviewers’ Health Beliefs and Habits Modify Responses to Sensitive Questions? A Study using Data Collected from Pregnant Women by Means of Computer-assisted Telephone Interviews. Anne-Marie Nybo Anderson, Jorn Olsen. American Journal of Epidemiology 2002: 155(1); 95-100.
Further reading – Nonresponse Bias
Response and nonresponse bias in oral health surveys. D Locker. Journal of Public Health Dent 2000: 6072-81. [Medline]
Separating Refusal Bias and Non-Contact Bias: Evidence from UK National Surveys. Peter J. Lynn, Paul Clarke, Institute for Social & Economic Research. Accessed on 2003-10-20. www.irc.essex.ac.uk/pubs/workpaps/wp2001-24.php
PEDAKSI: methodology for collecting data about survey non-respondents. Peter J. Lynn, Institute for Social & Economic Research. Accessed on 2002-February. www.irc.essex.ac.uk/pubs/workpaps/2002-05.php
Further reading – Response Rates
Response rates to mail surveys published in medical journals. D. A. Asch, M. K. Jedrziewski, N. A. Christakis. Journal Clinical Epidemiology 1997: 50(10); 1129-36. [Medline]
Reported response rates to mailed physician questionnaires. S. M. Cummings, L. A. Savitz, T. R. Konrad. Health Serv Res 2001: 35(6); 1347-55. [Medline]
Further reading – Reliability and Validity
seamonkey.ed.asu.edu/~alex/teaching/assessment/reliability.html Reliability and Validity by Chong Ho (Alex) Yu. This page discusses the issues surrounding reliability and validity.
trochim.human.cornell.edu/kb/measure.htm Measurement by Bill Trochim. This page discusses various research topics in psychology including the various types of validity.
www.yorku.ca/dept/psych/classics/Cronbach/construct.htm Construct Validity in Psychological Tests, Lee J. Cronbach and Paul E. Meehl (1955). First published in Psychological Bulletin, 52, 281-302. The full text of this classic paper on validity is available on the Internet.
Rating health information on the Internet: navigating to knowledge or to Babel? Jadad, A. R. and A. Gagliardi (1998). Jama 279(8): 611-4.
Developing a scale for measuring the barriers to condom use in Nigeria. Sunmola, Adegbenga M. Bull World Health Organ, 2001, vol.79 no.10. ISSN 0042-9686.
You can find an earlier version of this page on my original website.