I was asked by BMJ to review a paper that involved a qualitative data analysis. These reviews are confidential, so I don’t want to describe the paper in any detail. It is worthwhile, however, to note some of the standards that others have suggested for assessing the quality of a qualitative data analysis.
This is important for a peer-reviewer like myself. I need to be able to assess whether the authors have produced a result that is sufficiently rigorous to merit publication.
Perhaps the best paper in my files on this topic is
- Mays N, Pope C. Qualitative research in health care. Assessing quality in qualitative research. British Medical Journal 2000: 320(7226); 50-2. Aritlce is behind a pay wall.
The authors address a debate in the research community: should qualitative research be judged by the same standards as quantitative research. In particular, do concepts like validity, generalizability, and reliability apply in the same ways?
Those who would answer “no” to this question, the antirealists,
argue that qualitative research represents a distinctive paradigm and as such it cannot and should not be judged by conventional measures of validity, generalisability, and reliability. At its core, this position rejects naive realism—a belief that there is a single, unequivocal social reality or truth which is entirely independent of the researcher and of the research process; instead there are multiple perspectives of the world that are created and constructed in the research process.
The authors do not use the term, but the perspective of antirealists reminds me of post-modern philosophy. There are numerous post-modern critiques of evidence-based medicine, and I would like to summarize some of these articles in a separate web page.
On the opposite side of the fence are the relativists, those who would offer a conditional “yes” to the above question. According to the authors, relativists believe that
assessment criteria are feasible but that distinctive ones are required to evaluate qualitative research have put forward a range of different assessment schemes. In part, this is because the choice and relative importance of different criteria of quality depend on the topic and the purpose of the research.
The criteria used by relativists to assess the quality of a qualitative analysis include the following:
- Degree to which substantive and formal theory is produced and the degree of development of such theory,
- Novelty of the claims made from the theory,
- Consistency of the theoretical claims with the empirical data collected,
- Credibility of the account to those studied and to readers,
- Extent to which the description of the culture of the setting provides a basis for competent performance in the culture studied,
- Extent to which the findings are transferable to other settings,
- Reflexivity of the account—that is, the degree to which the effects of the research strategies on the findings are assessed or the amount of information about the research process that is provided to readers.
I personally do not find this list very helpful.
The authors then summarize a perspective of a third group, the subtle realists, who hold a position somewhere between antirealists and the relativists. Like the antirealists, this group does recognize that
there are multiple perspectives of the world that are created and constructed in the research process,
but does argue that
there is an underlying reality which can be studied
and that while research may not be able attain the “truth”, it is capable of representing the underlying reality.
The difference between attaining truth and representing an underlying relative is a subtle difference, and it is unclear to me whether this in an important difference. But the criteria that the authors propose for this group are promising.
- Triangulation. I have seen this term used narrowly to represent a series of research studies that alternate between qualitative and quantitative research, with each quantitative study building on the knowledge of the previous qualitative study and each qualitative study building on the knowledge of the previous quantitative study. The authors of this paper take a broader perspective. Triangulation is the use of multiple methods of data collection or the use of multiple sources of data.
- Respondent validation. If a qualitative researcher really made a hash of things, the people best able to point this out would be the participants whose interviews and comments served as the raw data for the qualitative data analysis. So you can validate a qualitative analysis by showing a summary of the qualitative study to the original sample and recording their comments and opinions about the summary using the same qualitative analysis tools.
- Transparency. If you provide clarity in the process by which various themes have been identified and developed, your reader should be able to make informed judgments about the extent to which your work was influenced by personal biases.
- Reflexivity. This is a term that the authors use without a good definition. One source defines it as an “awareness of the researcher’s contribution to the construction of meanings throughout the research process, and an acknowledgment of the impossibility of remaining ‘outside of’ one’s subject matter while conducting research.” When you are documenting your qualitative research study, you should therefore define your own prior assumptions and experience.
- Attention to negative data. The authors also define this as deviant case analysis. If you develop a theme in a qualitative research study, there will be some raw data that supports this theme and some that undermines this theme. By giving extra attention to the non-supportive data and trying to reconcile it with your results, you produce a produce that does a better job of representing the totality of the data rather than an isolated viewpoint.
- Fairness. Is there sufficient diversity in your sample to insure that an isolated viewpoint does not predominate.
The authors do point out that all of these criteria have limitations, but I believe it still represents a valuable and interesting list of perspectives to consider as you are reviewing a qualitative research study.
Outside resources:
- http://www.gla.ac.uk/departments/generalpractice/qualitative_research.PDF (Link is broken)
- http://www.wmich.edu/evalctr/checklists/qec/index.htm (Link is broken)
- [http://www.phru.nhs.uk/casp/casp_qualitative_tool.pdf (Link is broken)
- Trisha Greenhalgh. How to read a paper: Papers that go beyond numbers (qualitative research). BMJ 1997; 315:740. Article is behind a paywall
You can find an earlier version of this page on my old website.