What alternative medicine can teach us about evidence-based medicine

Steve Simon


(To be added)

This is a rough outline of a seminar I will present in a couple of days. It incorporates material from another talk,

Is the randomized trial the gold standard for research?

The title of this talk seems to be backwards. The question most people are interested in is, “What can evidence-based medicine teach us about alternative medicine?” It turns out that we can learn quite a bit. A quick search of PubMed for the term “alternative medicine” restricted (for my convenience) to article in PubMed Central and limited to meta-analyses and systematic overviews produced the following reference.

This is an open source journal, so I can include as much of it as I like on my web pages without worrying about copyright restrictions. If you are curious, you can read the full free text of this article on the web. There’s even a journal called evidence-based Complementary and Alternative Medicine, and you can find the full free text of articles like the following:

This is also an open access article, but the notice at the top of the article reminds us that “if an article is subsequently reproduced or disseminated not in its entirety but only in part or as a derivative work this must be clearly indicated.”

That’s an important reminder and I apologize for only including the abstracts of these two articles. It’s a bad habit to read only the abstract. The abstract often leaves out important details. Often the important limitations appear only in the paper itself. Sometimes, the outcome measures highlighted in the abstract are the ones that are statistically significant, rather than the ones that are clinically important.

There is a lot more that can be said about the empirical evidence for or against various alternative medicine approaches, but I want to turn the question around. I am interested in what alternative medicine can teach us about evidence-based medicine.

First it might help to define exactly what alternative medicine is. I don’t want to dwell on this point too much, but you can get a good understanding of what a person thinks about alternative medicine by how they define it.

You can define it by exclusion, (everything that they didn’t teach you in medical school), but that is a very squishy definition:

You can also define alternative medicine as anything that has not yet been proven scientifically.

That is also a squishy definition (what is scientific proof?) but also a recipe that allows conventional medicine to preempt any approaches which have been proven effective and to leave alternative medicine with only the failures and unproven approaches. Seems a bit unfair to me.

The Institute of Medicine has published a book: Complementary and Alternative Medicine (CAM) in the United States and they offer a nice definition from the National Center for Complementary and Integrative Health.

The comments in the NEJM editorial, though, are interesting because they highlight some of the controversies about alternative medicine. They argue that

There is similar antagonism in the alternative medicine community to the standards of medical research.

The term used most often by critics of evidence-based medicine is “reductionism.”

We should probably stop and define evidence-based medicine here. There are a variety of definitions, but my favorite is from an article by Sackett et al in the British Medical Journal.

It’s important to remember to include the values of the individual patient in any EBM decision. Suppose you are considering a treatment that has as a side effect impairs the production of sperm and reduces your fertility. Some men, would not even consider such a treatment. They have a strong desire to father their own children now or in the future, and they would sacrifice their own health in order to maintain their ability to produce children. Other people would be totally indifferent to this side effect. A man with a vasectomy does not worry to much about drugs that alter his sperm production. Others might actually perceive reduced fertility as a benefit rather than a side effect.

So who is right in all of this debate. It turns out that the critics are half right. Some of their complaints are just “sour grapes.” but they also highlight some areas for improvement, especially in the conduct of randomized clinical trials.

A balanced perspective on this controversy appears in (Mason 2002). They point out that:

They argue that randomized trials have to be adapted to the special features of CAM. In particular, they point out that the tendencies of randomized trials and CAM are often in conflict. Here is a paraphrase of their main argument.

Note that these are tendencies. Some randomized trials focus on more than one disease, but the tendency is to focus on a single disease. Some types of CAM are standardized, but the tendency is to offer individualized therapies.

It’s not just CAM that exhibits these conflicts, though. The Medical Research Council wrote a report in April 2000 (Source: bmj) that discusses the evaluation of complex interventions where it is difficult to isolate the individual components of the intervention. They mention several examples.

Does a physiotherapist contribute significantly to the management of knee injuries? This role goes beyond a simple sequence of exercises.

The package of care to treat a knee injury may be quite straightforward and easily definable - and therefore reproducible: “This series of exercise in this order with this frequency for this long, with the following changes at the following stages”. However, the physiotherapist may have, in addition to the exercises, a psychotherapy role in rebuilding the patient’s confidence, a training role teaching their spouse how to help with care or rehabilitation, and potentially significant influence via advice on the future health behaviour of the patient. Each of these elements may be an important contribution to the effectiveness of a physiotherapy intervention.

How does a stroke unit improve the quality of care for stroke patients? The concept of a stroke unit is difficult to standardize.

For example, although research suggests that stroke units work, what, exactly, is a stroke unit? What are the active ingredients that make it work? The physical set-up? The mix of care providers? The skills of the providers? The technologies available? The organisational arrangements?

How cognitive behavioral therapy works? This approach is highly individualistic.

Does success depend on the personality of the therapist? The personality, health status, social status, or other characteristic of the patient? The content of the therapy? The way it is delivered? The frequency of contact? The location of contact? The duration and the timing? What other components count?

Rather than arguing that randomized trials need to be adapted to the special needs of CAM, perhaps randomized trials should be adapted to meet the special needs of many types of medical interventions.

Furthermore, the claim that a practice is holistic should not be used as a blithely disregard evidence from an overly simplistic randomized trial. Perhaps the randomized trial can get to the heart of the issue by focusing on a single key dimension to the problem. A fourth grade student evaluated Therapeutic Touch (TT) for a science fair project. This project was highlighted on a Public Broadcasting Service show “Scientific American Frontiers” (no longer available on the web) and published in the April 1, 1998 issue of JAMA and received a lot of press coverage.

Therapeutic Touch is a therapy to improve health through the manipulation of the human energy field. There apparently is no physical touching. The official website on therapeutic touch describes it as:

“…an intentionally directed process of energy exchange during which the practitioner uses the hands as a focus to facilitate the healing process. It is a contemporary interpretation of several ancient healing practices. Therapeutic Touch is a scientifically-based practice founded on the premise that the human body, mind, emotions and intuition form a complex, dynamic energy field. The human energy field is governed by pattern and order. In health, the field is balanced, however in disease, the energy is characterized by imbalance and disorder.” (Source: jama)

Emily Rosa’s experiment was very simple, perhaps too simple. If practitioners of Therapeutic Touch are able to manipulate energy fields, they must first be able to detect energy fields. She would hold her hand above either the left or right hand of the practitioner and ask him/her to tell which hand. The choice of hand was randomly determined by a coin flip. A screen with two holes in it prevented the practitioner from seeing what was going on.

Emily Rosa got 21 experienced practitioners to agree to the test. They were right only 44% of the time. Did this simple experiment disprove the healing power of TT? Perhaps not. TT is a complex intervention and this experiment only looked at a single aspect of it.

The experiment does shift the burden of proof, however. Detection of energy fields is a fundamental aspect of TT that all other aspects of this therapy rely on. How can practitioners of TT manipulate energy fields that they cannot even detect? Any further research should be discontinued until practitioners of TT can demonstrate the ability to detect energy fields in a rigorous blinded study.

Larry Sarner (Emily Rosa’s step-father) makes much the same point in an article on the Quackwatch web site that responds to criticisms of the Rosa study. In particular, he responds to the criticism of reductionism:

There is, by the way, a huge financial incentive to demonstrate the ability to detect energy fields. The James Randi Education Foundation offers a one million dollar prize to anyone who can show, under carefully controlled conditions, evidence of any paranormal, supernatural, or occult power or event. James Randi himself says that TT as well as several other alternative medicine therapies (Iridology, Reiki, Homeopathy and Applied Kinesiology) would qualify for the challenge.

Do commercial ties influence research findings? There are many documented cases where money does alter the research. Perhaps the best understood conflict of interest involves the tobacco companies. Financial support from tobacco companies has a large and quantifiable impact on the findings of a study. Articles on passive smoking written by authors affiliated with the tobacco industry were far more likely to conclude that passive smoking was not harmful (Barnes 1998). A review of studies on the economic effects of laws restricting smoking (Scollo 2003) showed that tobacco affiliations were associated with greater use of subjective outcomes, a lower rate of peer review, and a greater tendency to report negative economic impacts.

Support or commercial ties with pharmaceutical companies can also be troublesome. At least thirty studies have examined whether authors with commercial ties come up with more favorable conclusions about the drugs they are studying. A review of these studies, (Lexchin 2003) showed that industry-financed studies were four times more likely to reach conclusions favorable to the company’s product when the researchers were supported by the drug company. The authors offered five possible explanations:

Another problem is that authors rarely disclose possible conflicts. A review of disclosure of conflicts of interest (Hussain 2001) calculated the rate of disclosure at 1.4% (52 out of 3,642), a number that is far too low to be credible. If authors fail to report potential conflicts of interest, it may be out of the stubborn beliefs that commercial ties only influence other people (Boyd 2003).

Charges of financial conflict of interest are sometimes a “red herring” that is intended to distract from a discussion of the merits of the research. Stephen Senn tells an interesting story about himself (Senn 2001) where such a charge was leveled. Stephen Senn is a famous statistician with over 190 publications. Because of his stellar reputation, he is widely sought out as a statistical consultant to the pharmaceutical industry. In a discussion with an academic researcher, though, Dr. Senn was informed that his “source of employment” meant that his recommendations about the proper analysis of crossover trials were worthless. It didn’t matter that Dr. Senn had written the definitive textbook on that very subject (Senn 1993).

So how should you approach a research article where the authors have declared a conflict of interest? You should be cautious, but not cynical. If the research is objective, well documented, and subject to external review, then you should not let financial conflict of interest exert a veto power over the findings. On the other hand, an editorial article or opinion piece written by an author with commercial ties to a product being discussed in the editorial is very troublesome (Angel 1996).

Is there an explicit assurance from the author that the industry support still allowed the author to independently assess the data and to publish the results without first getting approval from the sponsor? A reasonable review period by the sponsor is acceptable as long the final decision to publish rests with the author and not the sponsor. A 2001 revision to the statement on publication ethics from the International Committee of Medical Journal Editors (Davidoff 2001) highlights how important this assurance is.

You can find an earlier version of this page on my original website.