Evaluation of Medical
Information
Evaluating
information, medical and otherwise, on the Web or in print is difficult. It is
important to decide if any given information is worth reviewing, using or
remembering. Here are some general guidelines to help you evaluate medical
information:
Who is
the author? What are her or his credentials? What is the motive for presenting
the information?
Is the
information presented clearly in an unbiased manner?
Are information
sources cited or footnoted so you can find the original author or are sources
readily available in a standard textbook?
Is the
information valid? To decide this, one has to know a bit about the scientific
method.
A brief review of THE SCIENTIFIC METHOD
At the
heart of the conflict between modern medicine and many health food store
owners, naturopaths, some chiropractors, a few physicians, and a variety of
others who attack the institutions of science, including medicine, is the
question of what is acceptable as true or factual. Both sides make claims and
counter claims. How can you decide who is right?
Modern
medicine is based on scientific observation, research, and testing. That means,
before a given statement or theory can be accepted, say eating green peppers
cures warts, it has to be proven by a scientific study. In this case a statistically valid,
controlled, double-blind study would best prove that green peppers cure warts.
STATISTICALLY
VALID means a large number of people must take part to remove the likelihood of
chance happenings from skewing the results. If two people are being studied for
wart cures, and one of them has his wart cut off by a piece of falling glass,
it is going to make a much bigger difference in the result of the study than if
five hundred people are being studied and one of them loses his wart to flying
glass. The larger number makes the study statistically valid.
A
CONTROLLED study has two similar groups of subjects to be tested. In our
example case, say there are two groups of 250 people each, all with warts. Both
groups have similar ethnic mix, a similar diet, take similar medications, live
in similar areas, etc., but one group eats green peppers and the other does
not. The experimental group eats peppers, the control group does
not. If the pepper group has fewer warts at the end of the study than the
other group, the control group, then it probably was the peppers and not the
sun or the blood pressure medicine or the high altitude that made the warts
disappear.
The
DOUBLE BLIND part is to eliminate the possibility that personal bias might
influence the results of the warts study. Psychological influence is very
important in medicine, but our question is about green peppers and warts so we
have to think up a way to have the test subjects "blind" to whether
they are taking peppers or not. We could disguise the peppers with a lot of
other greens in salad, or grind them up and put them in capsules to be
swallowed. We'd have to feed the control group pepperless salad or give them
capsules filled with sugar (a placebo) so both groups are treated equally, and
the test subjects in both groups would be "blind" to the knowledge of
which group was getting the peppers. Now, who is going to decide when a wart is
gone? Warts don't just suddenly disappear. They often get smaller until there's
just a little spot on the skin--which might be there for several weeks. The
test subjects can't say when a wart is gone because gone to one person might
not be gone to another. The guy who administered the peppers can't tell because
he knows who got peppers and who didn't and his bias toward pepper eaters might
influence his judgment. What is needed is someone who is "blind" to
who had peppers and who didn't to look at all the warts and decide which are
worse, which are unchanged and which are gone. This is the second
"blind" person, the first were the test subjects, hence the term
DOUBLE BLIND.
So,
after the statistically valid, controlled, double-blind study is done and
published, several other people will repeat the experiment to confirm the
results. If they all come out with similar results, then a definite statement
can be made regarding the tested question or theory. That's a lot of work, time
and expense involved in this kind of ideal scientific investigation, but
universities, governmental agencies, biotech companies, and sometimes individuals go to all
this trouble because that's really the only way to prove something works or
doesn't work.
However,
sometimes a question or theory is impossible or impractical to test completely.
Will the chicken pox vaccine require a booster shot thirty years after original
series of two shots are given? Just exactly what does dioxin, that deadly
contaminant in some weed killers or other petroleum products, do to the human
fetus? Do heart attack victims do better in zero gravity? In these cases animal
models, less ideal experimental designs, computer models, or sometimes simply a
well reasoned and educated guess is the best we can do under the circumstances. However, none of these answers are as
reliable as the results from a statistically valid, controlled, double-blind
study.
Some
questions don't fit scientific investigation. Did you think the sunset last
night was pretty? Would Hillary make a good President? Will the Kurds ever
become peaceful people? What is your favorite color? These are subjective
questions, not measurable scientific questions, and therefore not testable by
the scientific method.
Another
kind of evidence or argument is sometimes presented in medical circles and more
often in the popular press or by non-medical people. That is the anecdote or
personal experience. I knew a mother who swore blowing in her baby's face would
stop the hiccoughs. My cousin tried it and it worked for her, so now I
recommend it to all my friends. How do we know these were normal or average
babies? Was it the blowing, or the distraction or the halitosis that stopped
the hiccoughs? What constitutes stopping the hiccoughs anyhow--a fifteen second
pause? This kind of argument, an anecdote, is less valid than a good scientific
study with acknowledged limitations. Sometimes the question is minor and
doesn't deserve the expense and time of a full-blown scientific inquiry. The
best way to get a baby to take liquid medicine is to put it in a small amount
of formula and let him or her suck it from a nipple. I recommend that to a lot
of mothers just from my experience with children and from talking to mothers,
but I have never seen a statistically valid, controlled, double blinded,
scientific study on the subject.
Laetrile
cures cancer. Autism is caused by childhood immunizations. Vitamin C will
prevent or cure a cold. These are all statements that are scientifically
testable and have been publicly proclaimed by seemingly responsible and
knowledgeable people. These claims have been widely spread by the popular media
and taken as fact by many people--some physicians included. The scientific and
responsible approach would be to prove these claims prior to a press release.
These claims are being, or have been, tested and answered by good scientific
studies. However, scientific studies take time to plan, fund, test, review and
publish. Often years go by between the original allegations and the scientific
studies to test the claim. Often
scientific results get less media attention than the original claims.
A short review of META-ANALYSIS
Meta-Analysis
is a statistical combination of different scientific studies of the same or
similar questions for the purpose of further defining results. For example,
there might be dozens of studies done to evaluate green peppers and warts, but
the results are not consistent or maybe they are even conflicting. To help us
decide on the effectiveness of green peppers for getting rid of warts we need a
Meta-Analysis study of all these studies. We ideally need to include the
studies that are well designed on scientific principles, do not show bias, and
are similar enough in design to be comparable. But, this is not an ideal world.
A wart study done including only plantar warts, or dog warts, or an endpoint
when the wart is gone for ten years might not be included. A study of expert or
grandmother opinions of green peppers in the treatment of warts would be
excluded for lack of solid results based on scientific principles. Sometimes
good studies are done but there is a single flaw in the design that makes it
less reliable than another study without that flaw. For instance, maybe an
investigator had a well controlled, double blind, crossover study of warts and
green peppers done on army recruits. In the population of army recruits, there
is an obvious sex and age bias, and also a perhaps not so obvious racial bias.
In Meta-Analysis such a study may not be tossed out because it is not perfect, but
the results would be statistically weighted to account for the fact the study
population is biased.
So in
Meta-Analysis, after including only scientifically valid results, and after all
the studies are evaluated and weighted on the strength of their evidence, the
results of all the remaining studies are combined to get a larger population of
test subjects. A larger test population increases the reliability of the
results. In Meta- Analysis, a scientifically perfect study gets more weight
than a flawed but useable study, but both results can be used in the final
analysis. Maybe in our green pepper example there are six studies that are
included with a total experimental subject population of 3000 individuals.
Those results are more reliable and applicable to the practice of medicine than
any single study, no matter how well designed, that has perhaps only 35
experimental subjects.
A quick statement on EVIDENCE-BASED PRACTICE
Evidence-Based
Practice is a catch phrase of the decade in medicine (or other fields such as
teaching), but it also is a good shift toward more valid information on which
to base medical decisions. Evidence-Based Practice uses scientific information,
including Meta-Analysis studies, to decide medical questions whenever possible.
Traditionally doctors made decisions largely based on their training or expert
opinion. Hopefully if the doctor was well trained and talked to reliable
experts, the decisions made were based in science. There is a better way. In
our example of whether eating green peppers gets rid of warts, we would look at
the scientific literature for our results rather than rely on our doctor's
opinion. Such results are based on scientific evidence that applies to our
question and therefore are more trustworthy that the opinion of one person.
That is Evidence-Based Practice.
(Brief) CONCLUSION
So,
regardless of what you read or whom you talk to about a medical question, you have
to ask, where did the information come from and how was the information
derived? And in turn, what do you accept as true?
For
further discussion on scientific evaluation of information, see the Wikapedia page on the
subject.
Ted
Humphry, M.D.
this
information last updated 1/09
return to More Resources for
Information
return to Doubts About Immunizations