Medical Statistics and Medical Errors, Part I

     The field of medical statistics is poorly understood by many doctors and by most patients. An article in the New England Journal of Medicine that surveyed over 100 articles published in reputable medical journals stated that over 50% of them used medical statistics incorrectly, and furthermore stated that this misuse invalidated the clinical conclusions of many of these articles. In this blog I will not discuss esoterica such as when to use a two-tailed t-test rather than a one-tailed test, or the uses and misuses of the Cox regression analysis. Instead I will try to point out fallacies in reasoning and statistics that should cause you to doubt a result quoted in the journal or in the newspapers.

     Let me begin by saying that except for out-and-out charlatans, no physician deliberately lies or misuses statistics in a paper. However, it is very easy to delude oneself when doing research. A classic example occurred to a physicist in France in the 1920's and 1930's who claimed to see special rays, called "N-rays" when a beam was passed through an aluminum prism. Only he was able to see these rays, which were subsequently proven to be nonexistent when an American physicist named Wood surreptitiously removed the prism and the rays were still seen by the French scientist. More recently, we can recall the Utah scientists who claimed to have produced "cold fusion" with a table-top experiment. In the distant past, I can refer you to the European horse, "Clever Hans", who was claimed by his owner to stamp his hoof four times when asked for the sum of two plus two, and was also observed to answer similar  arithmetical questions. It was later shown that the horse was (subconsciously) interpreting minute muscular twitches of his master when the number of hoof stomps reached the desired answer.

     Medical error can reach unimaginable heights. The Nobel Prize in Medicine was awarded to Dr. Egas Moniz in 1949 for work he did before WWII claiming to show that prefrontal lobotomies calmed and cured schizophrenic patients. There was no control group cited, and except for the surgery in "One Flew Over the Cuckoo's Nest" it is rarely if ever performed today. But schizophrenia was such a treatment-defying mental disease that the doctors became desperate to seize upon any procedure that seemed to promise a cure, including insulin-induced hypoglycemic shock. Again let me emphasize  that the doctors believed that they were helping the patient, even though a true blinded clinical study had never been done.

     I have already blogged twice about the statistical errors  inherent in meta-analyses of clinical studies, and have published a critique of meta-analysis in the Volume 81 edition of "Chance", a statistical journal published by the mathematics department of Middlebury College, Vt. The point I want to make is that the proper use of meta-analyses is to suggest hypotheses which then should be tested in a proper double-blind clinical study. It is irrational and a violation of statistical theory to combine a number of non-significant clinical studies and thereby to claim a statistically significant clinical result. I can assure my readers that no such approach was ever taken to decide the speed of light or the mean distance of the earth from the sun.

     I cannot emphasize enough the importance of paying attention only to articles and reports of clinical studies that are published in a refereed medical journal. The referee (and I have been both  a referee and an associate editor for both physics and medical journals) wants to ensure that only statistically accurate clinical reports are published. This is a valuable filter because the referee has no axe to grind, and (should be) free from any bias. Thus the referee ensures that the article makes sense, is statistically and scientifically accurate, has enough patients in the study to make the result clinically important as well as statistically correct, and contains enough information that any other medical scientist could repeat the exact study.

     Whenever you see a newspaper report about a medical study, you should try to at least get an abstract of the paper. The abstract is almost always available through PubMed.com, a library journal listing service managed by the National Institutes of Health. Many many times the newspaper, in an effort to simplify the scientific facts, misstates the results of the study. Needless to say, any TV interviews with a doctor should be taken with a large grain of salt, especially if he or she is publicizing a recently written book. Again, the gold standard is a refereed paper published in a reputable medical journal. And let us not forget how many times the clinical conclusions of a published article were refuted by the next clinical article that dealt with the same medical problem. That is why I always tell my patients to wait for and look for a second confirming article, and not to believe any medical "progress" that is written up in the National Enquirer.  

     As a final point, let me re-iterate the importance of differentiating between relative risk and absolute risk. The absolute risk is the benefit or risk you gain when compared with the total population of patients,  while the relative risk compares you to other patients with your identical clinical problem(s). Anticoagulation can reduce the risk of a stroke in patients with atrial fibrillation. The absolute risk of stroke in patients without anti-coagulation may be 5% in two years, and the absolute risk of stroke for patients who are anti-coagulated may be 2.5% in two years. You can therefore claim, with equal validity, that anti-coagulation reduces the absolute risk by 2.5% in two years, or reduces the relative risk by 50% in two years and most newspaper results and PR results will quote the 50% relative risk reduction because it is a larger number and therefore sounds more significant to the uninitiated.

     Let me close by citing verbs that should alert you to a non-scientific conclusion: if the results "suggest that", "can be interpreted as", "might mean that", "leads doctors to believe that", "indicates that",  "can support no other conclusion", then run for the nearest exit, because such phrases never precede a statement in a physics journal about the measured  half-life of the mu meson (2.2 microseconds at rest  for those of you who are interested in the result).