The Problem With Medical Clinical "Studies"

     I know that many readers are confused by or why one medical study refutes or contradicts a previous one,
or by how vehemently the purported results of a study are attacked by both physicians and patients who disagree with it. I will try to give some perspective and understanding about this, including why it happens. The practice of medicine is not a precise science, but rather a mixture of scientific principles, knowledge, art, intuition, luck,and  a certain je ne sais quoi. In addition,  you are dealing with people, all of whom have a tremendous number of "hidden variables", as Einstein would have put it.

     In a well-designed scientific experiment, the scientist keeps all the (known) variables fixed except one, and then he/she systematically varies the one, and looks to see how the system changes. He is seeking data, or verification of a scientific law or principle, but he/she should also be aware of the possible existence of the unexpected, and be prepared to recognize it when it happens, as did Rutherford when some of the alpha particles he shot at a gold foil were deflected/reflected sharply, and he thereby inferred that the atom had a hard nucleus surrounded by a shell of atoms. When Galileo used the first telescope to examine the skies, he immediately detected the moons of Jupiter, whose existence was hitherto unexpected. When Jenner observed that farm girls who became infected with cowpox (which scarified but was not fatal) never developed smallpox, he hit upon the idea of vaccination, and showed that deliberate inoculation with the fluid from the pustules of patients with cowpox prevented the recipients from infection with smallpox. (Of course, this was done years before Institutional Review Boards and the doctrine of Informed Consent had taken hold.)

     No one argues about the correct value for the speed of light, or Planck's Constant, or the diameter of the Earth, or the distance from the earth to the sun or that the core of the earth is liquid, or that the earth is round. (Actually there is a Flat Earth Society headquartered in London, but they have yet to construct a viable model of a flat earth than explains all modern observations.) What I am saying is that if there is vigorous, heated argument about a "fact", then either the data supporting the "fact" is not conclusive, or that some  people  have a strong vested interest or personal or political belief that makes the existence of the "fact" inconvenient for their belief system or is psychologically stressful. The concept that "ye shall know the truth and the truth shall make ye free" has been replaced by the concept that either the truth is not for all people (hence censorship), or that the truth might lead your thought patterns into dangerous realms (hence the insistence in the Middle Ages that the Earth was at the center of the universe, or the execution of Socrates by the democratic city of Athens for "leading the youth away from the true gods", or the removal of all mention of  Trotsky from the Soviet Encyclopedia by Stalin). I would briefly note the vociferous reaction to the announcement by one research group that according to their studies, annual mammograms of women between the ages of 40 to 49 did not seem to save lives from breast cancer as well as the present arguments for and against lowering the amount of salt in the diet. In medicine we still have the ongoing argument between Alcoholics Anonymous who insist that the only treatment for alcoholism is total abstinence, and other drug counselors who say that it is safe to reduce alcoholics to one or two drinks a day. Committees keep on lowering the "normal"value of fasting glucose, although no one has demonstrated any benefit in either reduced morbidity or reduced mortality by so doing.

     Now in physics we worked with inanimate objects, which we could construct as being identical and had no free will or volition. In medicine, we work with humans of all ages, sexes, color, heights and weights, with free will and various cultural and religious beliefs as well as various degrees of compliance with advice and/or medications, who may be taking some unknown herbal medicine or vitamin supplement, so the definition of "identical" is quite different.  (And remember that vitamins are a huge variable: they are not under FDA jurisdiction, and the labeled dose may well be incorrect. Several years ago The Medical Letter chemically measured the amount of Vitamin E in gelcaps labeled "Vitamin E 400 Units". They found the amount of vitamin E in caps from different manufacturers varied from zero to 1200 Units.) I will not discuss Hume's famous analysis of the fallacies inherent in the inductive method of reasoning, but I will note that retrospective studies generate inductive "laws" or suggestions, while double-blind studies, assuming the model is correct, are essentially generating deductive solutions. And overall, both types of studies must deal with the question of their statistical accuracy, so I always tell my patients to wait for the second clinical study, which very often contradicts the first. I have discussed the problems  with and limitations of meta-analyses in two previous blogs, so I will not discuss that here, except to reiterate that IMHO the results of any meta-analysis should be used to suggest a future double-blind experiment, and not to suggest a medical treatment.

     I will not discuss the proper application of statistics to clinical studies, since that would make this blog much too long. The most common error that retrospective and case control studies make is to mistake correlation for causation. The studies may show that patients with disease X have diet Y or do activity Z, but very often when we then do a controlled study and take a group of (hopefully) identical patients, and have half of them adopt diet Y or activity Z we find that this intervention does not affect the incidence of disease X. For instance, no forward study has ever demonstrated that a low fat diet reduces the incidence of cancer (which of course has not prevented the American Cancer Society from advocating a low fat diet as a cancer preventative). Similarly, no one has demonstrated that a low salt diet prevents high blood pressure or heart failure or heart attacks in healthy individuals. And we still don't know if the optimum diet is one, two, three, four or more meals a day, and whether or not the largest meal should be eaten in the AM, at noon, or in the PM. And once a recommendation attains the force of law, it is rarely rescinded. For instance many studies have shown that a pregnant woman can safely have one drink a day without adverse effects on the fetus, but the Surgeon General's warning remains on every bottle of alcohol sold in the U.S. We also have been told that reducing cholesterol reduces the rate of heart attacks, but cannot explain why Zetia, which reduced cholesterol, did not appear to provide this benefit.

     One example of the interpretation of treatment or intervention is the history of tuberculosis in the 20th century.  The incidence and death rate from TB started to decrease in the 1920's in the USA, even though we had no antibiotics that worked at that time. Some social reformers claimed that it was due to the reduction in crowding and filth secondary to the elimination of many tenements, but of course this could neither be proved nor disproved with the available data. Dr. Waksman discovered injectable streptomycin, which was the first anti-TB drug that worked. It had the unfortunate side effect of causing irreversable damage to the 8th cranial nerve, leading to severe deafness in the patients so treated. Then INH, or isoniazid was developed in the laboratory. One of the largest groups of patients treated were veterans in the Manhattan VA hospital with TB: 50% received streptomycin and 50% received INH. INH was shown to be superior to streptomycin with the added advantage of not causing deafness. The old-time medicine cabinets had a slot in the rear for the disposal of disposable razor blades. When they tore down the old VA hospital, thousands of INH tablets were found behind the medicine cabinets, so the INH was probably more effective than anyone dreamed.

     The problem of course with all medical studies is that in successive studies the groups are never identical, so that for instance in a double blind study of any treatment of children in the 50's we had one group of children, while the identical study done in the 70's would be done on children who had received fluoride to prevent tooth decay. Therefore there is no reason to expect a priori that successive medical intervention studies should show the same result, or that a study in London should give  the same result as a study in San Francisco. The laws of physics are invariant under a Galilean transformation, but people are not. The same study done in an equatorial country and in a northern European country may disagree for the same reason that children who grow up in the equatorial zones have a significantly  different incidence of multiple sclerosis than do children who grow up in temperate zones (as do females who did and did not have a cat as a pet before the age of 16). For all we know, we should do historical genetic testing on DNA from cheek swabs because patients whose distant ancestors came from Asia Minor may have a different response to disease and treatment than those whose ancestors came from the steppes of Russia. Or perhaps nearsighted people and farsighted people also have different disease coping mechanisms  or immune responses because of linkage on the same chromosome, and no one allows for that.No two people  are identical biologically, genetically, or in their immune defenses or in their propensity to develop diabetes, or heart disease or cancer. We even have the false belief that a teacher can teach a group of 35 first graders as well as she can teach a class of 15.

     Finally the placebo effect which has been shown to be as high as 30% is nowhere accounted for in either the control group or the treated group, and the kappa statistic is an unreliable method for detecting the effect of random coincidence, since the effect can sometimes calculate out as being greater than one, which violates its basic statistical assumption.

     In the words of Ralph Waldo Emerson: "No facts to me are sacred, none are profane. I simply experiment and observe......No truth is so sublime that it may not become trivial tomorrow in the light of new thoughts and facts".