Equator - Enhancing the Quality and Transparency of Health Research

Even General Motors’ design engineers know that to design a new car they must take into account every part that is essential for the vehicle. Further they must be intimately aware and take into account all national and state regulations that govern automobile design, construction and assembly. They have a check list.

This is important because of the human and economic costs that result if mistakes are made; a part omitted, or one used that does not meet standards generally agreed upon or required by law. At a minimum, having a check list ensures that the design engineers do not arrive at a final design having forgotten to include say a break line or spark plugs or a motor to drive the windshield wipers. Even a defect or missing part discovered on the assembly line or by the first customer leaving the showroom in the rain would result in millions of design and production dollars lost. Engineering a new vehicle is expensive, often a three or four year project costing millions of dollars; not to be wasted by sloppy execution.

When it come to health research the design engineers are called ‘researchers’ and the product being designed is a research protocol that will lead to an answer to an important question relating to human health. The stakes are as high for the research community (or pharmaceutical company sponsor) as in the previous example they are for General Motors. In addition, a flawed design of health research for prospective trials involving human beings has the immediate result that the human participants are taking risks (they may be in the control group and not be treated, or in the experimental group and be subjected to an untested new product) that are from the beginning of the trial futile, unnecessary, and are always counterproductive by their yield of unreliable results.

For example, many randomized clinical trials are so poorly designed that the results can not be interpreted. Most clinical trials are too small to detect any but very large differences between study subjects receiving a new treatment and those receiving a placebo or conventional treatment. There are very few trials, even for common diseases such as heart disease and cancer that contain enough study subjects to be able to say that treatment is better than placebo in prolonging life. <1> And if this were not worrying enough, a single flaw in the study protocols resulted often in substantially biased estimates of treatment effects. <2>

Not only is this an abuse of study subjects time and sometimes their health and of taxpayer or investor money, but the final product, released onto the markets of health care with messages of breakthrough treatments, or claims of health benefits or dangers of certain foods or of sunlight, or of cell phones or of whatever, lead to flawed diagnoses , flawed or dangerous treatments. Their widespread use wastes public and private money on ineffectual or dangerous products. Vioxx - a drug that was no better than standard pain relievers and was accompanied by a non-negligible increase in heart attacks produced $ 2.5 billion in revenue for Merck in 20004 <3>

Research design, like vehicle design, is relatively uncomplicated. We know how to do it. There are generally accepted standards in science as there are in engineering. In 1996, a group of health care researchers designed a check list for the design and reporting of randomized clinical trials called CONSORT. <4> This simple design standard has been adopted as a requirement for publication of the trials by the International Committee of Medical Journal Editors (ICMJE) and subsequently adopted by major medical and health science journals in the world.

In fact there are now generally accepted and proposed standards for 82 different types of research designs, from randomized clinical trials of new drugs, to case-control studies to determine causes of cancer, to molecular genomic analyses to investigate new links between disease and genes, and so on. And the list is growing. See www.equator-network.org
Yet many of these research designs and study protocols are not used; by researchers; by the ethics review panels to ensure that the research is ethical (how can it be ethical if it is of flawed design?); by sponsors who are funding the research; and at the end of it all by the health and medical journals that are publishing the results.

We can do better. Equator, under the leadership of Doug Altman at the University of Oxford has set about to create a global resource for health researchers by identifying guidelines for research design that have already been developed, by providing resources to scientists attempting to establish design standards for their particular types of research (already more than 80 such guidelines exist www.equator.network.org ), by developing support for ongoing evaluations of research design guidelines (because design standards in research, as in industry, change as technology changes), by encouraging journal editors to implement publishing standards for research design and use them in peer and editorial review and by publishing checklists so that readers can be sure that standards were followed and the research is valid and by providing a central resource of guidelines for use by researchers, funders, ethics committees, peer reviewers, editors, physicians and health care personnel and ultimately by patients and their families.

We must do better. Fatally flawed designs in health care research are much more damaging to the public than fatally flawed designs of automobiles, for they are exceedingly more difficult to detect. Thus flawed results are much more likely to be used by patients, inappropriately and often dangerously, for a much longer period of time than it would take for a automobile designed without breaks to be detected.

1. Yusuf S, Collins R, Peto R. Why do we need some large, simple randomized trials? Statistics Medicine 2006;3(4):409-20

2. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality estimates of treatment effects in controlled trials. JAMA, 1995;273:408-12

3. Barry Meier, New York Times, December 19, 20044.

4. Moher D, Schulz KF, Altman DG. The CONSORT statement: Revised recommendations for improving the quality of parallel-group randomised trials. Lancet 2001:357:1191-4