As a sweeping global pandemic, COVID-19 has prompted a swift response to develop a treatment that’s safe and effective. While the pace of learning has only accelerated since the end of 2019, an analysis of clinical trials demonstrates the importance of design when interpreting results.
Researchers at Johns Hopkins University Bloomberg School of Public Health examined 201 clinical trials of treatments for COVID-19. These trials were registered through late March both in the U.S. under ClinicalTrials.gov and internationally by the World Health Organization (WHO).
The analysis, published in the journal BMJ Open, found significant design shortcomings within the trials.
“Our findings shine a brighter light on the ways that trials are being designed, and call into question some designs that may fall fairly short,” said Dr. G. Caleb Alexander, study author and professor of epidemiology and medicine.
Shortcomings and Significance
Researchers concluded that many trials “lacked key features needed to optimize their scientific value,” according to the press release.
- Nearly half of the trials tested groups of fewer than 100 patients, which limits the usefulness of the findings.
- Two-thirds of the trials were “open-label,” meaning both doctors and patients knew who received treatment and who received a placebo. This type of design increases the potential for participants’ and doctors’ unconscious biases to influence their results.
- One-third of the trials failed to identify clear endpoints, such as survival or hospital discharge. Without endpoints, researchers cannot determine whether a treatment is a success or a failure.
- One-fourth of the trials failed to identify control groups from the comparator drug. Control groups are necessary to ensure the treatment works.
The study also reports that the vast majority of trials tested treatments used for other conditions that could potentially be repurposed for COVID-19 more quickly due to existing safety profiles and Food and Drug Administration (FDA) evaluation. While time is of the essence, these “off-label” uses require thorough safety tests and can carry additional risks.
“Many of these studies are likely to yield only preliminary evidence,” said Dr. Hemalkumar Mehta, study author and assistant professor of epidemiology at the Bloomberg School, in a release. “Given the urgency of identifying definitive evidence on potential COVID-19 treatments, this is an instance where we wish we did not have to say ‘further research is needed’ because of basic trial design shortcomings and small trials.”
Ann Collier, professor of medicine at the University of Washington School of Medicine, noted that the push for answers is fantastic on one hand, but makes for disastrous misinterpretation and loss of public trust on the other. Analysis that adds context is worth highlighting.
“With every decision of how to do a study and how to report on it, there are always pluses and minuses. I think it’s important to recognize that that’s the case,” Collier told MedTruth. “ By pointing out the facts, their description of the studies is very valuable.”
Context and continuing trials
Given the scope of the pandemic, study author Alexander isn’t surprised by the significant variance in trial design quality. He likens the registries to open marketplaces in which trials of all levels, from single hospitals to multinational investigations, can register.
He also points to limitations as a basis for avoiding harsh critique. For example, “blind” studies, where neither the doctor nor patient know who has received treatment or a placebo, are preferred, but this type of study requires more intensive resources and complex scientific protocol. But the most useful findings will come from rigorous studies, study author Mehta said.
“As the safety and effectiveness of new treatments are evaluated, it’s vital that we use the best science to do so,” Mehta said in the release. “It’s especially important at this juncture with many lives in the balance.”
Researchers noted that the number of registered clinical trials has risen to more than 2,000 since their snapshot survey, and new trials are registered on a daily basis.
“There are some very well-designed studies that are underway and that have been completed,” Alexander said. “We will continue to learn a great deal over the coming months regarding the safety and effectiveness of potential treatments for COVID-19.”