New Initiatives to Improve the Rigor and Reproducibility of Articles Published in Circulation Research
The rampant irreproducibility of scientific articles is arguably the most serious problem that bedevils biomedical research.1–12 It is a veritable plague that undermines the credibility of published studies, hinders clinical translation of basic work, and impedes the progress of medicine. Disquietingly, the problem seems to be getting worse rather than better.10 Although the causes of irreproducibility are multifarious, inadequate rigor is probably the most important2,11; thus, it will be impossible to augment reproducibility without augmenting methodological rigor. It is not the purpose of this editorial to revisit the nature, origins, mechanisms, and consequences of irreproducibility, all of which have been discussed innumerable times in recent years, both in the literature and in ad hoc workshops.1–12 There has been enough discussion; now it is time for action.
Much of this action must come from editors. Publication of methodologically flawed or suboptimal research can be limited by promulgating, and diligently applying, higher standards during editorial evaluation of submitted work. The editors of Circulation Research believe that the journal has a responsibility to implement initiatives that promote more rigorous and, therefore, more reproducible scientific work. It is for this reason that Circulation Research has published several reviews, editorials, Viewpoints, and News & Views articles focused on this issue.1,2,7–12 It is for this reason that the journal has joined other leading journals to promote reproducibility of biomedical research4 by endorsing the National Institutes of Health principles and guidelines for reporting preclinical research,6 and it is for this very reason that we have recently chosen to publish a study showing that only a minority of preclinical animal experiments reported in leading cardiovascular journals (including Circulation Research) adheres to basic criteria necessary to assure rigor: for example, it was found that during a 10-year span, blinding was reported in 33% of the articles examined, randomization in 22%, and a priori sample size calculation in 2%.7 The most troubling aspect of this report is that, with the exception of stroke research, there has been no improvement in the level of rigor in the past decade.7
The purpose of this editorial is to announce new initiatives aimed at enhancing the rigor, transparency, and reproducibility—and, thus, the overall robustness—of articles published in our journal. Three distinct initiatives will be implemented: checklists for rigor, measures to improve transparency, and a policy on refutations.
Checklists for Rigor
As of October 1, 2017, Circulation Research will require that authors fill out checklists to document the methodological rigor of preclinical work submitted for publication. Different checklists will be used for studies in animals and for studies in vitro because the challenges are different in the 2 settings. Although methodological weaknesses plague all domains of research, they seem to be particularly conspicuous in animal studies.2,11,13 Accordingly, major emphasis in the checklists will be given to the methodology and reporting of investigations in animals.
Checklists for Studies in Animals
Authors will be asked to complete 2 checklists: a short one on initial submission and a longer one if and when the article is resubmitted in revised form. These documents are modifications of similar documents that were recently implemented by Stroke3 and have resulted in improved adherence to standards of rigor.7,14 The short checklist (Table 1) is relatively simple and covers the basic methodological aspects that are considered essential to the validity of the work: description of animals used, use of randomization, use of blinding, specification of inclusion and exclusion criteria and whether such criteria were established a priori, reporting of animals excluded, and description of statistical methods. The long checklist (Table 2) is much more detailed, providing information that enables in-depth evaluation of the level of rigor (and, therefore, robustness) of the study.
The short checklist will be used by reviewers and editors to evaluate the article but will not be published. The long checklist will be published as an online supplement along with the article; its purpose is to assist not only reviewers and editors, but also readers, to fully evaluate the extent to which the work adheres to standards of rigor. By examining the long checklist, readers will be able to assess for themselves the strengths and weaknesses of the study and how robust the data and conclusions are. The long checklist is accompanied by an explanatory document (Table 3), modeled after that used by Stroke,3,14 which clarifies the standards of rigor outlined in the checklist and assists the authors in reporting them. We have elected to use 2 checklists because most articles submitted to Circulation Research are rejected at first submission; thus, requiring detailed information for every article that is submitted would be unnecessarily burdensome for the authors.
Checklist for Studies In Vitro
For molecular and cellular studies performed in vitro, authors will be required to fill out only one checklist if and when the article is resubmitted in a revised form. This document (checklist for in vitro studies; Table 4) will be provided as a downloadable PDF file on the journal webpage and will cover the fundamental aspects that determine the rigor and robustness of the work, including technical replicates, characteristics of the antibodies and cell lines used, data processing and conversions, presentation of Western blots and immunofluorescence images, etc. This checklist will not be published, but authors will be asked to include the corresponding information in the text or in an online supplement. As in the case of the animal studies, the checklist will be accompanied by an explanatory document (Table 5), modeled after that used by Stroke,3,14 which clarifies the standards of rigor outlined in the checklist and assists the authors in reporting them.
Although the new checklists clearly enunciate the desired standards of rigor, it is important to stress that failure to comply with some of the criteria outlined in these lists will not automatically or necessarily lead to rejection. Each study is different, and the weight that rigor has in the editorial decision must be individualized for each article. In the case of animal work, rigor is particularly important in validation studies—studies designed to verify the validity of a hypothesis in vivo (eg, whether a type of cell therapy, previously suggested to alleviate left ventricular dysfunction in exploratory studies, does indeed have such an effect). In contrast, some of the standards outlined in Tables 1 and 2 may not apply to exploratory, hypothesis-generating, or mechanistic animal studies (ie, studies where the primary goal is to explore the effect of an intervention in vivo for the first time, to obtain initial evidence in support of a hypothesis, or to elucidate a molecular or cellular mechanism). There are also gray zones. Many preclinical studies contain a combination of in vitro and in vivo data; the importance of the in vivo component varies greatly, and in some cases, these data may be peripheral to the main goal of the study. Thus, decisions on acceptance or rejection will continue to be made on a case-by-case basis depending on the specific content and features of each submitted article. If adherence to particular standards outlined in Tables 1, 2, and 4 was not appropriate or meaningful, we encourage the authors to address these issues in the text and to describe why this was the case.
Measures to Improve Transparency
In addition to the checklists, and in keeping with the goal of enhancing transparency,6 we ask authors to use online supplements to provide detailed experimental protocols and communicate any secondary results not reported in the main article, both for in vivo and in vitro studies. As stated previously on these pages,2,15 we expect all articles to describe the methods in sufficient detail to enable others to reproduce the work; this is usually achieved by publishing a detailed Methods section online. As recommended by the National Institutes of Health guidelines, authors should use online data supplements to describe “any similar experimental results that were omitted from the reporting for any reason, especially if the results do not support the main findings of the study” and “any outcomes or conditions that were measured or used and are not reported in the Results section.”6
Policy on Refutations
Circulation Research also adheres to the National Institutes of Health recommendation that if an article is published in our journal, the journal “assumes responsibility to consider publication of refutations of that article, according to usual standards of quality.”6 In other words, an article that contradicts previous findings will not be penalized for not being the first; if the article is of quality consistent with the standards of Circulation Research, it will receive the same priority as if it were the first.
The CAESAR Experience
An excellent example of the importance of methodological rigor can be found in the field of cardioprotection (limitation of myocardial infarct size), where thousands of therapies have been claimed to reduce infarct size during the past 5 decades, yet few have been reproducibly effective and (with the exception of early reperfusion) none has been translated into a clinical therapy.11 The major reason for this colossal fiasco has been insufficient methodological rigor and transparency.11 This problem was addressed in a workshop organized by the NHLBI in June 2003, which resulted in the implementation of a multicenter consortium (CAESAR) that conducted studies of putative cardioprotective therapies with a level of rigor comparable with that of randomized clinical trials; for example, with the use of randomization, blinding, a priori sample size and power calculation, etc.16,17 When these methods were implemented, the CAESAR investigators found that different laboratories using the same protocols were able to obtain similar results.16 It was also found that several therapies previously claimed to reduce infarct size were reproducibly ineffective in 3 different species, suggesting that an increased level of rigor eliminated false-positive results.18,19
Impact of the New Initiatives
The new checklists, transparency criteria, and refutations policy introduced herein are the most important changes in the editorial policies of Circulation Research in many decades. Their consequences will be profound and far reaching. It will take several years for their full impact to be realized, but this impact will be, I think, positive.
By promoting methodological rigor and transparency, these new guidelines will further enhance the already high quality of the work published in our journal, thereby augmenting its reproducibility and scientific utility. I expect that this will result in fewer controversies; therefore, less time, money, and effort will be consumed to resolve apparent conflicts in the literature. As fewer biased or inaccurate conclusions will be published, the number of positive studies will decrease dramatically. This, in turn, will enable investigators to focus on those ideas or hypotheses that are true (or likely to be true) rather than try to reproduce those that are not, which will speed up progress. Translation of preclinical results to clinical therapies will be facilitated because investigators will concentrate their efforts on those (few) interventions that are reproducibly effective in animal models and, thus, are more likely to be effective in patients. Importantly, there will be a moratorium, or possibly even a reversal, of the erosion of the credibility of research caused by irreproducible results.
Change is often difficult, and it seems likely that the guidelines enunciated in this article will meet with some opposition. For example, it will be argued that they constitute yet another burden on authors in our overregulated world. Although I personally agree that our society is overregulated and that excessive paperwork and regulations are oppressive and can stifle creativity, I am more concerned about publishing conclusions that may be biased or inaccurate. The damage caused by the introduction of erroneous ideas in the literature greatly outweighs the extra effort necessary to comply with the principles that underlie the checklists.
An unintended consequence may be that these new checklists will discourage authors from submitting articles to Circulation Research. It is hoped that this will not happen, but it is a risk that must be taken. The alternative (publishing conclusions that may be inaccurate) is much worse. We have made a deliberate effort to minimize the burden on authors. For example, the initial checklist for animal studies (Table 1) is quite simple. Only when an article is seriously considered for publication is the long, more burdensome checklist (Table 2) required.
Time-Table for Implementation
To give authors time to reformat their papers in accordance with the new guidelines, the checklists will not be required until October 1, 2017.
What will happen then? After the launch of the initiatives outlined herein, we do not foresee a sudden spike in the level of rigor of the papers that we publish, but rather, a gradual increase. Manuscripts submitted to Circulation Research cannot be expected to conform with the new guidelines overnight. Obviously, studies that have already been completed cannot be modified, and changing the design and conduct of ongoing or planned experiments will take time. In many cases, 2-3 years elapse from the time when a study is first conceived to its submission for publication to our journal. Accordingly, in the short term we do not expect that, to be acceptable for publication, papers will have to meet all or almost all of the criteria specified in the checklists. We do hope, however, that these criteria will motivate investigators to change their practice in order to achieve a higher level of rigor, such that, in time, the rigor criteria encapsulated in these checklists (or at least most of them) will be met.
If this happens, it will be a slow process; it would be unrealistic to assume otherwise. The important thing, in our opinion, is to start this process. While our ultimate goal is to augment rigor and reproducibility in those papers that are published in Circulation Research, our more immediate objective is to promote a shift in experimental procedures toward practices that are conducive to a higher level of scientific validity. We will not be surprised if, over the next several months, we will continue to receive and publish manuscripts that do not meet all the criteria listed in the checklists. We will, however, be disappointed if, a few years from now, the level of rigor of studies published in Circulation Research has not improved.
Enhancing rigor, important though it may be, is not the only motivation for our actions. Even if a study does not meet the rigor criteria listed herein, the checklists will be helpful to the readers because they will allow them to see at a glance how much a study deviated from the ideal level of rigor and, therefore, how robust its conclusions are. An additional immediate benefit of the checklists is that they will spur authors to better report the details of their studies, which will enhance reproducibility irrespective of the level of rigor. Even if the experimental procedures do not change, a more complete, organized, and detailed description of the experiment will make it easier to either reproduce or refute it.
The editors of Circulation Research are committed to upholding the highest standards of rigor and responsibility in both the conduct and the reporting of preclinical research. We hope that the new checklists and transparency criteria will clearly communicate the desired standards of rigor to authors and investigators while facilitating the journal’s overall mission of promoting robust, rigorous, and reproducible scientific inquiry and experimentation.
Although adding more requirements for publication of articles in Circulation Research is not desirable, in the final analysis, the editors do not have a choice. Publication of studies that are less than rigorous (and, thus, inconclusive or possibly even misleading) does not benefit anyone, nor does dissemination of concepts that may be biased or inaccurate. As Ramirez et al7 have pointed out, there has been no progress in rigor and transparency during the past decade, even after the promulgation of the National Institutes of Health guidelines6 and the publication of many high-profile articles on the topic. The status quo is not acceptable. If we, as editors and gate keepers of scientific reporting, do not take action to remedy this problem, who will?
In conclusion, I think the initiatives outlined herein will enhance the quality of research published in Circulation Research and facilitate progress in cardiovascular biology. At present, most animal studies do not meet all of the criteria of rigor listed in Tables 1 and 2. In time, however, I think that the principles that underlie these guidelines will permeate the scientific community and become ingrained in the design of cardiovascular research, to the point that complying with the criteria articulated in the checklists will come natural. It will take time, but I think it will happen. When that happens, there will be less hype, less glamor, and probably less media headlines, but there will also be more truth. And that—the pursuit of the truth—is the core mission of any scientific journal.
The opinions expressed in this article are not necessarily those of the editors or of the American Heart Association.
- © 2017 American Heart Association, Inc.
- Bolli R
- Bolli R
- Vahidy F,
- Schäbitz WR,
- Fisher M,
- Aronowski J
- 4.↵Journals unite for reproducibility. Nature. 2014;515–517. doi: 10.1038/515007a.
- 6.↵NIH. NIH Principles and Guidelines for Reporting Preclinical Research. https://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research Accessed June 1, 2017.
- Ramirez FD,
- Motazedian P,
- Jung RG,
- Di Santo P,
- MacDonald ZD,
- Moreland R,
- Simard T,
- Clancy AA,
- Russo JJ,
- Welch VA,
- Wells GA,
- Hibbert B
- Begley CG,
- Ioannidis JP
- Demer LL,
- Tintut Y,
- Nguyen KL,
- Hsiai T,
- Lee JT
- Williams R
- Bolli R,
- Becker L,
- Gross G,
- Mentzer R Jr.,
- Balshaw D,
- Lathrop DA
- Jones SP
- Minnerup J,
- Zentsch V,
- Schmidt A,
- Fisher M,
- Schäbitz WR
- Bolli R
- Jones SP,
- Tang XL,
- Guo Y,
- et al
- Kukreja R,
- Tang XL,
- Lefer D,
- et al
- Lefer D,
- Jones S,
- Steenbergen C,
- et al