The Cochrane Risk of Bias Tool for Non-Randomized Studies of Interventions shows potential value in systematic reviews in an assessment published this week in PLOS Medicine. The study, conducted by researchers at Dalla Lana School of Public Health at the University of Toronto, Ontario, Canada demonstrated that exclusion of studies with higher Risk of Bias (RoB) assessed by a new instrument developed by the international Cochrane Collaboration changed estimates of clinically pertinent outcomes in two published systematic reviews of cardiovascular risks associated with use of two popular classes of drugs.
Inclusion of studies at high RoB may lead to erroneous conclusions about the benefits and harms of medical interventions. Researchers undertaking systematic review and meta-analysis (SRMA) of randomized clinical trials usually measure RoB of the component studies using an established tool, but there is no widely-accepted measure of RoB for non-randomized studies. As a test of performance, the Toronto-based researchers applied the new Cochrane RoB instrument, called ACROBAT-NRSI, to 37 papers included in two widely cited systematic reviews of the use of thiazolidinediones ("glitazones" used to treat diabetes) and cyclooxygenase-2 (COX-2) inhibitors (used to treat inflammation), and risk of major cardiovascular events (myocardial infarction and heart failure) and death. Assessment took roughly 2.5 hours per study, and inter-rater agreement was moderate to substantial (overall Kappa 0.45 to 1.0). Of the 37 studies originally included in the meta-analyses, only eight had low RoB scored by ACROBAT-NRSI. Confining meta-analysis to studies with low RoB shifted the pooled odds ratio for myocardial infarction for rosiglitazone versus pioglitazone from 1.14 [95% CI 1.07-1.24] to 1.06 [95% CI 0.99-1.13]). Estimates of pooled relative risks (RR) of cardiovascular events with COX-2 inhibitors compared with no nonsteroidal anti-inflammatory drug changed little when analyses were confined to studies with low or moderate RoB, with the exception of an increased RR associated with ibuprofen (from 1.07 (95% CI 0.97-1.18) to 1.14 (95% CI 1.03-1.26)).
Because the study was conducted by a specific two-person team and considered only two systematic reviews, the findings may not be generalizable and further testing of the new Cochrane instrument is needed. Nonetheless, the findings indicate that at least two published meta-analyses included studies with high RoB. The authors state, "[e]xclusion of studies deemed to have unacceptably high RoB may impact the findings of pooled estimates of intervention effects, altering both the statistical and clinical significance of the results."
###
Research Article
Funding:
The authors received no specific funding for this work.
Competing Interests:
DH was a member of one of the working groups that contributed to the development of ACROBAT-NRSI. DH is co-author of one of the systematic reviews included in the study. No other members of the Cochrane Collaboration or ACROBAT-NRSI working groups were involved in the planning analysis, or interpretation of this study.
Citation:
Bilandzic A, Fitzpatrick T, Rosella L, Henry D (2016) Risk of Bias in Systematic Reviews of Non-Randomized Studies of Adverse Cardiovascular Effects of Thiazolidinediones and Cyclooxygenase-2 Inhibitors: Application of a New Cochrane Risk of Bias Tool. PLoS Med 13(4): e1001987. doi:10.1371/journal.pmed.1001987
Author Affiliations:
Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
Ontario Strategy for Patient-Oriented Research Support Unit, Toronto, Ontario, Canada
Public Health Ontario, Toronto, Ontario, Canada
Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada
Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
IN YOUR COVERAGE PLEASE USE THIS URL TO PROVIDE ACCESS TO THE FREELY AVAILABLE PAPER: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001987
Contact:
David Henry
CEO
Institute for Clinical Evaluative Sciences
Office of CEO
2075 Bayview Avenue
Toronto, Ontario
CANADA
14163716947
david.henry@utoronto.ca
Journal
PLoS Medicine