Searching the Literature

We searched for publicly available guidance reports and manuals prepared by regulatory bodies or organizations engaged in evidence synthesis for guidance related to network meta-analyses or MTCs. More specifically, we searched the following Web sites: (1) Agency for Healthcare Research and Quality (AHRQ, www.ahrq.gov); (2) Centre for Reviews and Dissemination (CRD, www.crd.york.ac.uk/crdweb/); (3) Cochrane Collaboration (www.cochrane.org/); (4) National Institute for Health and Clinical Excellence (NICE, www.nice.org.uk/); (5) International Society of Pharmacoeconomics and Outcomes Researchers (ISPOR, www.ispor.org/); (6) Drug Effectiveness Review Program (DERP) of the Oregon Health & Science University (OHSU) Center for Evidence-based Policy (www.ohsu.edu/xd/research/centers-institutes/evidence-based-policy-center/derp/?WT_rank=1/); (7) the Institute of Medicine (IOM, www.iom.edu/) and (8) all other current members of the International Network of Agencies for Health Technology Assessment (INAHTA) (as listed on the INAHTA Web site www.inahta.org/on December 26, 2011). Finally, we supplemented the above with a Google search (www.google.com).

Data Synthesis

Each identified relevant document was read by a single researcher in detail, and key statements were summarized into the following three categories:

• Guidance on “when to conduct” a network meta-analysis/MTC, including definitions of network meta-analysis and MTC, justification for conducting such analyses and assumptions that should be met.
• Guidance on “how to conduct” a network meta-analysis/MTC, including planning and design, analysis framework, statistical modeling, detection and handling of potential bias/inconsistency, assessment of model fit and sensitivity analysis.
• Guidance on “how to report and interpret” a network meta-analysis/MTC, including requirements or suggestions for reporting and displaying results, types of permissible conclusions, limitations of analysis.

Provided synthesis was not intended to be an exhaustive representation of the content of the source documents, but rather a summary. A selection of verbatim quotes from the source documents can be found in Appendix A.

Searching the Literature

A systematic literature search was conducted in Medline (2006 to July 31, 2011), the Centre for Reviews and Dissemination Databases (July 31, 2011) (including the Database of Abstracts and Reviews of Effects, Health Technology Assessment and the National Institute for Health Research Economic Evaluation Database), The Cochrane Library (July 31, 2011), and the American College of Physicians Journal Club (2006 to July 31, 2011). The search strategy in Appendix B was used.

Inclusion and Exclusion Criteria

Two independent investigators assessed reviews for inclusion in a parallel manner based on a priori defined criteria. Systematic reviews that met the following criteria were included: (1) compared the clinical effectiveness or safety of three or more interventions (any treatment, dose, treatment regimen or clinical procedure) based on RCTs; (2) utilized either Bayesian or Frequentist methods to conduct MTC; (3) published in full text; (4) published in the English language; and (5) published between January 1, 2006 and July 31, 2011. Of note, methodological publications that presented MTCs for illustrative purposes and cost-effectiveness analyses were not considered, nor were individual patient data meta-analyses. We included all interventions regardless if pharmacologic, behavioral, or procedural.

Data Extraction

Two reviewers used a standardized tool (Appendix C) to independently extract data; disagreements were resolved through discussion. For each included MTC, all published material including the manuscript, supplements, appendices, or external Web sites which the reader was referred to for additional data were used during data extraction. Therefore, the extraction of data in this project is predicated on the reporting of the information by the authors within these sources. When extracting data, we recorded what the authors reported without ourselves judging whether the methods were appropriate or not. If there was insufficient data from all available sources, we indicated “not reported” for that criterion on data extraction.

First, general characteristics were collected on the journals in which included MTCs were published. Characteristics included journal name, impact factor, allowance of supplements or appendices, and limitations on word, table, and figure counts. Secondly, general characteristics of each MTC were extracted including: (1) the number of authors and if any authors were considered to be methodologists, (2) the number and type of intervention comparisons made; (3) country and year in which the review was conducted; (4) funding source and affiliations; (5) number of printed pages and use of supplement or appendix; (6) the number of trials and patients in the analyses; (7) clinical area (e.g., cardiology, endocrinology, etc.); and (8) the network pattern. For the purposes of this project, we defined a methodologist as an individual with affiliation to a department of statistics, biostatistics, epidemiology, clinical epidemiology, or public health services, as determined by author information and affiliations listed in the publication.¹¹ The country in which a review was conducted was determined by the corresponding author's affiliation. The network pattern was determined by figures presented within the review. If a figure was not available investigators determined the pattern based on text descriptions of included trials.

We also extracted information regarding the methods used to conduct the MTC including (1) methods/models applied (e.g., Bayesian or Frequentist); (2) whether a fixed-effect or random-effects model was applied; (3) description of model parameters (e.g., choices of prior distributions in Bayesian analysis and assumptions in Frequentist analysis); (4) method for assessment of model fit; (5) methods for handling of potential bias, inconsistency and heterogeneity (e.g., qualitative or quantitative); (6) use of covariate adjustment in models; (7) whether the model accommodated multi-arm trials; (8) software utilized (WinBUGS, OpenBUGS, wrappers, R, commercial software such as SAS/STATA/SPSS); and (9) availability of code. Finally we extracted data concerning the reporting of results. This included (1) type of endpoint (e.g., continuous versus binary); (2) effect size (e.g., odds ratio, relative risk, risk difference, weighted mean difference) assessed; (3) measure of variance (e.g., confidence or credible intervals); (4) use of other methods to report results (e.g., probability of treatment being best, claims of equivalence or non-inferiority); and (5) format/presentation of results (e.g., text, tables, figures, such as figure of network of studies, raw data tables).

Data Synthesis

The general characteristics of journals and MTCs were summarized qualitatively. Categorical data is presented using frequencies and continuous data as means with standard deviations (SDs).

Composition of the Focus Group

Up to nine individuals were allowed to participate in this focus group. We randomly identified MTCs identified in part two of this project to invite, via email, either the first or corresponding author to participate in this group. If no response was obtained we sent a reminder email. If we still did not receive a response, we attempted to contact another author on the publication. After contacting two authors unsuccessfully, we selected another publication at random. Upon investigator-expressed interest, a link was sent to the investigator via email which redirected them to the Web-based tool SurveyMonkey©. The investigators were asked to complete questions in regard to the unique MTC which we identified in Part Two of this project (Appendix D).

We intended on participation in this group to be numerically similar between investigators who used Bayesian and Frequentist methods. However, the number of Frequentist MTCs identified in part two of this report was much fewer and author response was suboptimal. In an effort to identify additional models using Frequentist MTCs, we re-ran the original literature search from part two without the publication year limit. Although an additional model was identified the author declined participation in our group and therefore we chose to continue to invite investigators who used Bayesian methods until we met the target number of nine respondents.

Data Synthesis

Responses from members of the focus group were tallied, summarized, and reported in a de-identified format. Categorical data was summarized using frequencies and continuous data as means with SDs.