This report provides the results of a three-part methods project that aimed to first review existing guidance on network meta-analysis, secondly to identify previously published MTCs and summarize their characteristics, and finally, to gather insight from investigators who have conducted network meta-analyses using these methods.

Our review of publicly available guidance documents from various governmental and evidence synthesis groups found that the majority of these documents were typically written in a fashion applicable to network meta-analysis in general, and not specific to any one methodology type. In regards to methods used to conduct meta-analyses of networks of trials containing at least one closed loop, the two approaches typically discussed by guidance included the Bayesian and the Frequentist mixed methods approach initially described by Lumley. Guidance documents stressed that both these approaches have decreased internal validity because they compromise the positive impact of individual study randomization. Common limitations of the Lumley's Frequentist approach discussed by guidance documents included the approaches' inability to synthesize networks of studies lacking at least one closed loop, the fact that the method does not account for correlations that may exist between effect estimates when they are obtained from a single multi-arm study, and a weaknesses in situations where zero cells are common. These limitations can be addressed through special preparations such as using a small increment to address zero cells and adding steps to adjust for correlations between effect estimates. The Bayesian approach was often criticized for requiring specification of noninformative priors, its complexity to understand, and the need to use non-user-friendly software to implement. Guidance noted some similarities between the methods as well. Regardless of the approach discussed, guidance documents stressed the need for consistency of factors and event rates between direct and indirect trials and the importance of assessing for consistency/inconsistency and heterogeneity. The International Society of Pharmacoeconomics and Outcomes Researchers was the only group that attempted to comprehensively address how to conduct, interpret and report a network meta-analysis. Additional guidance on how to conduct, interpret and report a network meta-analysis is needed.

Our systematic review identified 42 unique MTCs that used either Bayesian or Frequentist methods. These MTC were published in 32 different journals, most of which with accompanying supplements. Of the 42 MTCs, the vast majority used Bayesian methods. Investigators could have chosen either Bayesian or Frequentist methods as both can accommodate close loop models. Despite the option, most investigators chose a Bayesian approach. Of the analyses that utilized Bayesian approach, there was a wide distribution of disease states evaluated although cardiology was the most common area. Most analyses evaluated pharmacologic interventions and were funded by industry or a government/foundation source. There was a large variance in printed pages number of the manuscript, although two included MTCs were affiliated reports without page limitation and were likely the contributing factor. The statistical code used in the analysis was rarely made available to the reader, despite the majority of journals allowing publication of a supplement or appendix, although raw outcomes data were more commonly published. A similar number of analyses used vague priors or did not specify whether priors were intended to be vague and few analyses used informative priors. However, it was uncommon for authors to report specific priors used. Most models used a random effects model. Unfortunately, data regarding the evaluation of convergence, heterogeneity, and inconsistency were inconsistently reported and often times not mentioned throughout the publication. From the analyses that reported evaluating these three characteristics, it appears that a broad range of methods are being utilized. We cannot say with certainty though that a lack of reporting means these characteristics were not evaluated. Perhaps with more clear guidance in the future, as to how to conduct and report these types of network meta-analyses, a more consistent approach may be taken. When investigators reported results of their findings, it was common that interventions were rank ordered based on the probability of the intervention being best for a given outcome. Rarely did authors conclude equivalence or non-inferiority of interventions based on network meta-analysis results. The most common types of outcomes evaluated were binary outcomes, measured with relative risks or odds ratios and 95% credible intervals.

As there were very few network meta-analyses identified by our systematic review that used Frequentist methods, summarizing similarities and differences across the analyses is difficult. Only nine analyses used these Frequentist type methods, despite the option of doing so amongst the majority of Bayesian MTCs. Unfortunately, we did not gain any insight as to the decisionmaking and opinions of the investigators of the Frequentist models because of a lack of response to our focus group invitation. All of the respondents had conducted a Bayesian MTC and therefore we could not compare and contrast the viewpoints between investigators who used Bayesian methods versus Frequentist methods.

The group of respondents did not appear to be new to Bayesian MTC methods as all had conducted at least two such analysis and appeared to be involved in a variety of steps in the process. However, it is unlikely the respondents were methodologists since most did not know how the code or prior distributions were chosen. Although we prefaced the questions with a list of terms and definitions for the respondents to use while answering the questions, we assume the respondents did in fact apply those definitions. Another potential limitation to this portion of the project is the one time correspondence with the investigator to obtain opinion. The process was not interactive and therefore a general consensus was not achieved in areas of discrepancy.

On average, the group felt the term “network meta-analysis” is used ambiguously and inconsistently in the medical literature, although did not feel the same about the terms “mixed treatment comparison” or “frequentist network meta-analysis.” In general, there were neutral opinions on average regarding network meta-analyses principles. However, clear majority was seen for the following: disagreement that investigators should consider restricting their search to the minimum number of interventions of interest when conducting a network meta-analysis; agreement that the combination of indirect and direct evidence adds valuable information that is not available from head-to-head comparisons; and agreement that network meta-analysis should provide a graphical depiction of the evidence network. The respondents identified several strengths and limitations of Bayesian MTC. Although most were unique statements, there was a common limitation suggested regarding the user friendliness of software used to run the analyses.

Respondents were asked specifically about their Bayesian MTC which we had identified in part two of this project. The most influential criteria in deciding to use Bayesian MTC, on average, were the method's ability to handle multi-arm studies and collaborator's or respondent's prior expertise and/or experience. The least influential criterion was the requirement to specify priors which are often arbitrary. Most respondents built the code from scratch or adapted the code from a previously published code. Unfortunately we did not gain insight as to how or why prior distributions were chosen rather what the priors chosen were.

Overall, further research is needed to build on this report and develop a set of practical guidelines for conducting MTCs, developed by all relevant stakeholders, including representatives from academia and industry. Such guidelines may also lead to standardized approaches to reporting MTCs. Future efforts should be made to continue to understand the rationale of investigators in their choice of Bayesian versus Frequentist methods to conduct MTCs.