Definitions/Terminology

“Terminology for indirect treatment comparisons, mixed treatment comparisons, and network meta-analysis varies in the literature.”^2,3

“The results of direct comparisons can be combined with those of indirect comparisons by using a mixed approach, known as a mixed treatment comparison (MTC).”¹⁸

“Methods are available for analysing, simultaneously, three or more different interventions in one meta-analysis. These are usually referred to as ‘multiple-treatments meta-analysis’ (‘MTM’), ‘network meta-analysis’, or ‘mixed treatment comparisons’ (‘MTC’) meta-analysis.”¹⁶

“Several [proposed global estimation methods] have been proposed in the literature to represent networks. Several statistical methods for estimating the parameters for these models (particularly those characterising treatment comparisons) have also been proposed. This diversity has resulted in the following classification:

• Estimation using Bayesian methods: Bayesian network meta analysis
  • Lu and Ades model
  • Model of Caldwell et al.
• Estimation using a mixed linear model
  • Lumley's network meta-analysis”¹⁸

“Also called mixed treatments comparison or multiple treatments comparison meta-analysis, network metaanalysis expands the scope of a conventional pair-wise meta-analysis by analyzing simultaneously both direct comparisons of interventions within randomized controlled trials (RCTs) and indirect comparisons across trials based on a common comparator (e.g., placebo or some standard treatment).”⁹

“Indirect and mixed treatment comparisons (MTC), also known as network meta-analysis, represent a recent development in evidence synthesis, particularly in decisionmaking contexts. Rather than pooling information on trials comparing treatments A and B, network meta-analysis combines data from randomised comparisons, A vs. B, A vs. C, A vs. D, B vs. D, and so on, to deliver an internally consistent set of estimates while respecting the randomisation in the evidence.”¹³

“Network analysis will be used to describe a single synthesised analysis in which more than one common reference is used to indirectly compare the proposed drug and its main comparator.”¹¹

“Multiple-treatments meta-analysis (MTM) is an extension to indirect comparisons that allows the combination of direct with indirect comparisons, and also the simultaneous analysis of the comparative effects of many interventions”¹⁶

“Mixed treatment comparisons, a special case of network meta-analysis, combine direct evidence and indirect evidence for particular pairwise comparisons, thereby synthesizing a greater share of the available evidence than traditional meta-analysis.”^2,3

“Mixed treatment comparisons (MTC), or network meta-analyses, are used to analyse studies with multiple intervention groups and to synthesise evidence across a series of studies in which different interventions were compared…They build a network of evidence that includes both direct evidence from head to head studies and indirect comparisons whereby interventions that have not been compared directly are linked through common comparators.”⁶

According to the HIQA, a multiple treatment comparison combines direct and indirect evidence to compare a technology to two or more other treatments; a network meta-analysis is appropriate for analysing a combination of direct and indirect evidence where there is at least one closed loop of evidence connecting the two technologies of interest, and a Bayesian mixed treatment comparison is appropriate for comparing multiple treatments using both direct and indirect evidence.”¹

“Often only one direct comparison trial is available. Quite often this trial has been designed with a lack of power. In other cases, the comparator may have been used in ways which are debatable. In such a situation a mixed approach, called a mixed treatment comparison11, in which the results of direct comparisons are compared with those of indirect comparisons, is very useful as it removes or confirms any reservations that one might have about direct comparison trials.”¹⁸

Justification

“In many clinical fields, competing treatments are assessed against placebo and direct comparisons are rare. Indirect comparisons can make it possible to estimate the relative efficacy and/or safety of therapies in relation to each other before any direct comparison trials are available.”¹⁸

“In the absence of randomized, controlled trials involving a direct comparison of all treatments of interest, indirect treatment comparisons and network meta-analysis provide useful evidence for judiciously selecting the best choice(s) of treatment.”^2,3

“In the absence of sufficient direct head-to-head evidence and presence of sufficient indirect evidence, indirect comparisons can be considered as an additional analytic tool.”⁴

“Direct comparisons are preferred over indirect comparisons; similarly, effectiveness and long-term or serious adverse event outcomes are preferred to efficacy and short-term tolerability outcomes.”⁵

In some cases, the choice of the comparator will be difficult due to, for instance, changes in prescription behaviour and therapeutic insights over time. The comparator defined at the time of the clinical trials may no longer be the relevant comparator at the time of the pharmacoeconomic evaluation. In this case, indirect comparisons and/or modelling may be required.¹⁹

Indirect comparisons are second best solutions and are only accepted if no single trial of appropriate quality or relevance to the Belgian target population has been performed and under specific conditions regarding the analyses. ¹⁹

If there are no clinical studies for a direct comparison with the pharmaceutical being assessed, or if these do not provide sufficient information about the additional benefit, indirect comparisons can be made in the dossier.²⁰

“Where relevant direct randomised trials (as defined in Part II, Subsection B.2) comparing the proposed drug directly with the main comparator are available, their analysis and presentation are preferred as the basis of the clinical evaluation (see Part II, Section B). However, in the absence of any such direct randomised trials, the second step in the hierarchy is to determine whether it is possible to present an indirect comparison based on two or more sets of randomised trials involving one or more common reference.”¹⁰

”In situations when both direct and indirect comparisons are available in a review, then unless there are design flaws in the head-to-head trials, the two approaches should be considered separately and the direct comparisons should take precedence as a basis for forming conclusions.”¹⁶

“Although it is often argued that indirect comparisons are needed when direct comparisons are not available, it is important to realize that both direct and indirect evidence contributes to the total body of evidence. The results from indirect evidence combined with the direct evidence may strengthen the assessment between treatments directly evaluated. Even when the results of the direct evidence are conclusive, combining them with the results of indirect estimates in a mixed treatment comparison (MTC) may yield a more refined and precise estimate of the interventions directly compared and broaden inference to the population sampled because it links and maximizes existing information within the network of treatment comparisons”^2,3

“Data from head-to-head RCTs should be presented in the reference-case analysis, if available. When head-to-head RCTs exist, evidence from mixed treatment comparison analyses may be presented if it is considered to add information that is not available from the head-to-head comparison. This mixed treatment comparison must be fully described and presented as additional to the reference-case analysis (a ‘mixed treatment comparison’ includes trials that compare the interventions head-to-head and indirectly). When multiple technologies are being appraised that have not been compared within a single RCT, data from a series of pairwise head-to-head RCTs should be presented. Consideration should also be given to presenting a combined analysis using a mixed treatment comparison framework if it is considered to add information that is not available from the head-to-head comparison. If data from head-to-head RCTs are not available, indirect treatment comparison methods should be used (an ‘indirect comparison’ is a synthesis of data from a network of trials). The principles of good practice for standard meta-analyses should also be followed in mixed and indirect treatment comparisons.”

“Pursuit of qualitative or quantitative indirect comparison is never required and decisions to do so must depend on consideration of clinical, methodological, and statistical heterogeneity levels across the individual studies.”⁵

“CRGs should be encouraged to identify existing IRs that compare multiple interventions and consider the feasibility of indirect comparisons and MTM.”¹⁷

“The large majority of [intervention reviews] that involve many interventions present meta-analyses of a series of pair-wise comparisons without a specific plan to integrate the various pieces of evidence. Statistical synthesis using MTM could be performed in many cases, provided that the assumptions of this approach are fulfilled.”¹⁷

Flow chart (Figure A-1) to select proper meta-analysis when comparing interventions, from the Health Information and Quality Authority.

Figure A-1

Selecting appropriate meta-analysis when comparing interventions. Adapted from: Health Information and Quality Authority. Guidelines for Evaluating the Clinical Effectiveness of Health Technologies in Ireland. Dublin: Health Information and Quality Authority; (more...)

Assumptions

“Many assumptions behind network meta-analysis methods appear to be similar to those made in standard pair-wise meta-analysis.”⁹

“The validity of an indirect comparison relies on the different subgroups of trials being similar, on average, in all other factors that may affect outcome.”¹⁶

“Indirect comparisons are often performed on the assumption of a constant relative treatment effect across varying baseline risks (with ‘adjustment’ for the event rate in the common reference group assumed to control for differences in baseline risk). This assumption however is also usually unverifiable unless there are large numbers of trials across the indirect comparison. It also inadequately incorporates all aspects that affect the exchangeability assumption.”¹¹

“For network meta-analysis, covariates that act as relative treatment effect modifiers must be similar across trials (or adjusted for using meta-regression). And, when it combines indirect evidence with direct evidence, network meta-analysis adds the assumption of consistency: The indirect evidence must be consistent with the direct evidence.”^2,3

“The major assumption of indirect and mixed treatment comparisons is that the direct and the indirect evidence are consistent. That is, the treatment effect dBC estimated by the BC trials, would be the same as the treatment effect estimated by the AC and AB trials if they had included B and C arms. If this is not the case the evidence is inconsistent. Factors such as (relative) treatment effects varying with disease severity may cause inconsistency (e.g. if the BC trials are done in patient populations with higher/lower baseline risks than the AB and AC trials and the treatments interact with baseline risk, the evidence will be inconsistent).”¹⁵

“MTC analysis requires a connected network; that is, for each treatment, there is a chain of pair-wise comparisons that connects it to every other treatment.”¹⁵

Planning/Design

“Objectives of network meta-analysis may include considering all relevant evidence, answering research questions in the absence of direct evidence, improving the precision of estimates by combining direct and indirect evidence, ranking treatments, and assessing the impact of certain components of the evidence network.^2,3”

“When a new [intervention reviews] seeks to compare multiple interventions (i.e. to determine a preferential ordering of three or more competing interventions for a particular outcome), this should be made explicit in the protocol, and appropriate methods should be planned and implemented.”¹⁷

“The principles of good practice for systematic reviews and meta-analyses should be carefully followed when conducting mixed and indirect treatment comparisons.”¹²

“To minimize error and ensure validity of findings from meta-analyses, the systematic review, whether it involves a standard, pair-wise meta-analysis or a network meta-analysis, must be designed rigorously and conducted carefully.⁹

“The literature search for a network meta-analysis builds the network, applying the same basic standards as for a meta-analysis leading to a direct comparison” [ispor]

“It may be difficult to identify all relevant comparators for the treatments of interest, and any search involves costs and tradeoffs. It may be efficient to proceed in stages, using one of the strategies developed by Hawkins et al.”^2,3

“The more interventions that are included in a MTM, the greater the potential gain in precision and the greater the ability to establish whether various sources of evidence ‘agree’ with each other. Therefore, it may sometimes be useful to include interventions that are not current candidates for clinical practice, such as placebo or no treatment, or interventions that are no longer recommended or available (‘legacy treatments’).”¹⁷

“Different specification of eligibility criteria may result in differences in the structure or extent of a network, leading to discrepant findings for network meta-analyses on the same topic. This is because different combinations of direct and indirect evidence, some independent and some overlapping, contribute to the comparisons and estimates of treatment effect. Certain interventions, for example, interventions that are no longer in use, or placebos, may not be of primary interest but may be included in the network meta-analysis if they provide information concerning the interventions of interest through indirect comparisons.”¹⁵

“To ensure that all relevant studies are identified, the network meta-analyst could search de novo for all relevant studies, but this would waste valuable resources if good systematic reviews with comprehensive searches already exist. To conserve valuable resources, one might consider using data identified through existing high quality systematic reviews of relevant pair-wise treatment comparisons provided the searches in the existing reviews are up-to-date.”¹⁵

“After demonstrating that no relevant direct randomised trials exist, broaden the literature search criteria to identify all randomised trials relevant for an indirect comparison of the proposed drug and the main comparator.” [PBAC]

“The network can be restricted to include the minimum number of comparisons required to enable an indirect comparison between the technologies of interest. Alternatively it can be expanded to include as many relevant comparators as possible.”¹

“Extending mixed treatment comparisons networks to include trial comparisons not of direct interest can reduce uncertainty in the comparisons of interest.”¹⁵

Analysis Framework

Indirect comparisons should be based on “adjusted” methods, which use the common control arm of RCTS as a way to “standardize” the comparison. Different methods of increasing complexity are available.¹⁹

“Network meta-analysis can be performed within a Frequentist or Bayesian framework.”^2,3

“The MTC method can be used to obtain measures of effect for each of the indicated patterns. The network meta-analysis method proposed by Lumley can compare treatments in a network geometry that contains at least one closed loop.”^7,8

“For syntheses where all trials are two-arm, there is no reason why frequentist methods should not be used, as long as statistically sound estimators are used and appropriate steps are taken to propagate parameter uncertainty, including correlations, through the decision model.”

“Various approaches for indirect treatment comparisons have been reviewed. The mixed treatment comparison approaches by Lu and Ades are elegant, but require information that may not be available. The challenge of Lumley's network meta-analysis is that it needs a data-based assessment of trial consistency; therefore, it requires information from a large number of different treatment comparisons.”^7,8

“The common Generalised Linear Model (GLM) framework can, of course, be applied in either frequentist or Bayesian contexts. However, Bayesian Markov Chain Monte Carlo (MCMC) has for many years been the mainstay of “comprehensive decision analysis”, because simulation from a Bayesian posterior distribution supplies both statistical estimation and inference, and a platform for probabilistic decisionmaking under uncertainty”¹³

“A major advantage of the Bayesian approach is that the method naturally leads to a decision framework that supports decisionmaking”^2,3

“Bayesian methods based on an evidence network because of the great flexibility of the model (allowing detailed and flexible modeling of data, which can be adjusted for particular cases), estimation of inconsistency, and ability to take account of multiarm trials.”¹⁸

“A particular advantage of using a Bayesian framework is that all interventions in the analysis can be ranked, using probabilistic, rather than crude, methods.”¹⁶

“For a network meta-analysis, a specific advantage is that the posterior probability distribution allows calculating the probability of which of the competing interventions is best and other probability statements. This aspect of a Bayesian analysis is providing information that is directly relevant to health-care decisionmakers (e.g., policymakers and health-care professionals/clinicians). Other advantages of a Bayesian meta-analysis include the straightforward way to make predictions and the possibility to incorporate different sources of uncertainty.”^2,3

“Because Binomial and Poisson likelihoods with zero cells are allowed, special precautions do not usually need to be taken in the case of the occasional trial with a zero cell count. This is a major strength of the Bayesian MCMC approach, because some popular Frequentist approaches for log odds ratios or log relative risks have to add an arbitrary constant, usually 0.5, to cells in order to obtain non-infinite estimates of treatment effects and non-infinite variance, but in so doing they generate biased estimates of effect size.”¹³

Statistical Modeling

“Results from the naïve approach, i.e. comparing simply the treatment arm of the RCTs as if they were one single trial, are completely untrustworthy.”¹⁹

“When evidence is combined using indirect or mixed treatment comparison frameworks, trial randomisation must be preserved. A comparison of the results from single treatment arms from different randomised trials is not acceptable unless the data are treated as observational and appropriate steps taken to adjust for possible bias and increased uncertainty.”¹²

“Extending network meta-analysis models with treatment-by-covariate interactions attempts to explain heterogeneity in relative treatment effects and estimates relative treatment effects for different levels of the covariate… Unfortunately, the number of studies in a network is often limited, and in such cases, adjustment by incorporating study-level covariates with meta-regression models may sometimes be questionable. In addition, aggregate-level covariate adjustment might produce ecological bias, limiting the interpretation of estimated results for subgroups.”^2,3

“If confounders are present in an indirect comparison, it is only possible to adjust for them via meta-regression. However, this would be an unusual situation because at least 10 trials per adjustment variable are required in order to achieve stability in the meta-regression results”¹¹

“Network meta-analysis can be performed with fixed- or random-effects models…If there is heterogeneity, however—variation in true (or underlying) relative treatment effects for a particular pairwise comparison—random-effects models must be used. A random-effects approach typically assumes that true relative effects across studies are considered exchangeable (i.e., the prior position of expecting underlying effects to be similar but not identical) and can be described as a sample from a normal distribution whose mean is the pooled relative effect and whose SD reflects the heterogeneity.”^2,3

“Because the standard error obtained from a fixed effect analysis will be too small if there is heterogeneity between trials (beyond random variation), and recognising that there may be additional heterogeneity in an indirect comparison compared to a direct comparison, the Working Group supports the conclusion in the 2005 Glenny AM, et al paper that a random effects method is preferred to a fixed effect method.”¹¹

“Choices of prior distributions are, to some extent, arbitrary…”^2,3

“We recommend vague or flat priors, such as N(0, 1002), throughout for μiand d1k.”¹³

“It has become standard practice to also set vague priors for the between-trial variances. For binomial with logit links models the usual practice is to place a Uniform prior on the standard deviation, for example σ ∼ Uniform(0,2).…An alternative approach, which was once popular but has since fallen out of favour, is to set a vague Gamma prior on the precision, for example 1/ σ2 ∼ Gamma(.001,.001).”¹³

“The parameters in the distributions of random effects have vague prior distributions: N(0, 106) for the dAk (independently) and Uniform(0, 2) for σ. These priors are common choices in such models.”^2,3

“Two further alternatives may be found useful when there is insufficient data to adequately estimate the between-trials variation. The first is the use of external data. If there is insufficient data in the metaanalysis, it may be reasonable to use an estimate for σ from a larger meta-analysis on the same trial outcome involving a similar treatment for the same condition. If there is no data on similar treatments and outcomes that can be used, an informative prior can be elicited from a clinician who knows the field.”¹³

“Particular care must be taken in checking convergence, and we suggest that at least three chains are run, starting from widely different (yet sensible) initial values…Posteriors should be examined visually for spikes and unwanted peculiarities, and both the initial “burn-in” and the posterior samples should be conservatively large and the number of iterations for both must be reported in the analysis.”¹³

“While the likelihood is not altered by a change in which treatment is taken to be “Treatment 1 [referent], the choice of the reference treatment can affect the posterior estimates because priors cannot be totally non-informative…Choice should therefore be based on ease of interpretation, with placebo or standard treatment usually taken as Treatment 1.”¹³

“It is incorrect to analyze the pairwise effects in a multiarm trial as if they came from separate studies.”^2,3

“If the network appropriately includes a multiarm trial, omitting it from the analysis may introduce bias. The analysis, then, must take into account the correlation among the effect estimates for the pairs of arms;”^2,3

“Shared parameter models allow the user to generate a single coherent synthesis when trials report results in different formats. For example some trials may report binomial data for each arm, while others report only the estimated log odds ratios and their variances; or some may report numbers of events and time at risk, while others give binomial data at given follow-up times.¹³

“The consistency of the comparative treatment effect across trials (and sets of trials) also depends upon whether the appropriate measure of effect is used.… If an appropriate measure of comparative treatment effect is used to minimise variation in comparative treatment effect within each and all sets of included randomised trials, the exchangeability assumption is more likely to be maintained.”¹¹

“…relative measures of comparative treatment effect are often a robust way of summarising the overall result of the evidence available in order to apply it to any subgroup with a particular baseline risk.”¹¹

“Whatever the method of analysis, the pooling of individual study results and indirect comparisons should be based on relative effect measures (e.g., OR, difference in change from baseline, hazards ratio) to preserve randomization.”^2,3

One advantage of the OR is that, because it is symmetrical around 1.0 (unlike the RR), the OR for harm is equal to the inverse of OR for benefit, and hence is consistently estimated regardless of how the research question is framed (eg in a study that is to measure survival, the researchers could use a null hypothesis of no difference in survival, or a null hypothesis of no difference in mortality).”¹¹

“If the underlying baseline risk is the same across the two sets of trials and the PBS population, then there it may be considered appropriate to use the directly synthesised RD as an absolute measure of comparative treatment effect… If the baseline risk is different, then the primary issue for the indirect comparison is whether the trials are similar in terms of potential confounders… If it is decided to proceed with the indirect comparison, then a ratio measure (OR or RR) is usually preferred to the RD, because as outlined above, it is considered that relative measures of comparative treatment effect have more often been observed to be constant across different baseline risks than absolute measures of comparative treatment effect.”¹¹

Assessment for and Handling of Potential Bias/Inconsistency

“Before comparing the proposed medicine with the main comparator, the comparability of the two sets of trials must be established.”²¹

“When direct evidence and indirect evidence are combined for a particular pairwise comparison, it is important that the indirect estimate is not biased and there is no discrepancy between the direct and indirect comparisons. Therefore, consistency between these direct and indirect comparisons should be accounted for.”^2,3

“Heterogeneity, inconsistency, and bias may propagate through a network of trials, and may affect the estimates differentially across regions of the network.”⁹

“The indirect comparison across trials does not have a randomisation step to allow the characteristics of the patients to differ only due to the play of chance.”¹⁰

“The indirect comparisons involved are not randomized comparisons, and may suffer the biases of observational studies, for example due to confounding”.¹⁶

“…it is important to remember that in a network meta-analysis of RCTs, the value of randomization does not hold across trials.… Hence, an ITC or network meta-analysis of RCTs is a form of observational evidence, but arguably less prone to confounding bias than is a cohort study(or any other observational design).”^2,3

“the mechanisms that potentially could create “bias” in indirect comparisons appear be to identical to those that cause heterogeneity in pair-wise metaanalysis.”¹⁴

“Inconsistency can be thought of as a conflict between “direct” evidence on a comparison between treatments B and C, and “indirect” evidence gained from AC and AB trials. Like heterogeneity, inconsistency is caused by effect-modifiers, and specifically by an imbalance in the distribution of effect modifiers in the direct and indirect evidence.”¹⁴

“Factors such as the total number of trials in a network, number of trials with more than two comparison arms, heterogeneity (i.e., clinical, methodological, and statistical variability within direct and indirect comparisons), inconsistency (i.e., discrepancy between direct and indirect comparisons), and bias may influence effect estimates obtained from network meta-analyses.”⁹

“In principle, the validity of indirect comparison relies on the invariance of treatment effects across study populations. However, in practice, trials can vary in numerous ways including population characteristics, interventions and cointerventions, length of followup, loss to followup, study quality, etc. Given the limited information in many publications and the inclusion of multiple treatments, the validity of indirect comparisons is often unverifiable. Moreover, indirect comparisons, like all other meta-analyses, essentially constitute an observational study, and residual confounding can always be present. Systematic differences in characteristics among trials in a network can bias indirect comparison results. In addition, all other considerations for meta-analyses, such as choice of effect measures or heterogeneity, also apply to indirect comparisons.”⁴

The ICWG report quotes Glenny et al.'s definition of inconsistency (they call it exchangeability), “… the two sets of trials should be exchangeable, in the sense that there is no reason to suppose that the results as a whole would be different had the various trialists kept the same protocol and patients, but chosen to study a different treatment comparison.”¹¹

“Most agencies to which the results of a network meta-analysis could be submitted currently require that direct estimates and indirect estimates be calculated separately and shown to be consistent before direct evidence and indirect evidence are combined.”^2,3

“…network meta-analysis relies on the randomization in the RCTs that compared the treatments directly. It also involves a similarity assumption: “Combining studies should only be considered if they are clinically and methodologically similar”. Nevertheless, “no commonly accepted standard [defines] which studies are ‘similar enough.”^2,3

“In a multiple treatment comparison involving both direct and indirect evidence, the evidence network can become very complex with many comparisons based on only one or two studies. With increasing complexity and greater numbers of treatments, the prospect of inconsistency increases. There is also a power trade-off between the number of pair-wise comparisons and the number of studies included in the analysis – too many comparisons with too few studies and the analysis may be underpowered to detect true differences.”¹

The ICWG report¹¹ provides an example framework for assessing the exchangeability assumption of a network meta-analysis. Assuming a indirect comparisons of treatments A and B through a common comparator C is being considered, ICWG first recommends for the AvC and BvC direct randomized trials:

• Assessment of the available trials for factors that may cause heterogeneity of the AvC and BvC comparative treatment effect
• Assessment of the event rates in the drug C populations
• Assessment of whether the measure of the comparative treatment effect for AvC and BvC is appropriate
• Assessment of evidence of the statistical homogeneity of the AvC and BvC comparative treatment effect across the available trials

Then for the BvA indirect comparison:

• Assessment across the sets of trials (i.e. the AvC and the BvC trials) for factors that may cause heterogeneity of the BvA comparative treatment effect
• Assessment of the event rates in the drug C populations across the sets of trials
• Assessment of whether the measure of the comparative treatment effect for BvA is appropriate
• Assessment of evidence of statistical homogeneity of the synthesized comparative treatment effect BvA across the sets of trials (only possible if BvA has been compared via multiple common references)

According to the CADTH, “Whether an indirect treatment comparison provides a valid estimate of the relative efficacy for an intervention of interest significantly depends on the fulfillment of this primary assumption. To determine whether or not this assumption is met, trials included in the indirect comparison can be assessed according to three criteria”:

• comparability of the linking treatment;
• comparability of patients/heterogeneity;
• methodological comparability of included trials

“…whichever [indirect comparison/network meta-analysis] method the investigators choose, they should assess the invariance of treatment effects across studies and appropriateness of the chosen method on a case-by-case basis, paying special attention to comparability across different sets of trials.”⁴

“Where direct and indirect evidence are combined, inconsistencies between the direct and indirect evidence must be assessed and reported.”¹

“Decisionmakers making use of results of network meta-analyses will need to assess whether the differences between treatments are most likely true or whether they can be explained by bias in the analysis. The internal validity of the analyses is contingent on three factors: 1) the appropriate identification of the studies that make up the evidence network, 2) the quality of the individual RCTs, and 3) the extent of confounding bias due to similarity and consistency violations.”^2,3

“Factors such as the total number of trials in a network, number of trials with more than two comparison arms, heterogeneity (i.e., clinical, methodological, and statistical variability within direct and indirect comparisons), inconsistency (i.e., discrepancy between direct and indirect comparisons), and bias may influence effect estimates obtained from network meta-analyses.”⁹

“Evaluation of homogeneity and consistency (if the network supports both direct and indirect comparisons) should be specified as components of the analysis and should reflect the risks and benefits of combining data for the particular research question”^2,3

“While it is essential to carry out tests for inconsistency, the issue should not be considered in an overly mechanical way… We emphasise that while tests for inconsistency must be carried out, they are inherently underpowered, and will often fail to detect it. Investigators must therefore also ask whether, if inconsistency is not detected, conclusions from combining direct and indirect evidence can be relied upon.”¹⁴

“…tests for statistical heterogeneity have low power, and therefore, even if statistical heterogeneity is not demonstrated, doubts will remain about its presence, particularly in the presence of obvious clinical differences across the AvC and BvC trials by a factor that is known to influence drugs B and/or A.”¹¹

“When analyzing a network of comparisons, the inconsistency of the network needs to be considered, as well as between-trial heterogeneity and sampling error. Large inconsistencies rule out a meta-analysis, small inconsistencies should add uncertainty to the results.”⁸

“A departure from consistency arises when the direct and indirect estimates of an effect differ…Researchers must evaluate departures from consistency and determine how to interpret them.”²

“The assumption of constant efficacy requires all trials included in the analysis to be equivalent and attempting to measure the same treatment effect – that is, the results of one set of trials (A vs. B) should be generalisable to the other set of trials (A vs. C). Determining whether the assumption of generalisability holds is a subjective assessment based on a detailed review of the included studies in both comparisons.”¹

“Disagreement between direct and indirect evidence must be fully investigated and it may preclude pooling data if the disagreement cannot be adequately explained.”¹

“When information on heterogeneity within the direct comparisons is available, consideration of it can form a preliminary step in a network meta-analysis, but one should first examine potential effect modifiers, because disparities among studies may preclude analysis of the network.”^2,3

“Consistency or coherence describes the situation that direct and indirect evidence agrees with each other, and when the evidence of a network of interventions is consistent, investigators could combine direct and indirect evidence using MTM models. Conversely, they should refrain from combining multiple sources of evidence from an incoherent network where there are substantial differences between direct and indirect evidence.”⁴

“Decisions should be based on coherent models that fit the data. Careful examination of different sources of evidence may reveal that some estimates are “corroborated” and others not. If inconsistency is detected, the entire network of evidence should be reconsidered from a clinical epidemiology viewpoint with respect to the presence of potential effect modifiers.”¹⁴

“Any adjustment in response to inconsistency is post hoc, which emphasizes the importance of identifying potential causes of heterogeneity of effect at the scoping stage, and potential internal biases in advance of synthesis”¹⁴

“Measures of inconsistency variance or incoherence variance are not recommended as indicators of inconsistency.”¹⁴

“Within a Bayesian framework a consistency model can be compared to an “inconsistency” model. Analyses of residual deviance can provide an “omnibus” test of global inconsistency, and can also help locate it.”¹⁴

“Node splitting is another effective method for comparing direct evidence to indirect evidence in complex networks.”¹⁴

Assessment of Model Fit

“In frequentist analyses, measures of model fit are similar to those for direct evidence and depend on the particular outcome measure. Bayesian analyses customarily use deviance (a likelihood-based measure)—the lower the residual deviance, the better the fit. For comparing models, the deviance information criterion (DIC) adds a penalty term, equal to the effective number of parameters in the model. If a model fits poorly, graphical techniques can aid more-detailed examination.”^2,3

“The goodness-of-fit can be estimated by calculating the difference between the deviance for the fitted model and the deviance for the saturated model (which fits the data perfectly). For example, the Akaike information criterion, which uses the likelihood function, the Bayesian information criterion, or deviance information criterion can all be used for model selection”^2,3

“…competing models should be compared in terms of their goodness-of-fit to the data, and residual deviance calculations may be provided to justify the study's choice of the base case model.”^2,3

“In this document we suggest that global DIC statistics and res D are consulted both to compare fixed and random effect models, and to ensure that overall fit is adequate.”¹³

“The choice of a fixed- or random-effects meta-analysis model, with or without covariate interactions, can be made by comparing different competing models regarding their goodness-of-fit to the data.”^2,3

Use of Sensitivity Analysis

“Investigators should conduct sensitivity analysis to check the assumptions of the indirect comparison. If the results are not robust to the assumptions, findings from indirect comparisons should be considered as inconclusive.”⁴

“Sensitivity analyses should focus on the areas of greatest uncertainty. Potential effect modifiers can be explored by stratifying on variations in study design or population. Comparisons between random-effects and fixed-effects analyses may be appropriate. Bayesian analyses should also explore the influence of choosing different prior distributions.”^2,3

“Choices of prior distributions are, to some extent, arbitrary, so they are often subjected to sensitivity analysis, which may be especially important for priors on heterogeneity in random-effects models.”^2,3

Interpretation

“Probability statements could be made about the effectiveness of each treatment [24]. For example, for each treatment, one can calculate the probability that the treatment is the best, second best, or third best among all treatments. Such probability statements should be interpreted carefully since the difference between treatments might be small and not clinically meaningful.”⁹

“Investigators should explicitly state assumptions underlying indirect comparisons and conduct sensitivity analysis to check those assumptions. If the results are not robust, findings from indirect comparisons should be considered inconclusive. Interpretation of findings should explicitly address these limitations.”⁴

In respect to Bayesian network meta-analysis, “Probability statements could be made about the effectiveness of each treatment.”⁹

“The external validity of the network meta-analysis will naturally be limited by the external validity of the RCTs included in the evidence network, and health-care decisionmakers will need to review whether results can be extrapolated to the population of interest.”^2,3

“Furthermore identification of the “best” or most appropriate treatment cannot be made on the basis of efficacy end points alone. To inform health-care decisionmaking for clinical treatment guidelines and reimbursement policies, the efficacy findings of a network meta-analysis must be interpreted in light of other available (observational) evidence and other characteristics of the competing interventions, such as safety and convenience”.^2,3

“The network of available evidence should be described and used to guide the selection of the method of meta-analysis. The selection of direct and indirect evidence must be clearly defined. The exclusion of relevant evidence, either direct or indirect, should be highlighted and justified. Where direct and indirect evidence are combined, inconsistencies between the direct and indirect evidence must be assessed and reported.”¹

“An approach based on a network of trials can incorporate both non-inferiority and superiority trials and so unify interpretation of the results of these different types of trials, without taking into account the non-inferiority margins used (which very frequently cannot be justified).”¹⁸

“There are two types of potential errors when interpreting the results of indirect comparisons, mainly those derived from networks of comparisons:

1. drawing conclusions of equivalent efficacy for two treatments when there is no statistically significant difference
2. and within an indirect comparison, establishing an incorrect hierarchy by naive comparison of point estimates.”¹⁸

Guidance from the Haute Autorite de Sante provides a brief “critical review guide” with six main sections:¹⁸

1. acceptability of the approach used;
2. search strategy and selection process for data contributing to the indirect comparison calculations;
3. clinical homogeneity of trials and stability of effects;
4. consistency of estimates;
5. degree of concordance of the result with that of existing direct comparisons;
6. correct interpretation of results in the proposed conclusions.

The NICE Decision Support Unit technical support document #7 provides a “reviewer checklist” for evidence synthesis reports, which addresses “issues specific to network synthesis” including:^13-15

“C1.

Adequacy of information on model specification and software implementation

C2.

Multi-arm trials

C3.

Connected and disconnected networks

C4.

Inconsistency”

Reporting

“In addition to the estimates of treatment effects, uncertainty, clinical and methodological characteristics, and potential biases within included trials must be conveyed.”⁹

“If the analyses were performed within a Bayesian framework, the choice of prior distributions for the model parameters should be defined.”^2,3

“Indicate software package used in the analysis and provide code (at least in an online appendix)”

“Evidence from a mixed treatment comparison may be presented in a variety of ways. The network of evidence may be presented in tabular form. It may also be presented diagrammatically as long as the direct and indirect treatment comparisons are clearly identified and the number of trials in each comparison is stated.”¹²

“In order to appreciate the value of a network meta-analysis, it is recommended that results of all (relevant) pairwise comparisons (as a reflection of the functional parameters) are presented as well.”^2,3

“It is critical to report all pairwise effect estimates together with the associated confidence or credible intervals, depending on the statistical model used (i.e., frequentist or Bayesian model).”⁹

“Investigators should make efforts to explain the differences between direct and indirect evidence based upon study characteristics.”⁴

“The heterogeneity between results of pairwise comparisons and inconsistencies between the direct and indirect evidence on the technologies should be reported.”¹²

“… the choice of an indirect instead of a direct head-to-head comparison between the study treatment and the comparator should be explained, together with the limitations of the indirect comparison.”¹⁹