Background

Decision-analytic models represent an explicit way to synthesise evidence currently available on the outcomes and costs of alternative (mutually exclusive) healthcare interventions. Usually their objective is to obtain a clear understanding of the relationship between incremental cost and effect in order to assess relative cost-effectiveness and to determine which interventions should be adopted given existing information. Given that the use of decision-analytic modelling for health technology assessment has increased exponentially in recent years, there is a need to consider how good practice in the field has been defined. Since the 1980s, several published guidelines have been available for those developing and evaluating decision-analytic models for health technology assessment. However, given the speed at which economic evaluation methodology has progressed, it is timely to review, critically appraise and consolidate those existing guidelines on the use of decision-analytic modelling in health technology assessment, and to identify key issues where guidance is lacking.

Objectives

• To identify and describe published guidelines for assessing the quality of decision-analytic models in health technology assessment.
• To develop a synthesised guideline and accompanying checklist using available good practice guidelines.
• To provide guidance on key theoretical, methodological and practical issues not yet covered in published guidelines. Two areas were identified in advance as priorities: literature searching for parameter estimation in decision models, and adjusting for bias in treatment effect estimates from observational studies used in decision models.
• To consider the implications of this research for what might be expected of future decision-analytic models relating to the National Institute for Clinical Excellence (NICE) technology appraisal process and health technology assessment in general.

Methods

The project consisted of four key elements:

• A systematic review of existing good practice guidelines was undertaken to identify and summarise guidelines currently available for assessing the quality of decision-analytic models that have been undertaken for health technology assessment. Areas of guidance that were relevant to economic evaluation in general, rather than decision-analytic modelling specifically, were omitted.
• A synthesised good practice guidance and accompanying checklist was developed. Each theme and subtheme from the review of guidelines was taken in turn, and its relevance was discussed in relation to the development of general guidelines for decision-analytic modelling in health technology assessment. Where previous guidelines were contradictory, a consensus decision was taken by the research team regarding the most appropriate item for the synthesised guideline. A checklist was constructed from the synthesised guideline, using the suggested headings and statements. This checklist was applied to three recent decision-analytic models undertaken as part of the NICE appraisal process.
• Two specific methods areas in decision modelling which have received relatively little consideration in the literature, were considered. They were selected on the basis of the team's experience, rather than any systematic review of the methods literature. The first method's topic is the identification of parameter estimates from published literature. Parameter searches were developed and piloted using a case-study model. The second topic relates to bias in parameter estimates; that is, how to adjust estimates of treatment effect from observational studies where there are risks of selection bias. A systematic literature review was conducted to identify those studies looking at quantification of bias in parameter estimates and the implication of this bias.
• The use of the guidance and checklist for future decision-analytic models developed as part of the NICE Technology Assessment Review (TAR) process was considered. Decision modelling is central to the NICE technology assessment process and it is essential to assess the quality of those models that are developed to inform the Appraisal Committee.

Results

Conclusions

The review of current guidelines showed that although authors may provide a consistent message regarding some aspects of modelling, in other areas conflicting attributes are presented in different guidelines.

A preliminary assessment showed that, in general, the checklist appears to perform well, in terms of identifying those aspects of the model that should be of particular concern to the reader. The checklist cannot, however, provide answers to the appropriateness of the model structure and structural assumptions, as these may be seen as a general problem with generic checklists and do not reflect any shortcoming with the synthesised guidance and checklist developed here. The assessment of the checklist, as well as feedback from the EAG, indicated the importance of its use in conjunction with a more general checklist or guidelines on economic evaluation.

The review of current guidance for good quality decision-analytic modelling for health technology assessment highlighted a number of methodological areas that have not received attention in the literature on good practice. There are a lot of these areas and, therefore, it was only possible to consider two specific methods areas in decision modelling: the identification of parameter estimates from published literature, and the issue of adjusting treatment effect estimates taken from observational studies for potential bias. Literature reviews showed that both of these areas are under-researched and are areas in which further research is needed.

Recommendations for research

This project has highlighted many areas where further methods research may be of value. In particular:

• A review of the literature is needed pertaining to the quantification of selection bias in non-controlled studies and in controlled observational studies.
• Empirical research is needed to define further the level of bias in the different non-RCT study designs.
• Studies are needed which compare results from RCTs with those from other non-randomised studies.
• There is a need to assess the strengths and weaknesses of alternative ways to adjust for bias in a decision model.
• Studies are needed to determine how to prioritise searching for parameter estimates. The value of information methods is worth consideration.

Publication

• Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess 2004;8(36). [PubMed: 15361314]

NHS R&D HTA Programme

The research findings from the NHS R&D Health Technology Assessment (HTA) Programme directly influence key decision-making bodies such as the National Institute for Clinical Excellence (NICE) and the National Screening Committee (NSC) who rely on HTA outputs to help raise standards of care. HTA findings also help to improve the quality of the service in the NHS indirectly in that they form a key component of the 'National Knowledge Service' that is being developed to improve the evidence of clinical practice throughout the NHS.

The HTA Programme was set up in 1993. Its role is to ensure that high-quality research information on the costs, effectiveness and broader impact of health technologies is produced in the most efficient way for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined to include all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care, rather than settings of care.

The HTA programme commissions research only on topics where it has identified key gaps in the evidence needed by the NHS. Suggestions for topics are actively sought from people working in the NHS, the public, consumer groups and professional bodies such as Royal Colleges and NHS Trusts.

Research suggestions are carefully considered by panels of independent experts (including consumers) whose advice results in a ranked list of recommended research priorities. The HTA Programme then commissions the research team best suited to undertake the work, in the manner most appropriate to find the relevant answers. Some projects may take only months, others need several years to answer the research questions adequately. They may involve synthesising existing evidence or designing a trial to produce new evidence where none currently exists.

Additionally, through its Technology Assessment Report (TAR) call-off contract, the HTA Programme is able to commission bespoke reports, principally for NICE, but also for other policy customers, such as a National Clinical Director. TARs bring together evidence on key aspects of the use of specific technologies and usually have to be completed within a limited time period.

Criteria for inclusion in the HTA monograph series

Reports are published in the HTA monograph series if (1) they have resulted from work commissioned for the HTA Programme, and (2) they are of a sufficiently high scientific quality as assessed by the referees and editors.

Reviews in Health Technology Assessment are termed 'systematic' when the account of the search, appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others.

The research reported in this monograph was commissioned and funded by the HTA Programme on behalf of NICEas project number 02/32/01. The authors have been wholly responsible for all data collection, analysis and interpretation and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors' report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

The views expressed in this publication are those of the authors and not necessarily those of the HTA Programme, NICE or the Department of Health.

HTA Programme Director: Professor Tom Walley

Series Editors: Dr Peter Davidson, Professor John Gabbay, Dr Chris Hyde, Dr Ruairidh Milne, Dr Rob Riemsma and Dr Ken Stein

Managing Editors: Sally Bailey and Caroline Ciupek