Methods for Literature Search

R Adams Dudley; Anne Frolich; David L Robinowitz; Jason A Talavera; Peter Broadhead; Harold S Luft

2Methods for Literature Search

Technical Expert Advisory Panel

For advice on the scope of the project, refinement of the key questions, and preparation of this technical review, we consulted technical experts in the following fields: employer purchasing strategies, provider performance assessment, consumer use of report cards and consumer preferences for health care information, risk adjustment, and economics. (See Appendix A, available at www.ahrq.gov/clinic/epcindex.htm.)

Target Audiences and Population

The decisionmakers addressed in this technical review are purchasers (both private purchasers such as employers and public purchasers such as the Centers for Medicare & Medicaid Services and State Medicaid programs), executives in health plans that must negotiate incentive arrangements with provider organizations or individual providers, executives in provider organizations that must negotiate incentive arrangements with providers, public health officials and other organizations interested in creating health care performance reports for public release, and policymakers. For the purpose of this report, provider organizations include all clinical health providers such as physicians, nurses, and hospitals. Public health officials and policymakers include those at the local, State, Federal, and international levels.

The ultimate target population of this report is the U.S. population at risk for morbidity or mortality resulting from quality problems in the provision of health care. We are interested in QBP strategies that affect the entire U.S. population—all members of which are at risk for receiving poor quality care—including those of all racial and ethnic backgrounds, all ages, and both genders.

Key Questions

We developed the key questions in collaboration with AHRQ, the Alliance (the nominating partner), and our Technical Expert Panel. The goal of these discussions was to identify the issues purchasers interested in QBP faced so that, if the available research offered conclusions about these aspects of QBP, the various stakeholders would be in a better position to select optimal approaches to QBP.

The key questions for which literature, ongoing research, or results from analyses were sought in preparation of this report were:

Choosing provider incentive strategies
1. What is the evidence on the extent to which health plans and employers use incentives to improve quality and efficiency?
2. Does the use of financial incentives for quality and efficiency actually increase the probability that patients receive high quality, efficient care?
3. Does the impact of financial incentives for quality and efficiency depend on:
  - The basis of the incentive (structure, process, outcome)?
  - The nature of the incentive (bonus, penalties or holdback, tiering or patient steerage/referral)?
  - To whom the incentive is targeted (plan vs. provider group vs. individual provider)?
  - The payer of the incentive (purchaser vs. plan vs. medical group)?
  - The magnitude of the incentive?
4. Does the use of nonfinancial incentives for quality and efficiency actually increase the probability that patients receive high quality, efficient care?
5. Does the impact of nonfinancial incentives for quality and efficiency depend on:
  - The basis of the incentive (structure, process, outcome)?
  - The nature of the incentive (public release of performance report vs. confidential performance report)?
Relationship between cost and quality
1. Does greater spending result in higher quality?
2. What are the cost savings for the health care provider and purchaser as a result of the quality improvement?
3. What are the cost savings associated with different approaches to preventing medical errors or otherwise improving quality?
4. What specific processes and structures result in quantifiable cost savings? Who realizes the savings? How should they be shared?
Policy and market context in which incentives are used
1. What contextual variables (e.g., provider supply, employer number and market share, health plan competition, organizational system/infrastructure, employee demographics) positively or negatively influence the effectiveness of financial and nonfinancial incentives for providers?

Literature Review Methods

Based on input from our expert advisors, our conceptual model, and practical considerations, we developed literature review methods that included: inclusion and exclusion criteria to identify potentially relevant articles, search strategies to retrieve articles, abstract review protocols, and a system of scoring published studies for completeness.

Inclusion and Exclusion Criteria

To be considered an article that provided evidence regarding one of the key questions above, the article had to address one of the predictor variables and either quality (as measured by processes or outcomes) or cost. In addition, the intervention in the trial had to be a strategy that could plausibly be introduced by a purchaser. Our focus was on articles that provided definitive primary data from randomized, controlled trials, but we also included systematic reviews to determine whether these contained any additional information not covered by the primary randomized, controlled trial reports.

We excluded articles that did not meet specific criteria in terms of the quality of the research and reporting. These were:

For interventional trials
- Intervention randomized
- Inclusion/exclusion criteria clear and appropriate
- Greater than 75% follow-up
- Note: two criteria usually used to judge the quality of a randomized, controlled trial—provision of placebo to the control group and blinding of the subjects—are not applicable in this situation
For systematic reviews
- Information source appropriate
- Information source adequately searched
- Inclusion/exclusion criteria clear and appropriate
- Data abstraction performed by at least 2 independent reviewers
- Principal measures of effect and the methods of combining results appropriate

Search Strategy

The objective of our search strategy was to identify all published QBP randomized trials and all ongoing research into QBP strategies. For the literature review, we used standard search strategies involving the querying of two online databases (MEDLINE^® and Cochrane) using key words, followed by evaluation of the bibliographies of relevant articles, Web sites of relevant organizations (especially of funding agencies providing project summaries and of employer organizations pursuing QBP), and reference lists provided by our Technical Expert Panel (Table 1).

Table

Table 1. Information sources for literature review and catalog of ongoing research.

Database Searches

To identify potentially relevant articles in the medical literature, we searched MEDLINE^® and Cochrane databases and references provided by our Expert Advisors.

MEDLINE ^® search strategies. We searched MEDLINE^® (January 1980 to December 15, 2003) for English language articles using the search terms described in Table 2. Some citations were reviewed and articles were retrieved in more than one of the searches listed below.

Table

Table 2. MEDLINE^® searches to identify potentially relevant primary data.

Cochrane search strategies. We searched the Cochrane databases from January 1, 1990 through December 15, 2003 (OVID, Evidence Based Medicine Reviews Multifile) using the search terms described in Table 3.

Table

Table 3. Search terms and citations for Cochrane databases.

Abstract Review

To identify potentially relevant articles for focused searching, at least two investigators (to ensure consistent application of the inclusion and exclusion criteria) reviewed each citation and, whenever an abstract was available, the abstract. Discrepancies in inclusion were resolved by discussion and re-review.

Evaluating Published Articles for Completeness of Reporting

We assessed each of the published articles for their completeness in reporting the factors we identified in our conceptual model that could influence a provider's response to incentives. Specifically, we scored them for the inclusion (or not) of descriptions of the elements in Table 4. We also recorded the type of care (preventive care, acute care, or chronic care) to which the quality measured pertained.

Table

Table 4. Evaluating randomized controlled trials for completeness of reporting.

Identifying Ongoing Research

Based on input from our expert advisors, our conceptual model, and practical considerations, we developed methods to catalog ongoing research into QBP that involved specifying: inclusion and exclusion criteria to identify potentially relevant research projects, search strategies to retrieve project abstracts, abstract review protocols, and a system of describing the study design of ongoing research projects.