NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Dudley RA, Frolich A, Robinowitz DL, et al. Strategies To Support Quality-based Purchasing: A Review of the Evidence. Rockville (MD): Agency for Healthcare Research and Quality (US); 2004 Jul. (Technical Reviews, No. 10.)
Technical Expert Advisory Panel
For advice on the scope of the project, refinement of the key questions, and preparation of this technical review, we consulted technical experts in the following fields: employer purchasing strategies, provider performance assessment, consumer use of report cards and consumer preferences for health care information, risk adjustment, and economics. (See Appendix A, available at www.ahrq.gov/clinic/epcindex.htm.)
Target Audiences and Population
The decisionmakers addressed in this technical review are purchasers (both private purchasers such as employers and public purchasers such as the Centers for Medicare & Medicaid Services and State Medicaid programs), executives in health plans that must negotiate incentive arrangements with provider organizations or individual providers, executives in provider organizations that must negotiate incentive arrangements with providers, public health officials and other organizations interested in creating health care performance reports for public release, and policymakers. For the purpose of this report, provider organizations include all clinical health providers such as physicians, nurses, and hospitals. Public health officials and policymakers include those at the local, State, Federal, and international levels.
The ultimate target population of this report is the U.S. population at risk for morbidity or mortality resulting from quality problems in the provision of health care. We are interested in QBP strategies that affect the entire U.S. population—all members of which are at risk for receiving poor quality care—including those of all racial and ethnic backgrounds, all ages, and both genders.
Key Questions
We developed the key questions in collaboration with AHRQ, the Alliance (the nominating partner), and our Technical Expert Panel. The goal of these discussions was to identify the issues purchasers interested in QBP faced so that, if the available research offered conclusions about these aspects of QBP, the various stakeholders would be in a better position to select optimal approaches to QBP.
The key questions for which literature, ongoing research, or results from analyses were sought in preparation of this report were:
- Choosing provider incentive strategies
- What is the evidence on the extent to which health plans and employers use incentives to improve quality and efficiency?
- Does the use of financial incentives for quality and efficiency actually increase the probability that patients receive high quality, efficient care?
- Does the impact of financial incentives for quality and efficiency depend on:
- The basis of the incentive (structure, process, outcome)?
- The nature of the incentive (bonus, penalties or holdback, tiering or patient steerage/referral)?
- To whom the incentive is targeted (plan vs. provider group vs. individual provider)?
- The payer of the incentive (purchaser vs. plan vs. medical group)?
- The magnitude of the incentive?
- Does the use of nonfinancial incentives for quality and efficiency actually increase the probability that patients receive high quality, efficient care?
- Does the impact of nonfinancial incentives for quality and efficiency depend on:
- The basis of the incentive (structure, process, outcome)?
- The nature of the incentive (public release of performance report vs. confidential performance report)?
- Relationship between cost and quality
- Does greater spending result in higher quality?
- What are the cost savings for the health care provider and purchaser as a result of the quality improvement?
- What are the cost savings associated with different approaches to preventing medical errors or otherwise improving quality?
- What specific processes and structures result in quantifiable cost savings? Who realizes the savings? How should they be shared?
- Policy and market context in which incentives are used
- What contextual variables (e.g., provider supply, employer number and market share, health plan competition, organizational system/infrastructure, employee demographics) positively or negatively influence the effectiveness of financial and nonfinancial incentives for providers?
Literature Review Methods
Based on input from our expert advisors, our conceptual model, and practical considerations, we developed literature review methods that included: inclusion and exclusion criteria to identify potentially relevant articles, search strategies to retrieve articles, abstract review protocols, and a system of scoring published studies for completeness.
Inclusion and Exclusion Criteria
To be considered an article that provided evidence regarding one of the key questions above, the article had to address one of the predictor variables and either quality (as measured by processes or outcomes) or cost. In addition, the intervention in the trial had to be a strategy that could plausibly be introduced by a purchaser. Our focus was on articles that provided definitive primary data from randomized, controlled trials, but we also included systematic reviews to determine whether these contained any additional information not covered by the primary randomized, controlled trial reports.
We excluded articles that did not meet specific criteria in terms of the quality of the research and reporting. These were:
- For interventional trials
- Intervention randomized
- Inclusion/exclusion criteria clear and appropriate
- Greater than 75% follow-up
- Note: two criteria usually used to judge the quality of a randomized, controlled trial—provision of placebo to the control group and blinding of the subjects—are not applicable in this situation
- For systematic reviews
- Information source appropriate
- Information source adequately searched
- Inclusion/exclusion criteria clear and appropriate
- Data abstraction performed by at least 2 independent reviewers
- Principal measures of effect and the methods of combining results appropriate
Search Strategy
The objective of our search strategy was to identify all published QBP randomized trials and all ongoing research into QBP strategies. For the literature review, we used standard search strategies involving the querying of two online databases (MEDLINE® and Cochrane) using key words, followed by evaluation of the bibliographies of relevant articles, Web sites of relevant organizations (especially of funding agencies providing project summaries and of employer organizations pursuing QBP), and reference lists provided by our Technical Expert Panel (Table 1).
Database Searches
To identify potentially relevant articles in the medical literature, we searched MEDLINE® and Cochrane databases and references provided by our Expert Advisors.
MEDLINE ® search strategies. We searched MEDLINE® (January 1980 to December 15, 2003) for English language articles using the search terms described in Table 2. Some citations were reviewed and articles were retrieved in more than one of the searches listed below.
Cochrane search strategies. We searched the Cochrane databases from January 1, 1990 through December 15, 2003 (OVID, Evidence Based Medicine Reviews Multifile) using the search terms described in Table 3.
Abstract Review
To identify potentially relevant articles for focused searching, at least two investigators (to ensure consistent application of the inclusion and exclusion criteria) reviewed each citation and, whenever an abstract was available, the abstract. Discrepancies in inclusion were resolved by discussion and re-review.
Evaluating Published Articles for Completeness of Reporting
We assessed each of the published articles for their completeness in reporting the factors we identified in our conceptual model that could influence a provider's response to incentives. Specifically, we scored them for the inclusion (or not) of descriptions of the elements in Table 4. We also recorded the type of care (preventive care, acute care, or chronic care) to which the quality measured pertained.
Identifying Ongoing Research
Based on input from our expert advisors, our conceptual model, and practical considerations, we developed methods to catalog ongoing research into QBP that involved specifying: inclusion and exclusion criteria to identify potentially relevant research projects, search strategies to retrieve project abstracts, abstract review protocols, and a system of describing the study design of ongoing research projects.
Inclusion and Exclusion Criteria
Since the search for ongoing research focused on projects not yet reported in the literature, the criteria for identifying relevant projects focused on the planned intervention. Two types of research potentially met our inclusion criteria: projects designed as randomized controlled trials, or projects with interventions using QBP methods as described above (i.e., payment or performance reporting strategies) and applied at the community level (or in a broader geographic region, such as a State) that included historical or contemporaneous non-randomized control groups.
Search Strategy
We searched online health services research databases (HSRProj and AHRQ's Grants-On-Line Database or GOLD). We also searched the Web sites of other funders or coordinators of projects (e.g., the Leapfrog Group at www.leapfroggroup.org/RewardingResults/). Finally, we inquired of staff at AHRQ, the Robert Wood Johnson Foundation, the California HealthCare Foundation, and the Commonwealth Fund whether there was ongoing research that met our inclusion criteria being funded by those organizations. Table 5 lists our information sources for this aspect of the report.
Database Searches
We searched the two available databases for ongoing health services research, using a similar search strategy for each (Tables 6 and 7). We accessed HSRProj through the National Library of Medicine's Gateway database at gateway.nlm.nih.gov/gw/Cmd and GOLD at www.gold.ahrq.gov.
GOLD search strategies. We searched GOLD through February 15, 2004 for grants funded by AHRQ using the categories described in Table 6. Through our combination of searches, we eventually evaluated all projects in GOLD.
HSRProj search strategies. We searched the HSRProj database through February 15, 2004 using the categories described in Table 7.
Grant Abstract Review
Two investigators reviewed the abstracts of projects identified from the database searches to assess relevance to the technical review. Discrepancies in inclusion were resolved by discussion and re-review and by discussion with project officers at funding agencies or with the principal investigator of the project under consideration.
Describing the Study Design of Ongoing Research
For each research project, we interviewed either project staff (usually the principal investigator) or the project officer to determine the study design. We obtained information about the intervention—performance measures and incentives used—and the control group. The information sought is described in Table 8.
Tables
Table 1. Information sources for literature review and catalog of ongoing research
| Goal of Search | Databases searched | Relevant Organizations (for Web-based searches) |
|---|---|---|
| Identify randomized, controlled trials of quality-based purchasing strategies | MEDLINE® | AHRQ |
| Cochrane | Robert Wood Johnson Foundation | |
| California HealthCare Foundation | ||
| Commonwealth Fund | ||
| National Business Coalition on Health | ||
| Leapfrog Group |
Table 2. MEDLINE® searches to identify potentially relevant primary data
| Search Terms | Citations reviewed | Articles retrieved |
|---|---|---|
| “pay” AND “quality” AND “measurement” | 80 | 1 |
| “incentive” AND “quality” AND “measurement” | 195 | 5 |
| “financial incentive” AND “quality” AND “efficiency” | 125 | 11 |
| “provider supply” AND “incentive” | 15 | 0 |
| “quality” AND “error” AND “safety” AND “cost*” | 16 | 0 |
| “pay” AND “performance” | 389 | 2 |
| “pay” AND “incentive” AND “quality” | 79 | 3 |
| “pay” AND “quality” AND “measurement” AND “Randomized Controlled Trial” [Publication Type] | 8 | 1 |
| “incentive” AND “quality” AND “measurement” AND “Randomized Controlled Trial” [Publication Type] | 13 | 2 |
| “financial incentive” AND “quality” AND “efficiency” AND “Randomized Controlled Trial” [Publication Type] | 1 | 1 |
| “provider supply” AND “incentive” AND “Randomized Controlled Trial” [Publication Type] | 0 | 0 |
| “quality” AND “error” AND “safety” AND “cost*” AND “Randomized Controlled Trial” [Publication Type] | 0 | 0 |
| “pay” AND “performance” AND “Randomized Controlled Trial” [Publication Type] | 6 | 1 |
| “pay” AND “incentive” AND “quality” AND “Randomized Controlled Trial” [Publication Type] | 1 | 1 |
| “incentive” AND “quality” AND “Randomized Controlled Trial” [Publication Type] | 42 | 2 |
| “pay” AND “quality” AND “Randomized Controlled Trial” [Publication Type] | 26 | 2 |
| “value” AND “incentive” AND “Randomized Controlled Trial” [Publication Type] | 49 | 0 |
| “value” AND “pay” AND “Randomized Controlled Trial” [Publication Type] | 10 | 0 |
| “Insurance, Health, Reimbursement” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 72 | 6 |
| “Medicare Payment Advisory Commission” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 0 | 0 |
| “Physician Payment Review Commission” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 0 | 0 |
| “Prospective Payment Assessment Commission” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 1 | 0 |
| “Prospective Payment System” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 28 | 1 |
| “Salaries and Fringe Benefits” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 78 | 1 |
| “Single-Payer System” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 2 | 0 |
| “Fee-for-Service Plans” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 11 | 1 |
| “Reimbursement Mechanisms” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 66 | 6 |
| “Reimbursement, Incentive” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 10 | 4 |
| “Cost and Cost Analysis” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 2,561 | 9 |
| “Medical Errors” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 678 | 0 |
| “Medication Errors” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 17 | 0 |
| “Management Quality Circles” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 6 | 0 |
| “Professional Review Organizations” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 3 | 0 |
| “Quality Assurance, Health Care” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 586 | 14 |
| “Quality Control” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 161 | 1 |
| “Quality Indicators, Health Care” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 22 | 0 |
| “Total Quality Management” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 45 | 2 |
| “United States Agency for Healthcare Research and Quality” [MESH] AND “Randomized Controlled Trial” [Publication Type] | 11 | 0 |
| Total Articles | 5413 | 76 |
- *
The use of the asterisk expands search terms such that all combinations of terms with the phrase preceding the asterisk will be returned in the search (e.g., cost* returns searches for cost, costs, etc.).
MESH = Medical Subject Heading
Table 3. Search terms and citations for Cochrane databases
| Search terms | Citations reviewed | Articles retrieved |
|---|---|---|
| Pay | 6 | 2 |
| Incentive | 4 | 0 |
| Efficiency | 74 | 0 |
| Safety | 264 | 0 |
| Cost | 210 | 2 |
| Error | 12 | 0 |
| Performance | 60 | 0 |
| Value | 95 | 0 |
| Insurance | 0 | 0 |
| Reimbursement | 0 | 0 |
| Total | 725 | 4 |
- *
The use of the asterisk expands search terms such that all combinations of terms with the phrase preceding the asterisk will be returned in the search (e.g., cost* returns searches for cost, costs, cost effectiveness, etc.).
Table 4. Evaluating randomized controlled trials for completeness of reporting
| Domain of the Conceptual Model | Specific Variable |
|---|---|
| Financial Characteristics of Incentive | Recipient: individual provider vs. provider group |
| Revenue potential: magnitude of the financial incentive | |
| Revenue potential: incentive as a proportion of total income | |
| Impact on cost: direct costs and opportunity costs of complying | |
| Nonfinancial Characteristics of Incentive | Perceived attainability: how easy/difficult it is to accomplish the task of the incentive |
| Performance domain measured: structure, process, outcome | |
| Predisposing Factors | Financial characteristics of the environment: proportion of income from: fee for service, salary, capitation |
| Financial characteristics of the environment: number of other financial incentives in place | |
| Provider characteristics: demographics, specialty, and other immutable factors | |
| Provider characteristics: workload, proportion of patients if service where incentive relevant | |
| Market characteristics: community initiatives or performance standards | |
| Enabling Factors | Organizational characteristics: size, type of practice, specialty, etc. |
| Organizational characteristics: capabilities such as information systems, use of guidelines and feedback, etc. | |
| Organizational characteristics: leadership, culture, etc. | |
| Patient characteristics: demographics and other immutable factors | |
| Patient characteristics: type of insurance, benefits structure | |
Table 5. Information sources for the catalog of ongoing research
| Goal of Search | Databases searched | Relevant Organizations (for Web-based searches and staff interviews) |
|---|---|---|
| Identify ongoing research evaluating quality-based purchasing strategies | GOLD (http://www | AHRQ |
| Leapfrog Group | ||
| Robert Wood Johnson Foundation | ||
| California HealthCare Foundation | ||
| Commonwealth Fund |
Table 6. Search terms and citations for GOLD
| Search by Category | Grants reviewed | Grants retrieved |
|---|---|---|
| Quality Outcomes | 319 | 2 |
| Quality Measures | 189 | 2 |
| Quality Improvement | 256 | 2 |
| Managed Care/Market Forces | 98 | 1 |
| Payment Strategies | 22 | 1 |
| Cost | 121 | 0 |
| New Knowledge | 374 | 2 |
| Total Grants | 1379 | 10 |
Table 7. Search terms and citations for HSRProj database
| Search terms | Grant abstracts reviewed | Grants retrieved |
|---|---|---|
| Pay | 49 | 1 |
| Incentive | 165 | 6 |
| Efficiency | 144 | 2 |
| Safety | 374 | 4 |
| Error | 160 | 1 |
| Performance | 546 | 7 |
| Value | 219 | 6 |
| Reimbursement | 136 | 2 |
| Total Grants | 1793 | 29 |
- *
The use of the asterisk expands search terms such that all combinations of terms with the phrase preceding the asterisk will be returned in the search (e.g., cost* returns searches for cost, costs, cost effectiveness, etc.).
Table 8. Design information sought about ongoing research
| Design Issue | Examples of Possible Responses |
|---|---|
| Patient Population from an Insurance Perspective | Privately Insured, Medicare, Medicaid, or multiple populations |
| Health Plan Setting | Health maintenance organization, preferred provider organization, point of service |
| Control Group | Randomized controlled trial vs. non-randomly selected contemporaneous control vs. historical control |
| Incentive Structure | Describe financial or reputational gains from superior performance |
| Performance Measures | Participation vs. clinical performance (for the latter, describe determinants of performance assessment, including weighting given when multiple measures are used) |
| Evaluation Plan/Goals | Assess determinants of participation in the program, catalog incentives used, test impact of incentives on clinical performance |