Articles Identified

Our literature searches identified 5,045 unique candidate articles for inclusion in our literature review (Figure 4). Of these, 4,882 were eliminated after review of their abstracts. The reasons for exclusion were: 4,861 because they were not relevant to the key questions, 14 because they were cost effectiveness studies or decision analyses that provided no primary data about the questions, and 7 because they had dependent variables that were “quality” in an abstract sense—responses to a questionnaire or survey about what the provider would do if presented with a hypothetical patient—rather than actual measurement of quality performance.

Figure

Figure 4. Articles Identified By Systematic Searches.

The remaining 163 articles underwent full text review, which eliminated another 101 that were not relevant to the study question. Of the 62 studies that were relevant, only 15 were good quality. Of these, 9 were interventional studies (randomized controlled trials)^{18, 32–34, 37, 38, 43, 59–61} and 6 were systematic reviews (Figure 4).^{20, 46, 62–65} Of the nine randomized controlled trials, eight used specific financial incentives as the intervention;^{18, 32–34, 37, 38, 43, 59, 60} one used specific reputational incentives as the intervention.⁶¹

Completeness of Reports of Randomized Controlled Trials of Incentives

Trials of specific financial incentives. In every article reporting the results of a randomized controlled trial of performance-based payment incentives, there were significant variables from our conceptual model that were either not reported at all or that were incompletely described. In Table 9 we show the completeness in the reporting of the eight trials of specific performance-based payment.^{18, 32–34, 37, 38, 43, 59, 60}

Table

Table 9. Evaluating Randomized, Controlled Trials for Completeness of Reporting.

The only variables that were reported in all trials were characteristics of the incentive itself: the recipient (although even this was sometimes ambiguous between individual provider versus provider group), the magnitude of the incentive, and the domain of performance measured. Several potentially critical variables were never reported in any trial, including payment incentive as a proportion of total income and the costs of complying with the incentive and most enabling factors at the organizational level.

A trial of reputational incentives. There was a single trial of reputational incentives, in which Hibbard et al. report on the response of hospitals in south central Wisconsin to public release of performance data and compare this response to other Wisconsin hospitals who were randomly assigned to receive either a confidential report or no report at all.⁶¹ We did not include this study in Table 9 because some of the elements of that table are not applicable to or not measurable for reputational incentives (e.g., most of the financial characteristics variables) and others were not applicable to the specifics of the study (e.g., market characteristics will vary when a study is done statewide). In this article, however, there was no explicit consideration of whether response to the incentive varied with differences among hospitals in terms of enabling or predisposing factors, which were not measured.

Results of Randomized Controlled Trials of Performance-based Payment

The eight trials of performance-based payment were neither consistent in their design of the independent variable (the financial incentive offered) nor comparable in terms of their dependent variable (the performance indicator measured). Thus, we present their results as a function of several of the variables within the conceptual model (those that are actually reported for all papers). Note that among these eight trials there were ten hypotheses tested, because one study had two intervention arms (a fee-for-service arm and a bonus arm) compared to controls³⁷ and one had two dependent variables (smoking cessation processes and smoking cessation outcomes).⁶⁰

Recipient of incentive. In four studies, the recipient of the incentive was an individual provider,^{32, 33, 37, 43, 59, 60} while in the other four the recipient was the provider group or could be either an individual provider or a group.^{18, 34, 38} Among the studies targeting individual providers, there were five positive and two negative results; among the studies in which the target was always or could be the provider group, there were one positive and two negative results. (In general, we use the term “positive” to mean an effect in the desired direction—the incentive worked—and “negative” to mean there was no significant effect of the incentive on the outcome measure.)

In seven studies, with a total of nine dependent variables, the target of the incentive was a physician. Of the nine dependent variables assessed, five showed a significant relationship to the incentive in the expected direction, four showed no significant change after the incentive was introduced.^{18, 34, 37, 38, 43, 59, 60} A single study (reported in two papers) involved non-physician recipients (pharmacists) and was positive.^{32, 33}

Magnitude of the incentive. Incentives ranged in magnitude from $0.80/flu shot³⁴ to a bonus of up to $10,000 per clinic per year.⁶⁰ There was no consistent relationship between the magnitude of the incentive and response, and in fact the largest single incentive (the bonus of up to $10,000) was ineffective.⁶⁰ The two studies in which the provider faced significant uncertainty about whether they could achieve success—in each case because the incentive was tied to performance relative to other groups, and this benchmark was unknown during the time when performance was measured—were negative.^{18, 38}

Structure of the incentive. Five studies (with five outcomes) assessed fee-for-service incentives to improve quality,^{32–34, 37, 43, 59} while four studies (with five outcomes) evaluated the impact of bonuses tied to performance.^{18, 37, 42, 60} Among the studies of fee-for-service, four were positive and one was negative. With bonuses tied to performance, there were two positive results and three negative.

Performance domain measured. Among the articles included, there were seven studies of preventive care with nine dependent variables assessed. Among these nine outcomes, five were positive and four negative. The single study addressing chronic care was positive.⁶¹

Patient factors. Authors did not report the burden adherence would place on patients in any of the articles we found. However, in a general sense, we found that incentives to achieve performance were more effective when the indicator to be followed required less patient cooperation (e.g., receiving vaccinations or answering questions about smoking) than when significant patient cooperation was needed (e.g., to quit smoking, Table 10).

Table

Table 10. Available results by conceptual model domains tested.

Synopses of the available studies. As there were so few available studies, we are able to include synopses of each in this report. Rather than use the original abstracts, which varied in structure and content, we have put each into a uniform format. The eight randomized controlled trials of performance-based payment, presented in alphabetical order by first author, were:

• Christensen DB, Holmes G, Fassett WE, et al. Influence of a financial incentive on cognitive services: CARE project design/implementation. J Am Pharm Assoc. Sep-Oct 1999;39(5):629–639.
  and
• Christensen DB, Hansen RW. Characteristics of pharmacies and pharmacists associated with the provision of cognitive services in the community setting. J Am Pharm Assoc. Sep-Oct 1999;39(5):640–649.
  Setting and Design: This study took place in Washington State from February 1994 - September 1995. Incentives were offered by the Washington State Cognitive Activities and Reimbursement Effectiveness Project to community pharmacies that served primarily ambulatory patients, were not a part of a staff-model health maintenance organization, and dispensed at least 50 prescriptions per month to ambulatory Medicaid recipients to improve performance. The treatment group (n=110) performed and documented cognitive services (CS) provided to Medicaid recipients, received a fee for each intervention of $4 or $6 dollars depending on whether the CS lasted greater than six minutes, and received a monthly stipend of $40/month for their participation in the demonstration. Control pharmacies (n=90) received a monthly participation stipend of $40/month, but performed and documented CS interventions without additional reimbursement. A silent control group (group C, n=100) neither received additional payment nor documented CS interventions. Performance was measured over 19 months, ensuring a minimum 12 month observation period for each pharmacy.
  Results: At baseline, differences in operating characteristics between groups were minor and nonsignificant. Over the study period, the incentive group performed significantly more CS than the control group. Factors associated with the provision of any CS by pharmacists included perceptions of how burdensome the task of documenting CS was and the percentage of sales from prescriptions.
• Davidson SM, Manheim LM, Werner SM, Hohlen MM, Yudkowsky BK, Fleming GV. Prepayment with office-based physicians in publicly funded programs: results from the Children's Medicaid Program. Pediatrics. Apr 1992;89(4 Pt 2):761–767.
  Setting and Design: This study took place in Suffolk County, New York, from July 1983 through December 1985. The Health Care Financing Administration and the John A. Hartford Foundation offered incentives to individual primary care physicians in private office based practices. All 140 primary care physicians who treated Medicaid children and had more than $2000/year in billings were invited to participate and 80 agreed. Physicians were randomly assigned to augmented fee-for-service (n=40) at nearly double the usual New York Medicaid rates in return a commitment to meet performance goals or capitation (n=40) and compared to physicians operating under conventional Medicaid arrangements. (We do not report on the capitation arm herein as comparisons of capitation to fee-for-service were not within the scope of this report.) The payment groups were evaluated in comparison to children enrolled in the regular Children's Medicaid Program and the patients who refused to be included in the study to see if there was any difference between the groups. Performance was measured over 29 months.
  Results: There was no difference in the rates of compliance with well-child care recommendations between the augmented fee-for-service group and the control group. Emergency room visit rates and hospitalization rates also were not significantly different.
• Fairbrother G, Siegel MJ, Friedman S, Kory PD, Butts GC. Impact of financial incentives on documented immunization rates in the inner city: results of a randomized controlled trial. Ambul Pediatr. 2001;1(4):206–212.
  Setting and Design: This study took place in New York City, NY from July 1997 to December 1998. Incentives were offered to individual inner-city physicians with the highest rates of poverty and proportions of Medicaid-enrolled children among their patients to determine the effect of two financial incentives—bonus and enhanced fee-for-service—on documented immunization rates during a second period of observation. Physicians assigned to the bonus with feedback group (n=24) could receive $1000 and $2500 for improvements in immunization rates of 30% and 45% from baseline, $5000 for reaching 80% up to date (UTD) coverage, and $7500 for reaching 90% UTD coverage for immunizations against diphtheria and tetanus toxoids and pertussis vaccine (DTP), Haemophilus influenzae type b vaccine (Hib), polio vaccine, and rubella vaccine (MMR). The investigators also determined the percentage of visits in the past four months that were missed opportunities to immunize (MOI) and to increase the average number of vaccinations per child given on a date of visit versus no office visit scheduled. Physicians assigned to the enhanced fee-for-service group (EFF, n=12) received $5/vaccine administered within 30 days of coming due and $15/visit at which all vaccines were administered. The control group (n=21) received feedback on their performance with respect to lead, anemia and overall UTD screening and $100 for participation in a concluding interview. The incentives were given in 4-month intervals. Performance was measured over 16 months.
  Results: Overall UTD coverage increased in the two groups receiving financial incentives. UTD coverage improved significantly within the bonus group compared to the control between time 1 and time 3, and in the EFF group at time 4. The average number of immunizations recorded in the chart increased significantly for children in the bonus group between time 1 and time 2, but not for children in the EFF group relative to the control. The MOI for sick visits were high, ranging from 89–92% and did not change significantly for the EFF group, whereas they decreased significantly at time 3 for the bonus group relative to the control. Seventy-one percent of the visits were sick visits, thus a change in this category will have an overall effect.
• Hickson GB, Altemeier WA, Perrin JM. Physician reimbursement by salary or fee-for-service: effect on physician practice behavior in a randomized prospective study. Pediatrics. Sep 1987;80(3):344–350.
  Setting and Design: This study took place at the Vanderbilt University Pediatric Residents Continuity Clinic in Nashville, Tennessee from September 1983 to June 1994. Incentives were offered by the study to 18 medical residents. Nine were randomized to receive $2/patient visit and nine were randomized to a control group that received the expected average compensation of $20/month to determine the effect of augmented fee-for-service on physician behavior. Prior to data collection residents completed a questionnaire to monitor interest in outpatient practice and a variety of other questions. Performance was measured over nine months.
  Results: Due to the small sample size, randomization failed to equalize physician interest because the nine physicians in the control group were more likely to plan a career in private practice than the fee-for-service group. Fee-for-service physicians did not have significantly more patient visits, but fee-for-service patients experienced greater continuity of care (more often saw their regular physician when they came to clinic) and fewer ER visits than patients enrolled to salaried physicians. There were 22% more per capita visits by patients using fee-for-service than by patients with control physicians, almost entirely due to well-child visits. Although initial and follow-up visits for illness were not different, fee-for-service patients averaged 43% more well child visits.
• Hillman AL, Ripley K, Goldfarb N, Nuamah I, Weiner J, Lusk E. Physician financial incentives and feedback: failure to increase cancer screening in Medicaid managed care. Am J Public Health. Nov 1998;88(11):1699–1701.
  Setting and Design: This study took place in Philadelphia, PA in 1993-1995. Incentives were offered by a Medicaid managed care organization structured like an independent practice association with provider sites paid by capitation, to the largest primary care sites stratified by practice type (solo/group) to ensure sufficient representation of each. The randomly selected intervention sites (n=26) were eligible to receive a full bonus (20% of capitation) for the three intervention sites with the highest compliance scores, the three next highest scores and the three improving the most from the previous audit both received partial bonuses (10% of capitation). The in order to increase their rates of compliance in mammography, Pap smear, and colorectal screening for all female members fifty years of age and older. In addition to bonuses, the intervention group received feedback. The control group (n=26) received no intervention and no feedback. Bonuses ranged from $570 to $1260 per site. Chart audits were performed at baseline and every six months for 1.5 years
  Results: There was no significant difference between intervention and control groups by type of practice, specialty, or patient panel size. Baseline compliance scores were relatively low and did not differ significantly between study groups, although group practices had consistently higher compliance scores than solo practices. There was a significant improvement over time in performance for both intervention and control groups, but there was no significant difference between the groups. A subanalysis comparing aware and unaware intervention sites showed no significant between-group differences.
• Hillman AL, Ripley K, Goldfarb N, Weiner J, Nuamah I, Lusk E. The use of physician financial incentives and feedback to improve pediatric preventive care in Medicaid managed care. Pediatrics. Oct 1999;104(4 Pt 1):931–935.
  Setting and Design: This study took place in Philadelphia, PA in 1993-1995. Incentives were offered by a Medicaid managed care organization structured like an IPA with provider sites paid by capitation, to primary care physicians with at least twenty-five pediatric members younger than seven. After stratification by practice type (solo/group), the primary care sites were randomly assigned into one of three groups to assess whether feedback coupled with financial incentives could improve pediatric preventative care. The three arms included a feedback only group (n=17) where physicians received written feedback about compliance scores, a feedback and incentive group (n=19) where physicians received feedback and a financial bonus when compliance criteria were met, and a control group (n=17) with no feedback and no incentive. Preventive care guidelines were distributed to providers in all three study groups. Chart audits were performed for practice sites in all three groups at 6-month intervals. Eligibility for bonuses in the feedback and financial incentives group was based on a total compliance score of 20% for each indicator. The three sites with the highest total compliance received a full bonus (20% of the sites total 6-month capitation for pediatric members less than seven years of age). The three next best scoring sites received a partial bonus (10% of the sites total 6-month capitation for pediatric members less than seven years of age). The three sites showing the most improvement from the last audit the partial bonus if their total compliance score increased by at least 10%. Performance was measured at baseline and every six months for 1.5 years.
  Results: Bonuses paid out during the course of the study ranged from $772–$4682 with an average of $2000. Thirteen of nineteen sites received a bonus. At baseline no significant differences were observed. Compliance with pediatric preventive care improved dramatically in the study period. Repeated measures analysis of variance demonstrated a significant increase in all three study groups throughout the time in total compliance (56%–73%), as well as scores for immunizations (62%–79%) and other preventive care (54%–71%). However no significant differences were observed between the intervention groups and the control, nor were there any interaction (group-by-time) effects.
• Kouides RW, Bennett NM, Lewis B, Cappuccio JD, Barker WH, LaForce FM. Performance-based physician reimbursement and influenza immunization rates in the elderly. The Primary-Care Physicians of Monroe County. Am J Prev Med. Feb 1998;14(2):89–95.
  Setting and Design: This study took place in Rochester, New York and surrounding Monroe County from September 1991 to January 1992. Incentives were offered by the Medicare Influenza Vaccine Demonstration Project to providers or group practices who provided primary care to at least fifty patients sixty-five years and older, participated in the Medicare Demonstration Project, and used target-based poster method for tracking immunizations. Physicians were randomized by practice group to the control (n=27) or the incentive group (n=27), which was eligible for reimbursement above the standard $8 fee per immunization if immunization rates above 70% or 85% were achieved. If a final immunization rate of 70% was attained, the physician received an additional 10% reimbursement—$.80/shot given in the office. If a final immunization rate of 85% was attained, the physician received an additional 20% reimbursement—$1.60/shot given in the office. Immunizations given outside the office were included in the percent immunized, but were not given the incentive. Performance was measured over three months.
  Results: At baseline there were no statistically significant differences between the control and incentive groups. The median change in immunization rate was significant (10.3%) in the incentive group and not significant (3.5%) in the control group. In the incentive group, 52% of practices attained the 70% immunization target level, with 15% attaining the target level of 85%. In the control group, 44% of practices attained the 70% immunization target level, with 7% attaining the target level of 85%. Individual physician performance within group practices was quite variable.
• Roski J, Jeddeloh R, An L, et al. The impact of financial incentives and a patient registry on preventive care quality: increasing provider adherence to evidence-based smoking cessation practice guidelines. Prev Med. Mar 2003;36(3):291–299.
  Setting and Design: This study took place in the Minnesota from May 1999 to December 2000. Incentives were offered by the Allina Health System to forty clinics providing primary care service (family practice, internal medicine, obstetrics/gynecology) in a large multispecialty group practice to improve performance. The three experimental conditions were represented by financial incentives for reaching preset clinical performance targets combined with access to a centralized smoker registry and intervention system (Incentive + Registry group, n=10), financial incentives for reaching preset clinical performance targets (Incentive group, n=15), and no intervention except the distribution of printed versions of the smoking cessation guidelines (Control group, n=15). The two clinical performance targets were 75% of adult patients having their tobacco status clearly identified at each visit and documented in their medical records and 65% of smokers having received ongoing in-office counseling (measured as advice to quit given at last visit). Actual smoking cessation rates were a secondary endpoint. Incentive amounts were based on the number of providers per clinic. Clinics with one to seven providers could receive $5000 and clinics with eight or more providers were eligible for a $10000 bonus. Clinics who reached or exceeded only one of the two performance goals were eligible for half the incentive. The Incentive + Registry group received weekly updates on their referral activity during the past week and their referral activity to date. The Incentive + Registry group was able to compare the referral patterns of their site to other clinics. Performance was measured over nineteen months.
  Results: At baseline no differences were found between the groups. Identification of patients' tobacco use status statistically significantly improved in all groups but was statistically significantly higher in the two incentive groups (14.4% in the Incentive group and 8.1% in the Incentive + Registry group vs. 6.2% in the control group). However, ongoing in-office counseling and actual quit rates did not differ significantly between the groups.

Results of Randomized Controlled Trials of Reputational Incentives

There was only one randomized controlled trial of reputational incentives.⁶¹ This study showed that hospitals with low performance scores were more likely to engage in quality improvement activities. This was especially true for hospitals whose performance was released to the public (as opposed to being kept confidential). As this is the only study of this type, we include a synopsis of it below:

• Hibbard JH, Stockard J, Tusler M. Does publicizing hospital performance stimulate quality improvement efforts? Health Aff (Millwood). Mar-Apr 2003;22(2):84–94.
  Setting and Design: This study took place in Wisconsin and concluded in May 2002. The study evaluates the impact of a public hospital performance report on subsequent hospital quality improvement efforts. The report on hospital safety was produced and disseminated by the Alliance, a large employer-purchasing cooperative in the Madison, Wisconsin area. The report, Quality Counts, compared the performance of twenty-four hospitals in south central Wisconsin. Two summary indices of adverse events (deaths and complications) occurring within the broad categories of surgery and non-surgery were included, along with indices in three individual clinical areas: hip/knee surgery, cardiac care, maternity care. Hospitals were rated as better than expected (fewer deaths and complications), as expected, or worse than expected. The primary intervention group was the twenty-four hospitals in south central Wisconsin in the Alliance service area. These hospitals were in the public report, were not randomly selected, and received a more detailed report on their performance. The other ninety-eight hospitals in Wisconsin were randomly assigned to either the secondary intervention that received a private report on their performance or the control condition that did not receive anything.
  Results: On average, public, private, and no report hospitals were slightly negative about the idea of publicizing hospital performance. There were statistically significant differences among the respondents toward the validity of the public report, its appropriateness for the public's use, and its value for quality improvement. Public report hospitals were most negative and those with private reports were most positive. Low-scoring public-report hospitals show the highest level of quality improvement activities, the private-report hospitals an intermediate level, and the no report hospitals the lowest level and the differences among the hospitals in the three study conditions were statistically significant. Most of the hospitals were optimistic that they could improve their scores through attention to quality improvement within the next two years.

Ongoing Randomized Controlled Trials

We identified no currently ongoing randomized controlled trials of QBP strategies from any funding source.

Interventional Trials With Non-Randomized Designs

There were 18 ongoing research projects in which there was a QBP intervention without randomization (Tables 11 and 12). For many of these, the exact nature of the performance measures and the incentive were still being determined. For some, the study design is observational; that is, health plans are making decisions about incentives without input from the investigators, but the investigators are assessing the response.

Table

Table 11. Ongoing quality-based purchasing research: Projects in the Rewarding Results initiative.

Table

Table 12. Ongoing quality-based purchasing research: Other QBP projects.

The single largest initiative is Rewarding Results, which has components funded by the Robert Wood Johnson Foundation, the California HealthCare Foundation, and AHRQ. Therefore, we first list the Rewarding Results projects (Table 11), then list separately other ongoing QBP research (Table 12).

Topics covered. These projects will provide some important additional information about QBP. Several studies (particularly those by Rabson et al. and Epstein et al.) will describe the type and frequency of use of QBP strategies. Several projects (most of the Rewarding Results projects, plus Young et al., Braun et al. and Callahan et al.) will investigate provider reactions to incentives in terms of willingness to participate in programs and awareness of the incentives offered. In addition, Braun et al. and Young et al. will obtain quantitative and qualitative information about attitudes towards incentives used and performance targets set (such as salience, clinical validity, and whether the performance measures were within the providers' scope of control). These studies may be useful for understanding providers' motivation to respond and organizational decisionmaking when incentives are offered. Still other projects (particularly Sofaer et al.) will report on the tools used to communicate incentives, rather than the provider or consumer response to the incentive.

Quantitative assessments of the impact of incentive interventions. The Rewarding Results projects and several others (Delbanco et al., Rosenthal et al., and Epstein et al.) will provide assessments of the impact of incentives on traditional performance measures of structure, process, and outcomes. While none of these are randomized and all involve organizations that self-select to adopt or participate in incentive programs, taken together they will provide preliminary evaluations of QBP in Medicaid, Medicare, and commercial insurance settings and will cover many different approaches to incentives. For instance, in the Integrated Healthcare Association project alone, the health plans have adopted financial incentives that vary in structure from increases in capitation to augmented payments per encounter (and also range widely within these approaches; e.g., there is greater than two-fold variation in the magnitude of the capitation increase available across plans). One of these studies (Rosenthal et al.'s “Determining Whether Pay-for-Performance Incentives Improve Health Care Quality in Medical Groups”) will investigate whether the provision of incentives for specific indicators also lead to improvement in domains of performance that are not included in the incentive measure set (or, alternatively, worsening in these measures if the non-incentivized areas of performance are subsequently neglected). In addition to these that are ongoing, the Centers for Medicare & Medicaid Services has a project in development with Premier Healthcare Informatics that will include both financial and reputational incentives for hospitals. While the dissemination of performance data has already begun, the evaluation plan for this project is still under development (see: www.cms.hhs.gov/researchers/demos/phqidemo.asp).

Among the interventional studies, there are also some major differences in the characteristics of the incentives themselves between the prior literature and the ongoing research. For instance, the ongoing studies involve actual health plans or government programs making an ongoing commitment to an incentive strategy, rather than a researcher making a short-term payment intervention (which was the situation in the prior studies). Similarly, all the studies included in the literature review above involved incentives directed at only a small number (usually just one) performance indicator for a single condition or type of patient. However, all the ongoing interventional studies we identified involve multiple measures (often ten or more) across a variety of conditions and distinct patient populations. Both these factors—that the incentive comes from a health plan or government program that expresses a longer-term commitment to the strategy and that there are multiple indicators—may should increase the probability that providers will believe that investments in quality improvement (such as installing a new information system) can be recouped relative to previously studied incentive strategies.