An official website of the United States government
NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
D.J. Wilkie, PhD, RN, FAAN, Y. Yao, PhD, M.O. Ezenwa, PhD, RN, A. Gill, T. Hipp, MBA, BSN, RN, R. Shea, BA, J. Miller, MPH, K. Frank, MS, RN, N. Nardi, BSN, RN, CHPN, M. Murray, MS, RN, CHPN, J. Glendenning, APRN, CHPN, J. Perez, RN, J. Carrasco, BS, D. Shuey, BA, Z. Zhao, PhD, V. Angulo, BA, M.L. Suarez, PhD, T. McCurry, MD, J. Martin, MD, A. Butler, MD, Z.W. Wang, PhD, and R.E. Molokie, MD.
Author Information and AffiliationsUnrelieved cancer pain at the end of life is a major health problem that interferes with patient-centered goals for their last days. Building on a successful approach researched in outpatient oncology settings, we tested a system-level intervention (PAINRelieveIt) that included an internet-based tool with patient-reported pain outcomes (in English and Spanish languages), decision support for clinicians (English), and multimedia education tailored to each cancer patient and lay caregiver (in English and Spanish languages).
Specific aims were to compare usual hospice care with PAINRelieveIt groups for effects on (1) patient outcomes (analgesic adherence [primary outcome]; worst pain intensity; satisfaction; and misconceptions about pain and pain management) and lay caregiver outcomes (misconceptions about pain and pain management, known as pain control barriers) in a diverse sample of cancer patients receiving hospice care and their lay caregivers; and (2) nurse outcomes (obtained appropriate analgesics for patients) in a sample of hospice nurses.
Using a 1-week pretest/posttest, stepped-wedge randomized design in 234 patients receiving home-level hospice care provided by 2 Chicago-area hospices and 231 of their lay caregivers, we compared effects of usual hospice care and PAINRelieveIt on pain outcomes. The average age of the 234 patients was 68.4 (SD, 14.0) years; 49% were male and 51% were from ethnic or racial minorities. The 231 caregivers were younger (mean, 53.2 years, SD; 15.0 years) and predominately minority (54%) and female (74%). The tablet-based PAINRelieveIt included valid and reliable pain tools (PAINReportIt, which includes analgesic adherence in the past 24 hours, and Barriers Questionnaire-13 [BQ-13]); a summary of the patient's pain data (PAINReportIt Summary); a summary of the patient's pain data with decision support for hospice nurses to obtain recommendations for algorithm-based analgesic therapies (PAINConsultN); and multimedia education tailored to the patient's and lay caregiver's misconceptions about pain and pain management (PainUCope). Patients' and caregivers' answers were automatically stored in an electronic database, from which the system generated a PAINReportIt Summary (control group: PAINConsultN and experimental group: PAINUCope). All patients received usual hospice care. All patients/caregivers completed PAINReportIt and BQ-13 at pretest and 1 week later (posttest); patients also completed parts of both tools daily. Via daily email updates, the control group hospice nurses received access to a PAINReportIt Summary and experimental group hospice nurses received access to a PAINConsultN. Additionally, experimental group patients/caregivers were asked to view multimedia educational materials via PAINUCope to help patients report pain and adhere to prescribed analgesics (analgesic adherence from PAINReportIt).
At posttest, neither analgesic adherence nor satisfaction with pain level differed significantly between the usual care group and experimental group. The posttest worst pain intensity was 6.05 ± 2.24 for the control group and 6.63 ± 2.12 for the experimental group, which was statistically significantly higher for the experimental group; the difference (0.70 [95% CI, 0.12-1.27]) is not clinically meaningful. There was nearly universal availability of prescriptions for strong opioids and adjuvant analgesics for neuropathic pain in both groups. Lay caregivers' pain control barriers (P = .01) were significantly lower in the experimental group compared with the usual care group (mean difference controlling for baseline is 0.38; 95% CI, 0.08-0.67), but the patients' pain control barriers did not differ significantly by study group.
Unanticipated imbalance in study group size occurred given the ever-changing clinical environment of hospice care. Other limitations included the nurses' low access to the intervention reports for their patients and the ceiling effect for appropriateness of the analgesics for type of pain that may have occurred because medications were placed in the home at admission for use during active dying.
Over the 1-week study, both intervention groups had high availability of appropriate prescriptions, yet in both groups a high proportion of patients were not satisfied with their pain levels, reporting moderate to severe worst pain, and were not completely adherent to their opioid analgesics (primary study outcome). Better understanding of the lack of change in the patient outcomes is important for improving patient-centered care of the patient with cancer who is dying.
Unrelieved cancer pain at the end of life is a major health problem that interferes with achieving patient-centered goals for the last days. Pain is a frequent companion of the nearly 2 million Americans diagnosed with cancer annually (1.7 million cases expected in 2017).1 In fact, pain has been reported as the most feared cancer symptom.2 Pain is particularly challenging not only for the estimated 600 920 Americans who will die from their cancers in 20171 but also their lay caregivers. Recent findings from 4 hospitals indicate that only half of dying hospitalized patients achieved their desired pain control outcomes.3 Unrelieved cancer pain is distressing to the patient and family, diminishes the dying patient's quality of life, interferes with life-closure activities, and is a common reason for hospice admissions. Although we used numerous approaches to improve cancer pain management,4-9 effects are unknown for system-level changes that utilize the internet to improve pain assessment and management for hospice patients with cancer. Recent availability of PAINRelieveIt, an internet-based program designed to include prior innovations focused on improving cancer pain management, offers an opportunity to improve pain control. The purpose of this study was to compare usual hospice care and PAINRelieveIt groups for effects on patient, lay caregiver, and hospice nurse outcomes.
Hospice care is known for managing pain during the end-of-life transition, but many (27%-85%) hospice patients with cancer have inadequately controlled pain.10-12 Hospice patients with cancer commonly experience pain at higher levels (87%) than the general hospice population,13 and about 10% of patients are admitted with severe, uncontrollable pain.14 Unfortunately, McMillan11 found that, even after 3 weeks of hospice care, many cancer patients (42%) reported that less than half of their pain was relieved. Others15 noted similar inadequate pain control in a study of 1306 patients receiving services from 13 hospices across the United States. For too long, pain control has been inadequate for the patient who is dying, and a new paradigm is desperately needed.
Our recent findings from 2 hospices provide important insights for improving pain control of dying patients with cancer.16 Like others, we found that the average worst pain intensity during the previous 24 hours was severe. Unlike others, though, our surprising finding was that 90% of the patients had appropriate analgesic prescriptions based on worst pain intensity and nearly all had appropriate analgesic prescriptions based on current pain intensity. Appropriate for their pain level, opioid analgesics were readily available to all patients, but of the 62% of 192 patients who may have had neuropathic pain (due to neural tissue damage or alteration)—which may respond to adjuvant drugs but is less responsive to opioids—only 21% had adjuvant analgesics prescribed that could have been appropriate for their type of pain. In addition to this gap in analgesic prescribing, we found that, on average, the patients were not consuming all of their available analgesics.16 There is an enormous gap between the scientific evidence (calling for adjuvant analgesics for neuropathic pain) and the usual hospice pain care that devotes insufficient attention to assessing and prescribing for neuropathic pain and to the urgent need to help patients adhere to prescribed analgesics to improve pain outcomes in hospice. Fortunately, the PAINConsultN intervention delivered to the hospice nurses could provide the patient assessment data and clinical decision support information that would lead hospice physicians to prescribe adjuvant analgesics for patients. The PAINUCope intervention for patients and lay caregivers could improve adherence to prescribed analgesics by reducing misconceptions about pain and pain treatments. Together, the 2 interventions (with a PAINReportIt tool and collectively known as PAINRelieveIt) have the potential to address the gap and improve pain control for hospice patients with cancer.
It is well known that cancer pain can be adequately controlled for most cancer patients (85%-95%) with appropriate pain assessment and judicious use of existing opioid, nonopioid, and adjuvant analgesic therapies, but in many clinical practice settings this outcome remains elusive.17-21 A complex set of pain management behaviors is needed to translate these findings to the hospice setting. First, the hospice nurse must continuously and systematically measure the multiple pain dimensions and adequately communicate the data to the hospice physician, who will recognize the need for appropriate analgesic prescriptions given the patient's specific type of pain. Second, patients and lay caregivers must have sufficient understanding about analgesics and treatment of adverse effects to enable the patient's adherence to the therapies. These conditions for successful pain control are complex and involve adequate knowledge, attitudes, and behavioral skills (potential pain control barriers) among a triad including the dying patient, lay caregiver, and nurse/hospice physician team.
Considering the complexity of cancer pain management in hospice settings, we based the theoretical rationale for the study on pain as a multidimensional phenomenon,22 which is an extension of the Gate Control Theory of Pain23 with its recognition of the sensory, affective, cognitive, and behavioral contributions to pain transmission and inhibition within the spinal cord and brain. Current understanding of pain mechanisms also recognizes the important contributions of the peripheral and autonomic nervous systems to pain and its control.24,25 The complexity of cancer pain mechanisms often requires multidimensional cancer pain measurement26 and analgesic therapies inclusive of opioid (eg, morphine), nonopioid (eg, nonsteroidal anti-inflammatory agents), and adjuvant (eg, tricyclic antidepressants, gabapentin, haloperidol) drugs.
Management of cancer pain as a multidimensional experience in the hospice system context, however, is affected by patient-level, lay caregiver–level, and nurse-level barriers. Physician-level barriers might also affect pain control in hospice, but we recently found that 90% to 96% of hospice patients with cancer had appropriate nonopioid and opioid analgesic prescriptions,16 an important finding that guided our focus on the hospice nurse. The nurse is the key health professional in contact with hospice patients with cancer as they die in their homes. Our findings indicate that the nurse's focus on pain intensity drives the cancer pain management process, which means that opioids are readily available in the home; however, cancer pain is more complex than its intensity and changing the focus beyond pain intensity requires helping the nurse recognize the need for adjuvant analgesics. Since typical hospice pain assessment forms include few items about neuropathic pain and patients lack the language to describe their pain without a word-list aid, this gap may be part of the problem of inadequate pain control in hospice. Fortunately, the PAINRelieveIt intervention gives the appropriate pain language to patients and provides decision support to the hospice nurse regarding adjuvant analgesic prescriptions (drugs, dosages, intervals, escalation parameters) as just-in-time information. The hospice nurse can then provide follow-up and communication of new information to the hospice physician.
Also, we based the behavioral component of our research on the Johnson Behavioral System Model,27 which supports the notion that the interventions (computer programs) could teach or model new behaviors when patients', lay caregivers', and nurses' behavioral choices are limited or informed by misconceptions.28 Consistent with the Johnson Behavioral System Model, our interventions are intended to alter the behavioral choices of patients and lay caregivers (PAINReportIt,5,6,9,29-31 PAINUCope4,7,8) as well as nurses (PAINReportIt, PAINConsultN9). If the behavioral choices are enacted as indicated in the conceptual model (Figure 1), then the outcomes will be produced by virtue of theories related to pain mechanisms, pharmacokinetics, and pharmacodynamics, which underpin the PAINConsultN intervention. The PAINUCope intervention, on the other hand, is intended to help the patient learn about therapeutic options that are available to control the type of pain he or she is experiencing and overcome his or her barriers to pain control (eg, engage in new behavioral choices that promote pain control).27
Conceptual Model: Computerized Intervention (PAINRelieveIt) for Patient–Caregiver Dyads and Nurses to Promote Relief of Cancer-Related Pain in Hospice Settings.
Barriers to adequate control of cancer pain have been well known for more than 2 decades. Unfortunately, these barriers have been difficult to overcome because of their complexity, including (1) patients' reluctance to report pain,32 (2) patients' hesitation to take analgesic medications,33 (3) inadequate providers' pain assessments focused nearly exclusively on pain intensity but typically not on other pain dimensions,34-36 and (4) providers' lack of knowledge about a vast number of pain management options and especially when to use specific adjuvant analgesics.35 Some scholars view these barriers as psychological, sociological, or moral issues. We view them as misconceptions that can be addressed by education and behavioral choices that promote pain control. Other researchers have studied educational interventions to overcome one or more of these barriers and produced improvements in patient outcomes within an institution,37-39 at home,40-42 or across multiple institutions in selected populations.8 Also, investigators have demonstrated improvements in provider outcomes, such as knowledge and attitudes.43,44 Unfortunately, these interventions have not been replicated, have not been amenable to adoption in practice settings, or have shown no evidence that the outcomes were sustained beyond a short period. One reason that these educational interventions have not been translated into practice may be related to the high cost of one-on-one delivery. Our intervention, however, has potential to overcome the gap related to these real-world translation issues because it uses computer technology and processes that would be highly scalable in practice settings, such as hospices, as was confirmed by stakeholders in our study planning sessions.
Based on the prior research4-9,12,16,30,45,46 and available technological innovations offered by the internet, the specific aims of the study were to compare usual hospice care with PAINRelieveIt groups for effects on (1) patient outcomes (analgesic adherence [primary outcome]; worst pain intensity; satisfaction; and misconceptions about pain and pain management) and lay caregiver outcomes (pain misconceptions) in a diverse sample of cancer patient–caregiver dyads receiving hospice care; and (2) nurse outcomes (obtained appropriate analgesics for patients) in a sample of hospice nurses.
We hypothesized that at posttest, compared with the usual care group, the PAINRelieveIt group would (1) report decreased scores for worst pain intensity and pain misconceptions, (2) have increased analgesic adherence (primary outcome), and (3) have a larger proportion who reported satisfaction with pain intensity and whose nurses obtained prescriptions from the hospice physicians for appropriate analgesics for the patients' pain.
We engaged physician, nurse, and patient stakeholders through regular planning meetings in the initial design and planning of this study, and in determining how to conduct the study throughout the entire study period and dissemination phase. The study planning and implementation built on a long history of collaboration among the researchers and both hospices. The patient stakeholders were hospice volunteers with many years of practical experience working with hospice patients and, most important, had been caregivers of hospice patients. Engaging with the clinician and patient stakeholders helped circumvent some of the challenges encountered regularly while approaching and trying to recruit hospice patients and their caregivers into the study. The patient stakeholders offered input about approaching the patients and caregivers and strategies to retain the study participants. The nurse stakeholders ensured that the hospice researchers (HRs) had access to weekly interdisciplinary team meetings during which new patients and potential study participants were discussed. The nurse stakeholders were instrumental in ensuring that hospice nurses received training on study procedures and knew how to access and interpret the clinician reports generated by the PAINRelieveIt program, and they allowed hospice nurses to participate in the study as part of their regular work flow. All the stakeholders participated in dissemination efforts. Patient, nurse, and physician stakeholders have all been presenters or co-presenters at various conferences and are coauthors on 2 published articles.47,48
The study's inclusion criteria required that the patient (1) be admitted to home care level of hospice service; (2) have a diagnosis of cancer; (3) have experienced worst pain in the previous 24 hours of ≥3 on a scale of 0 to 10; (4) be able to speak, read, and write English or Spanish; (5) be 18 years of age or older; (6) have a lay caregiver who was 18 years of age or older and willing to participate; and (7) have a life expectancy of at least 10 days at the time of study enrollment, as suggested by a Palliative Performance Scale (PPS) score of 30 or higher. Many patients with cancer join hospice very close to death, and we used a PPS of 30 as the minimum score to try to ensure that they could complete the 1-week study. The inclusion criteria for the lay caregiver required that the person (1) be designated as the primary lay caregiver for the participating patient during the study period; (2) speak, read, and write English or Spanish; and (3) be 18 years of age or older. Patients and lay caregivers were excluded if they (1) were legally blind or deaf, or (2) had cognitive or physical impairments that would make it impossible to communicate or to complete study instruments at enrollment or during the study.
All patients admitted to the participating hospices with a cancer diagnosis were screened for inclusion and exclusion criteria by a researcher who was also a hospice employee. All patients who fulfilled the criteria were approached for participation. Since all admitted cancer patients were screened according to specific criteria, and approached if deemed eligible, we made every effort to avoid selection bias. We routinely identified, recruited, and worked to retain study participants representative of the spectrum of the population of interest and always ensured that data were collected thoroughly and systematically from all study participants. One of the participating hospices served a large population of underrepresented minorities, which facilitated their inclusion.
As Figure 2 shows, we screened a total of 3533 cancer patients—1177 were eligible to participate, 262 consented to participate and enrolled in the study, and 28 either did not complete the baseline or died within 7 days of the baseline and were thus excluded from the primary analysis. Of the 2356 dyads that did not meet eligibility criteria, the number excluded for each inclusion and exclusion criterion is detailed in Figure 2. We did not keep an exact count of the reasons for declining to participate because in many cases the patients or caregivers simply refused to meet with the researcher, but when reasons were stated they included (1) the caregiver not wanting the patient to participate, thinking it would be too much for the patient; (2) the patient being overwhelmed, anxious, not feeling comfortable using computers, or having too many other things going on at the end of life, like time dedicated to family and friends; or (3) the patient not being interested. Other reasons for patients being excluded involved researchers being unable to meet patients for enrollment before their health declined too much or they died. Inability to meet with potential participants was due to the patient feeling ill or too tired to meet with the researcher, the patient having other commitments, or inability to schedule a visit with both the patient and caregiver at the same time.
PAINRelieveIt Hospice Study of Cancer Patient–Caregiver Dyads: CONSORT Flow Diagram.
Among the 28 enrolled but excluded patients, 5 provided only a minimal amount of data at baseline and no outcome data, meaning that their exclusion would have minimal impact on analysis; and 23 patients died within 7 days of the baseline, the time frame for our outcome measures. They were excluded from the primary analysis that relies on multiple imputation because imputing patient outcomes missing due to patient death is not meaningful. Sensitivity analysis including all 262 patients shows that excluding these 28 patients did not change our conclusion. Table 1 presents patient characteristics with comparisons for the analytic and excluded samples. We found no significant difference between the analytic and excluded samples.
Patient Characteristics for Included and Excluded Samples (N = 262).
The analytic sample of 234 patient participants included 115 (49%) men and 119 (51%) women who were about 68 years of age on average, with a median age of 67 (Table 1). The sample was ethnically and racially diverse; 18% were Hispanic and 51% were racial minorities (Table 1). Patient (Table 2) and caregiver (Table 3) characteristics were not significantly different between the control and experimental groups.
Patient Characteristics: Comparisons by Control and Experimental Groups (N = 234).
Caregiver Characteristics: Comparisons by Control and Experimental Groups (N = 231).
We conducted the study in the private residences of cancer patients receiving home-level care from Rainbow Hospice and Palliative Care and JourneyCare (formerly Horizon Hospice and Palliative Care), 2 Chicago-area nonprofit hospice programs. Horizon was Chicago's first hospice, beginning service in 1978, while Rainbow began shortly thereafter in 1981. At the time of the study, the payer mix for JourneyCare was 72% Medicare, 16% Medicaid, 9% commercial insurance, and 3% other. The Rainbow payer mix for was 93% Medicare, 3% Medicaid, 3% commercial insurance, and 1% private pay. For cancer patients, both hospices had a median length of stay of 19 days, which was typical of the national data. A total of 831 adult hospice patients were admitted to JourneyCare the year before the study began, and 58% of the patients had a diagnosis of cancer (n = 482). Rainbow admitted 2031 patients in 2011, and 37% had a cancer diagnosis (n = 751). Therefore, annually the 2 hospices were expected to have 2862 patients available for screening and 1233 to have cancer. For care delivery, both hospices were divided into teams with a physician, nurses, home health aides, a social worker, a chaplain, and volunteers. Rainbow had 4 teams and JourneyCare had 3 teams during the study. Assignment of a patient to a particular team was according to geographic location of the patients. Typical for hospice care, the number of cancer patients seen by each team varied due to different factors: for example, different populations in the geographic areas seen by each team.
This study was a 7-day randomized clinical trial (RCT) of patients receiving home-level hospice care and their caregivers. The trial included pretest/posttest measures to determine the effect of the PAINRelieveIt interventions on selected health care outcomes, as well as data collection of pain, symptoms, and medications on a daily basis. The study was approved by the institutional review boards at the University of Illinois at Chicago (UIC) and the University of Florida.
We reduced the number of teams from 10 to 7 between grant submission and initiation of the study. Therefore, most patients from minority groups would have been assigned to 1 team, which prompted change from the planned simple cluster-randomized design to a specific type of cluster randomization, a stepped-wedge randomization design.49 This change was necessary because we considered it unethical to randomize a team with mostly minority patients to a control group for the duration of the study.
Therefore, we implemented a stepped-wedge randomization design (Figure 3) in which all the clinical teams from both hospices, 7 in total, contributed both control and experimental subjects. In the first step, at the beginning of the study, all 7 teams were assigned to the control condition (usual care). In the second step, new patients enrolled from 1 team from each hospice (2 teams in total) were assigned to receive the experimental intervention and patients from these teams continued to receive the intervention until the end of the study, while the other 5 teams continued to receive the control condition. At the third step, new participants from 2 additional teams, 1 from each hospice, began to receive the experimental condition, while the other 3 teams continued to receive the control condition. This continued until all 7 teams were in the experimental condition. At each step, the teams transitioning into the experimental intervention were randomly chosen. The study statistician, who had no patient contact or role in data collection, completed the randomization using a random number generator and shared with the study trainers the teams that were to be transitioned to the experimental condition just before training occurred for the nurses on the randomly selected teams. Other team members, including the recruiters and the data collectors, were blind to the team randomization status until after all data were collected for the last study participant and data cleaning decisions were finalized.
Stepped-Wedge Study Design.
The usual home-level care at each of the hospices was provided by teams comprising hospice nurses, nursing assistants, home health aides, pharmacists, dieticians, social workers, therapists (speech, physical, occupational), bereavement counselors, chaplains, physicians, and volunteers as well as clinical and administrative support staff members. An essential part of every hospice team was the patient; his or her primary physician; and lay caregivers, who were often family members. The professional staff members at both hospices provided usual care with priority attention to pain assessment and management. The goal was for patients to have pain control within 72 hours of admission, but our recent findings indicate that goal was not always met. After the hospice nurse completed the initial pain assessment, he or she consulted with the hospice medical director or the patient's primary medical provider to obtain prescriptions for the analgesics, which typically included opioid (eg, morphine, fentanyl), nonopioid (eg, nonsteroidal anti-inflammatory agents), and adjuvant (eg, tricyclic antidepressants, gabapentin) drugs. The hospice nurse obtained needed changes in the prescriptions over time by consulting with the hospice medical director or the patient's primary medical provider. The nurse provided all cancer patients an emergency medication pack for as-needed use; it included immediate-release opioids (eg, morphine) for uncontrolled pain and medications for agitation (eg, haloperidol, which also has documented adjuvant analgesic properties).50,51 Our recent findings indicate that in these 2 hospices, the opioid prescriptions were highly appropriate for the patients' pain (90% of the patients had appropriate analgesic prescriptions based on worst pain intensity, and 96% had appropriate analgesic prescriptions based on current pain intensity). The cost of all analgesic medications was included in the hospice benefit, and the hospice program arranged for the analgesics to be delivered to the patient's home. Therefore, access to medications at home was not a barrier to pain control for hospice patients with cancer.
After baseline data collection, we used Nursing Consult LLC's PAINRelieveIt software,6 which includes (1) PAINReportIt with screens to collect pain, medications, and misconception data (both groups); (2) the intervention for the nurse clinicians, a summary of the patient's pain data, PAINReportIt Summary (control group) and PAINConsultN (experimental group); and (3) the intervention for the patients and lay caregivers, PAINUCope (experimental group). This innovative program was the first computerized, multidimensional, self-report measure of pain with clinician decision support for analgesic prescriptions and multimedia patient education tailored to the patient's misconceptions and pain.
PAINReportIt, a computerized version of the McGill Pain Questionnaire (MPQ), allowed the patient to share his or her expertise about how the pain felt; only the patient knew this information, but he or she may have lacked the language to share it with others. In the busy practice environment, most clinicians lacked sufficient skill to quickly coax the details about the pain that would help them to prescribe the appropriate pain medications. PAINReportIt guided the patient to report this information to the computer and then the computer summarized it for the clinician's expert interpretation regarding needed analgesics.
PAINReportIt also measured the patient's misconceptions about pain and its management. Based on the patient's pain and misconceptions, the tailored, multimedia education (PAINUCope) was intended to reduce misconceptions, which would help the patient adhere to prescribed analgesics.
Via the PAINConsultN component, the nurse received the pain report data along with suggestions for therapies that were consistent with the patient's pain and based on clinical guidelines for managing pain and adverse effects.21,52-58
For this community-based hospice study, the secure link to PAINConsultN was transmitted to the hospice nurse via an email that allowed him or her to click the link to a secure server and supply a password to review the recommendations and changes in the patient's pain. The nurse, as part of usual practice, was encouraged to validate the pain information and communicate with the hospice physician about prescriptions to improve the patient's pain control. In prior research,5-7,9,12,45 PAINRelieveIt was implemented with patients and oncologists or oncology nurse practitioners and paper PAINConsultNs. Such a delivery system was inconsistent with the geographical challenges of home-based hospice care. Fortunately, advances in wireless internet access allowed electronic delivery that fitted better with the hospice care delivery system.
Furthermore, in the home hospice environment, lay caregivers play important roles in pain management, especially as patients' health status declines. Considering the impact of lay caregivers on cancer pain control was an important focus for patient outcome research.42,59 We modified PAINReportIt to measure the lay caregiver's pain and pain management misconceptions. We also modified PAINUCope to allow independent or joint viewing by the patient and lay caregiver to enable the dyad to be educated together about behaviors more likely to lead to pain control, such as supporting the patient's adherence to the pain therapies.
The HR contacted the patients and caregivers who were deemed eligible to schedule a home visit, explained in detail the study procedures, obtained informed consent for those patient–caregiver dyads interested in joining the study, and obtained baseline measures for consented dyads. The HR, while remaining blind to study condition (control or experimental), trained the participants on the daily data collection procedure, which was identical for both study conditions. Participants assigned to the experimental condition received access to PAINUCope, while control subjects had access to computer games. After baseline data collection (day 0), daily data were collected for another 6 days, followed by posttest (day 7). A research specialist (RS) from UIC monitored the daily data collection and telephoned subjects who had not completed the daily data collection to remind them to complete it, provide assistance as needed, or collect the data over the phone if the patient preferred that option. In this case, the RS entered the patient's reported data directly into the program. However, sometimes patients felt too ill to complete the daily data, or they were unreachable by phone. At the end of the study week, the HR returned to the home to obtain posttest measures. Following these procedures, the hospice patients reported information about their current pain and least and worst pain in the past 24 hours, while both the HR and UIC RS ensured that data from the subjects were collected thoroughly and systematically. Data were collected using internet-enabled Samsung tablets, and were transmitted directly to the UIC College of Nursing secure network via the wireless internet connection.
We met monthly with the hospice physicians to discuss their concerns about the study. Also, the lead investigator met with the nurses on each team after the last team advanced to the intervention step to discuss their impressions and concerns about accessing the PAINReportIt Summary and PAINConsultN intervention and its content. Based on the nurses' feedback, we implemented additional monitoring and messages delivered to the nurses' cell phones if access did not occur by 3:00 pm each day. Additionally, we sent additional weekly reports to the nurses to communicate their success rate in accessing the PAINReportIt Summary and PAINConsultN intervention reports. This information was not communicated to the nurse managers.
The PAINReportIt software program was first developed with an electronic MPQ (1970 version)60 and included questions about analgesics used by the patient as well as a shortened version of the pain Barriers Questionnaire (BQ).33 PAINReportIt was designed as an interactive, touch-screen method for assessment of pain. It was designed for self-administration and required little or no patient computer experience and minimal or no provider time for administration, even for hospice patients with PPS scores of 40 or higher who completed it in 19 minutes, on average.61 Directions for self-administration allow the patient to read instructions on-screen and practice making all types of selection responses that are available in the program. Dr Ronald Melzack owns the copyright for the MPQ and authorized the modifications, presentation format, and computerized use of the MPQ. The following have been reported for the MPQ: concurrent (r = 0.31-0.40),62 predictive (67%-77%),63 and construct validity (3 factors)62,64,65; alternate forms (0.72)66,67 and test-retest reliability (0.70-0.90), and sensitivity.60,66 The MPQ is well known as a well-validated pain assessment tool68 that has been easily used by patients with cancer.4,69,70 Paper-and-pencil MPQ compared with the PAINReportIt previously was shown to be equivalent.9,71 Also, cognitive interview approaches demonstrated strong validity for PAINReportIt.30
We obtained 5 outcome measures from PAINReportIt data. The measures all have sufficient validity and reliability and use in the hospice population:
Our team's 20-year history of working collaboratively with patients and professional stakeholders in the planning and execution of our patient-centered research was a strong enabler of this study from its inception. Specifically for this study, we engaged physician, nurse, and patient stakeholders through regular discussion and planning meetings in the initial design and planning as well as in determining how to conduct the study throughout the entire study period, including dissemination of findings. Furthermore, the study planning and implementation built on a long history of collaboration among the researchers and both hospices. The patient stakeholders were hospice volunteers with many years of practical experience working with hospice patients and, most important, had been caregivers of hospice patients. In addition, one of the patient stakeholders had been totally paralyzed and was in severe ongoing pain and close to death as a result of Guillain-Barre syndrome, giving him direct experience of being a very ill patient with unrelieved pain. Engaging with the clinician and patient stakeholders increased the efficiency of the study by helping circumvent some of the challenges encountered while approaching and trying to recruit hospice patients and their caregivers into the study. The patient stakeholders offered input about approaching the patients and caregivers and strategies to retain the study participants. One example was the stakeholders' advice to develop a video showing previous participants' testimonials about their positive experience with the study and ease of using the tablet computer, to reduce some of the patients' and caregivers' concerns about study participation.
The nurse stakeholders ensured that the HRs had access to weekly interdisciplinary team meetings during which new patients and potential study participants were discussed. They were instrumental in ensuring that hospice nurses received training on study procedures and knew how to access and interpret the clinician reports generated by the PAINRelieveIt program for their patients participating in the study, and in allowing hospice nurses to participate in the study as part of their regular work flow.
All the stakeholders (patients, nurses, and physicians) regularly participated in dissemination efforts. Patient, nurse, and physician stakeholders have all been presenters or co-presenters at various hospice, palliative care, and oncology conferences. The stakeholders were also coauthors on 2 published articles47,48 and have presented at several meetings to report study implementation and baseline findings. During the implementation phase, all stakeholders were paid for their time and were valued as members of the research team.
The power of a stepped-wedge design depends on the difference between teams as measured by the intraclass correlation coefficient (ICC). With our study design of 7 teams over 5 stages with an average number of 7 dyads per team per stage, we expected the sample power to be lowest when ICC was 0.1. We analyzed the power of the proposed sample assuming a 0.05 level of type I error probability and a worst-case scenario of ICC = 0.1. In our previous hospice study of 110 patients,5 we found a mean adherence rate of 43.6% with an SD of 38.9%.16 Based on these values, we estimated that the proposed sample would enable us to detect with 80% power a mean posttest group difference of 23%, which we believed to be quite attainable with the proposed intervention.83,84
The programmer or statistician exported the data from the Structured Query Language database and the statistician conducted analyses using the statistical software R.85 We computed descriptive statistics, including mean, SD, frequency, and percentage for demographic data as well as for pain intensity, analgesic adherence, and other outcome measures at baseline. We used independent t tests for comparison of continuous variables between 2 groups. For categorical variables, we used chi-square tests and Fisher tests. We set statistical significance at P < .05.
To estimate intervention effects, we performed regression analysis, more specifically binary logistic regression for patient satisfaction with pain level and linear regression for other outcomes (worst pain, analgesic adherence, and patient and caregiver BQ scores), controlling for study teams. Given the stepped-wedge design of this study, we used natural cubic splines with 5 degrees of freedom to model potential time trends. For outcomes measured twice (baseline and posttest), including patient and caregiver BQ scores, we included baseline values of the corresponding variable as control covariates; the main effect of the group assignment represents the intervention effect. For outcomes that we collected daily, such as worst pain, analgesic adherence, and pain satisfaction, we treated the baseline values as dependent variables and used random-effects terms to account for between-patient differences. The intervention effects for these longitudinal outcomes were represented by the interaction between group and time (coded as 0 for day 0 and 1 for afterward). We used Wald's tests to determine the statistical significance of intervention effects. In all our regression analyses, control group was the reference group. For this analysis, the intervention effect was of main interest and we report only coefficient estimates for them in the “Results” section.
To process missing data, we used multiple imputations, which produced multiple completed data sets, upon which inferences were performed and then aggregated. The amount of missing data across our study measures varied—25% for patient self-reported medication data, 26% for worst pain intensity, 25% for pain satisfaction, 13% for patient pain misconception questionnaire items, 15% for caregiver pain misconception questionnaire items, and less than 1% for hospice medication data. Because being too tired or too ill is one of the reasons cited by patients for not entering daily data, missing at random assumption most likely was violated. To assess the robustness of our analysis under missing not at random conditions, we relied on postprocessing of imputations to conduct sensitivity analysis based on the pattern mixture method. In particular, we re-ran the multiple imputation, making adjustment to the models used to impute the key outcome measures using various missing not at random assumptions where the missing observations were worse or better than would be predicted by a model based on the missing at random assumption.
We previously reported other detailed specifics of the study's adherence to PCORI's Methodology Standards.86 From study conceptualization through implementation, analysis, and reporting, we were attentive to rigorous and patient-centered research processes.
Some changes were introduced to the protocol during the study period. The most significant change was the modification of the randomization procedure from a simple cluster-randomized design to the stepped-wedge design. We also changed the eligibility criterion for PPS from 40% to 30% to increase subject recruitment. The fact that our attrition rates were lower than we originally projected supported this change. We changed the sample size from a sample that completed the study of 250 dyads to 192 dyads. The study accrual was slower than we had predicted, and new power analysis conducted using data available at the time the change was requested indicated enough power with 192 completed dyads. Finally, 3 investigators changed their faculty status at UIC and assumed positions at the University of Florida, which we added as a performance site. The investigators retained positions at UIC that allowed them to continue their role as originally approved.
As stated above, the specific aims of the study were to compare usual hospice care with PAINRelieveIt groups for effects on (1) patient outcomes (analgesic adherence [primary outcome]; worst pain intensity; satisfaction; and misconceptions about pain and pain management) and lay caregiver outcomes (pain misconceptions) in a diverse sample of cancer patient–caregiver dyads receiving hospice care; and (2) nurse outcomes (obtained appropriate analgesics for patients) in a sample of hospice nurses. We hypothesized that at posttest, compared with the usual care group, the PAINRelieveIt group would (1) report decreased scores for worst pain intensity and pain misconceptions, (2) have increased analgesic adherence (primary outcome), and (3) have a larger proportion who report satisfaction with pain intensity and whose nurses obtained appropriate analgesics for the patients' pain.
For our data analysis, we excluded 5 patients who did not complete the baseline and 23 patients who died within 7 days of the baseline, for reasons explained earlier. These 28 patients accounted for 11% of the 262 patients that consented and were randomized. By treatment condition, 9% of control and 13% of experimental condition were excluded, with the group difference statistically insignificant (P = .48). The patient data for analysis included baseline (day 0), posttest (day 7), and daily log data (days 1-6) from 234 patients. At the baseline, patients completed surveys on demographics, pain barriers, pain assessment, symptom distress, computer acceptability, and medication taken. Patients completed he same array of surveys, with the exception of demographics, at posttest. In addition, patients completed pain assessment and analgesic medication surveys each day they were in the study.
We collected a large number of variables each day. Not including pain locations and associated location-specific pain descriptors and aggregating the medication-related variables into a single adherence variable, 109 variables were entered daily. Of these, the outcomes of interest are the worst pain (1 variable), pain satisfaction (1 variable), and the analgesic adherence (1 variable). The other variables, while not the focus of our analysis, may provide useful information for the outcome variables that can be utilized in our missing data processing based on multiple imputation. Given the large number of available variables, we eliminated variables with low correlations (below 0.3) to outcome variables and created a data set for imputation containing the outcome variables, variables related to pain (current pain, least pain past 24 hours, optimal pain goal, tolerable pain goal, amount of time past 24 hours pain tolerable, 4 pain rating index components, pain pattern score), as well as individual pain descriptors with correlation (>0.3) with outcome variables (frightening, intense, unbearable, nauseating, constant). Although this set of data still contains 31 variables, it was much more manageable than the original daily log data of 109 variables and facilitated multiple imputation.
On days that patients accessed PAINReportIt, the percentage of missing daily log data is low (6.0%), but considering the missing data caused by patients not accessing PAINReportIt, the amount of missing daily data rises to 26.4%. The main reasons cited by patients and caregivers for not accessing PAINReportIt on a study day included that the patient was too tired, feeling too ill, or had other commitments with family and friends consistent with activities at the end of life. When we merged the daily data with the baseline and posttest data, the overall percentage of missing data was a moderate level of 22.5%.
Comparison of baseline measures for patients indicated no statistically significant differences by control and experimental groups (Table 4), except that the control group had higher patient pain control barriers (2.75 ± 0.81 vs 2.44 ± 0.86; P = .01). Fewer than 4 of 10 patients reported being satisfied with their pain levels, which ranged from mild at its least and severe at its worst in the previous 24 hours. At the time the patients completed baseline measures, their pain was moderate in intensity. At baseline, almost all (99.6%) of the patients were prescribed opioids for moderate to severe pain and 97.0% of the patients were prescribed adjuvant analgesics for neuropathic pain; therefore, the variable of the appropriateness of analgesic prescription was not further considered . On average, the patients' ATC analgesic adherence rate was only 63% and their pain control barriers scores indicated they had misconceptions about pain that could interfere with successful management of their pain.
Comparison of Baseline Measures for Patients and Caregivers by Control and Experimental Groups.
Caregivers, too, at baseline had misconceptions (pain control barriers) about pain that could interfere with successful management of the patients' pain. These pain control barriers at baseline, however, were not significantly different by control and experimental groups (Table 4).
Compared with baseline, a smaller proportion of both groups reported being satisfied with their pain levels postbaseline (33% for the control and 31% for the experimental group; Figure 4). Binary logistic regression showed that there was no significant difference between groups (Table 5).
Proportion of Sample Satisfied With Pain Level Over Time by Control and Experimental Groups.
Regression Analysis.
As Figure 5 shows, the patients' worst pain intensity averaged over daily measures at postbaseline was significantly lower than at baseline for the control group (mean, 6.05 ± 2.24; P < .01) but not for the experimental group (mean, 6.63 ± 2.12; P = .10). Regression analysis confirms that the daily worst pain intensity decreased postbaseline, but the experimental group showed less improvement postbaseline than the control group (P = .02).
Average Scores for Worst Pain Intensity in Past 24 Hours Over Time by Control and Experimental Groups.
The pattern of analgesic adherence rates across time is displayed in Figure 6 and shows that both groups increased to 70% adherence, but on different days (day 2 for the experimental group and day 5 for the control group). Averaging over postbaseline days, the adherence rate was 66% (62%-71%) for the control group and 67% (61%-74%) for the experimental group. These adherence rates were not statistically different between groups (Table 5).
Adherence Rates for Scheduled Analgesics Over Time by Control and Experimental Groups.
At posttest, the patients' pain control barriers scores (scores on pain misconception questionnaire BQ-13, possible score range of 0-5, with higher scores indicating more misconceptions and pain control barriers) decreased slightly in the control group to a mean of 2.72 ± 0.79 and increased slightly in the experimental to a mean of 2.56 ± 0.87. Regression analysis controlling for the pretest pain control barriers scores showed that the difference between the 2 arms was not significant (P = .316) (Table 5).
The caregiver outcome at posttest, pain control barriers scores, increased for the control group to a mean of 2.97 ± 0.81 and decreased for the experimental group to a mean of 2.67 ± 0.82. Regression analysis controlling for the pretest pain control barriers scores showed that the group effect was significant (P = .01), with the caregivers in the experimental group reporting lower pain control barriers scores than the caregivers in the control group (Table 5).
PAINUCope videos were available to patients and caregivers assigned to the intervention group. On average, patients viewed 62% of the PAINUCope videos on pain misconceptions. The caregivers on average viewed 66% of the PAINUCope videos on pain misconceptions.
Nurses of the control group patients had access to PAINReportIt Summary. On average, they viewed 37% of the summary reports. Nurses of the intervention group patients had access to PAINConsultN decision support messages. On average, they viewed 34% of the reports. The hospice physicians always indicated that the nurses did not disclose to them the patient's study participation. They repeatedly stated they did not know who was or was not participating. The nurses who accessed PAINReportIt Summary or PAINConsultN reported that they found the information useful, that their patients had more pain than they were communicating to the nurses, and that they appreciated the suggestions for improving pain control. Those not accessing the interventions reported that they had lost their password, missed the email among all their messages, or had little time to access the information because they considered it optional given that they knew their patients.
We conducted pattern mixture analysis assuming that missing outcome observations were up to 50% worse than expected under the missing at random assumption. Our findings did not change substantively under these missing not at random assumptions.
In this 7-day RCT of an internet-based system that included pain assessment, patient and lay caregiver education, and just-in-time decision support for professional caregivers, we demonstrated that it is possible to implement the system change in 2 urban-area hospices. The goal of the intervention was to reduce pain for dying cancer patients, through improving appropriateness of pain medication prescription as well as lowering patient and caregiver misconceptions about pain control to enhance patient analgesic adherence. Most of the effects, however, were not as hypothesized. The significant reduction in lay caregivers' pain control barriers but not patients' pain control barriers is an important finding with implications across the care trajectory after a cancer diagnosis. The nearly universal availability of prescriptions for strong opioids and adjuvant analgesics for neuropathic pain is inconsistent with the high proportion of patients not satisfied with their pain levels, their severe worst pain, and their less than complete adherence to analgesics that we found in this study. The inconsistencies and lack of change in the patient outcomes also provide important insights for improving patient-centered care of the patient with cancer who is dying.
An important finding of this study is the high proportion of the sample in both groups that was not satisfied with their pain level; specifically 64% and 67% of the control group and 61% and 69% of the experimental group were not satisfied at baseline and posttest, respectively. That more patients were not satisfied in both groups over time may be an effect of them monitoring their pain daily and recognizing that it was not improving. That more, but not statistically significant, experimental group patients (relative to control) were not satisfied with the pain level at posttest may be an artifact of the PAINUCope intervention that provided education that cancer pain can and should be relieved. Until recently, it was rare to see such high proportions of patients reporting that they were not satisfied regarding pain.72 The question in our data set focuses on satisfaction with pain level instead of satisfaction with pain management because of a paradox that patients reported high pain but also reported satisfaction with pain management. The change in the item to focus on satisfaction with pain level reverses the paradox, and patients with cancer72 or sickle cell disease87 who report severe pain rarely report being satisfied. In this hospice sample, the severe pain intensity is consistent with the high proportion of the sample reporting they were not satisfied with their pain level.
The distributions for age, gender, and types of cancer within the sample were typical of hospice patients with cancer. The minority status distribution of the sample, however, was not typical. Our sample included more patients from minority groups than is typical of patients with cancer who are served by hospices nationwide.88 The generalizability of the study findings is limited to dying patients with cancer who are served by hospices similar to those that participated in this trial.
An important implication of our findings is that in the hospice setting where the median length of stay before death continues to be around 19 days for cancer patients, the target for an educational intervention focused on pain control barriers may need to be the caregiver rather than the patient. Previously, the PAINUCope educational intervention significantly reduced patient pain control barriers when it was administered during outpatient oncology care. Since the median length of stay in hospice has been around 19 days for many years, many patients receiving hospice care may be too ill to benefit from even short educational programs that they can view from home at their own pace.
The study was not powered for subpopulation analyses. Therefore, we did not conduct comparisons for subpopulations.
In addition to the lack of intervention effects for the patient outcomes, other limitations of the study merit consideration. The “luck of the draw” in randomization of hospice teams to the steps of the stepped-wedge design produced an unanticipated imbalance in study group size. Given the ever-changing clinical environment of hospice care, 1 team provided more referrals to the study than the other teams. Unfortunately, this team was randomized to the last step and provided more control group participants than originally planned. In addition, recruitment in some teams slowed down over time, leading to a slower recruitment pace during the intervention stages. It is unknown if study results would have differed had the referral patterns been more consistent among the hospice teams. Also, other changes in the administration and referrals from the community to the hospices occurred, including 1 hospice merging with 2 others to create the largest nonprofit hospice in the United States (teams were not changed); a new administration of the other hospice early in the study with subsequent process changes; a new hospice forming in the area that significantly reduced referrals to 1 hospice; and new institutional alliances that also changed referral patterns to both hospices. Although every effort was made to mitigate the effects of all these changes on the study by active involvement of the stakeholders during the monthly team meetings, it is unknown what, if any, effects the environmental context had on study outcomes.
An important limitation in the study was the nurses' low access of the PAINReportIt Summary and PAINConsultN reports for their patients. This finding was unexpected given the nurses' enthusiasm about the study and feedback from the stakeholders. The mitigation actions were implemented after the last team stepped up to the intervention, and it is clear they are necessary but are probably not sufficient. In future implementation studies, the success reports need to feed back into the care system with supervisors involved when report access does not improve. Another important limitation was the lack of documentation for the nurses' communications with the hospice physicians about the changes in analgesics. This issue is not addressed easily, but full access to the hospice electronic medication record could help document the dosage and interval changes that might have occurred. The ceiling effect for appropriateness of the analgesics for type of pain was unexpected and may have been related to the role of medications in the home for use during active dying. Finally, given the multiple outcomes studied in this RCT, it is not clear whether the positive caregiver findings on pain misconception scores is an artifact.
Future research focusing on patients with cancer and their lay caregivers should occur early in the cancer experience when patients are able to participate in short learning activities, or the caregiver should be the main target if the education has not occurred before hospice enrollment. When palliative care is implemented earlier in the cancer trajectory simultaneously with antitumor treatments, the PAINUCope intervention could be a tool to change patient and lay caregiver misconceptions about pain management. Such changes in misconception would be expected to lead to improved analgesic adherence and reduced worst pain intensity.
The lack of group effect or effects favoring the control condition were not as expected and require additional exploration to better understand directions for further system changes directed at improving pain control of dying cancer patients who receive home hospice care. Additional examination of the data set to explore intervention process issues is warranted. A helpful step would be to compare our control group pain intensity scores with the hospice record pain intensity scores for the cancer patients served during the study period and those who did not participate in the study. Since our control condition (PAINReportIt Summary) allowed the patient to communicate pain data daily via the internet-based system, it is possible that it was an intervention itself and that the added intervention of the decision support (PAINConsultN and PAINUCope) is not needed. We think this effect may contribute somewhat to our nonsignificant group effects, as well as the statistically significant but clinically not meaningful advantage of pain observed in the control group at posttest relative to the experimental group. The worst pain in the past 24 hours was severe, about 7 on a 0 to 10 scale for patients in both groups; this value is not consistent with comfortable dying, especially if this intensity of pain persists for extended periods within the 24 hours. PAINReportIt includes a variable focused on the amount of time that the pain was beyond the patient's tolerable level. Additional analysis of the data set should include examination of this variable, including the group comparison as well as its relation to pain outcome measures reported here, including pain intensity and satisfaction, to better understand the context of pain control for both groups. Also, additional analysis of the data set for the set of analgesic medications used daily and symptoms reported could provide additional information about the dynamic context of patient and lay caregiver behavior that could have affected study findings. Such analysis will require a qualitative approach to fully understand the patterns in analgesic use given worst pain and satisfaction with pain levels. This analysis was beyond the scope of this RCT, but could provide important information for guiding next steps to change hospice care systems to improve pain control.
Additional clarification is needed to explain the low analgesic adherence rates by patients (at least 30% less than they could have taken) when they had severe worst pain intensity in the past 24 hours and pain levels with which they were not satisfied. As we found in this study, our hospice patients were prescribed the types of analgesics that are known for controlling pain, but they were not taking the prescribed opioid medications as often as they could. It is possible that they were taking as-needed analgesics instead and were confused about subsequent dosing schedules. It is also possible that their medication regimens were overly complex and in need of simplification to improve adherence, reduce worst pain intensity, and increase satisfaction with pain level. Additional research with mixed methods approaches is needed to better understand the patient-centered context of pain management during home hospice care.
In summary, this RCT was a negative trial for the patient outcomes, including our primary outcome of ATC analgesic adherence, but positive for the caregiver outcome, a secondary outcome. Despite the lack of support for the patient-centered hypotheses, the trial clearly demonstrates the feasibility of implementing the innovative set of interventions with tablet-based pain assessment and patient and caregiver education as well as email-delivered or cell phone–delivered links for patient-centered clinician decision support via the internet as part of home hospice care. Additional research would need to implement these innovations in usual hospice care settings for patients expected to have lengths of stay that would allow study of the patient and caregiver outcome effects over a longer study period, using implementation science or comparative effectiveness strategies.
We thank the patients with cancer and their caregivers for participating in this study and the hospice administrators and staff for supporting this study.
Research reported in this report was [partially] funded through a Patient-Centered Outcomes Research Institute® (PCORI®) Award (#CDR-1306-03556) Further information available at: https://www.pcori.org/research-results/2013/testing-tablet-based-software-help-reduce-hospice-patients-pain
Wilkie DJ, Yao Y, Ezenwa MO, et al. (2019). Testing Tablet-Based Software to Help Reduce Hospice Patients' Pain. Patient-Centered Outcomes Research Institute (PCORI). https://doi.org/10.25302/8.2019.IH.13046553
The [views, statements, opinions] presented in this report are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.
This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License which permits noncommercial use and distribution provided the original author(s) and source are credited. (See https://creativecommons.org/licenses/by-nc-nd/4.0/
Your browsing activity is empty.
Activity recording is turned off.
See more...
