Background:

Although they represent a standard of evidence, randomized controlled trials (RCTs) often fall short because of insufficient or unrepresentative enrollment, and many needed trials are never conducted. This leaves gaps in evidence to inform patient care decisions and creates a need for a method to facilitate RCTs in usual care settings.

As medical therapies become increasingly less satisfactory for patients with osteoarthritis, an average of 680 886 patients receive surgical knee replacement per year in the United States. Yet, there have been no substantial comparative effectiveness RCTs of medical vs surgical total knee replacement (TKR). The question about TKR for knee osteoarthritis is suitable for exploring a method that would facilitate the conduct of comparative effectiveness RCTs by assisting discernment of patient-specific equipoise between treatments.

Clinical equipoise is a prerequisite for enrollment into an RCT; likewise, mathematical equipoise is the use of mathematical models to predict and compare patient-specific outcomes of alternative treatment options that should be considered when enrolling patients into an RCT. When the predictions are similar, suggesting equipoise, then random treatment assignment may be justified, and the patient may feel more comfortable enrolling in the RCT. When the predictions suggest one treatment is better than another, trial enrollment may be inappropriate, but the predictions still can inform clinical decision-making.

Objectives:

This project aimed to use mathematical equipoise for making patient-specific comparisons of alternative treatment outcomes of TKR vs nonsurgical treatment of knee osteoarthritis as a way to consider enrollment into a comparative effectiveness RCT.

Methods:

We first obtained the views of patient stakeholders with knee osteoarthritis to identify key pain and physical function outcomes. After creating a consolidated database from non-RCT sources of knee osteoarthritis outcomes, and adjusting for the inherent differences between the databases, we developed multivariable mathematical models that predict patient-specific pain and physical function outcomes for TKR or nonsurgical treatment. We then developed the Knee Osteoarthritis Mathematical Equipoise Tool (KOMET) user interface based on these models to discern patient-specific equipoise. We pilot tested the interface to assess usability and responsiveness to the needs of patients and physicians and its adequacy for supporting shared decision-making, both for RCT enrollment and for treatment.

Results:

We incorporated KOMET regression models into prototype KOMET decision support software, which we successfully pilot tested in a range of clinics. Patients found it very helpful in making treatment decisions, but only 7 of the 12 understood the concept of equipoise.

Conclusions:

This project demonstrated the use of mathematical equipoise as a method for providing patient-specific decision support for shared patient-physician decision-making for selecting between alternative treatments and considering enrollment into a comparative effectiveness RCT.

Limitations and Subpopulation Considerations:

Although largely accomplishing its intended objectives, as an early stage in the development of mathematical equipoise decision support, this project has limitations related to the available clinical data, the modeling methods and variables, and the prototype software. The next step will be to conduct a larger-scale test, and then to implement it for its intended use—the conduct of a comparative effectiveness trial in usual care settings.

Selection of Outcomes

We chose the 2 outcome scales on which we built our models, the Western Ontario and McMaster Universities Arthritis Index (WOMAC) and 12-item Short Form Survey (SF-12) physical component scores, after discussions with clinician and patient stakeholders. Factors considered included the time frame of the outcome beyond surgery and the meaningfulness to someone making a decision about surgery, taking into account constraints imposed by our available data sources. Stakeholders were strongly supportive of using both the pain and functional outcome scores, as both were part of patients' decision-making processes.

Modeling Database Creation

We created a modeling database from 4 data sets, matching patients who had surgical treatment with ones who had nonsurgical treatment. To guide our choice of the variables on which they would be matched, we gathered input from clinicians on the research team, clinician and patient stakeholders, and results from prior published literature. Variable choices for these models were informed by the needs of stakeholders who would use decision support for knee osteoarthritis, focusing on their views about the representation of pain and functional outcomes.

Predictive Model Development and Results

We provided all stakeholders with an orientation to the modeling process to foster their ability to provide input on selection of candidate variables for model development. Interaction terms in the statistical models allow differences in predicted benefit for different patients, so receiving input on plausible interactions was important. The candidate primary and interaction variables included in the model selection process were those stakeholders considered important, plausible, and easily and reliably provided. We considered outcome variables based on stakeholder ranking of how much the variable would be related to pain and functional outcomes 1 year in the future.

We sought clinician and patient stakeholder input on the clinical significance of the results of predictive modeling. As the project evolved, the research team and stakeholders concluded that many of the variables under consideration were too burdensome to collect or too difficult to ascertain. To accommodate this, we adjusted models that did not have significant impact on performance characteristics.

User Interface Development and Testing

Both clinician and patient stakeholders contributed extensively to the design of the user interface of the decision support application. They reviewed its presentation of outcome predictions and its usability. Their recommendations led to improvements in the wording and ordering of the questions, instructions, and display of predicted outcomes.

Selection of Data Sets and Description of Outcomes

To create the modeling database, we considered a range of knee osteoarthritis databases (briefly described below) as well as the scales used in these databases: the WOMAC (for pain) and the SF-12 (for functional status). We selected 3 of the databases (Multicenter Osteoarthritis Study [MOST], Osteoarthritis Initiative [OAI], and Canadian Osteoarthritis Research Program [CORP]) because they are large, well established, and publicly available epidemiological studies of knee osteoarthritis. The 2 additional databases are knee osteoarthritis registries (New England Baptist Hospital [NEBH] and Tufts Medical Center [TMC]) determined to have adequate cases and the required indexes, and that were available from collaborating organizations.

Evaluation of Registry Variables

We used a consensus process involving clinician investigators and stakeholders to select variables for model development. First, clinicians were asked to rank variables based on their impact on (1) predicting prognosis for pain or function, with or without surgery, and/or (2) predicting assignment to medical or surgical treatment (ie, indications or contraindications for treatment).

They a priori ranked each variable from A to D:

1. Variables that almost certainly must be included in the model; eg, age
2. Variables that would be desirable to have established risk factors for the outcome; eg, body mass index (BMI)
3. Variables that would be desirable to have for exploratory analyses; eg, history of falls over the past 12 months
4. Variables not likely to be needed; eg, family history of arthritis

Finally, a few variables were ranked by clinicians for importance and ease of collection using a scale of 1 to 10, with 10 being very important or very hard to collect. We collapsed the importance rankings into 3 categories: not at all important (1-3), fairly important (4-7), and very important (8-10). Clinicians ranked most of the variables as easy to collect. We included in the modeling database the final list of variables deemed as fairly important and very important.

Creating the Modeling Database

The database for creating KOMET models included 2 types of registries. Two databases, MOST and OAI, had data collected on knee osteoarthritis at fixed intervals per their protocols. During the course of follow-up, some patients had TKR and continued to be followed afterward. The 2 other registries, NEBH and TMC, were from hospitals that collected baseline and follow-up data only on their patients who had TKR.

For this project, our target sample was patients who had knee osteoarthritis and had reached the clinical stage at which they would be deciding whether to have TKR. Lacking a cohort of such patients randomized to the medical or surgical options, we used data from patients who had TKR and matched them to patients (knees) who did not have TKR but who had similar characteristics. Where possible, we matched non-TKR knees to TKR knees within the same database (OAI, MOST). We matched TKR knees from the NEBH and TMC registries to non-TKR knees from MOST and OAI based on the best match. In practice, we created a database in which we used the knee as the unit of analysis, and we conducted matching based on characteristics of the knee and the patient. Thereby, we created a study sample of patients who would or could be considering this therapeutic choice.

For the MOST and OAI registries, we identified all knees that underwent TKR and then designated the data collected at the closest previous visit as the baseline visit for that TKR. We then extracted baseline data on these TKR knees from the patients' registry data, including demographics, knee characteristics, comorbidities, mental and physical function, and other clinical features. To find non-TKR control knees, we created a subdatabase of all knee visits from all patients, excluding any that occurred after a TKR. We then used a greedy matching computer algorithm³⁸ to select control knees for each TKR knee (within the same database, OAI or MOST). It should be noted that the variables used for matching differed among the databases, based on data availability. As a guide to determine variables to use for matching, we used input from research team clinicians, stakeholders, and the literature. For matching, we converted continuous variables to categories. We loosely based categories on Riddle et al, which presented an algorithm to judge the appropriateness of TKR.³⁹ Our research team considered the factors used in that algorithm as reasonable factors to match on where possible. Categories were ordered, and we did not allow matches beyond 1 category of difference. We did not always require exact matches because we did not want to lose patients who had TKR from the model-building sample, and we could statistically adjust for differences between the TKR and non-TKR groups in the modeling process. Thereby, we matched each TKR knee in OAI with a similar non-TKR knee in OAI based on values of matching variables at baseline. The same was true for MOST.

Because the TMC and NEBH samples included only TKR participants, we drew their matched non-TKR controls from a pooled data set of knee visits from the OAI and MOST registries.

We established exclusion criteria based on discussions with the research team members and applied them before we performed modeling. We excluded any knee that did not have follow-up information (9 months to 5 years after the baseline visit or TKR) on the same knee in the same state (TKR vs non-TKR). If a knee visit was a candidate control but had TKR at some point between that visit and a follow-up at least 9 months later, we excluded it from the pool of non-TKR knees used for matching. If a knee had TKR but did not have pre-TKR baseline data within 12 months of the TKR, we excluded that TKR. If a knee had TKR, we excluded the contralateral knee from the pool of non-TKR knees used as controls. If a patient had TKR on 2 knees, >90 days apart, we excluded both knees; with an interval of >90 days, we were concerned that the 1-year evaluation of pain and function for the first knee could still be during the recovery period of the surgery for the second knee, which would confound the assessment of the outcome. If bilateral surgery was completed on 2 knees within 90 days of each other, we used the first knee or randomly chose 1 if both knees were completed on the same day. There was 1 exception in the MOST data for which 14 patients were counted twice, including and following each bilateral knee separately. In the full database, 104 other patients were counted 2 times (92 patients) or 3 times (12 patients). Overall, 1322 patients contributed data for 1452 matched knees for these analyses. We did allow single patients to contribute both a control and TKR knee when surgeries were far enough apart in time to allow full follow-up on each independently. We also allowed OAI and MOST control knees to be reused for the matching process for TKR knees from the NEBH and TMC registries. See Appendix A for details and limitations of this approach.

On the matched data set, we compared baseline characteristics between knees with and without TKR, using chi-square tests and t tests. To account for missing data, we used multiple imputation, creating 10 imputed data sets for each study source. We also compared baseline characteristics on imputed data sets as we used these for model development. We adjusted P values from the analysis of the multiple imputation data set to account for imputation variability.⁴⁰ We used SAS software for these analyses using the model information (MI) procedure to impute the data and MIANALYZE to process the results of analyses on the imputed data. See Appendix B.⁴¹

Creating Predictive Models for Outcomes

We conducted analyses using SAS for Windows (Version 9.4 TS Level 1M2; SAS Institute, 2002-2012) and SAS Enterprise Guide (Version 7.13 HF3; SAS Institute, 2016).

We developed a multivariable linear regression model to predict the 1-year knee pain outcome based on the WOMAC score or, when a database lacked WOMAC items, using an estimated WOMAC score, as described in Appendix C. Our approach was to develop the model using a set of matched TKR to non-TKR knees from the OAI database and then to validate/test it on a set of matched TKR to non-TKR knees from the MOST database. We then pooled the OAI and MOST data sets and built a new model, starting with variables used in the model developed in the OAI data and tested on the MOST data. We also rederived models on a database that included all 4 data sets (OAI, MOST, NEBH, and TMC). We used a similar variable selection process but with a more limited set of candidate predictor variables because NEBH and TMC did not capture as many variables as the OAI and MOST registries. We repeated this entire process for the functional outcome (SF-12 physical component at 1 year). To create models that could provide predicted estimates of 1-year knee pain and 1-year function, with and without TKR, for any patient based on their characteristics, all models included an indicator variable for treatment type. We explored covariates and interactions of treatment type with covariates in the different phases of the modeling process. We did not adjust for matching in the linear regression during modeling because the purpose of matching was to create a reasonably balanced study sample, and covariates in the models could account for remaining imbalances between groups.⁴² We describe further details of our approach in Appendix D.

Prototype Decision Support Software Development, Interface Design, and Usability Testing

The goal of software development and usability testing was to translate the results of the predictive models into easily understood, patient-specific reports with predictions of 1-year outcomes that could be produced in real time in the course of clinical care, for shared treatment decision-making and, if appropriate, enrollment into an RCT.

Study Design and Database Creation

The final database included 1452 knees (726 with TKR and 726 without) of 1322 patients. Of patients, 91% (1204) had a single knee included in the database, 8% (106) had 2 knees used or a single knee used 2 times, and 1% (12) had knees used 3 times. We matched TKR knees from OAI to control knees from OAI, and we matched TKR knees from MOST to controls from MOST. Because NEBH and TMC included only TKR knees, we drew the controls for those databases from non-TKR knees from OAI and MOST. In the final matched database, the relative contributions of TKR knees were OAI, 252; MOST, 154; NEBH, 248; and TMC, 72. For the control knees, contributions were OAI, 472, and MOST, 254. Figure 1 and Appendix F: Figures 1a to 1d provide breakdowns of how we selected the final analysis sample from each database in CONSORT-type figures.

Figure 1

Description of Final Analysis Sample Selection.

Study Sample

We compared distributions of variables used for the matching process between TKR and non-TKR knees for each data source; these results are presented in Appendix F: Table 1a. They confirmed that the matching algorithm had worked. In each database, characteristics used for matching were well balanced between the TKR and non-TKR knees. Baseline characteristics considered for the modeling process, and of interest to clinicians and stakeholders, were comparable between TKR and non-TKR knees, as presented in Appendix F: Table 1b. This also was true of the variables used in the final multivariable models using the imputed data, as shown in Appendix F: Table 1c.

Baseline characteristics and outcomes at follow-up of the matched study sample are summarized in Table 1. Approximately 40% were men, the mean age was 65 years, and the mean BMI was 31. On the 0 to 100 pain scale (100 indicating extreme pain), the mean baseline knee pain was significantly higher in the TKR group than in the non-TRK group (mean, 45.6 vs 40.5; P ≤ .01), despite efforts to match on this variable (categorized). Comparisons of mean baseline SF-12 scores between TKR and non-TKR groups showed better physical and mental function in the non-TKR groups than in the TKR groups, with the difference being significant for physical function (mean, 37.2 vs 38.6; P = .008). Overall, at follow-up there was less knee pain and better physical function in the TKR groups than in the non-TKR groups. Irrespective of significance, we used all variables listed in Table 1 in building the multivariable models of long-term (approximately 1-year) outcomes.

Table 1

Description of Pooled Study Sample Used for Model Derivation for n = 1462 Matched Knees (Imputed Data).

Model Development

We used linear regression to model the 2 outcomes, the WOMAC knee pain scale (rescaled 0-100; see Appendix C: WOMAC Knee Pain, Part II) and the SF-12 physical functioning component score.

Based on the methods described above, we chose these outcomes (including timing), before building models, following repeated discussions with clinician and patient stakeholders and the research team. We chose 1 year as the target follow-up time to have a time point beyond the recovery time from surgery, estimated as up to 9 months. Stakeholders felt benefits of surgery were stable beyond that time point. To address inconsistencies and gaps, we allowed for use data from up to 5 years past baseline in which there was no closer time to 1 year for a knee.

Stakeholders were strongly supportive of using both the pain and functional outcomes in patients' decision-making processes, although the outcomes were not of equal importance to all patient stakeholders. As the project progressed, the team continued to receive more input from patient and clinician stakeholders, which influenced modeling, an example of which is described in Appendix G.

Summary of Multivariable Models (Table 2, Figure 2, and Appendix H: Table 2c): Appendix H

Table 2c shows a summary of variables included in all 4 final models (P1, P2, F1, F2) and the distribution of each variable in the pooled databases. In the earlier phases of this project, we hoped our P1 and F1 models would have better performance because we had a larger pool of variables (although fewer patients) to use for the modeling process. As the project evolved, the research team realized that many of the variables under consideration were burdensome to collect and/or difficult to capture consistently. In the end we decided to use only models P2 and F2—which we built on the data sets that had more patients (OAI, MOST, NEBC, TMC) but fewer independent variables—for the development of the software. The coefficients for these models are presented in Table 2. Although neither model was validated in an independent database, we believe the models have sufficient performance, based on variables consistent with clinical understanding and importance such that they are reasonable for use in this demonstration project. Based on the results of testing our OAI model on the MOST data, we are optimistic the models can be useful in patients similar to those used to develop the models. These patients, who are presumably at the point of deciding whether to have TKR, have characteristics similar to those shown in Table 1.

Table 2

Final Models for 1-Year Knee Pain (P2) and SF-12 Physical Function (F2).

We used these models to estimate 1-year knee pain and physical functioning for the treatment each participant actually underwent (TKR or non-TKR) and also for their counterfactual situation, as if they received the alternative treatment. In other words, we calculated 2 predicted values for each participant in our database (that we used to make our models). One prediction assumed participants received TKR and the other prediction assumed they did not. These data allowed us to predict the difference in pain and function outcomes for each patient under 2 courses of treatment (TKR vs non-TKR). The distribution of predicted differences in pain and function with and without TKR is shown in Figure 2. The figure shows that there was a range of predicted improvement with TKR, and those patients predicted to have benefit in knee pain may not have been the same as those for whom benefit in physical functioning is predicted. In this project's database, 9% of participants had a predicted gain in function of TKR vs non-TKR of at least 8 SF-12 physical function points and a predicted reduction in knee pain of at least 20 points (on WOMAC scale of 0-100). At the other end of the spectrum, 6% had predicted gains in physical function of <4 points and reduction of knee pain of <10 points. Only 2% had larger gains in physical function and smaller improvements in pain. Figure 2 also shows sample participants from each of the 9 combinations of estimated knee pain and physical function change. Examples of participants with the most-, mid-, and least-estimated reduction of pain as well as their gain in function with 95% prediction intervals for the estimates are shown in Table 3. Participants with higher baseline knee pain had the largest predicted reductions in knee pain with TKR vs non-TKR. Younger patients with lower SF-12 scores had the largest predicted benefits in physical function with TKR vs not having TKR. These differences in estimated benefits between participants are because of the interaction terms included in the multivariable models.

Figure 2

Mosaic Plot Showing Distribution of Predicted Differences (TKR vs Non-TKR) for 1-Year Knee Pain and SF-12 Physical Function in Pooled Data (n = 1452).

Table 3

Estimated Outcomes for a Sample of Cases.

We ran into statistical questions regarding the use of the proposed linear model with 1-year outcomes, specifically knee pain, for which the scores do not have normal distributions and adjustment for covariates still produced a model in which the resulting residuals (the difference in predicted and observed values) still had skewed distributions. We explored alternative nonlinear models with little gain in model performance and ultimately used the linear form of the model. See Appendix I.

Prototype Decision Support Software Development, Interface Design, and Usability Testing

The KOMET development process resulted in the creation of 1 web-based application for clinicians (http://medicalequipoise.com/tkrclinician) and 1 for patients (http://medicalequipoise.com/tkrpatient). Both applications are composed of an analytics software library that also could be embedded into an EHR system.

The applications underwent user testing to assess the ease of data collection through the web-based questionnaire and users' ability to understand the outcome predictions when presented in data tables, graphs, and as dynamic text. We also tested depictions of prediction uncertainty and definitions of mathematical equipoise.

All users were able to easily enter demographic data and complete the questionnaire with only minor questions or comments. We initially presented users with a table and bar graphs describing current and predicted pain and function outcomes (Appendix J: Figure 1). After initial testing, we refined the report to provide a dynamic text description (Appendix J: Figure 2). This change improved users' ability to identify their current pain and function scores and the predicted 1-year outcome scores with surgical and nonsurgical treatments.

The combined pain and function plot proved to be less intuitive. Many users immediately understood that the single data point represented both the pain and function outcome predictions, but others struggled to describe the data represented by the graph (Appendix J: Figure 3).

User testing led to improvements in the way predicted outcome uncertainty was communicated. The degree of uncertainty around the predicted pain and function outcomes, initially represented by whiskers on the bar chart (Appendix J: Figure 1), was not understood by users. We changed the chart by using shading within the bar that faded at the edges and added a dynamic text explanation describing the range of possible values. (Appendix J: Figure 2). This improved user understanding. Analogously, for the combined pain and function plot, we changed the uncertainty around the prediction from a dotted circle around a data point (Appendix J: Figure 3) to a shaded circle (Appendix J: Figure 4). A limitation of our depiction of the results, not of the interface per se, is that our methodology made separate statistical models for 1-year knee pain and physical function; in reality, these 2 outcomes are likely related. Therefore, our uncertainty regions may still not be accurately capturing, and are likely overestimating, the joint-prediction areas. The true uncertainty region would be a subset of the circle if pain and functioning were dependent.

Based on these uncertainty estimates regarding the predictions and based on the mathematical equipoise approach, we used KOMET to identify patients for whom enrollment in a randomized clinical trial might be appropriate. For the purpose of demonstration, we defined mathematical equipoise as a condition when pain and functioning outcome predictions with nonsurgical care and TKR are relatively close and fall within each other's circles of zones of uncertainty—that is, their circles of uncertainty overlap. These circles are created when the pain and function outcome predictions are presented as point estimates on a 2-dimensional graph with pain on the vertical axis (y) and function on the horizontal axis (x). The uncertainty circle is defined by the shaded area extending around each of the point estimates and represents the uncertainty associated with the predictions. In Appendix J: Figure 1, the blue diamond represents the outcome prediction point estimate for nonsurgical care and the green circle represents the point estimate for TKR. The large shaded blue and green overlapping circles are around the 95% confidence intervals of the pain and function point estimates and represent the uncertainty associated with the predictions. When we computed the mathematical distance between the nonsurgical and TKR predictions the resulting distance between the 2 coordinates on the pain and function graph was 43. Empirically, based on the input of rheumatologists, orthopedists, and primary care clinician stakeholders, and after reviewing the 95% CI for a sample of cases, we selected the distance of less than or equal to 20 to flag the possible presence of equipoise during the usability testing.

When mathematical equipoise was present, an alert appeared on the user interface's results page and a patient contact and screening form was made available to the clinician. The form could be used to begin the clinical trial recruitment process.

We asked participants about usefulness of the information for decision-making. Each stated that the tool was helpful or somewhat helpful. All wanted to discuss the results with their physicians.

We used the combined pain and function predicted outcomes plot to discuss the idea of equipoise in the context of random assignment of treatments in an RCT. We showed users 3 sample graphs depicting the predicted outcomes of the 2 potential treatments, with small, moderate, and large amounts of overlap between the 2 circles that depicted uncertainty around the predicted point value (see Appendix L). We assumed that if patients perceived greater overlap in the predicted outcomes between the 2 treatments, then they would be more likely to consider being randomized to 1 of the treatments. Only 7 of 12 users shown these scenarios responded that they understood the concept of equipoise. Because of their personal preferences, some users rejected the option of surgery despite predictions suggesting dramatic reductions in pain and improvement in physical functioning. Other patients indicated they would consider randomization only if the burden of surgery promised a far better outcome than nonsurgical treatment. Overall, users did not respond to the depiction of the circles of uncertainty scenarios as we expected.

We conducted patient and clinician user testing to understand KOMET use in the clinical setting during regularly scheduled clinic visits and research-specific visits. There were significant challenges in allocating adequate time for the patient to complete the decision support tool and for the clinician and patient to discuss the results and implications for decision-making. We determined that future dissemination should include patients completing the tool before their visit to allow the patient and clinician more time during the visit to discuss the results, the patient's priorities and choices, and treatment options.

Through the efforts of our research team, stakeholders, and design consultants, we were able to develop a software program that users found helpful in shared clinical decision-making. Although the final prototype seemed attractive and easy to use, there will need to be further refinements for routine clinical care use and enrollment into RCTs.

Study Results in Context

In deciding between treatment options and deciding whether to participate in a clinical study, the patient is the ultimate decision maker. Ideally, these determinations will be made with ample consultation and support by relevant clinicians. In this context, methods to share information and support a shared conversation about these decisions can be very helpful. Decision aids explicitly intended for shared patient-clinician decision-making have been shown to improve patient knowledge, patients' satisfaction with decision-making, and agreement between choices for treatment and their health outcome preferences, among other positive effects.⁴³ The same kind of shared decision-making is justified in the decision as to whether to participate in an RCT. In developing KOMET, we sought to develop decision support that could support both a clinical decision and a decision to participate in an RCT—in this case, the decision between surgical (TKR) and nonsurgical treatment of knee osteoarthritis.

There are 2 general contexts for the results of this project: (1) the development of mathematical equipoise as a basis for decision support and (2) the state of evidence for treatment decisions for knee osteoarthritis. We discussed the latter of these 2 in the introduction; although knee replacement surgery for osteoarthritis is very common in this country, until a relatively small RCT was conducted as this project was being completed there were no RCT data to directly inform this treatment choice.⁹ Although KOMET does not provide new data, it presents those data available at the time of its development in a potentially helpful way. More to the general point of this project, KOMET is intended to help generate the needed RCT data for knee osteoarthritis to add to extant evidence. Thus, the main context for the results of this project is for the development of the mathematical equipoise method.

Mathematical equipoise is based on the use of mathematical models that serve as clinical predictive instruments to predict patient-specific outcomes of treatment options, which then can be compared. By doing so, in a sense we are discerning patient-specific equipoise. When the predictions are not discernibly or importantly different, which can suggest equipoise between options, enrollment in an RCT that compares the treatments can be considered. When the predictions suggest one treatment is likely to have better outcomes then trial enrollment would not be appropriate. When this is the case, however, this identification of a potentially superior treatment can inform patient-clinician decision-making, thereby constituting an approach for enrolling RCT participants that also supports clinical decision-making for those not to be enrolled in an RCT.

Our original examples of this approach used predictive model outcomes of acute myocardial infarction that were built using RCT data, which are ideal sources of data for making predictive models. However, for many treatments there are no prior RCTs—and indeed these are the very conditions for which RCTs and, in particular, clinical effectiveness trials are needed.

For the widespread use of mathematical equipoise to help fill in gaps in RCTs, predictive models for these conditions will need to be built on data from clinical registries, EHRs, and other non-RCT data. Therefore, the purpose of this project was to further develop the method by applying it to an important clinical treatment question for which there were essentially no prior RCTs. With >680 886 TKRs done each year in the United States for knee osteoarthritis,⁴⁴ we considered this an important question for patients and society, and a good opportunity to test whether this approach could work in this challenging but common situation.

To do this, we created a consolidated database from non-RCT sources on knee osteoarthritis outcomes on which we created predictive models of the outcomes of surgical knee replacement and nonsurgical treatments. The choices for variables for these models were informed by the needs of stakeholders who would use decision support for knee osteoarthritis, with focus on their views on the representation of pain and functional outcomes. We then developed multivariable mathematical models that predict patient-specific outcomes of surgical and nonsurgical treatment, using statistical and analytic methods to adjust, to the extent possible, for the inherent biases in the databases. We also performed a variety of analyses to understand how to best model and represent the predicted outcomes. We incorporated these models into a stakeholder-informed prototype decision support software for potential incorporation into EHRs. KOMET exemplifies a tool that can be used to provide shared decision support for RCT inclusion and clinical care that is responsive to the perspectives and needs of patients and clinicians in supporting shared decision-making for RCT enrollment and treatment.

We believe the impact of such a method on the field of CER could be substantial. The impact of CER is based on evidence generation, which leads to evidence synthesis, interpretation, application, dissemination, implementation in widespread practice, and then feedback for the generation of new evidence. This entire chain rests on having unbiased generalizable, ideally RCT, evidence. Were there such a method for patient-centered enrollment into RCTs that could be incorporated into EHRs, far more targeted comparative effectiveness trials could be conducted, more diverse clinical sites could be included, and more representative patients could be enrolled. This would lead to results that are applicable to more special groups and to more care settings. This would also facilitate clinicians' and the public's understanding of, and enrolling into, clinical trials, which could help improve the public's engagement in the biomedical research enterprise. Additionally, clinical trial duration, a scientifically and financially important component of drug development pipeline time, could be much shorter. If instead of only 10% of eligible patients being enrolled enrollment pace was increased by up to tenfold, trials would finish much faster. All these advantages should result in better clinical trials and greater impact on the public's health.

If successful in demonstrating that this method has applicability to the many important conditions for which RCTs have not yet been conducted, providing onsite real-time decision support for trial enrollment, it could transform how comparative effectiveness research could be conducted across the spectrum of health care. This would address the failure of current clinical trial approaches to enroll sufficient numbers of patients, facilitate the need to identify and ethically handle treatment of all potential participants, and engage a conversation between clinician and patients based on data specific to that patient. Thereby, it could have use in broad areas of clinical care and could help enable the great promise of CER in improving clinical care.

Uptake of Study Results

In this project, based on input from multiple stakeholders and potential users, we implemented prototype KOMET software and tested it in clinical settings. Although it is not ready for widespread implementation in its user interface, content, and connectability to EHRs, it did function as intended and thus is an important step in the ultimate goal of clinical use. We believe its promise is sufficient to warrant further development with the explicit intent of being a tool that can be implemented in clinical settings and linked to EHRs, to serve both treatment and RCT enrollment purposes. Toward that end, we will seek further opportunities to move toward that goal.

Study Limitations

Although largely accomplishing its intended objectives, as an early stage in the development of mathematical equipoise for shared clinical decision-making, this project has many limitations related to the available data, the modeling methods, the model variables, and the prototype software.

An important limitation of our approach is that the models were created on potentially biased data. Although we sought data from studies that had both surgical and medical treatment of knee osteoarthritis, 2 of our studies fit that requirement while 2 other registries were of only 1 treatment (surgery). Both types of sources provide challenges for creating comparable patients who underwent the 2 treatments, which is needed to make accurate models of the 2 treatments. In contrast, our prior clinical predictive models, including the first examples of mathematical equipoise, used data from RCTs. This allowed for representation of the alternative treatment effects based on comparable samples undergoing the treatment choices, providing confidence that the effects and outcomes represented by the multivariable models would reflect the actual treatment effects and not differences in the underlying characteristics of patients receiving the alternative treatments. However, for mathematical equipoise to serve its intended purpose of facilitating RCTs of treatments for which none have yet been conducted, its models will need to be made on non-RCT data. Therefore, for this project, we intentionally chose a condition for which our only data were from registries that posed challenges for making models that could be based on comparable patients receiving the 2 treatments. We undertook many checks to maximize the comparability and to accurately represent effects despite the likely biased samples. For example, we chose to use matching for our study design but acknowledge that while 1:1 matching improves control of confounding and enabled us to create a hypothetical sample of non-TKR patients who could be considered as TKR eligible, this approach does not use data from all available participants and may therefore have the cost of less precision. While we believe KOMET models have very good performance, despite the challenge of the available data, additional sources of data for this approach should be developed.

The modeling methods also have limitations. Although such multivariable regression as we used have advantages over some more computer-intensive methods like those used for machine learning, including the clearly interpretable variable coefficients and robustness that is more resistant to overfitting than some computer-driven methods,⁴⁵ larger databases on which more corrections might be made (eg, via the use of propensity scores) and newer computer methods might advance the level of models that might be created. Indeed, we believe this approach will benefit greatly from such advances in modeling.

Another limitation is that neither model was validated in an independent database; we simply did not have sufficient data to support model development and to still have enough for a test database. However, we believe the performance of the models and their variables are consistent with clinical understanding and importance and are reasonable for use in this demonstration project. Nonetheless, testing on an independent data set will be an important future priority.

Beyond the methods, the modeling variables we used have limitations. While in general, based on the collection of important variables in the available databases and based on published clinical evidence and input by stakeholders, we believe we used very credible variables to represent independent and dependent (treatment outcome) variables, there is one about which we have reservations. The functional outcome we used, based on our wish to capture a holistic physical function of the patient, was based on the SF-12 functional scale. In looking at the results of KOMET predictions, we noticed that pain is often substantially changed by surgery but function tends to have a relatively modest improvement. In discussing this with patients, we wondered if we would have better captured their meaningful knee functional improvement if we had used a more knee-specific function rather than overall physical functioning. We hoped to address this limitation by performing further analyses of the treatment outcomes for the subsample of patients in our databases for whom we have a knee-specific functional scale, the KSS, as a treatment outcome. The results of these exploratory analyses, presented in Appendix M, suggest that the WOMAC knee pain tracks well with other measures of knee pain and symptoms and, in particular, Knee injury and Osteoarthritis Outcome Score (KOOS) knee pain. The SF-12 physical component score, while positively correlated, does not track as strongly with other knee-related quality of life and function variables. These results are somewhat to be expected in that, while there may be overlap in physical function and knee-related function, they are not the same thing. Our meetings with stakeholders suggest both overall and knee-related function are important, and we have come to believe future work to develop predictions of the more specific knee-related function would be useful to both patient and clinical stakeholders.

Certainly, as a prototype the KOMET software has limitations. The creation of full-featured, user-friendly, robust software is beyond the scope of this project. Our prototype has significant distances to go in these and other dimensions before it could be used in routine care. Nonetheless, we believe it is quite attractive and functional and, in the context of its intended role in this project, a successful product of this project.

Finally, in putting the use of KOMET in the context of clinical decision-making, this approach does not consider how the patient might feel about the outcome states (pain and function). This would involve translating the WOMAC and SF-12 scores into familiar terms for patients and making sure the idea of overlap of predictions that suggests equivalence are all understood. Also, it would include ensuring that these features are readily incorporated into patients' understanding and decision-making for their own and shared consideration. Beyond these user issues, as additional information, patients would have to know about the downsides and potential complications of surgery, delays of surgery, and adverse consequences of other treatments. Thus, while KOMET provides an important foundation for the shared decision-making process, to provide complete and optimal support additional work is needed in many dimensions.

Future Research

The limitations listed above suggest areas for future research. Approaches must be developed that lessen the biases inherent in clinical registry data. Although having more data, such as might be obtained from EHR data warehouses and other wide sources of clinical data, will not eliminate biases, finding ways to mitigate the biases using selection and sampling methods and other approaches will be extremely important for work on mathematical equipoise, as well as for many other efforts to harvest clinically important insights from clinical data. Beyond developing such methods, validation of these approaches will be crucial.

In future efforts of this type, we would like to have more complete accounting for ancillary issues and complications. For example, in the 1 RCT done to date,⁹ serious adverse events were more common in the TKR group than in the nonsurgical treatment group (8 vs 1 involving the index knee [P = .05] and 24 vs 6 overall [P = .005]), with the 2 most common serious adverse events involving the index knee having deep venous thrombosis (in 3 patients) and stiffness requiring brisement force (in 3 patients). Unfortunately, we did not have access to such information in the databases available to us. In this Danish study, ⁹ adverse events that occurred before the 12-month follow-up were identified in hospital records, by self-report at follow-up visits, and by the physiotherapist and were then categorized. In future work exploring mathematical equipoise, in an analogous way, we intend to methodically collect such data.

Modeling clinical outcomes based on data is evolving rapidly, and increasingly sophisticated computer-based methods, such as artificial intelligence and machine learning, are being applied to analysis of clinical data. Although computer-based algorithms have a tendency to overfit,⁴⁵ compromising their generalizability to new populations, methods are advancing and an investigation of best methods is certainly warranted.

As indicated above, we believe the functional outcome we used for physical function might benefit from being a more knee-specific outcome variable.^46,47 There are examples in other diseases in which, for specific conditions, disease-specific outcomes are more useful than more general functional outcomes, such as we used.^48,49 We believe additional research that uses a more specific functional outcome would be worth conducting.

In terms of the prototype software, it is clear more research is needed for this and similar decision support that provides full-featured, user-friendly, interoperable, robust software. Badly needed will be attractive and functional software for this and similar purposes.

Finally, in developing and testing such decision support software, we will need to further investigate how to better understand, make clear, and use the patient-specific determination of equipoise that could be the basis of a comparative effectiveness RCT. We believe the method has important advantages for such studies, but before it can be widely deployed and used it must be fully understood and transparent to all stakeholders. We look forward to advancing this work.

Appendix A.

Matching (PDF, 113K)

Appendix B.

Missing Data (PDF, 105K)

Appendix C.

WOMAC Pain Score (PDF, 177K)

Appendix D.

The Process for Creating Predictive Models for the Pain and Function Outcomes (PDF, 141K)

Appendix E.

User Interface Testing (PDF, 263K)

Appendix F.

Supplementary Figures and Tables (PDF, 430K)

Appendix G.

Example of Stakeholder Influence on the Model Development Process (PDF, 139K)

Appendix H.

Supplementary Tables 2a to 2c (PDF, 196K)

Appendix I.

Exploration of Linear and Alternative Regression Models for WOMAC Knee Pain (PDF, 428K)

Appendix J.

Supplementary Figures 1 to 4 (PDF, 405K)

Appendix K.

Mathematical Equipoise Between Pain and Function Predictions With Nonsurgical Care or Total Knee Replacement (PDF, 223K)

Appendix L.

Three Simple Graphs With Small, Moderate, and Large Amounts of “Uncertainty Circle” Overlap (PDF, 184K)

Appendix M.

Consistency of Knee Pain and Function Outcomes Used for Models With Other Measure of Knee Pain and Function (PDF, 1.3M)