NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Karanikolos M, Adib K, Azzopardi Muscat Net al., authors; Figueras J, Karanikolos M, Guanais F, et al., editors. Assessing health system performance: Proof of concept for a HSPA dashboard of key indicators [Internet]. Copenhagen (Denmark): European Observatory on Health Systems and Policies; 2023. (Policy Brief, No. 60.)

1Introduction

Policy-makers need to have an in-depth understanding of how their health system is working in order to improve it. Health system performance assessment (HSPA) gives them the details they need. It is a process of monitoring, evaluating, communicating, and reviewing the extent to which aspects of a health system meet their key objectives and support overarching health system goals, as measured through a comprehensive set of indicators. However, HSPA exercises often rely on understanding of a multitude of metrics. These can be overwhelming for policy-makers who need to focus on a subset of relevant indicators to support policy decision-making.

This policy brief operationalizes the existing WHO-Observatory global HSPA framework (Rajan et al., 2023) and the renewed OECD HSPA framework (OECD, 2024) to address the need for prioritizing information derived from the volume of HSPA metrics. It builds on the existing health system assessment metrics and methodologies, such as OECD’s Health Statistics database and Health at a Glance reports (see Box 1.1), the WHO Regional Office for Europe’s Health for All database (see Box 1.2) and the Observatory’s body of work on HSPA indicators in the Health Systems in Transition reviews (see Box 1.3) and frameworks (Smith et al., 2009; Papanicolas & Smith, 2013; Papanicolas et al., 2022).

This policy brief takes health workforce, digital health, people-centredness and health system objectives and dimensions of access and quality as “test” areas and explores how a given policy question can be addressed through a careful selection of a targeted (sub)set of indicators that could allow policy-makers to navigate performance much more easily. It also advances the notion of the organization of indicators around policy questions rather than around the mechanics of the system in place. It suggests that certain high-level indicators can help policy-makers to highlight if there are systemic problems, or if the health system performance lags behind other countries, and to point out where they need more detailed information.

The selected “tracer indicators” do not aim to provide all the answers but rather act as an entry point and serve as a valuable diagnostic tool and a way of identifying appropriate types of intervention, as well as working through the implications of a given option. As a result, the brief provides a clear approach to make health systems performance “policy-makers friendly” and captures the relationships between the functions and subfunctions, assessment areas, intermediate objectives and final goals of the health system.

Current HSPA approaches may make it difficult for policy-makers to prioritize action

The selection of indicators is often determined by the availability of data at the national and international levels, rather than by relevance and usefulness for health policy-makers. This can lead to comprehensive but cumbersome HSPA exercises which, however, may lack context, analysis or interpretation. While data availability is a prerequisite, this focus on relevance is crucial because extensive HSPA exercises often leave policy-makers with an overwhelming amount of information that does not help to identify priorities for health system strengthening and says nothing about what possible policy options might have the optimal impact on health system transformation. This is true even for well conducted health system performance assessments that collect high-quality data. Without clarity on the policy objectives or their implications for health system reform, policy-makers cannot identify the options for addressing them or monitor progress of the key performance issues. Establishing a set of key tracer indicators that speak to health system performance at a broad level, organizing them around the concrete policy questions and providing clear guidance on how to interpret them may help policy-makers “see the wood for the trees”.

Tracer indicators can support policy-makers to assess and address health systems performance

Health system tracer indicators are specific high-level metrics that are indicative of broader aspects of the health system’s functioning and can be used to “trace” (identify and assess) health system performance. They help policy-makers to pinpoint performance issues, understand them in the national context, identify areas for improvement, and monitor the impact of policies and interventions at the health system level, as well as to track progress over time. The idea of tracer indicators then is not just to make the volumes of data more manageable. They are an opportunity to shift the focus of performance assessment from very particular inputs and “micro” processes within a particular health system function to a broader perspective that looks at the system level and at how the health system addresses its intermediate objectives and final goals.

HSPA Frameworks allow policy-makers to see performance linkages between areas measured by indicators

The tracer indicators selected in this brief sit within existing HSPA frameworks, such as the WHO-Observatory global HSPA framework (Papanicolas et al., 2022) or the OECD renewed HSPA framework (OECD, 2024). These frameworks serve as blueprints that link the performance of various discrete health system functions to performance of the health system as a whole, as demonstrated in Health at a Glance (OECD, 2023a). Crucially, these HSPA frameworks can be used to track indicators across functions, subfunctions, assessment areas and health system goals through performance linkages, maintaining the policy-makers’ vantage point on wider health system performance issues.

The WHO-Observatory global HSPA framework and the OECD HSPA framework are aligned and complementary in that they build on common elements and either can be used to assess health systems performance in the context of current pressing health system and broader societal challenges (see Appendices 1 and 2). The former provides detailed assessment areas and highlights dynamics of performance pathways between health system functions and goals. The latter emphasizes the key policy elements, showing broader categories and the main relationships that can be zoomed in as needed. Common elements appear in both frameworks, with different levels of emphasis.

How does the WHO-Observatory global HSPA framework work?

The WHO has played a pivotal role in health system performance assessment for several decades. Notably, its landmark World Health Report 2000 (WHO, 2000) laid the foundation for a comprehensive framework that specifically linked health system inputs to the overarching goals of the health system, providing a clear methodology for their measurement. An integral element of the 2000 report was the incorporation of “responsiveness” as a fundamental health system goal, positioned alongside the traditional objectives of health improvement and financial protection, thereby bringing in not only “objective” metrics into performance thinking but also subjective and experiential aspects.

The WHO-Observatory global HSPA framework evolved upon this foundation, expanding and refining it two decades later (Papanicolas et al., 2022). In 2023 the WHO-Observatory framework has been renewed (Rajan et al., 2023); it signposts performance assessment also in light of pandemics, conflict, climate change and other recent major developments. This has resulted in a decided emphasis on environmental sustainability, digital health, data governance, people-centredness, and participation. A key novel element is the elaboration of societal goals, and the recognition of the health system’s contribution to them.

In this framework (see Fig. 1.1), each of the four health system functions from the 2000 report (governance, resource generation, financing, service delivery) has been delineated into subfunctions (see Table 1.1). In addition, the linkages between inputs (functions and subfunctions) and outcomes (health system goals) are fleshed out, distinguishing between intermediate objectives and final goals. The intermediate objectives are direct outcomes of the service delivery function (quality, with its subdimensions of effectiveness, safety, and user experience, and access) while the final goals (health improvement, people-centredness, financial protection) are achieved at the systems level principally through the intermediate objectives. Equity and efficiency traverse both intermediate objectives and final goals, depending on whether the analysis is at the service delivery or health system level. All of these health system goals contribute to societal well-being, described through the paradigm of sustainability, which considers a balance of economic, social and environmental domains in pursuit of improved well-being. ‘Economic’ relates to sustainable economic development, ‘social’ to social cohesion and ‘environment’ to environmental sustainability.

Central to this framework remains its prioritization of people and patients through their inclusion in a governance subfunction (population and civil society participation), as an outcome of service delivery quality (user experience), and a final goal (people-centredness). The concept of people-centredness is seen as a twenty-first-century version of the responsiveness concept, i.e. a measure of how far the system is perceived as responsive to people’s and patients’ needs and views.

The renewed WHO-Observatory framework also places a nuanced focus on governance, an aspect often overlooked in indicator development and health system assessments. Governance is crucial because it offers policy-makers a potent lever from within the health system to go outside the system (depicted in a dotted line in Fig. 1.1), to collaborate across sectors, and to act on the health determinants. A nuanced view on governance can also be seen in the differentiation between governance linked to other functions (for example, health workforce planning, service delivery facility management) and system-level governance issues (human rights legislation, national health policies) (see Table 1.1).

Fig. 1.2 illustrates the application of the WHO-Observatory global HSPA framework and some of the performance pathways. Here, the novel element of assessment areas adds value by defining key characteristics which are essential for gauging the performance of the functions as well as their broader impact on overall health system performance. They are linked to each of the subfunctions and functions and are measurable through relevant indicators (see Appendix, Fig. A1.1). A policy question can thus be anchored to a concrete subfunction or an assessment area, with the framework guiding the user to root causes of the policy issue (within the functions) or towards the impact of the policy issue on system performance (objectives and goals). More importantly, visualizing the linkages helps to anchor the framework in a policy-maker’s day-to-day reality, helping to craft a viable policy solution.

How does the OECD renewed HSPA framework work?

For three decades, the OECD has helped countries to identify the key principles of high-performing health systems and to assess health system performance based on internationally comparable health indicators. This work has contributed to, and been guided by, the development of conceptual frameworks for health system performance developed by the OECD over time (Hurst & Jee-Hughes, 2001; Kelley & Hurst, 2006; Carinci et al., 2015). The original OECD framework was last updated in 2015, with a focus at the time on quality and outcomes. Since then, health systems have been called upon to become more centred around people’s needs and expectations, with enduring challenges pressing them on a daily basis. All of this has fostered a major shift in the broader health policy context, and expectations of what a health system performance assessment framework should measure have evolved substantially.

The OECD renewed HSPA framework is also in the final stages of adoption (Fig. 1.3). It places people at the centre of health systems and incorporates new key health system objectives such as sustainability (from both the economic and environmental perspectives) and resilience. It also stresses more clearly the interconnectedness and potential trade-offs across different health systems dimensions (such as balancing efficiency and equity or sustainability and resilience).

The renewed OECD HSPA framework focuses on the health system and does not serve as a conceptual model for determinants of health but acknowledges that the overall context plays a significant role in shaping the functions of all health systems, by either facilitating or restricting their performance.

In the framework, the outcomes of health systems correspond to the consequences of a health system’s activities, policies, and interventions on the health and well-being of the population. The framework places people’s health needs and preferences at the core of health systems and as such, people-centredness is regarded as an objective of health systems, as well as being instrumental to achieving other policy objectives. The section on health systems resources and characteristics covers the “structural” elements of health systems, i.e. the inputs necessary to enable the health system to function and the context in which it operates. It includes the following six pillars:

  • expenditure and financing;
  • workforce;
  • technologies and pharmaceuticals;
  • governance;
  • data and digital; and
  • knowledge and innovation.

The framework’s section on healthcare services and public health interventions includes all activities that fall under healthcare, such as curative care, long-term care, mental health care, and palliative care, while also including prevention and health promotion interventions, such as screening, vaccination and public health campaigns.

The renewed OECD framework also includes four “cross-cutting” dimensions of health system performance, namely efficiency and equity on one side, and sustainability and resilience on the other. The reason that these are cross-cutting is that they do not belong to one particular block in the framework but relate to them all.

Some relations between concepts are acknowledged in the renewed framework. For example, although different terminology is used, Donabedian’s model of structure, process, and outcomes (Donabedian, 1988) remains visible in the framework through the relationships between health system resources, characteristics, and policy (structure); healthcare services and public health interventions (process); and individual and population health (outcomes). Yet the framework remains high level. It shows the main elements in relation to each other at a higher level and is not intended to detail all possible conceptual relationships.

A policy brief as a proof of concept and a step towards a performance dashboard

There is scope to populate the HSPA frameworks with tracer indicators but this is a major undertaking that requires experts from the different strands to agree which indicators matter most and what they mean. As a proof of concept (see Box 1.4), this policy brief focuses on selected policy questions within the workforce and digital health subfunctions and the health system goal of people-centredness. These areas were chosen in light of the lessons from the COVID-19 pandemic (Sagan et al., 2021; OECD, 2023b), which highlighted the critical importance of the healthcare workforce, the system-level transformation brought by the rapid uptake of digital health, and the importance of people-centred systems in order to gain people’s trust. Access and quality were chosen to examine the high-level performance of service delivery and interaction of all health system functions and to illustrate how health system tracer indicators can be used to identify and map system-level performance and issues.

This policy brief has sought to identify key performance indicators that work as tracers for the healthcare workforce, digital health, people-centredness, and access and quality as outcomes of service delivery (primarily using literature review and WHO and OECD expert groups and working parties as described in Box 1.5). This document is in coherence with the programmes and outcomes of the WHO’s Tallinn Charter 15th Anniversary Health Systems Conference: Trust and Transformation – Resilient and Sustainable Health Systems for the Future and the OECD’s Health Ministerial Meeting: Better Policies for More Resilient Health Systems. Its ambition is to support the future development of a HSPA dashboard of indicators that will offer policy-makers easy access to key information including guidance on interpretation, and a clear sense of health systems’ relative strengths and weaknesses (across the system and in comparison to other countries).

References

Figures

Figure 1.1. WHO-Observatory global HSPA framework.

Figure 1.1WHO-Observatory global HSPA framework

Source: Rajan et al (2023)

Figure 1.2. WHO-Observatory global HSPA framework (expanded version).

Figure 1.2WHO-Observatory global HSPA framework (expanded version)

Source: Rajan et al (2023)

Figure 1.3. 2024 renewed OECD HSPA framework.

Figure 1.32024 renewed OECD HSPA framework

Source: OECD (2024)

Tables

Table 1.1Functions and subfunctions of the WHO-Observatory global HSPA framework

FUNCTIONSUB-FUNCTIONS
Governance
  • Policy and vision
  • Multisectoral collaboration
  • Population and civil society participation
  • Information and (digital) knowledge
  • Legislation and regulation
Resource generation
  • Health workforce
  • Infrastructure and medical equipment
  • Pharmaceuticals and consumables
  • Governance of resource generation
Financing
  • Revenue raising
  • Pooling
  • Purchasing
  • Governance of financing
Service delivery
  • Public health
  • Primary care
  • Specialist care
  • Governance of service delivery

Boxes

Box 1.1OECD Health Statistics and Health at a Glance reports

Since the mid-1980s, the OECD has published health statistics on health expenditure and different aspects of performance (OECD, 1985), and the first electronic edition of the OECD health database was released in 1991.

Since 2001, the OECD has been releasing Health at a Glance, a publication presenting a comprehensive set of key indicators on population health and health system performance in OECD countries and partner countries. These indicators are underpinned by a conceptual HSPA framework first developed in the context of the OECD workstream on healthcare quality and outcomes. The latest edition of Health at a Glance was released in November 2023 (OECD, 2023a). Since 2010, the OECD has also released every even year regional editions of Health at a Glance, including Health at a Glance: Europe jointly with the European Commission covering 38 European countries (OECD/European Union, 2022). Other editions are also available for the Asia-Pacific region and the Latin America and Caribbean region.

All versions of Health at a Glance utilize the OECD Health Statistics database. This database gathers information through two annual joint data collections by the OECD, Eurostat and WHO on health accounts (for health expenditure and financing data) and non-monetary healthcare statistics (for health workforce and healthcare resources and activities data), as well as separate data collections from the OECD (for example for healthcare quality indicators and access to care indicators), Eurostat (such as population-based surveys such as EU-SILC and EHIS), and other sources.

Alongside indicator-by-indicator analysis, an overview chapter in the OECD-wide edition of Health at a Glance summarizes the comparative performance of countries and major trends across key dimensions and indicators.

Explore Health at a Glance and OECD Health Statistics by visiting: https://www.oecd.org/health/health-at-a-glance/ https://www.oecd.org/health/health-at-a-glance-europe https://www.oecd.org/health/health-data.htm

Box 1.2WHO Regional Office for Europe’s Health for All database of health system indicators

Since the mid-1980s, Member States of the WHO European Region have been reporting essential health-related statistics to the European Health For All database (HFA-DB), making it one of WHO’s oldest sources of data. The HFA databases play a pivotal role in consolidating indicators from major monitoring frameworks for the WHO European Region, such as the European Programme of Work Measurement Framework and the Sustainable Development Goals (SDGs). Encompassing a broad spectrum of data, these indicators span basic demographics, health status, health determinants, risk factors, healthcare resources, expenditures, and more.

The comprehensive nature of the HFA databases facilitates a better understanding of health-related trends and patterns, making it serve as a key player in shaping evidence-based decisions, at both national and international levels. To maintain the relevance and timeliness of the data, the HFA-DB undergoes annual updates, collecting information from Member States and other international sources, with the latest being produced in October 2023. These updates ensure that the information presented represents the most current snapshot at the time of publication.

Explore the HFA-DB indicators interactively by visiting: https://gateway.euro.who.int/en/datasets/european-health-for-all-database/

Box 1.3European Observatory’s Health system in Transition series for country monitoring

Since 1998, as part of its Health Systems in Transition (HiT) series, the European Observatory on Health Systems and Policies has systematically described the functioning of health systems in countries, as well as reform and policy initiatives in progress or under development. The HiT health system reviews cover the countries of the WHO European Region as well as some additional OECD countries. They are updated on a regular basis.

The reviews are based on a template that provides detailed methodology for assessing health systems in a comprehensive and comparable way (Rechel, Maresso & van Ginnekin, 2019). The initial chapters describe the details and evaluate the health system organization and governance, financing, human and physical resources, service provision and the latest reforms. The assessment chapter focuses specifically on how health systems perform over time and compared to other countries. It explores governance, and follows on to analyse the accessibility and quality of health services (including primary and specialist care, as well as service integration), equity and efficiency, and the degree of financial protection, using multiple sources of information, including the OECD, Eurostat and WHO databases, as well as expert knowledge.

The HiT series is complemented by Health System Summaries – concise, engaging and policy-friendly reports on the main elements of a country’s health system. They analyse core evidence and data on the organization, financing and delivery of healthcare, and provide insights into key reforms and the varied challenges testing the performance of the health system.

Explore the Health Systems in Transition reports and the Health System Summaries by visiting: https://eurohealthobservatory.who.int/publications/health-systems-reviews

Box 1.4Scope of this brief: a summary

  • This brief is a proof of concept that intends to test how a suite of relevant, high-level tracer indicators from the practical perspective of decision-makers looking to monitor or improve performance can be developed.
  • It takes a policy question as the starting point to determine what indicators can help understand the performance of a health system function or subfunction, and their impact on other elements of a health system.
  • It demonstrates how a selected set of tracer indicators linked to the HSPA frameworks can help decision-makers explore a policy question, identify performance issues, and make better informed policy decisions.
  • It uses tracer indicators to offer insights into potential causes, flag limitations and discuss policy options, as well as serving to support, evaluate and monitor the impact of new policy initiatives.
  • Finally, it establishes the foundation for the development of a policy-focused HSPA dashboard of key indicators.

Box 1.5Policy questions and key performance indicators: an approach and methods

The brief’s identification of key performance metrics starts by establishing a policy question and then selecting a set of tracer indicators that provide system-level information to address that question.

The notion of using “markers” or tracers in health systems is not novel, however, but dates back to Kessner, Kalk & Singer (1973), who advocated using tracer conditions (frequent, well defined conditions with known epidemiology that have agreed pathways of appropriate care) to assess the quality of health services. The concept was taken forward by Nolte, Bain & McKee (2006), using diabetes as a tracer condition in international benchmarking of health systems with a diabetes mortality-to-incidence ratio seen as a simple way of differentiating quality of care for people with diabetes in different countries. There was a clear understanding that tracers could not give precise measures of the scale of a problem but their value as indicators of potential problems and as prompts for further investigation was recognized.

This brief builds on previous thinking and aims to identify some tracer indicators for the key areas of workforce, digital health and people-centredness, as well as access and quality as outcomes of service delivery. The choices of these areas and the elements explored under those headings were based on several factors, including: a) policy priorities presented at the Tallinn Conference and the OECD’s 23 January 2023 Meeting of Health Ministers (which in turn were defined in consultation with an extensive range of policy-makers and other stakeholders); b) practical experience of policy-makers and health system transformation during and post-pandemic; c) the insights offered by the international initiatives engaged in the development of the HSPA frameworks (including UHC 2030 Technical Working Group), which highlighted how connections work across functions, subfunctions, assessment areas, and intermediate and final goals; and d) consideration was given to the availability of comparable data, albeit working on the premise that lack of comparable data alone should not preclude inclusion of an indicator that is relevant (rather, this should serve as a call for improved data collection).

The tracer indicators were identified based on a HSPA literature review, including examination of indicators presented in the OECD’s Health Statistics database and Health at a Glance and the WHO’s Health for All database and reports on the workforce (WHO Regional Office for Europe, 2022) and digital health (WHO Regional Office for Europe, 2023), expert consensus within WHO and OECD working groups, as well as wider consultation. The selected areas and tracer indicators are not exhaustive and the indicators are intended to offer a glimpse at performance, monitoring at system level, and benchmarking where possible. They are not seen as a tool to address all the specifics of the processes within health system functioning but rather as a starting point for identifying key methodological principles and shortcomings, and to present domains that need further analysis.

The brief serves as a proof of concept and establishes the foundations for further work on populating the HSPA frameworks with tracer indicators and on policy dashboards.

Image app1.f1
© World Health Organization, 2023 (acting as the host organization for, and secretariat of the European Observatory on Health Systems and Policies) and OECD)
Bookshelf ID: NBK609570
Contents