Evaluating Integrated Care Demonstration Sites

• Note
• Introduction
• Hypotheses to be Tested Using the Demonstration Sites
• Integrated Care Initiatives and the HFA
• Focus of the Evaluations
• Approach to Evaluating the Current Initiatives
• Skills Necessary to Perform the Evaluation
• References

Note

The material contained in this paper is based on work conducted when the author was a member of the Midland Division of the Health Funding Authority. However, the views expressed are entirely the authors and do not necessarily reflect those of the HFA.

Introduction

In New Zealand, as in other countries around the world, the responsibilities for purchasing and providing health services are separated. The Health Funding Authority’s (HFA’s) role is to purchase services from providers to achieve the maximum health gain for the people of New Zealand within the available budget. This includes purchasing primary care from general practitioners, second and tertiary care from Crown Health Enterprises (CHEs) and private hospitals, pharmaceuticals from dispensers, and maternity services from midwives. Although each provider’s main concern is to deliver high quality care, the ultimate responsibility for ensuring that patients have access to the appropriate types of services remains with the HFA.

The structure of the health sector creates a number of challenges for the purchaser, including making sure that patient care is co-ordinated, ensuring no unnecessary or duplicate services are purchased, and monitoring to ensure that patients have access to the appropriate types of services. Although it is too early to tell how the newly re-organised HFA will approach purchasing, it has announced that integrating care across providers will be an integral part of its new focus. Central to this effort is its integrated care initiative   ¹   in which eighteen projects are to be designated as demonstration sites. The stated intent of the initiative is to examine a number of ‘hypotheses’ regarding the integration of care in New Zealand as described in the Guidelines for the Development of Integration Demonstration Projects.  ²  

Calls to integrate health services are not new in New Zealand.  ³   Numerous providers over the years have sought to improve the co-ordination of patient care across primary and secondary care in order to provide seamless coverage. However, the HFA’s initiative differs from those previous attempts in several important ways. First, the HFA has more tools at its disposal than previous provider groups. Whereas provider groups must rely upon voluntary attempts to communicate and co-operate with each other, the HFA can use financial incentives and purchasing agreements to achieve the desired results. Thus, this allows the HFA greater flexibility in seeking ways to achieve co-ordination of care among providers.

A second and potentially more important distinction is that, as the purchaser of health services, the HFA is concerned with providing services within a limited budget. Managing its financial risk is a stated objective of integrated care initiatives.  ⁴   Furthermore, the definition of integrated care adopted by the HFA  ⁴   is sufficiently broad to encompass projects ranging from the promotion of clinical guidelines for treatment of a specific illness to the total transfer of a capitated budget for all services in a given area. Thus, the potential exists for adopting initiatives that are fundamentally different from the current purchasing arrangements. This is supported by statements from both the Minister of Health   ⁴   and the HFA.  ⁵  

The current initiative is an opportunity to pilot a range of different approaches to integration. However, identifying the successful approaches projects requires a thorough evaluation of the initiatives. The primary role of the evaluation is to identify the impact of the integrated care projects on access to health services in New Zealand. Evaluating integrated care requires the HFA to examine factors such as the effectiveness of care, patients’ access to and satisfaction with health services and the cost of the care. Thus, in many ways, the motivation and approach to evaluation are similar to other types of programme assessment.

Yet the reasons for evaluating integrated care initiatives go well beyond those associated with most projects, and include the following:

a) To identify successful approaches for achieving integration
The purpose of the demonstration sites is to guide the HFA in determining the future of the health sector. The initiatives will only be successful if the lessons learned from the projects can guide the choice of future initiatives. To accomplish this, the HFA must identify those aspects of the projects that have led to successful integration and those that have not. Such a determination requires a more thorough evaluation of the projects than is normally conducted.

b) To understand changes to the health sector
The HFA recognises that the integrated care initiatives have the potential to significantly alter the structure of the health sector. The impact of the restructuring is likely to be felt by those who deliver health services, including providers, hospital and clinical administrators, nurse practitioners and midwives, Maori providers, and the general public. As a result, the evaluation must focus upon a wide range of groups and at many different levels.

c) To anticipate political risks to the HFA
The changes to the health sector create a number of political risks that the HFA must monitor closely. Anticipating the political risks requires an understanding of how the integrated care initiatives are likely to affect the health sector. An evaluation will provide the information necessary for the HFA to make well-informed policy decisions.

The purpose of this paper is to provide insight into the likely focus and challenges facing those evaluating the integrated care initiatives. The next section describes the hypotheses that are to be tested using the integrated-care demonstration sites. Although the final decision regarding the structure of the evaluation has yet to be announced, there are some unique problems that must be taken into account. The challenges facing those evaluating the integrated-care demonstration sites are outlined in the next section, and an example of how the evaluation team might be structured is presented.

Hypotheses to be Tested Using the Demonstration Sites

In its Discussion Paper on Integrated Care, the HFA stated its intent to test national hypotheses regarding integrated care.  ⁴   The specific hypotheses to be tested were described in the Guidelines for the Development of Integration Demonstration Projects.  ²   They include:

Hypothesis 1
“Decision making guidelines for service provision improve health outcomes and are cost effective.”

Hypothesis 2
“Contracting strategies that align incentives and promote collaboration across traditional service boundaries improve health outcomes and are more cost effective.”

Hypothesis 3
“Integrating service delivery is dependent upon integrating funding streams.”

Hypothesis 4
“Budget responsibility for a wide range of primary care services improves health outcomes and is cost effective.”

Hypothesis 5
“Budget responsibility for a specified bundle of services (eg, mental health or disease states such as asthma or diabetes) improves outcomes and is cost effective.”

Hypothesis 6
“Budget responsibility for a full range of services improves health outcomes and is cost effective.”

Hypothesis 7
“Integrated service networks run by Maori for Maori are more effective in contributing to improved health outcomes than mainstream integrated service networks.”

Hypothesis 8
“Consumers who make a choice about health service options have improved health outcomes and make more cost-effective choices.”

The breadth of the hypotheses is evidence of the HFA’s desire to entertain innovative approaches toward integration. Yet comparing the initiatives to test the hypothesis is likely to be complicated. For instance, suppose it was thought that a computerised system providing primary care providers with patient utilisation of secondary care services would facilitate the integration of care. There are a number of different ways this system could be implemented. One option would be for the HFA to contract directly with an information technology group to provide the information system. An alternate approach would be for the HFA to transfer both primary and secondary care budgets to primary care providers, thus giving them an incentive to fund and use the system. In each case the basic approach to integration is the same – provide better information to providers – but the projects differ in their basic assumptions regarding what is needed for integration to actually occur.

Integrated Care Initiatives and the HFA

As mentioned above, one aspect that distinguishes the integrated care demonstration projects from other health initiatives is their potential to change the very nature of the purchasing environment. In transferring the financial risk, the HFA is also transferring purchasing responsibilities. It is this aspect of the integrated care initiatives that has the potential for altering the basic nature of the health care system. Many of the difficulties that arise in testing the hypotheses occur because the evaluation must focus on both the clinical and health policy outcomes.

The extent to which the role of the HFA changes depends upon the contractual arrangement between the HFA and the provider organisations. There are five types of relationships that can arise:  ⁶  

1. HFA purchases specific services
One option the HFA has for achieving integration is to purchase the specific services and support systems it sees as necessary in order for integration to occur. For instance, suppose it were proposed that a computerised system tracks care for diabetics in a specific area and the results from the tracking are to be distributed to providers (with the goal of improving co-ordination among providers). This style of project would involve the HFA identifying the type of system needed, and then contracting directly for the establishment of that service. The HFA might consult with providers during the planning phase in order to ensure that the service is valued and will lead to improved delivery of health services. However, the ultimate responsibility for purchasing the project would lie with the HFA.

2. HFA contracts with providers to met specific performance criteria
A second type of integrated care arrangement involves the HFA contracting with providers for specific patient outcomes. This approach differs from the previous one in that the HFA would not identify the specific way in which integration was to be achieved, but rather would provide an incentive to the necessary parties and allow them to determine the mechanism for achieving these objectives. This would require the HFA to identify the relevant parties and then contract directly with them.

3. HFA contracts with ICO’s to met specific performance criteria
Closely related to the integrated care approach discussed above is the situation in which the HFA would contract directly with an integrated care organisation (ICO) to provide services. The ICO would then contract with individual providers for their services, with the HFA paying bonuses and other incentives directly to the ICO, which would then redistribute the profits to the providers.

4. HFA transfers purchasing responsibility and risk to ICO
Increases in medical expenditures can result from inefficient care (eg, inappropriate or unnecessary services being provided) or from increases in the cost of required health services. Providing incentives to providers (2) or ICO’s (3) according to their performance on a set of criteria is one strategy for ensuring that all procedures are provided in an efficient manner and that care is integrated. However, in each case the HFA would remain ultimately responsible for purchasing health since it pays for any increases in costs.

An alternative arrangement is for the HFA to pass budgets to an ICO, the HFA paying a set fee to an ICO for each patient enrolled with its providers. This fee could cover all services (ie, primary, secondary, mental health, disability) or only part of the patient’s health care. Unlike the previous approach, the fee would not be determined according to whether or not a set performance criteria is met, but would be predetermined. The distinguishing feature would be that the ICO holds the risk of budget overruns, paying for health care out of this budget and keeping any remaining profits. Thus, the ICO would determine the performance criteria and purchases health services. The HFA’s role would be reduced to monitoring outcomes to ensure quality care is being provided.

5. HFA transfers purchasing responsibility to users of the system
The final option is for the HFA to transfer the budget directly to the patient. This would remove the purchasing responsibility totally from the HFA (other than for public health measures). The HFA would remain the funder of health services by providing individuals with a “health voucher” but individuals would be given great latitude in choosing the types of services they wished to purchase. No longer would the HFA or an ICO have to determine the performance criteria as it would be left to the patient to decide the types of services to purchase.

Integrated care proposals of the first type listed above are likely to have little impact on the health sector. This is not to say that they cannot lead to successful integration of care, but rather that the purchasing environment would not be dramatically altered. The second and third types of integrated care arrangements represent a change in the way services are purchased, but the ultimate responsibility for purchasing would remain with the HFA. The last two types, however, represent a devolution of purchasing responsibility to either an ICO (as in the fourth type) or individuals (in the fifth type).

It is this potential for altering the basic health structure that makes evaluation of integrated care unlike most other evaluations. As such, the evaluations must focus on a much broader range of issues than is normally considered.

Focus of the Evaluations

The above discussion highlights an important point: Evaluating integrated care initiatives is much more complicated then a typical programme assessment. The evaluations must focus upon the effects at three different levels:

Level 1: Effect of integrated care on provider/patient relationship
At the most basic level, integrated care involves improving people’s access to health services. This will result in a change in the way providers offer services to meet the patient’s needs. The evaluations must identify how providers change the way they deliver care to their patients and the effect of these changes on access to services.

Level 2: Organisational structure
Integrated care is said to occur when different groups of providers co-ordinate their services to patients. This co-ordination would require an organisational structure in order to implement these changes. A second focus of the integrated care evaluations is to identify the types of organisations that allow integration to occur.

Level 3: Impact on the Health Sector
Integrated care projects have the potential to significantly alter the purchasing environment by transferring many functions now performed by the HFA to either third parties or individuals. It is this devolution of purchasing that creates the greatest difference between types of proposals. In transferring the financial risk, the HFA is also transferring the purchasing responsibilities. It is this aspect of the integrated care initiatives that has the potential for altering the basic nature of the health care system. Thus, the evaluations must consider the impact of the initiatives within the context of the New Zealand health system.

Typical evaluations of health initiatives focus only upon the first of these levels. Evaluating the initiatives on the other two levels requires not only an understanding of evaluation methodology, but also the ability to place the evaluation within the context of New Zealand’s health system.

Approach to Evaluating the Current Initiatives

The typical approach to evaluating the effects of a health policy is to compare the results from one area where the policy is introduced with another where no policy is in place. Evaluating the current initiatives is in New Zealand is complicated by the manner in which the initiatives have been developed. Rather than dictating to providers the types of projects it will fund, the HFA has opted to allow the providers to develop their own initiatives. This approach has the advantage of allowing the health professionals who actually work with the patients to develop approaches for integrating care.

The downside of this approach is that it makes it difficult to test national hypotheses. The initiatives emerging from this process are likely to differ significantly in their scope, targeted population, specific interventions, geographical location, structure of the regional health sector, and health needs of the population. As a result, each project will require its own unique evaluation. The HFA is therefore faced with the need to tailor the evaluation to fit the specific proposal while simultaneously allowing the national hypotheses to be tested.

The potential lack of comparability between sites means that the HFA must alter its approach to evaluating the proposals. Rather than compare two sites directly, the HFA must first establish a general framework for understanding the impact of initiatives. This framework must place the integrated care arrangements within the context of New Zealand’s health sector. Out of this framework will emerge the risks and benefits from each type of initiative at each of these three levels. The evaluation team should then use the actual evaluations to inform the debate about which approaches are and which are not successful in achieving an integration of care.

Skills Necessary to Perform the Evaluation

The evaluation of the initiatives requires three stages: (1) design of the evaluation; (2) monitoring and gathering data; and (3) interpreting the results from the initiatives. The skills necessary to perform each of these tasks vary greatly. Designing the evaluation requires a specialised set of skills, including clinical expertise (public health medicine, medical demography, health economics and management, research design, and epidemiology). The first step is to identify the specific areas to monitor. This will depend greatly upon the scope of the project. For instance, a project examining the use of clinical guidelines will require little organisational change (level 2 above) and have minimal impact on the health sector (level 3). On the other hand, an initiative that transfers both secondary and primary budgets to a private company has the potential to significantly alter patient care (level 1), involve a unique organisational structure (level 2), and will have implications for the structure of the health sector (level 3). Thus, the projects will differ greatly in the resources needed to evaluate the outcomes.

Interpreting the results from the evaluations also requires a highly skilled group that can understand not only how to evaluate specific projects, but also how the integrated care initiatives fit into the health structure of New Zealand. Specifically, this group would have to: (1) develop conceptual frameworks for each of the three levels; (2) identify the types of information that must be collected for each project; (3) analyse the results from the evaluations; and (4) interpret the impact of the initiatives within the context of New Zealand’s health sector. While there are strong arguments for housing the evaluation team outside the HFA, the team must work closely with the integrated care group within the HFA. That is, the HFA will be responsible for working with the providers to ensure the proper information is being collected and that the information to is accurate. The HFA must understand the evaluation design, the constraints facing by the providers, have the information systems necessary to collect and manage the incoming data, and develop working relationships with the providers.

References