The Convergence of Evidence-based Health Care and Quality Improvement

• Introduction
• Evidence-based Management Decisions
• Why do we Not Have Evidence-based Quality Programmes?
• Methodological Difficulties in Evaluating Quality Programmes
• Research Designs for Evaluating Quality Programmes
• How to Assess Research into Quality Programmes
• Conclusion
• References

Introduction

The purpose of evidence-based clinical practice is to decide the best treatment for a patient by using research evidence and clinical experience. This has always been an aim of medical practice: the “evidence-based practice” movement contributes new methods for finding relevant clinical research publications, and for assessing the validity of the research for a particular patient’s problem. The movement is part of a wider trend, termed “evidence-based health care”, which has received a boost from new developments in information systems, and from governments hoping to contain costs (and avoid rationing) by increasing the effectiveness of health care treatments and services.  ¹  

Evidence-based clinical practice, like the quality movement, is viewed by some as another fashion. It has its limitations, especially when viewed as a solution to all problems and applied indiscriminately by new converts. One of the limitations is a narrow view of evidence which rules out the findings from much research into the effectiveness of services, policies and health promotion programmes. However, the principles of basing decisions on research evidence are of general importance in health care and are having an increasing impact on clinicians’ thinking and practice.

From another direction, the “quality movement” has had an increasing impact on health care. This movement is perhaps less unified in terms of methods and theories. A feature of the quality movement is an emphasis on measurement and “fact-based quality improvement”. This aspect of the quality movement has not always had the attention it deserves in health care, which is one reason for the critical view many doctors hold of some quality initiatives  ²  ). Another reason is that quality methods have only relatively recently been applied to clinical care rather than to what clinicians have viewed as “trivial” or “irrelevant” issues. This has been a vicious circle – where clinicians have seen quality methods as not relevant, they have not become involved in programmes nor applied the methods to clinical issues. In part, this has been due to poor training in the early programmes.

More quality specialist and clinicians are recognising the compatibility between evidence-based health care and quality improvement at the clinical level in both philosophy and methods. These two approaches have converged in guideline development, audit, pathway analysis and in other areas. It is now easier to teach doctors about quality methods by showing the similarities between medical research and “applied organisational research”, which is one way of describing some approaches to quality improvement. The evidence-based health care movement has made it easier to gain doctors’ interest in examining systems of care and in making fact-based improvements – if they have the time and resources.

Evidence-based Management Decisions

This convergence has, however, begun to make managers’ lives more difficult and has highlighted a weakness of many quality programmes in health care organisations: the lack of an evidence-base for the programme. Management decisions are rarely evidence-based. This is partly because there are often more political and value considerations involved in management decisions than in clinical decisions. There is also a lack relevant research and a lack of skills to access and assess management research but, perhaps most of all, a lack of an “evidence-based attitude” on the part of managers. Management methods such as certain budgeting techniques, or personnel appraisal methods are “management technologies”. Such technologies, like medical technologies, need to be evaluated for their effectiveness, but often using different evaluation methods.  ³   Grol notes the lack of research into interventions to organisations which are intended to produce improvements in the quality of care,

. . . research on many interesting strategies is lacking. The effects are largely unknown of organisational development, team building, re-engineering complex care processes with many care providers involved, enhancing leadership in institutions, changing tasks and responsibilities of care providers, or introducing specific financial incentives and economic policies on causing changes in practice performance.  ⁴  

Nowhere is the absence of an evidence-based approach to management more apparent than in planning or reviewing a quality programme. It is true that managers and quality specialists sometimes gather data about personnel attitudes to quality, or about where the main quality problems are when they are planning a quality programme. But they rarely use research into which methods or approaches worked best elsewhere or learn the lessons from research into failed or successful projects in their own organisation. There is a need to develop management technology transfer capabilities in health care.  ⁵  

The argument of this paper is that future quality programmes need to be more evidence-based, for two reasons. First, to make such programmes more successful by basing them on more reliable knowledge; and, secondly, to more fully involve clinicians. Clinicians have rightly criticised such programmes for failing to practice what they preach – for failing to view the quality programme itself as a service to health personnel who are customers for it, and for failing to gather and use data about the progress of the programme in the organisation.

Definition: Evidence-Based Quality Programme
An evidence based quality programme is one which is planned using research findings about the effectiveness of different quality methods in comparable organisations, as well as evidence from within an organisation about the progress and effects of its own quality activities.

Why do we Not Have Evidence-based Quality Programmes?

What are some of the reasons why managers and quality specialists have tended not to use evidence in designing programmes? One reason is that it takes time to gather and analyse evidence about our own organisation and its quality performance. Yet these costs must be viewed in relation to the savings to be made from discontinuing ineffective projects and the gains that will come from more effective interventions to the organisation. Secondly, evidence tends not to be used because managers recognise that the progress of the programmes depends on many political and “non-rational” factors, and that evidence from the past or from others may be of little use in their own situation. However, gaining data about and understanding the actual process of implementation is necessary in order to make realistic changes to programmes, either to accommodate the political interests or to change them. Thirdly, there is a fear that the evidence will reveal that the programme has had little effect: it takes time for the effects to have an impact. Those working on the programme need to believe they are making a difference if they are to continue to devote the time and energy which such work needs.

There are also reasons why there is a tendency not to use evidence about quality programmes carried out elsewhere. Managers and quality specialists do not have the time or skills to search for reported evaluations, or to assess the scientific quality and relevance of the reports. Many lack the expertise to assess non-experimental and social science research in health care, and many evaluations of quality programmes use such designs. The reports themselves are also at fault: many are of poor quality and do not describe the quality programme or the health care organisation in sufficient detail for others to decide whether the findings are relevant to their own organisation, or how to replicate the interventions. Perhaps the most important reason is that few evaluations produce conclusive evidence; evaluating a hospital quality programme is more difficult than evaluating a treatment or a service.

Methodological Difficulties in Evaluating Quality Programmes

If managers and others are to make a greater use of internal or external evaluations of quality programmes, then they need to understand some of the difficulties evaluating these interventions and the research designs which have been used. A quality programme is not like a drug treatment; it is a set of activities for introducing methods for ensuring the quality of services provided by a health care organisation, such as a hospital. As well as clinical and other services to patients, these services include internal clinical and administrative services from one department to another and services to referring doctors and others outside the hospital. A programme may be a carefully planned and implemented set of training and organisational changes which are co-ordinated, or it may be a series of unrelated activities which, in retrospect, are described as a quality programme. In evaluation terms, a quality programme is a multiple component intervention to a health care organisation and one which aims to improve the quality of these services.  ⁶   In the health services we are more familiar with methods for evaluating treatment interventions to patients than with those for evaluating social interventions (“management technologies”) to complex health care organisations operating in a volatile and changing political context. Full experimental evaluation designs such as randomised controlled trails (RCT) are not possible.

The task of evaluating a hospital quality programme poses challenges to both experimentalist and non-experimentalist or “naturalistic” evaluators. The first challenge is that the intervention is difficult to specify. Although the aims of some quality programmes may be similar, the details of a quality programme in each organisation are different. Even total quality management programmes, which are supposed to follow similar principles, are different in the details of their implementation. Indeed it is thought that one determinant of success is the ability of management to adapt the principles of quality to their local circumstances and to design a unique programme which is suited to the culture, history and business environment of the organisation. Quality programmes are interventions which are often different from the planned intervention and are changed as they proceed. Many evaluation reports can be criticised for not giving a good description of the hospital quality programme, but this is not just a failing of the report: skilled researchers who wish to describe quality programmes face a number of problems identifying exactly what the intervention is or was. None of the features of the intervention (the quality programme), of the object of the intervention (health care organisations) and of the context (the political, financial and other circumstances affecting a hospital over a period of years) can be controlled and make it difficult to be sure that measured outcomes are solely produced by the quality programme. These problems make it difficult for others to be sure that they would get the same results if they used the same intervention, even where the evaluation gives a clear description of the details of the particular programmes which were evaluated. These problems do not mean that experimental principles should be abandoned entirely in favour of descriptive “process” designs with, at most, speculations about causality. Similar challenges face those evaluating health promotion programmes, and evaluations in this field have developed methods, concepts and models of mechanisms which can be drawn upon to improve evaluations of quality programmes and of other management technologies.  ⁷  

Research Designs for Evaluating Quality Programmes

In spite of these problems, a number of internal and external evaluations of quality programmes have been carried out. A review of research for the evaluation of the Norwegian Total Quality Management Programme found that the reported evaluations could be classified into the following ten designs: Surveys; Single case descriptions; Single case audit; Multiple case descriptions; Multiple case descriptions with case control(s); Single case, with evidence of effects; Multiple case, comparison of effects; Multiple case with control(s) and comparison of effects; Multiple case, comparison of standard measures of effects; and Action evaluations. Each type of design is appropriate for particular circumstances.  ²  

Can managers or their assistants learn the skills to assess such research, with the variety of different designs and data gathering methods used? One way to help managers to assess relevant research is to grade the type of design in a hierarchy, with designs which give more certain knowledge of the effects of the programme at the top of the hierarchy – a technique used in evidence-based medicine. For some questions and types of research such a hierarchy is useful and appropriate, and one has been used for assessing research into outcomes of quality programmes.  ²   However, there are limitations to this approach, even if one accepts the assumptions of the experimentalist paradigm. Many studies examine different stakeholders’ perceptions, not just of outcome but also of how the management change was implemented over time, and in the case of services, patients’ experiences during their contact. Controlling for confounding variables prospectively or retrospectively by statistical analysis is not the only way to judge whether other factors apart from the intervention have an influence. In addition, these hierarchies are of designs only: evidence from a poorly conducted but highly rated design can be less valid than evidence from a design with less good controls for confounders which has been carried out meticulously.

A second way to assess research is to use a general purpose assessment system for most types of research into quality programmes. The section ‘How to Assess Research into Quality Programmes’ below, notes how managers can use evidence from different types of research to formulate their own programmes and to improve the methods they use for collecting data about the progress of their programmes. These ideas are also useful for reviewing other projects within an organisation, such as the implementation of a policy to reduce absenteeism or a training programme.

What is the evidence of the effects of a quality programme? This is a common and important question. Sometimes evidence is available about the effects of a quality project, in part because measurement is an important part of such a project. But there is a surprising lack of evidence about the effects of large scale quality programmes such as those in hospitals. In the following, “effects” will be taken to mean the imputed effects of the quality programme: the ultimate effects on patients (“patient effects”) and intermediate- or short-term effects on health personnel and organisation (“provider effects”) in terms of a measured or perceived change.

Quality theories predict that a properly implemented quality programme will result, after a period of time (eg 3–5 years), in increased patient satisfaction, better medical outcomes, and lower resource utilisation (ie increased patient-, professional-, and management-quality.  ⁸   Most theories posit a mechanism involving, first, changes to personnel attitudes and behaviour arising from training and management action; and then both health personnel and management making changes to organisation and processes, which ultimately produce changes in quality of patient care and a reduction in resource utilisation. “Effects” are thus those predicted by quality theories and intended by management who implement quality programmes.

Data gathering and analysis
To assess an evaluation which used one of the ten designs noted above, we need to consider the data gathering and analysis methods which were used. Evaluations of quality programmes differ in the methods they use for data gathering and analysis. Even if they use the same methods, two studies can still produce data of different quality in terms of the reliability and validity of the data. To assess a report, users need to assess the quality of the data as well as the “power” of the design: a less powerful design may produce evidence of higher quality if the data gathering was more rigorous than the data gathering used within a more powerful design.

One way to assess data quality is to judge whether the methods chosen were most likely to produce the data needed to answer the study questions or to test hypotheses.  ^9  ,  3   This criterion of “data method appropriateness” is closely related to the choice of design. A second criterion against which to assess data quality is to judge whether the procedures for data gathering and analysis followed are generally accepted as being correct for the method chosen. Procedures for ensuring validity and reliability are different for quantitative and qualitative data, but there are still agreed procedures for the correct use of different qualitative methods.  ¹⁰   A third criterion for assessing data quality is whether multiple sources of data were used.

How to Assess Research into Quality Programmes

Given the problems evaluating quality programmes, and the variety of designs used, how are managers and others to proceed if they wish to create evidence-based quality programmes? Here we will concentrate on the use of reported evaluations rather than consider how to create an internal evaluation process. One approach is to ask experts to search for and assess evaluation reports for their validity and relevance to the health care organisation in question. (Few studies of this type are listed in the MEDLINE database: the relevant databases to search are ASSIA (Applied Social Sciences Indexes and Abstracts), the Health and Health Administration Index (also HealthStar, PNA Research and Anbar Management Intelligence), and the Leeds University HELMIS data base). However, many managers and quality specialists do not have access to this evaluation expertise, and many experts are only knowledgeable about a few types of evaluation approaches. A further disadvantage is that by using experts to assess evaluations managers and quality specialists miss an opportunity to learn how to develop internal systems for evaluating their own quality programmes: assessing others’ evaluations is a good way to decide ways in which to improve your own organisation’s self-evaluation processes.

The proposed approach is that managers and others use the following guideline questions to assess evaluation reports of quality programmes and other organisational interventions. However the questions should be applied remembering the following: