Evaluating Quality and Quality Measures for Comparison

• Abstract
• Introduction
• Quality Evaluation
• Organisational Quality Assessment
  • Organisational assessment schemes
  • Discussion
  • Research
• Comparative Quality Measurement
  • Examples of quality indicator systems and projects
  • Discussion
  • Lessons from experience collecting and using quality indicators
• Conclusions
• References

Abstract

By 2005 all health care organisations in Europe will be required to take part in a quality evaluation scheme and to collect data about the quality of their services.

This prediction was made recently at a workshop of Nordic quality experts. The pressures on health care organisations to assess quality are increasing, and there are many assessment, certification, accreditation and measurement schemes in use. Which is best? What evidence is there that any have been effective? How should a hospital or region introduce such a scheme? There are many proponents for different schemes, and an increasing amount of experience to help answer these questions, but little research.

This paper gives a simple overview for non-specialists of the different quality evaluation and indicator schemes for inspection and improvement. It draws on the experiences of quality specialists and leaders in each Nordic country who have applied the schemes in public hospitals and health care services.

How a scheme is introduced and used may be more important than which particular scheme is chosen. This is one conclusion of the Nordic workshop. Other conclusions are that clinicians must be involved, simplicity and low cost must be balanced with scientific validity and credibility for clinicians and there must be research into the different schemes to discover their costs and benefits to health care.

Key words: quality, evaluation, measurement, indicators, audit.

Introduction

Assessing and measuring the quality of a health service is expensive and time-consuming, particularly without suitable information technology. Why bother? One reason is that health services are required to by government or by health care purchasers, or soon will be. Other reasons are that the time and money spent can be worth it if they help to prioritise quality improvement work, increase health workers’ understanding of quality, and identify providers who could be a danger to patients. But do any of the schemes fulfil these functions? Have the schemes themselves been assessed?

Health care leaders and advisers need to know which scheme to choose, if they have a choice. Each scheme has strengths and weaknesses, and needs to be carefully introduced if an organisation is to gain some benefit for the time and money invested. The purpose of this paper is to describe the different approaches and note some experiences with each scheme - unfortunately there is little research evidence about cost benefits or about how best to introduce a scheme. This paper draws on a three-day meeting of Nordic quality specialists who presented and discussed their colleagues’s experience in each Nordic country as well as published papers on the subject.

Quality Evaluation

Audit and peer review schemes for medical specialties have been running for some time and are well-known within health services. Examples are the UK confidential enquiry into peri-operative deaths ¹ and the Swedish GP peer-review audits ² . There are also audit and peer review schemes for the nursing and therapy professions and for laboratories. However, in the 1980s more schemes were tested that assessed organisation and management of health care rather than the clinical content. Examples are the European Foundation for Quality Award, the ISO 9000 certification, the Joint Commission on Accreditation of Health Care Organisation scheme (JCAHO) and the Kings Fund Organisational Audit. There are versions of all of these in use in most of the Nordic countries and in many other European countries.

Other examples of quality evaluation methods are those for collecting and comparing quality indicators such as the Maryland Hospitals Comparative Indicator project, the UK performance indicators and the JCAHO indicator scheme, as well as local benchmarking schemes and the medical registers or data bases run by the medical profession in the Nordic countries.

There is a range of approaches for quality evaluation, each scheme being developed within a specific context and designed for a particular purpose. Grouping them all together as "quality evaluation" tends to obscure the differences. However, these schemes are being used increasingly and used for a variety of purposes. They appear to meet the growing need for some type of external evaluation of quality of care: to balance the existing exclusive focus on economic performance, to respond to current concerns about provider quality and, last but not least, for methods to help providers to prioritise areas for improvement and to learn about what needs to be done to prove and improve quality.

"Quality evaluation" is a term used in this paper to refer to two main approaches:

Organisational (or clinical) quality assessment or audit: comparing a statement about what a service should do with what the service actually does, usually through an external assessment of a service.

Comparative quality measurement: comparing measures of quality of the same service at different times or of the services to other services at the same time.

Although different approaches, both assessment and measurement schemes are ways of evaluating the quality of a service and are increasingly being used together. In 1999, the JCAHO required health facilities as part of their accreditation, to collect and report a specified set of quality indicators. Other organisational assessment schemes are also developing quality indicators to be used by assessed organisations.

Organisational Quality Assessment

This paper uses the term "organisational quality assessment" to cover a range of accreditation, certification and external and self-assessment schemes. Clinical quality assessment schemes such as medical specialty peer review will not be considered, although, it will be argued, a health care service such as a hospital needs both organisational and clinical external quality assessment.

Organisational quality assessment schemes are ways of assessing the quality of a health care service. The schemes are usually run by an organisation that is independent of the service being assessed. Most schemes are run by non-governmental private organisations employing full- or part-time assessors, who may also be health personnel with positions in a health service. These schemes are voluntary in most of Europe at present, whilst the JCAHO accreditation scheme in the USA is compulsory in order for hospitals to gain federal funding for patients. Examples of compulsory government schemes in Europe are the inspections carried out by the Norwegian Fylkeslege and the UK Commission for Health Improvement.

The schemes concentrate on whether a service has the resources, structures and processes which experts believe are necessary to produce good quality outcomes and experiences for patients. The term "believe" is used here because there is little certain knowledge about the relations between outcomes and organisation structure and process variables. An example is the Kings Fund Organisational Audit scheme which assesses whether an organisation is meeting defined standards for personnel, procedures, equipment and in other areas which experts and professionals have decided are essential for the provision of good quality care.

The two elements of these schemes are, first, the process by which a service is assessed and, secondly, the content of the scheme - the standards or elements covered. Most assessments involve external assessors visiting an organisation and gathering evidence to assess the extent to which its service meets these standards or assessment criteria. Different schemes allow differing amounts of subjective judgement to the assessors. In the Nordic countries most schemes are used by providers to get external and objective feedback about risk areas and to help decide improvement priorities. Elsewhere some schemes are used by purchasers to decide whether or not the service meets standards that the purchaser thinks are the minimum necessary to provide good care for patients.

Organisational assessment schemes

State inspection schemes: these are run by national or regional governmental agencies which inspect or accredit health care organisations. Examples are the "minimum legal control" assessments by the Norwegian Fylkeslege, which assesses whether a service is meeting legal requirements and keeping necessary documentation, and the UK Commission for Health Improvement, which audits services every four years or whenever a serious quality problem is suspected.

Accreditation: The main difference between accreditation and the other schemes is that an accreditation agency awards a "credit" to a health care service to indicate it has met defined standards. In France all health care providers must be accredited before they can provide services, whilst in the US accreditation is necessary for receiving government funding for patients covered by federal funds. There are one or more accreditation schemes operating in most countries. The schemes may be run by a government agency (France) and compulsory for providers, or voluntary and run by a non-governmental agency, most of which are financed from fees from the health care service when it is accredited ^3-4 .

The most well-known scheme is the US Joint Commission for the Accreditation of Healthcare Organisations ⁵ , which has an international division. This scheme is being used by the Copenhagen hospitals in Denmark. In the UK there are over 30 accreditation schemes: the Kings Fund Organisational Audit ⁶ is widely used and became an accreditation scheme in the late 1990s (the "Health Quality Service"). There are Swedish and Finnish versions.

Baldridge based awards: The most well-known in Europe is the European quality award ⁷ which has assessment criteria covering the following performance areas: leadership, people, policy and strategy, partnership and resources, processes, people results, customer results, society results and key performance results. There is a version of the US Malcolm Baldridge Quality Award for health care. The Nordic countries have the Norwegian Quality Prize and the Swedish Quality Development Leadership ("QDL") scheme for health care. These award schemes are not quality systems but are increasingly being used by hospitals as a basis for building their quality systems.

ISO 9000: This scheme certifies whether an organisation meets the standards required of a quality management system. It has been used mostly in laboratory and non-clinical support services. It has not been popular with clinical services or entire hospitals, mainly because the technical language has not been easy for health service personnel to understand and apply. This approach has been summarised as, "Write what you should do, do as you write, and write when you don’t do it right."

Clinical quality peer-review: These concentrate less on the administration of care and more on whether providers meet clinical standards of care. Most are run by the medical profession and vary from systematic assessment with defined standards to more informal peer review visits which offer assessors more scope to decide which areas require attention. A nursing scheme has been developed and tested in the UK ⁸ .

Discussion

These organisational assessment schemes differ in their purposes and in terms of the items they concentrate on or their "content": the Baldridge based awards pay more attention to outcomes (eg, customer satisfaction results and trends), the Kings Fund to organisational audit and ISO 9000 schemes pay more attention to inputs or structure and to processes. ISO 9000 schemes concentrate on assessing the quality system rather than assessing standards of care and of the organisation of care.

Do the schemes assure quality? There is no guarantee that an organisation which is well-assessed will always provide high quality care, only that the organisation meets standards which are thought to be necessary for that level of care. Anecdotal reports suggest that some organisations that are certified or well assessed do not give high quality care, but there is little research evidence of the quality results of using different schemes or of comparisons of assessed with non-assessed organisations.

An interesting development is assessment of specific systems of care or the "chain of care" for specific patient groups such as stroke or cancer patients. These assessments consider how well various services are provided and co-ordinated to meet all the needs of a patient and to give continuity of care throughout an illness episode.

Research

Considering the amount of time and money spent on organisational assessment, and the significance of the issue to governments, it is surprising that there has been no research into the cost-effectiveness of these schemes. Neither is there research which would help providers to decide how best to introduce a scheme and to maximise the benefits from its use. This is especially surprising as quality management emphasises a "fact based" approach.

Different types of research must be carried out. First, a review of all the studies that have been done and a summary of the evidence. Secondly, research into the results of using a scheme compared to not using one and research comparing the benefits of different schemes. We need to know what difference a scheme makes for health care personnel and patients, and the costs of using it. Thirdly, research into how organisations implement different schemes, which tests the hypothesis that the way in which an organisation implements a scheme is more important for quality improvement than the particular scheme they choose. In addition, there is a need to exchange experiences: the Nordic network and the Accreditation Federation of the International Society for Quality ⁹ are both forums which allow for this.

Investigations into the results of organisational assessments will be helped by the increasing use of quality indicators, the subject of the next part of the paper: it will be easier to study the impact of using a particular organisational assessment scheme by using the quality indicators which are being collected.

Comparative Quality Measurement

This paper considers quality indicators for comparing the same service at different times, or different services at the same time. Quality indicators can be used for research, quality improvement or management and accountability. Examples of quality indicators are nurse/patient ratios, waiting times, rates of patient withdrawal from treatment, medication error, infection for units or whole hospitals, unplanned re-admission and mortality 30 days after surgery.

The terms quality "indicator" and "measure" are often used interchangeably, but, strictly, a measure is a more certain representation of quality than an indicator. An indicator is a method of measurement that gives data that may be related to the quality of a service.

In choosing which indicator to use, the key question is who will use the data and what will they use it for? Indicators are used for five main purposes:

1. Research: data about patient outcomes are collected to study treatment effectiveness and data about service structure and activities can be used to study the link between outcomes and service organisation.
2. Clinical improvement: clinicians and teams can compare their results with last year’s and with others’ similar services to judge the effects of changes or to discover why others get better results.
3. Referrer and patient choice: patients and their referrers increasingly want information about waiting times, outcomes and what the experience of undergoing a treatment is like to decide which provider to select or whether to refuse treatment.
4. Resource management: purchasers and provider managers need to have data about quality as well as costs and the number of patients treated in order to decide which provider or clinician is using resources in the best way.
5. Transparency and accountability: politicians and the public have become less trusting of providers and want all providers’ activities to be more open and visible. There is a growing feeling that the public needs more protection and that providers should be inspected and be more accountable.

Types of indicator

Quality indicators can be categorised in terms of:

1. Input/structure (eg, number of personnel per patient, personnel qualifications), process (eg, number of medication errors, waiting times) or outcome (eg, mortality, quality of life). Generally, input and structure indicators are less directly related to quality than are process indicators, which in turn are less directly related to quality than outcome indicators.
2. Level of care - hospital-wide (eg, hospital infection rates), departmental, team or clinician, care programme, disease/treatment group or individual patient.
3. Subject of the indicator (eg, patient complaint, medication error, waiting time, unscheduled readmission rates).

A number of papers further discuss issues in designing, collecting and interpreting quality indicators ^10-15 . This paper concentrates on quality indicator systems for comparison - a set of indicators covering a whole hospital or a department or a patient group.

Examples of quality indicator systems and projects

Maryland Hospitals
In 1985 seven US hospital leaders arranged for data to be collected on 10 in-patient care indicators in order to compare their services ¹⁶ . These included the now-familiar set of indicators also used in many other schemes: rates of hospital acquired infections, surgical wound infections, mortality, caesarian sections and unplanned readmissions. By 1994, over 900 US, five UK and one Japanese hospital had voluntarily joined this "research project" and the indicator set was expanded to include more specialised original measures. The data are still being collected and sent by the hospitals to the central project office in Maryland and reports returned showing comparisons to similar services. This system has not been used in the Nordic countries but a description of the early UK experience can be found in Thompson et al ¹⁷ .

UK indicators
The UK has used quality indicators as part of its performance indicators set for the public national health service (NHS). In 1994 Scotland published a set of indicators which had been calculated from data collected in 1990-93 The set was a mixture of 17 health authority, public health and hospital indicators including readmissions, surgical mortality, stroke mortality and re-operations for transurethral prostactomy. Providers were consulted before publication. Publication was carefully managed and included a government warning that apparent differences in performance might be due to many factors apart from quality of care. The results of publication were less dramatic than expected. England followed with a set of much criticised "league table" performance indicators for hospitals and health authorities, which were to be revised to include 15 clinical indicators but, following consultation, only six were used in 1999: surgical mortality, mortality following fractured hip or heart attack, unplanned readmission, returning to home after stroke and following treatment for a fractured hip ¹⁸ . Sweden considered publishing a similar set but decided not to in 1998 after consulting experts and the medical profession.

Nordic registers
Sweden has the longest running and most extensive set of medical specialty and disease registers or databases in the Nordic countries. The registers are initiated and run by doctors but the data is available to the governmental inspection agency. Most hospital departments contribute to the relevant registers which have a high credibility and are valuable to researchers and clinicians in everyday practice. A description may be found in Garpenby & Carlsson ¹⁹ .

Norwegian National Quality Indicators Projects
In 1995 a group of five regional hospital directors and one county hospital director formed groups of health professionals, both clinical and administrative, which were to devise a set of indicators to allow voluntary comparisons of quality between hospitals ²⁰ . Five indicators were tested and are still in use: infection rates, turnover of healthworkers, time from discharge from hospital to mailing of discharge letter to general practitioner, cancelled elective operation rates and adverse drug events. Some indicators use existing data, others, such as the operation cancellations indicator, involve special collection.

A new national indicator project will collect 5 to 10 indicators in 2001, which will increase in time to 20-30 indicators. These include those developed in the above regional project and, in addition: the number of personnel sick days, pre-operative waiting times for hip fractures, waiting times for elective patients, 30-day survival for hip procedures, acute myocardial infarction, bypass and stroke procedures as well as patient-reported outcomes from treatment and patient injuries through falls.

Danish National Quality Indicators
Project Denmark set up a project in 1999 intended to develop a nationwide set of quality indicators for the "three dimensions of quality" ²¹ . The project initially focused on six areas: stroke, hip fracture, schizophrenia, acute surgery, heart insufficiency and cancer ²² . The project involves doctors and nurses deciding which process and outcome indicators are to be used for each disease and which data about patients to collect to allow interpretation of the data. Clinicians will be consulted before the indicators are made public. There is a strong research orientation to this project. Although this, and clinicians’ involvement, are thought necessary to ensure credibility and validity, it remains be seen which compromises will be necessary to keep data collection costs to a reasonable level for a nationwide system with different hospital data collection systems.

Joint commission indicators
The JCAHO required accredited organisations to submit indicators from 1999, most of which are patient clinical outcome indicators ²³ . This system has been tested over six years. It involves risk adjustment for comorbidity and other patient confounders and allows comparison between similar units. The indicators cover most specialties and include: perioperative complications and mortality, caesarean section rates, low birth weight rates, indicators for cardiovascular care, oncology, trauma, medication use and infection control and rates for ventilated inpatients who develop pneumonia. The intention is to carry out focused accreditation surveys for those areas that may show poorer performance, rather than full surveys for organisations which generally perform well on the indicators.

Quality Indicator Systems - Models Voluntary hospital comparison schemes (eg, Maryland and Norwegian regional indicators project) Required by higher authorities (eg, UK National government or the JCAHO accreditation agency) Clinician initiated, run and owned (eg, Nordic registers) Collaborative "research model" (eg, The Danish National Indicators Project)

Discussion

Some indicators or indicator systems are better for some purposes than others. For example, most quality of life indicators used for research are too expensive to use for routine inspection and accountability purposes. When choosing which indicator to use it is important to be clear who the indicator is for and which decisions they make that could be better informed by using the indicator. Indicator systems that try to serve too many purposes often do not serve any one purpose very well. This paper focuses on comparisons for accountability, management and local quality improvement, rather than data collection for research, although these different purposes are related.

Does an indicator or a measure truly represent the quality of a service? It depends which indicator is considered. Here we note one much discussed measure: mortality or survival rates. Are mortality rates after surgery a representation of the quality of a service? This depends on the quality of the data (do the methods of collection ensure validity and reliability? were the data collected properly?) and on whether causes other than the quality of the service could produce the phenomena which the measure represents. Age, sex and other patient characteristics affect the chances of a patient dying after surgery, independent of the quality of care they receive. It is possible to control for patient characteristics by gathering data about patients and estimating the chances of death based on past data.

When comparing services, or the same service over time, it is difficult to be sure that a measure truly reflects differences in the quality of two or more services or differences in the quality of the same service at different times. The registered rates may increase due to better reporting, or the rates may be affected by changes in the type of patients rather than changes in the quality of the service. Research studies can control for these variables and ensure accurate measures, but even researchers often have difficulty ensuring that the measures are valid and reliable, as well as in interpreting the results.

Do the systems work?

This question returns to the important earlier questions, who are the data for and which decisions are to be better informed by these data? Different data users will have different criteria for judging whether the system "works". Government and politicians may judge a system in terms of whether it allows cost-effective identification of dangerous providers or encourages quality improvement. Clinicians may judge it in terms of whether it is scientifically valid and reliable, truly reflects the quality of care they give and can be used to decide which changes to make to their practice and organisation of care.

As with research into organisation assessment, the research in this area is sparse. Thompson ⁵ notes, "Much of the current evidence on the effectiveness of performance indicators is based on observational or experiential data, although much of the policy agenda in this area seems to be based at its worst on dogma". There is some research into whether better outcomes do in fact result from providing data to clinicians about their outcomes or from publishing data. Apart from the early New York publication of mortality rates, most publication schemes have been found to have little effect on patient choice behaviour, provider behaviour or outcome performance. The 1990 New York state publication of mortality rates for heart surgery was thought to lead to a 40% lower mortality over four years. However a 1995 survey of Pennsylvania cardiologists found the consumer guide to coronary artery bypass graft surgery to be "of little or no influence" in choices of surgeon and not much used by consumers ²⁴ . The USA Health Care Financing Administration published hospital mortality rates in 1988 but the publication was stopped in 1995 because of criticisms of the validity of the data and the general view that publication did not stimulate improvement but caused defensiveness and fear amongst providers.

Many health care researchers and experts are critical of the usefulness of many quality indicators for identifying poor quality providers or for quality improvement. Experts have serious doubts about the validity and cost-effectiveness of comparing quality indicators. A 1995 UK study found that acute myocardial infarction outcome data did not show "gross failures of care" ²⁵ . A 1996 US study showed that outcome data did not identify poor quality hospitals ²⁶ . So far, there is no evidence that the benefits of collecting and using quality indicators justify the time and cost of collection, outside research and quality improvement projects. This, however, does not mean that there is evidence that any of the schemes are not worth the costs - there is an increasing number of independent reports of the usefulness of some schemes, eg ThOmpson et al ¹⁶ .

 

Four "golden rules"of quality data collection Do not collect data unless you are sure no one else has. Do not invent a new measure when a proven one will do. Measure what is important not what is easy to measure. Do not collect data about things you cannot influence or where confounders make interpretation impossible.

Lessons from experience collecting and using quality indicators

Experience with different schemes points to a number of issues and challenges in collecting quality data.

Which indicators should be used? How accurate do the data need to be? These are a key questions for most providers and can only be answed by being clear about who the data is for and which decisions the data are meant to inform. Indicators for clinicians to assess treatments or the organisation of care must be more accurate and comprehensive than indicators assisting management to identify possible problem areas. One guideline is to choose indicators which research has found to show large variations and which suggest action should be taken with providers.

In terms of ease of collection, indicators of input, structure and process are easier to collect but are less valid for some purposes than indicators of outcome. Indicators of outcome are difficult to interpret, especially without data about patient characteristics, because health is affected by many factors other than the quality of care. Knowledge about the relationship between outcome and other variables is uncertain.

Other than in a strictly controlled research project it is difficult and costly to overcome the technical problems of ensuring data are recorded, recorded in the same way at different times and places (reliability) and that a measure is valid, sensitive and specific. There are also problems ensuring enough data about patient characteristics are collected to allow for interpretation of outcomes and control of patient variables.

How accurate do the indicators need to be for the purpose? Can we afford a higher level accuracy and is it technically feasible at this time, given the information technology and skills we have? There must be a balance between technical quality and the cost and time of collection, even in research projects. How much does the extra accuracy and/or data cost and is it worth this extra? Are the indicators useless or misleading without this extra information? There is a tension between ensuring measures are valid, reliable, credible and useful to clinicians, and ensuring that they are easy to collect and report on and are of low cost. Some approaches to quality indicators are more like research data collection, with large amounts of data collected about outcomes (including quality of life) and about patient characteristics (eg, the Danish national indicator project).

Another lesson from experience using indicators is that if provider clinicians and managers are to use them, they must be involved in development of quality indicators or standards and in decisions about implementation. This can take much time, and agreement may not be reached, but voluntary systems do not ensure that poor quality providers are involved. Further, hospitals have a variety of information systems which often cannot collect the data required. It is usually necesary to agree on one type of software. However, where existing data or software can be used, it is possible to quickly introduce an indicator.

Health workers have little interest in spending patient contact time collecting information for others. Health workers need to understand and believe in the value and use of the data they are involved in collecting. At least some of these data need to be of use to health workers and actionable. Data reported too long after the collection (eg, over six months) is of little use for many practical purposes.

So far, quality indicators for government inspection and to identify poor providers have had little credibilty with providers and are thought by researchers and professionals to be unreliable and invalid. Yet there is increasing public and political pressure to collect and to publish these data, pressure which will not decrease in the coming years. There are good arguments for and against publication of different quality indicators. Non-publication helps to gain co-operation from health professionals and, often, accurate data. Some argue that publication ensures validity and that the argument that the public do not understand the data is patronising. There is evidence that publication can result in providers not giving accurate data and that it does not have a large effect on provider or patient behaviour.

What is missing from most schemes? Indicators of "system of care quality": many quality problems occur "between services" and are due to failures in communication and co-ordination. These problems are increasing with ongoing changes such as the addition of new providers. Indicators of patient co-ordination and continuity over an episode of care involving different providers are required to balance the increasing attention which will be given to measurable aspects of within-service processes. This balance will ensuure that attention remains focussed on the system of care quality for the patient ²⁷ .

In addition, there is a need to collect indicators of "public health quality": whether a service reaches those who most need it and provides preventative and health promotion services.

Conclusions

Although there are doubts about the effectiveness of accreditation, it is likely that all health care organisations in Europe will soon be required to take part in some form of external evaluation and to report data about quality of care. The results of at least some of these assessments are also likely to be publicly available and to effect decisions which are important to providers, such as those about financing, referrals and patient choice. How valid these schemes are and whether the costs are worth the benefits are still open questions.

There are many types of quality evaluation schemes, which are used for different purposes, but none are clearly to be preferred as the single required scheme for a particular country. This verdict is not likely to change with more research into the cost benefits of different schemes, although such research is greatly needed. What appears to be more important is how management makes use of the scheme: that they plan and project manage the introduction, and that they act on the findings of the assessment or the data comparisons.

At present, many hospitals take part in external medical specialty peer review for some of their services, and a few also use ISO certification for support services as well as taking part in hospital-wide organisational assessment schemes such as the EFQM or organisational audit. In addition, they may be required to take part in a state inspection or accreditation scheme. This mixed model is likely to continue, although there may be problems due to the different "quality language" used by each scheme in different parts of the organisation and to the amount of time taken to meet these different requirements.

It is likely that one scheme may accept the results of an assessment made by another as evidence towards the assessment - there is an increasing degree of substitution and conversion between schemes, in part due to improvements following criticisms. New information technology will reduce the time taken for documentation and assessment and for quality indicator collection and analysis. It will take time for all health care organisations to introduce this technology although they may be forced to meet external requirements. If so, it is questionable whether the gains in quality of care that might result would be greater than gains that would result from other uses of the allocated funds.

There is an increasing use of quality indicators as part of organisational assessment schemes, as well as for routine reporting to management or inspection agencies and for quality projects. There is no doubt that quality improvement has been slowed by a lack of quality measures, the absence of a culture that values such data and by the time, costs and complications of gathering and interpreting the data. Experts have serious doubts about the validity and cost-effectiveness of comparing simple quality indicators, but the public and politicians think quality can and should be measured and the data made public.

There is a need for research into the cost-benefits of different schemes, how to introduce a scheme and how to ensure that participating organisations benefit from taking part. The quality movement emphasises a "fact-based" approach to improvement, which is also compatible with evidence-based medicine. It is curious that such expensive schemes have not been evaluated and that little research evidence is available to those considering which scheme to use and how best to introduce it.

References