.jpg){kind=logo}
- Introduction
- What are We All Trying to Achieve?
- Can Both Standards and Quality Improvement Processes Coexist in the Same Programme? Where Should the Balance Lie between Standards and Improvement?
- Who Sets the Indicators and are They Transportable Across Health Care Sectors or Nations?
- Setting the Performance Standards?
- How Close Can We Get to a Common Framework for Accreditation Processes?
- How Have We Applied this in New Zealand General Practice?
- Conclusion
- References
Introduction
"Medical Audit has become a popular method of quality assurance. Not every one is convinced of the effectiveness of this approach. The emphasis is all too often on comparing performance data with standards presented by policy makers, and not on the improvement of performance in the practice, based on the motivation of care providers."
From the preface to ’Quality Improvement by Peer Review’ by Richard Grol and Martin Lawrence, Oxford General Practice Series, No 32, 1995.
How far have we progressed from this point?
Early performance improvement initiatives in general practice in New Zealand were based on Quality Improvement principles. Since 1995, when the new Medical Practitioners Act was passed, the increasing legislative requirements and the requirements for our publicly funded system to be more overtly accountable to the consumer have created a need to incorporate minimum standards into New Zealand’s quality processes in health care.
The relationship of quality improvement to minimum standards and the subsequent choice of indicators for these has been a matter of considerable debate. We have been faced with some fundamental questions:
- Can both standards and continuous quality improvement (CQI) processes coexist in the same programme?
- Where is the balance between standards and CQI?
- Who determines the indicators around which the standards are set?
- Who sets the performance expectation, ie, the performance standard?
- What level of evidence is needed to demonstrate that meeting the standard affects outcome?
- Can common indicators and standards be set across the health care industry or is there a necessary variation in them between, for example, rest homes and general practice?
- Can common standards be set across national boundaries?
This paper reviews the debate which has taken place in relation to these questions and contributes to the discussion of Charles Shaw, who suggests there are common concerns in several areas about how standards should be defined and measured and about how effectively the existing systems support internal quality improvement.
Shaw also suggests that, since there are common arguments for consistency within nations and within trading regions, countries would have to adopt a common assessment process and core standards if there is to be consistency and compatibility at the national or international level. While this is logical, and possibly desirable, I will explain that the ability to do this is limited to a few clinical indicators.
What are We All Trying to Achieve?
External assessment of performance and assistance with subsequent improvement is commonly referred to as accreditation. The International Society for Quality in Healthcare (ISQua) represents organisations that, collectively, probably accredit more health care organisations around the world than any other similar group. Its members range from the Joint Commission on Accreditation of Healthcare Organizations (JCAHO), which, alone, accredits 19,000 health care organisations in the US and around the world, to the New Zealand based Quality Health New Zealand. ISQua understands accreditation. It describes the purpose of accreditation in its Federation Operating rules (1998) as:
Accreditation is a self-assessment and external peer review process used by health care organisations to accurately assess their level of performance in relation to established standards and to implement ways to continuously improve the health care system.
Can Both Standards and Quality Improvement Processes Coexist in the Same Programme? Where Should the Balance Lie between Standards and Improvement?
In January 2000, one of the New Zealand GP magazines published a somewhat tongue-in-cheek article by a Brisbane GP, Dr Warwick Carter, called "606 ticks and one cross". 1 Dr Carter describes the experience of being in a practice undergoing accreditation. I believe his comments should be considered by all of us who are responsible for accreditation programmes, as the ensuing reality is not always what we had planned for.
Carter refers to a number of improvements made between the initial self-assessment and the subsequent external peer review process. Cloth towels were removed, leaving only the recommended paper ones. The 1.2m distance between steriliser and sink was cleared. Receptionists were reminded to keep patient files face down on the front desk. The junior receptionist rostered on that shift was replaced by the practice’s most experienced one. When the process was complete, Dr Warwick felt that the practice had probably scored well and may have only failed one item. "Would it be the one that failed the entire practice?", he asked.
It is very difficult to develop a culture of continuous improvement when the assessment is focussed, at least in the eyes of the assessed, on a set of standards applied in a pass / fail approach. How many of the changes in Dr Warwick’s practice will be long-lasting? Will the surgery staff have been left with the motivation to go on examining their practice with a view to doing better or will they now feel that "we are OK", bring the cloth towels back and forget about the 1.2m distance between sink and autoclave?
Even where an accreditation process is successful in creating real change through an improvement philosophy and process, do we know the size and significance of subsequent improvement? What about subsequent assessments? Is there further improvement once an organisation has reached the minimum standard required or is this just maintained? Is this minimum standard now the ultimate improvement goal? If Dr Carter’s practice did score 606 / 607, what would be the incentive for such a high scoring practice to remain a leader and maintain a CQI focus?
There is some information on CQI programmes in general practices. Back in 1995, in his William Pickles Lecture, "Professional Development in general practice: problems, puzzles and paradigms", Pendleton raised the issue of strategic planning workshops in practices and their role in creating focussed change. 2 Lawrence and Packwood 3 and, later, Hilary Hearnshaw 4 developed these concepts further, working with practices on problem identification and team-based problem solving. In 1998, Wilcock and Campion-Smith summarised the factors which were identified as being important for the uptake of CQI: 5
- leadership
- involvement of the whole team
- a perceived need by the staff to find better ways of meeting the needs of their patients or clients
- a methodology that includes system review
- the views of all the team are heard and valued
- the tacit knowledge of each team member is acknowledged
- there is a shared understanding within the team of each other’s roles.
These factors are often lacking in the busy and stressed environments of general practices. In 1996, the Goodfellow Quality Assurance Unit in the Department of General Practice at The University of Auckland School of Medicine developed a Practice Consultancy Programme to assist practices to develop and utilise these factors in order that they could identify and achieve quality goals.
The Goodfellow Quality Assurance Unit also surveyed a sample of Auckland general practices to establish perceived barriers to quality improvement. These were predominantly system issues. The biggest ones were reported as perceived lack of time and heavy workload, staff problems, issues with other doctor colleagues and coping with New Zealand’s frequent major health reforms. Clinical management was not included in the top 13 topics.
The Unit worked with 62 practices, providing five hours of sponsored time to each. The first hour was spent identifying the underlying problems and deciding what would be achieved if these problems were corrected. This turned negative problems into positive goals and, hence, defined the aims. Two sessions of two hours for each aim were then spent in project planning using the principles outlined above. Results were quickly apparent, with spontaneous comments being made about improvement in morale, more useful meetings, better communication and a greater zest for quality.
In order for change, and especially sustainable change, in the quality of service delivery to occur, it is necessary to do more than measure performance against a fixed set of indicators. We must enable practice staff to interpret the information, define their aims, and deal with their barriers.
As O’Leary and O’Leary (JCAHO) wrote in 1999 6 "…excellence in performance is what most health care organisations want for themselves and their patients. CQI offers them the opportunity to reach this lofty goal."
Who Sets the Indicators and are They Transportable Across Health Care Sectors or Nations?
The term "standards" is used by some to refer, loosely, to a set of measures, and by others to refer, more specifically, to levels of performance within criteria. The reason why we are making the measurements is often forgotten altogether. A lack of specificity about the terminology of indicators, criteria and, in particular, standards causes considerable confusion in discussions about measurement of quality.
The clearest description I have found for the terms comes from the WONCA working party on quality in Family Practice (EQuiP) as reported by Lawrence and Olesen in 1997. 7 Like Charles Shaw, they argue that in order to assess or compare health care quality nationally or internationally, it is first necessary to agree indicators of performance.
Since these indicators not only need to measure current performance but, equally importantly, any change in performance over time, they must be capable of measuring change. EQuiP thus modified the US Institute of Medicine’s definition of "a quantitative measure for monitoring care" in order to make it more useful in evaluating change and, hopefully improvement, in performance. The EQuiP definition is: ’a measurable element of practice performance for which there is evidence or consensus that it can be used to assess the quality, and hence change in quality, of care provided.’
Indicators must however do more than just be a measure of change. They need to be relevant to those experiencing quality. Let us consider the initial contact of a patient with a general practice. To the patient, important aspects of high quality care at this stage might include physical and telephone, or possibly e-mail, access to the service. Whereas, to the receptionist it is more important to have appointment options to offer and to the doctor the ability to obtain an accurate history of the problem might be the most important factor.
Thus, the indicator has different relevance to each stakeholder. It therefore becomes important that each stakeholder is represented in the process of indicator choice. Charles Shaw rightly refers to extending clinical indicators to system indicators, and by including the organisation’s staff in indicator choice, these system indicators are given greater importance and higher priority. It is particularly pleasing to note the increasing recognition that patients and the community are also stakeholders and should be actively involved in the choices of indicators for their services.
If indicators are to be useful, their various characteristics, such as reliability, face and content validity, clarity and measurability need to be understood by the group. They must also address all critical aspects of the organisation where quality requirements exist. These are referred to as domains of quality. Such domains need to be defined. There are several models that can help with this 8 , 9 , 10 but there is no evidence that one is any better than another. Assessment tools therefore use a variety of models. Since stakeholders are unlikely to have an existing understanding of the characteristics of indicators and the models for choosing domains, their team should include a person with this knowledge or experience.
The consequence of such a variety of input is the introduction of a wide range of values. When this is coupled with a lack of evidence to guide selection of indicators, it is not surprising that the final choice varies between assessment tools. This variance will be even more obvious between health care sectors, such as residential care and general practice, nations and cultures. It seems unlikely that a single set of common indicators can be developed.
This variability in final indicator choice, is reflected in ISQua reporting that at its 1999 Indicators Meeting in Melbourne:
a draft set of principles for indicators was generally agreed with the proviso being made that recognition be made of slightly different principles applying to accountability and to performance improvement. Given the diverse range of stakeholders and objectives for indicators, it was agreed to defer a set of international indicators…groups suggested were indicators for hospital care, primary care, community health. 11
Nevertheless, indicators will have the same purpose regardless of assessment tool. They must address similar overall quality domains. The characteristics which define their use, and the process by which this is done, could be incorporated in a standardised approach for tool development that is common across health sectors and nations.
Setting the Performance Standards?
Choosing Criteria
For performance to be measured, it must be possible to state very clearly whether a point of expectation, the standard, has been reached. Indicators, therefore, must be capable of being broken down into components or criteria that have sufficient specificity to allow assessors to say whether or not this specified level of performance has been met. In New Zealand, we have followed the advice of EQuiP and used Donabedian’s definition of a criterion, 8 which is "a criterion is a discrete, clearly definable and measurable phenomenon, in some specifiable way relevant to the definition of quality. It must be so clearly defined that we can say whether it is present or not." We have chosen several criteria for each indicator.
Who makes these choices? In the case of some clinical indicators there is an evidence base demonstrating a very strong link between process (glycaemic control in people with diabetes) and outcomes (fewer microvascular complications). 12 Thus for certain indicators the choice of criteria is straightforward, ie, when these indicators are used, they can incorporate the same criteria.
However, with respect to the system indicators, referred to above, which are the most frequent type of indicator chosen, there is much less evidence for a link between a change in their process measures (eg, how often one has a practice meeting) and altering outcomes (better staff morale). Indeed, the supporting evidence is often the consensus of a "few wise people". In this situation, it is hard to make one criterion more important than another. The deciding factor then becomes the value set of those making the choice. This naturally varies from tool to tool.
Setting the Standards
The potential for consistency becomes even more challenged when dealing with standards. The EQuiP group is very specific in its use of the term. It chooses Richard Baker and Robin Fraser’s definition: ’A standard is the percentage of events that should comply with the criterion.’ 13
This definition works well. In the case of the indicator of glycemic control, HbA1c, and its criterion ’is the HbA1c below 7%?’, the frequency of achieving this in a target population can be reported without confusion.
Setting the expectation for this frequency or the required level of performance (the standard) must take into account a number of factors. For example, the length of time an improvement programme has been running, the resources available and the attitude of the community (and hence the patient) to the importance of treatment, will all affect the ability of a provider to reach a required performance level.
Thus, even though many evidence-based clinical indicators can have consistent criteria between health sectors and between nations, this is unlikely to be true of their standards. For indicators based on consensus evidence, the chance of achieving consistency of criteria or standards is considerably lower.
How Close Can We Get to a Common Framework for Accreditation Processes?
Consistency, at the international level, of indicators, criteria and standards would have the benefits outlined by Shaw. However, as shown above, there are substantial barriers to this. From the discussion above, it could be concluded that:
- A set of "domains of quality" could well be agreed upon at international level.
- The characteristics of a good indicator are already widely agreed, though, again, there are some detailed variations. 14 , 15 International agreement on these characteristics would give consistency to the construction of locally chosen indicators. A bank of these could be kept and made available to those developing new tools to review and utilise where appropriate. JCAHO has already made a start on this with its National Library of Health Indicators. 15
- For evidence-based indicators, which are usually the clinical ones, international indicators and criteria could be suggested and added to the indicator and criteria library. However, for consensus-based indicators, and even some clinical indicators, both the indicators and their criteria tend to be chosen on values. It may have to be accepted that inter-sector and international consistency and comparison is not yet possible in these areas.
- The standards for achievement will almost always need to be set at local level.
- The information developed through analysis of the consequent audit data is the basis for the Quality Improvement component of Accreditation. While more work needs to be done to establish the most effective way of achieving this, it should be a goal for any group developing an international framework to specify the current knowledge level in this topic and make appropriate recommendations.
How Have We Applied this in New Zealand General Practice?
In New Zealand, we have set a series of 5 criteria for each of the 48 indicators that cover all the domains suggested by the Goodfellow Quality Assurance Unit. Some of these indicators include minimum standards, reflecting legislative requirements and the criteria that are regarded by the practice standards working Party as critical. Other criteria and their standards are goals that give practices a range of ideas about what could be achieved and not necessarily what should or must be achieved. They allow practices to assess the gaps between where it is desirable to be and what the present situation is.
In 1999, the Royal New Zealand College of General Practitioners (RNZCGP) piloted the draft assessment tool and process in 20 practices. The evaluation suggested that the CQI approach was the most valuable aspect of the visits. 16 This was a similar finding to that of Eliasson et al 17 who described four practices where the doctors reviewed each other against 13 topics they had themselves chosen. The GPs at first acted as observers, gathering the initial data and comparing it to their own practices. Six weeks later, they acted as tutors, providing feedback and facilitating a meeting to decide on priorities and what could be done to improve these priority areas. They concluded that "the feedback session, performed shortly after the visit is essential."
The RNZCGP will field trial the accreditation programme in approximately 100 general practices. The intention is to include a facilitated discussion on the day of the external review, which will assist members of the practice in identifying achievable goals for improvement and setting realistic action plans to achieve them. The assessors will be taught basic facilitation skills and to be mindful of the findings of Wilcock and Campion-Smith. 5 Assessor training for this field trial of 100 practices starts in July 2000.
Only after the evaluation of the field trial will it be clear whether this process has helped practices reach "the lofty goal of excellence in performance" referred to by O’Leary and O’Leary. 6 Our vision of achieving this is emphasised by the name of the accreditation programme, "Aiming for Excellence in General Practice".
Conclusion
Since Grol and Lawrence wrote their preface (1995), accreditation has come a long way. It has been a rapid journey and one in which they have been individual leaders. We have clearly moved on from medical audit. Standards are no longer presented by policy makers but are rather developed jointly with multidisciplinary stakeholders and technical advisors. For instance, the Practice Standards team in New Zealand comprised a practice manager, a practice nurse, rural and urban doctors along with consumer, and cultural representatives. It liaised closely with the funder and an academic quality improvement organisation.
However, despite the best intentions of many groups involved in accreditation, the linkage with CQI alluded to by Grol and Lawrence has been a hard goal to achieve. This is disappointing because it is probably our best opportunity for practices to make sustainable and meaningful improvements. Only by understanding CQI can practice members move on from seeing accreditation as a gate through which they have to pass, to seeing it as a door to that never-ending path towards quality.
New Zealand General Practice is the next group to attempt to achieve this combination. If we are successful, we will have combined the acknowledged need for minimum standards with the powerful process of CQI.
References
- Carter, W. New Zealand General Practitioner, 26 January 2000
- Pendleton D. Professional development in general practice: problems, puzzles and paradigms. Br J Gen Pract 1995; 45:377–381
- Lawrence M, Packwood T. Adapting total quality management for general practice: evaluation of a programme. Quality in Health Care 1996; 5:151–158
- Hearnshaw H, Reddish S, Peddie D, et al. Introducing a quality improvement programme to primary health care teams. Quality in Health Care 1998; 7:200–8
- Wilcock P, Campion-Smith C. Editorial. Never mind the quality, feel the improvement. Quality in Health Care. 1998; 7:181
- O’Leary DS, O’Leary MR, From quality assurance to quality improvement. The Joint Commission on Accreditation of Healthcare Organizations and Emergency Care. Emerg Med Clin North Am 1992 Aug; 10(3): 477–492
- Lawrence M, Olesen F, for the EQuiP working party on indicators. Indicators of quality in health care. Eu J Gen Pract September 1997; 3: 103–108
- Donabedian A. The criteria and standards of quality. In explorations in quality assessment and monitoring. Vol II. Ann Arbor: Health Administration Press; 1982
- Maxwell RJ, Quality assessment in health. BMJ 1984; 288:1470–2
- Ovreitveit J. Health Service Quality: an introduction to quality methods for health services. Oxford: Blackwell Science Ltd; 1992
- ISQua International Indicators Initiative Melbourne; 1999. http://www.isqua.org.au/ISQUAPAGES/ProgramsEvents.html#ALPHA
- O’Connor P, Spann S, Woolf S. Care of adults with type 2 diabetes mellitus: a review of the evidence. J Fam Pract 1998; 47(suppl):S13–S22
- Baker R, Fraser RC. Development of review criteria: linking guidelines and assessment of quality. Br Med J 1995; 311:370–373
- Department of Health. A first class service: quality in the new NHS. 1998. http://www.open.gov.uk/
- Joint Commission on Accreditation of Healthcare Organizations in the USA. National Library of Health Indicators. Www.jcaho.org/perfmeas/mlhi/appendc.htm
- RNZCGP Practice Standards Pilot, Summary. Report to the Health Funding Authority; 1999
- Eliasson G, et al. Facilitating quality improvement in primary health care by practice visiting. Quality in Health Care 1998; 7: 48 - 54
