.jpg){kind=logo}
This article is also available as a PDF file.
Abstract
Introduction
Background to Audit and Peer Review
Implementation of Software to Support Clinical Audit
Discussion
Conclusion
Acknowledgements
References
This paper discusses the current thinking in Knowledge Management and how it may enhance the Clinical Audit and Peer Review process. Selected Knowledge Management theories are used to explore the value of peer review in achieving better outcomes from audits (i.e. better uptake of recommendations and, by inference, better care and outcomes for patients).
Through analysis of Knowledge Management theory and the example of a technology implementation to support Clinical Audit, it is argued not only that knowledge is an active process that cannot be packaged up and distributed, but also that much of what we label knowledge is really only information.
This proposition has implications for all environments that are trying to manage knowledge and in particular for the use of technology in Health. If we want to promulgate the outcomes of research, audits and the like, the challenge becomes how can we best provide the environment where that trusted information is helped through the process of becoming knowledge that is used to improve clinical practice.
The examination of Knowledge Management theories and how they can be applied with a positive effect in Health is of considerable interest. With drives for best practice, continuous quality improvement and reducing error paramount in the sector, questions are asked about why clinical staff don’t use guidelines, embrace clinical pathways, or adopt the recommendations of investigations and audits [1]. Why is the knowledge available seemingly not applied to practice?
Unlike industry production lines, the knowledge required to provide good health care is not as simple as using the right materials according to the right instructions with the right skill. A production pathway lends itself to being clearly ‘known’. It can be planned, documented, tested, adjusted, taught and with practice performed over and over with predictable results. Healthcare, however, as a human endeavour and with people as the raw material, is not at all as straightforward. Treatment outcome is dependent on so many variables including the treatment plan, the physical and psychological person and the environment they exist in. Each patient journey is unique. The question arises then, how can the clinical knowledge gained from such multiple experiences of the health care process, be best harnessed for lasting value outside of the person who experienced it?
The following sections provide some background to audit and peer review then outline an implementation project for technology to support clinical audit. Selected Knowledge Management theories are then explored and related to this context. Success factors and lessons learned from the implementation are also given.
2. Background to Audit and Peer Review
Clinical Audit is a long standing mechanism for monitoring professional practice, with Florence Nightingale often cred-ited with conducting the first audit in the mid 1800s [2]. There are various definitions of audit. In New Zealand it is defined by the Royal Australasian College of Surgeons as a cycle composed of several stages: Determine Scope; Select Standards; Collect Data; Present and Interpret Results with Peer Review and Make Changes and Monitor Progress [3 p8]. In the Health system in New Zealand, auditing clinical practice is not mandatory in the same way as it is in some countries, such as the UK, though it is expected. There is a demand from professional bodies, health payers and providers, and the public, for clinicians to review their practice and adapt it in response to the knowledge gained by doing so. This is performed in varying ways across organisations and professions, some more sophisticated than others.
There is considerable literature on Clinical Audit, with much of it bemoaning the poor outcomes of Audits and pondering on the continuing problem of how to embed the resulting evidence into practice [1,3,4,5,6].
Peer Review is defined by the New Zealand Ministry of Health (MoH) as “an evaluation of the performance of individuals or groups of practitioners by members of the same profession whose status is similar to the status of those delivering the care. It may be formal or informal and can include any occasion in which practitioners are in learning situations with other colleagues” [7 p.xi.]. Thus, peer review is an activity in its own right as well as a component part of the audit cycle as described above.
3. Implementation of Software to Support Clinical Audit
3.1. Overview
The implementation under discussion was one workstream of a large District Health Board project to implement Concerto Clinical Portal, bringing together a suite of underlying applications including Clinical Documents, Electronic Discharge Summaries and Digital x-rays, as well as Audit. Two surgical departments, General Surgery and Orthopaedics, were in scope for the Audit application. These departments already carried out regular meetings, where data on the number of cases and complications was presented and specific cases of interest discussed. These meetings provided the forum for peer review. In both specialties the process relied on data being captured by largely manual efforts, resulting in a high risk of incomplete and inaccurate records.
The first objective of the project was to deliver a tool to capture the relevant data, whilst minimising data entry for clinicians and ensuring consistency of information across repositories. The second implementation would take the initial design and adapt it to ensure it met the needs of the other specialty. A mechanism for reporting was also part of the scope. The following discussion focuses on the first implementation for General Surgery.
The goals of the audit project for General Surgery were to:
- Support the electronic data collection for the Clinical Audit and Peer Review process
- Replace the incumbent stand-alone system in General Surgery
- Design for the future - provide a (technology) blueprint for other specialties
Figure 1 below is copied from the project material and shows the cycle that was designed and used during the project. This is similar to the MoH cycle described above [3].
It is of note that the Collect Data and Present Data activities were in scope for the project team, whilst the Peer Review and Change Practice elements were the responsibility of the clinical leaders in each specialty.
The usual software design steps were taken:
- Design – documenting the requirements for the new system in such a way that a technical person could take them and write a system specification.
- Develop – the technical coding/configuration to deliver the requirements in the chosen application.
- Test – a program of testing to ensure 1) requirements are met and 2) there are no bugs or otherwise unplanned behaviour in the technology and 3) the technology works in the manner expected.
- Deploy – the steps to install the application into the live environment and provide users with the capability to interact with it – understand the process and have access and training on the system.
3.2. Design features
The Audit application was seamlessly integrated within Concerto, meaning that users could readily access it via the Clinical Portal. In summary, any patient admission to General Surgery triggered enrolment in the Audit application, and data collection forms were automatically generated based on admission, visits to the operating theatre and discharge. Once all the forms were completed and finalised, the data was available for reporting.
Information from the patient management and theatre system (PMS) was drawn through to the forms to provide foundation data on the event and ensure the Audit information correlated with other reports derived from the PMS data. Relevant audit forms appeared on the relevant registrar’s ‘to-do’ list, based on the consultant they associated themselves with in Concerto. Users could not create or cancel enrolments, this was automated to ensure all patients admitted to the specialty were counted, regardless of whether they had surgery, or were transferred to another specialty.
A number of code sets were developed, to minimise the amount of free text and facilitate reporting. This included co-morbidities and complications of surgery, which were a key component. The Complications code set was based on the Royal Australasian College of Surgeons (RACS) standards with clarity on definitions and severity being provided through training and an information booklet.
3.2.1. Process for data collection
Design and configuration emphasised maximising the use of existing data, and facilitating the promptness and accuracy of the clinical data entry. Users were given a specified number of days to complete each form before it attained a status of overdue, overdue forms were reported on so they could be tracked and managed. Any erroneous information drawn from the PMS was not editable within the Audit application (e.g. patient admitted to the wrong specialty, or with the wrong surgical procedure). Although this strategy was painful to begin with, the long term outcome was that this enhanced the accuracy and timeliness of information in all systems. For example, registrars made sure changes to the responsible consultant were appropriately communicated to the ward clerks to update the PMS, and that the correct surgical procedure was recorded in the operating theatre.
Responsibility for completing the Audit forms lay with the registrars with the rationale that they had better information than house surgeons and were senior enough to be accountable. The option was given for consultants to have a final quality assurance step on the data before it was submitted for reporting but this was deemed unnecessary.
There were two access groups - approximately 35 ‘Users’ of the Audit application in each specialty (all the consultants and registrars) with the ability to view and enter data case by case and search to find audit episodes by consultant and date. Alterations or additions to data already finalised, or cancellations of entire episodes could only be done by “Administrators’ of which there were three, the consultant leading the audit in each specialty and the Decision Support team member.
3.2.2. Presentation of the information captured
User access to enter data and view individual patient’s audit records did not grant access to aggregated information. This was provided by a suite of reports, which included a log of operations each doctor had performed or assisted with. Access to the reports and to the database itself was tightly controlled, in recognition of the sensitive nature of the information. The Department of General Surgery gained approval for the information to be protected under the Health Practitioners Competence Assurance Act 2003. The specification and writing of the reports, and the way the information was presented took considerable effort, and required refining as time went on. Later on, data was also made available via an Online Analytical Processing (OLAP) Cube which allowed ad hoc slicing and dicing of data, similar to that provided by pivot tables.
At the meetings, cases discharged in the previous fortnight were reported on. The meeting was in two parts, the first spent reviewing and discussing the aggregated information - ie the type and number of cases by consultant and the number and severity of complications that occurred for those patients. The second part of the meeting was spent on specific case presentations where the registrar detailed the case history, which was then discussed by the group. The selection of cases for presentation was based on a number of factors; for example unplanned returns to theatre or other complications, unexpected death, or general clinical interest, perhaps due to rarity or an unexplained increase in occurrence of a particular condition.
The conversations at the audit meetings were not documented. This was deliberate to ensure the free flow of conversation and honest peer review. As time went on the meetings became a forum where questions could not only be asked about various aspects of a case, but explored though conversations with other experts, with frankness and the added benefit of hindsight. Over time, the breadth and depth of conversations increased, with more engagement as participants became comfortable and saw more and more value in the information sharing. It is of note that the meetings, originally monthly, soon became fortnightly due to the wealth of information worthy of discussion, and since have become weekly for the same reason.
There were additional channels where items of merit could be formally carried forward, such as decisions to review clinical guidelines or policies. Topics for literature review via journal club were identified and issues taken to other services or clinical leaders for discussion and potentially inclusion in the other specialties’ review processes.
3.3. Knowledge Management theory
It is evident in the literature that theories of Knowledge Management are continuing to evolve. There is much to be read about what exactly knowledge is and how to manage it [8,9,10,11]. Here we focus primarily on two concepts arising in the literature – that of knowledge being personal and existing only in one’s head (which is agreed with) and the commonly accepted definitions of tacit and explicit knowledge (which is challenged).
Snowden [12], in discussing Acts of Knowing, suggests that although Knowledge Management became popular in the mid 90s, in hindsight concludes it was more content or information that was being managed. He asserts that knowledge is not a ‘thing’ but an active process. This shift in thinking Snowden calls the third age of Knowledge Management and it changes the understanding of the word knowledge. This view is supported and enlarged on by others, for example, Whittaker and Van Beveren [13] assert that knowledge is not a transferable commodity at all, “but rather a constructivist mechanism established from access to people, relationship building and developing shared understandings” (p298). McElroy’s second generation Knowledge Management theory is largely about process and people [9] and Stacey [14] asserts that “knowledge is continuously replicated and potentially transformed in the communicative interactions between people” (p.222).
In this theoretical discussion about the nature of knowledge, comes the debate about explicit and tacit knowledge. The common understanding is that explicit knowledge is written and therefore can be readily shared, whereas tacit knowledge is embodied, residing in people’s heads and therefore much less accessible [14]. Knowledge Management efforts are often focussed on how to get tacit knowledge to the status of explicit knowledge. Nonaka and von Krogh [15] provide an extensive review and analysis focussed on these two types of knowledge, suggesting that they exist along a continuum, with ‘knowledge conversion’ occurring to transform one into the other. There continues much debate around this and they acknowledge the view that there are times when tacit knowledge by its very nature of being embedded in the tactile experience, remains tacit forever. A key point Nonaka and von Krogh make is that by acknowledging tacit knowledge in organisational knowledge creation, it has changed the propensity to focus on information management, both in theory and gradually, in practice. This acknowledgement that tacit knowledge is equally worthy of attention is relevant for this discussion, especially when linked to Whittaker and Van Beveren assertions [13] that knowledge is not a transferable commodity at all.
If one accepts these meanings of knowledge as intangible, residing in heads, or even more ethereal, not residing anywhere, but being a product of interaction, it makes the debate about explicit and tacit knowledge a different one. The proposition here is that there is no such thing as explicit knowledge. Artefacts that we consider explicit knowledge - research; guidelines; pathways, audit results and so on – are not knowledge at all, they are simply more information.
If this proposition is accepted, there are wide implications for any Knowledge Management endeavour. Instead of directing energy into how to get that which is in heads written down, and implementing tools to ensure it can be shared, we need to attend to almost the opposite – how we turn what is commonly termed explicit (written) into tacit (in our heads), because then, and only then will it affect what we do and how we behave.
4.1.1. Bringing Knowledge Management, Peer Review and Clinical Audit together
Whilst peer review activity is clearly a component of the audit cycle [3], it does not seem to be an explicit driver of audit topics. Determining the scope and subject of an audit in the first place, is not well described in the literature. Who decides what should be audited? Who plans the detail of the audit and how the results are evaluated, discussed, published and put forward as a reason for change? Who is expected to change? There seems to be an issue of ownership here that Knowledge Management theory can enlighten. If the information resulting from audits is not asked for and it is not trusted, then it is little surprise that it does not change practice.
Snowden [12] asserts that people in organisations make up a complex environment and that this environment will have a natural flow of knowledge creation which is related to the desire to problem solve. People naturally incline toward groups that facilitate knowledge processing in order to solve a particular problem where they ‘pull’ the knowledge they want rather than have arbitrary information ‘pushed’ at them. Malterud contends that clinical knowledge consists of interpretive action and interaction—factors that involve communication, opinions, and experiences [16]. It has also been shown that problem based learning is more effective than other forms of learning [17]. Davidson and Voss talk about buyers and sellers of knowledge [10], where the buyers want knowledge, generally based on problems or questions they want answered. The ‘pull’ engendered when a community wants to solve a problem is more effective to enhance both the transfer of knowledge and putting that into action; than pushing arbitrary knowledge at groups.
Drawing these factors together puts increasing emphasis on the role of clinical leadership in the context of Peer Review and similar forums. If we can ensure that audits are generated through these forums (the place of tacit knowledge) then Snowden’s problem solving ‘pull’ for information is enacted. The subsequent results (information) are fed back into the group where this new information can be brought together and viewed in the context of the experiences of the group, and with hindsight. The outcome of this is much more likely to result in new knowledge, which is then acted out in clinical practice.
4.1.2. Project Success factors
From a project implementation perspective, there were some key success factors:
- A partnership between the project team and the clinical specialty with a collaborative approach to software design was successful, with the clinician able to consult and then make decisions promptly which were not then changed. An approach of getting something workable implemented rather than seeking perfection was practical, and ensured the project met the timelines.
- Clear vision and ownership for the data capture process meant the implementation team could focus on the technical aspects and not be diverted by managing business process. The process and instruction materials were designed collaboratively, but accountability for the process sat with the clinical team.
- Organisational commitment to provide resources after the project officially finished recognised that implementing the technology is not the end of the work. Once the technology was implemented, continued commitment was required, driving the quality of data entry, reviewing reports, selecting cases for review and facilitating the meetings. It was also important to facilitate the correction of erroneous data quickly, once identified, so the data collection process could move on without delay and there was faith in the quality of the data. Work on reports and support for data analysis also continued well after the project finished.
4.1.3. Lessons Learned
In terms of Lessons Learned, on reflection there were three that stood out.
- Focussed clinical effort and leadership was put forward during the project to ensure the people and process aspects of the implementation were addressed as well as the technology. The outcome was robust, timely information that was owned by the specialty, and trusted, that could be fed in to the peer review sessions. However, the potential for undertaking focussed audit projects remains untapped. Active management of this information, to ensure it is converted to knowledge and used to effect change for the benefit of patients, is required. Continued attention to audit projects, however, is no small undertaking. Audits require planning and resourcing as any other project. Monitoring and evaluation is essential to promote uptake of recommendations and measure subsequent improvement. To ensure ongoing value for patients from implementations of tools to support Audit, ongoing commitment to this should be clear at the start, potentially forming part of the initial business case for the software implementation.
- Often with technology, there is an assumption that good data entry means ready access to information once the system is in use. It seems there is rarely enough effort put in to the reporting requirements at the outset (personal experience). There are three implications for this. The first is that resources run out, meaning the recipients of the software are left feeling short changed- all this effort goes into data entry but I can’t get the reports I need. The second is that the database design may not be such that it lends itself to linking relevant pieces of data together. Data items that from the clinical perspective are obviously related may not seem so to the person developing the solution, resulting in a database structure that puts limitations on (or adds extra work to) how information can be retrieved. The third is that decisions on the software design may be found to be less than ideal once the system is in use and reports are finally available. If more thought is put into the desired reporting outcomes early in the design phase, these risks may be minimised. This project did a reasonable job of the reporting component, but it could have been better. The lesson is to ensure attention to the reporting needs in the early stages of both planning the project and the software design, and to seek advice from business intelligence experts before the design is finalised.
- Lastly, there was tight integration of the clinical software with the Patient Management System which was designed to minimise data entry and ensure ‘one source of truth’. This was an agreed principle from early on, but meant that the accuracy of this component of information was highly dependent on other people and processes. Late data entry or inaccuracies such as admitting the patient to the wrong consultant had ramifications on both the quality of the data and the surgical staff’s data entry processes. The positive side of this was that the implementation uncovered these practices allowing them to be addressed, but it was unforeseen, and caused some challenges along the way. The lesson here is to widen the communications about the new use for the data so staff can understand the implications, but also to be prepared for this eventuality, as there will inevitably be some issues of this nature. The implementation of any clinical system invariably uncovers anomalies in the administrative data which have been largely invisible or deemed unimportant before.
Many writers acknowledge the people and process elements of Knowledge Management and this is just as applicable in health as business – perhaps even more so with healthcare being such a human endeavour. If we re-conceptualise the common meaning of explicit knowledge as only information, and accept the redefinition of knowledge as a process, then this provides clear direction for Knowledge Management efforts - we should think more about how we effectively manage raw information through the process of people coming to know something, and how we evaluate whether that transition has been successful.
Alongside the information management activities must come a culture that facilitates the process of knowing. Forums such as Peer Review groups are critical in facilitating the transformation of key information such as audit and research results into knowledge; however it is not as simple as just making time for a meeting. The challenge is to ensure forums are available that provide the information, leadership, time and culture that allows for safe discussion that can result in increased knowledge that guides practice. Furthermore, enduring clinical leadership is required, to drive audit projects that ensure that health professionals are doing the best they can to improve practice according to the evidence available.
In this example of software implementation to support clinical audit, the question of when and how the information to hand is turned into ‘knowledge’ is interesting to debate. Notwithstanding the obvious benefits of gaining new information and hearing the advice and experiences of expert colleagues, the opportunity to review collective practice scientifically and measure the effectiveness of an introduced change has not yet been made use of. Like most technology implementations, the success lies in the people and process changes that are enabled by technology – not by the technology itself.
A worthwhile next step would be to select audit topics as described above, i.e. chosen and managed by the Peer Review group, and once the results are available, combine this with research activities to assess the effect it has on the subsequent uptake of any recommended outcomes. Only with this clear process for selection of audit topics, introduction of change where appropriate and measurement of the outcomes of that change, can we be sure new knowledge is really improving health outcomes.
Thanks to David Theobald for his assistance in reviewing this paper, and to the DHB concerned for supporting its publication.
[1] Prasad KRS, Reddy KTV. Auditing the Audit Cycle. Clinical Governance: An International Journal 2004; 9 (2) pp. 110-114.
[2] Davies HTO, Avoiding Common Pitfalls: Developing Effective Clinical Audit. Hospital Medicine 1999; 60 (10) 748-750
[3] Surgical Audit and Peer Review: A Guide by the Royal Australasian College of Surgeons.3rd ed. Retrieved on 29.8.10 from www.surgeons.org/publications.
[4] Palmer C. Clinical Governance: Breathing new life into Clinical Audit. Advances in Psychiatric Treatment 2002; 8. 470–476
[5] Johnston G, Crombie IK, Davies HTO, Alder EM, Millard A. Reviewing audit: barriers and facilitating factors for effective clinical audit. Quality in Health Care. 2000;9:23–36
[6] Goodman NW. Clinical Governance. British Medical Journal 317 Dec 1998 p1725 -1727
[7] Ministry of Health, Wellington, New Zealand. Towards Clinical Excellence: An Introduction to Clinical Audit, Peer Review and Other Clinical Practice Improvement Activities. 2002
[8] Tan J, Wen HJ, Awad N. Examining chaos theory in action. Health Care and Services Delivery Systems as Complex Adaptive Systems. May 2005/Vol. 48, No. 5 Communications of the ACM
[9] McElroy MW. The New Knowledge Management: Complexity, learning, and sustainable innovation. Amsterdam; Boston: KMCI Press: Butterworth-Heinemann; 2003
[10] Davidson C, Voss P. Knowledge management: An introduction to creating competitive advantage from intellectual capital. Auckland, N.Z.: Tandem Press; 2002
[11] Parent,R. St-Jaques, D, Believau, J. Knowledge Transfer Strategy: An emerging Classification System. In Abou-Zeid, E. (ED) Knowledge Management and Business Strategies. Electronic reproduction. Boulder, Colo. : NetLibrary, 2007
[12] Snowden D. Complex acts of knowing: paradox and descriptive self-awareness. Bulletin of the American Society for Information Science and Technology. 2003; 29(4):23-28
[13] Whittaker J, Van Beveren, J. Social capital, an important ingredient to effective knowledge sharing: Meditute, a case study. Clinical Knowledge Management: Opportunities and Challenges Chapter 17 p. 297-314
[14] Stacey RD. Complex Responsive Processes in Organisations; Learning and knowledge creation. London, New York: Routledge; 2001
[15] Nonaka I, von Krogh G. Perspective-Tacit knowledge and knowledge conversion: Controversy and advancement in organizational knowledge creation theory. Organzation Science. 2009;20(3):635-52
[16] Malterud, K. The art and science of clinical knowledge: evidence beyond measures and numbers. The Lancet August 2001 358 (9279), pp 397-400
[17] Wyatt J. Clinical Knowledge and Practice in the Information Age: A Handbook for Health Professionals. London: RSM Press, 2001. Book Preview via Google Books
