Implementing an Electronic Clinical Activity Capture System for a Hospital Pharmacy Department - a Case Report in New Zealand

This article is also available as a PDF file.

Abstract
Introduction
Objective
Methods to implementation
Results
Discussion
Conclusion
Acknowledgements
References

Abstract
Aim: To implement an electronic clinical activity capture system for a hospital clinical pharmacy department.
Introduction: Health service departments are increasingly being held accountable for providing high quality, efficient yet cost effective care. In order to do this, there must be robust information systems in place to capture relevant and measurable data on service workflow and activities.
Case report: This case report discusses the implementation process of such a system at the Middlemore Hospital Clinical Pharmacy Department from the period of January 2008 through to July 2009, highlighting key steps and barriers during this process. Selected examples of data and reports produced are also described showing the types of information that can be generated for the purpose of quality improvement and administration.
Discussion: Implementation of a new electronic clinical activity capture system requires a multifaceted approach which must address legal, privacy, security, technological and human factors. This case report describes key steps in the implementation process and may help other departments which are undergoing such processes.

1.    Introduction
1.1.    Background
Studies have shown the devastating and costly effects of medication errors [1-3]. The Hospital Clinical Pharmacy (HCP) profession has increased the diversity of services provided to target and improve this important facet of health care with some good results [4-6]. Clinical pharmacy is a speciality field of pharmacy which has moved away from the traditional tasks of compounding and distributing medicines, to more cognitive clinical activities, providing services focused on optimising medication health outcomes in patients [7]. Such changes in service provision often involve – but are not limited to - reviewing the appropriateness of medications prescribed, recommending and advising on changes to drug therapy, providing patient and staff medication education, and medication safety initiatives [8, 9]. These changes in service provision aim to optimise medication related outcomes for patients throughout the Medication Management Process (MMP) [10].

1.2.    Why was the project initiated?
The concept of clinical governance [11] and trends in healthcare has meant that just like any other health service, the HCP service at Middlemore Hospital (MMH) must continually demonstrate its efficacy, quality and efficiency in a resource scarce environment. Prior to the implementation of an electronic clinical activity Data Capture System (DCS); data was primarily reported monthly to administrative management staff via qualitative means.  This approach was adequate for project based staff. However, for clinical pharmacists who dealt primarily with patients and other health professionals in the ward (i.e. “by-the-bedside” care); there was a lack of clinical activity and workflow information available to guide resource allocation decisions and administrative purposes. Previous attempts to collect data resulted in information that could not be easily communicated and understood by key stakeholders such as non-clinical administrators or other health professionals who may not necessarily understand the nuances of HCPs and services provided. This has led to suggestions that Clinical Pharmacists’ (CPs) contribution to patient care was under-recognised [12].

Therefore, a DCS was needed to assist pharmacy managers to justify the value of the service, increase the recognition of HCP’s contribution to patient care, performance manage staff, facilitate administrative and workforce planning and identify areas for improvement [13-17]. Having a robust system to capture data on workflow process and clinical activities was a way to provide some of this data [18]. Furthermore, since CPs concentrate exclusively on improving the quality and safe use of medicines (e.g. through intervening on medication errors and potential adverse drug events), capturing information on CPs’ workflow activities may also help to improve the quality of medicines use by proactively indicating trends and patterns of medicines use and error.

1.3.    Aim of the project
The aim of the project was to implement a DCS system that could measure CPs’ workflow activity and contribution to patient care at one site - Middlemore Hospital. This case report discusses key experiences during the acquisition and implementation processes.  The system chosen and some of the sample reports generated are also briefly discussed.

2.    Objective
2.1.    Requirements of the chosen system
The initial scoping and consultation with various pharmacy staff identified that the following requirements were essential for the project’s end users:

• Usability - Ease of use with fast data entry and minimal impact on current workflow process whilst providing the information required (including ability to model financial savings associated with data)
• Interoperability - Potential for integration and interoperability with existing and future systems (e.g. linkage to laboratory results, Patient Information Systems (PIMs) or medication management systems)
• Adaptability - Ability to manipulate system depending on type of data required
• Security and privacy – Compliance with New Zealand (NZ) requirements (e.g. the HIPC (Health Information Privacy Code 1994 [19]).
• Maintenance and training – Preferably undertaken by an external agent to minimise impact from internal staff changes
• Cost – An important factor for consideration although no initial budget was set

3.    Methods to implementation
3.1.    Evaluation process and the system chosen
A literature review and an evaluation of commercial and existing DCSs systems available at that time was undertaken (two commercial products from the United States (US) and one product from Australia, three pharmacy specific systems currently used at other District Health Boards (DHBs) in NZ, two non-pharmacy specific systems that were used at MMH). The evaluation revealed that no one system was able to deliver all these requirements. Some of the systems were either too costly for implementation, lacked the required features or were too time consuming to input data. Some were not user-friendly, had no room for future interoperability between systems, produced limited management reports or were not designed specifically for HCP use so significant alternations to the system would be required. An in-house built system was deemed too costly and resource consuming (i.e. clinical staff expertise input, time to build, etc). The results of the evaluation were presented to HCP management and CP staff with the decision to utilise the commercially available product Quantifi© by Pharmacy OneSource Inc (previously known as HealthProLink). This system appeared to best fulfil the criteria’s for MMH purposes.

The chosen system was a customisable clinical documentation and reporting tool that was built specifically for HCP and permitted future upgrades and interoperability with current and potential systems via HL7 (version 2.1-3.0) messaging (see Figure 1 for the interface system and note that the servers were held offshore). Software by this company has been reputedly used at over 1200 hospitals across the world [20]. From the demonstrations and trial versions, the system appeared to be relatively easy and quick to input whilst providing data that were thought to be required at the time. Users (i.e. CPs) accessed Quantifi© via a web browser. Personal Digital Assistants (PDAs) were a less preferred option due to the lack of hardware and the significant additional costs that would have been required to attain new hardware, however, this remained a possible option if required. The Quantifi© system had an optional interface that enabled Quantifi© to receive and process HL7-formatted ADT messages from MMH’s existing patient management system [21]. The system was HIPAA (Health Insurance Portability and Accountability Act) compliant, helped fulfil the JCAHO (Joint Commission on Accreditation of Healthcare Organizations) in the US and appeared to satisfy security concerns from the local Information Technology (IT) department. The product required an annual subscription to the system and this included training, maintenance, support and the production of custom reports by the vendor.  At that time, Quantifi was deemed to be the system that most suited MMH needs and the procurement process was undertaken.

3.2.    Managing unintended consequences during the procurement and implementation process
3.2.1.    Procurement process
As previously described, the author had utilised a mixed methodology to identify potential systems from various commercial and NZ built systems. Once the decision was made to use the Quantifi© program, the procurement process was undertaken facing two main barriers, 1) resources to attain the program and 2) legal, privacy and security issues.  A “New Software Service Request” form was completed with support from HCP management to initiate engagement with the IT department (Health Alliance (hA)). The tender process was bypassed due to the limited number of systems available for consideration. However, a business proposal was written and presented to the Health Information Committee (HIC) at MMH and also sent to the Asset and Capital Committee. Support from senior management staff from both HCP and the Quality Improvement Unit permitted additional momentum for the attainment of resources to fund this project.  

3.2.2.    Legal, privacy and security issues
One of the other main concerns was regarding the actual technical and physical security safeguards in place to protect the information during transmission and storage in NZ and the US respectively. The vendor had indicated that from a physical security aspect, the server is held in a secure facility with a high degree of technical and physical security [21]. Whilst at a network level (see Figure 1), the browser based transaction employs Secure Socket Layer (SSL) technology using both server authentication and 128-bit data encryption [21]. If the ADT option was also taken, then data would have been transferred over a point-to-point Virtual Private Network (VPN) and/or using the CDC’s (Centre for Disease Control and Prevention’s) PHIN-MS (Private Health Information Networks- Messaging System) encrypted messaging system [21].

From an application security perspective, Quantifi© utilised a variety of methods such as unique user identification number, minimum use of cookies, auto-expiring systems, etc and was HIPAA compliant (i.e. must have the ability to audit and review user information) [21]. From the local IT aspect, it appeared that the systems and encryption in place would be acceptable in terms of technically safeguarding the information from disclosure. However, it was noted that although the system was HIPAA compliant it was still necessary to ensure compliance with New Zealand law, in particular the requirements of the Health Information Privacy Code 1994.
An assessment of privacy and contractual issues between the vendor and MMH was completed in consultation with the legal advisors. As can be seen in Figure 1, if the ADT option was taken, patient identifiable information (e.g. name, NHI, weight, etc) would have interfaced with the vendor and held in offshore servers. Given the potential transfer of patient identifiable information off shore, careful thought was given to the extent of information required for the system and the safeguards that could be instigated.  In discussion with the vendors, it was determined that certain fields were not mandatory and could be blocked.  Contractual commitments were also established to prohibit the vendor from disclosing any information from the system or undertaking any data mining.  Agreement was also reached that the contract would be governed by New Zealand law. Consideration was given to the potential ability of the American government to access data under the provisions of the Patriot Act and further information on this aspect was provided by the vendor’s attorneys.  The New Zealand State Services Commission and Government Communications Security Bureau [22-25] guidelines were also taken into account in terms to determine best practice arrangements for holding sensitive information offshore.

Following this advice, only non-identifiable information was to be transferred. Consequently, the patient name field was blocked and a unique identifier different from the NHI was used to identify each patient.  All other information fields remained. The resulting information generated from the DCS would still provide HCP with the data required about workload and clinical activities even without direct patient details. The unique identifier was generated manually by CPs and would allow patient details to be traced at MMH when required. The system was subsequently purchased in February 2009.

3.2.3.    Implementation process issues
Despite the benefits associated with IT systems, there were many reports of failure during implementation [26, 27]. Staff resistance to change, insufficient training, lack of understanding of the new technology, limitation in time spent training due to constraints on regular work, insufficient IT support and technology not suiting requirements were some problems commonly associated with the technology [28-30]. The evidence suggests that failure of IT projects appear to be most often due to social and organisational factors rather than technological aspects [31-33].

The implementation of the DCS project therefore, concentrated on these social factors. Getting staff engagement and ownership was extremely difficult at the start. Comments such as “we’ve done this all before”, what’s the point” were commonly heard when the DCS system implementation was signalled. The resistance may have been historical in nature with such systems failing to be sustained. One CP staff member seemed to sum it up by saying that “the previous systems were just too clunky and time consuming to use, and it didn’t seem to mean anything as we never got to see the results”. This user feedback was extremely valuable as the key to ensuring the success of implementation primarily rested with the ease of usability. Exploring other District Health Boards (DHBs) HCP departments and discussion with other allied health professionals in MMH revealed that staff spent an average equivalent of 20-30 minutes entering data per day. This was used as a guide for ensuring that the Quantifi® system met such expectations. In addition, the HCP management team were informed and engaged to ensure that staff were provided resources (i.e. computer space and rostered time to input data) to allow input of information into the DCS. In this way, management made it easier for staff and hence provide some respite from the addition of another task to an already busy workload.

Feelings of being uninformed, and lacking the opportunity for meaningful participation seemed to reflect what is already known from the literature [30, 32]. In the health care sector where staff are highly professionalised and autonomous [34], end users must be sought to comment, discuss and involved in all parts of the decision making process. The literature suggests that despite difficulties with the time and effort required to collect data, most CPs deemed such data collection as important [35]. This was also witnessed in MMH. Most of the CPs understood the purpose of establishing such a system and agreed ideologically in the importance of data capture. Even though there was initial resistance, there appeared to be an attitudinal and cultural change. The use of two clinical champions, bottom-up decision making, leadership from management, discussion intra- and inter-professionally via allied health presentations and chat groups, have all appeared to contribute to the perception of importance to the DCS to the end users. It was interesting that the roll out and Go-Live was initially planned to be one HCP medical team at a time, however, close to the Go-Live date, all the teams wanted to start at the same time as the pilot testing results became available and benefits were seen.

3.3.    Customisation of the DCS system to meet end user requirements
Key stakeholders (i.e. HCP management and senior staff, Quality Improvement Unit staff and CPs) were sought to identify the type of information required from the DCS system. There was consensus amongst the key stakeholders that the DCS system must demonstrate CP’s contribution to patient care and be able to justify the HCP service. However, none of the key stakeholders were able to identify the exact key performance indicators (KPIs) required to evaluate the HCP service. As a consequence, a literature review was undertaken to identify the KPIs most aligned with evidence based effectiveness. Bond et al suggested that CP activities such as pharmacist being part of medical rounds, drug histories on admission and provision of drug information were correlated with the most significant benefits in terms of mortality and cost savings [6]. Following discussions with the HCP management team, and support from evidence based literature and pragmatic considerations, the DCS system was customised based upon two main sets of data:

• Quick Interventions – A summative capture of a clinical pharmacist’s daily workflow activity and service provision (e.g. Number of medicine histories completed, number of drug charts reviewed, time spent in the ward, etc)
• Interventions – Specific data on recommendations made by CPs to optimise patient outcomes

4.    Results
4.1.    Implementation and Go-Live
The Quantifi® program was implemented into the HCP department at MMH and went live on the 15 July 2009, providing information for key stakeholders on CPs’ workflow activity. There were a total of 25 users which included by-the-bedside clinical staff, project staff and various management staff. There were many different types of reports that could be produced; only selected reports are presented here for information. Reports were generated via the DCS’s standard reports and custom reports were analysed using Microsoft Excel.

From the management and administration perspective, the DCS was able to produce a report to represent the number of workflow activities completed over a desired period of time. For example in Figure 2, where the vertical axis indicates the total number of events and the horizontal axis indicates the week of the year. From here, one could identify the number of medication histories completed over time. The horizontal axis could easily be converted with a denominator (e.g. per 1000 patient bed days, per Full Time Equivalents, etc) in order to produce a run chart which could allow for comparison and track trends. For example, it may help to identify staffs that were particularly efficient at certain tasks and hence it may be beneficial to learn from their techniques. Similarly poorer performing staff may be identified as staff that may benefit from extra training. This form of data allowed pharmacy managers to identify the productivity of their staff and observe the general performance of the department.

Figure 3 shows a line graph of the specific drugs involved in the interventions made by pharmacists. This type of graph identifies the occurrence of the drugs involved in the interventions and may help facilitate knowledge discovery purposes. By identifying trends in the common medications associated with interventions, it may allow a proactive approach to prevent further error. Data could also be broken down further and analysed according to the wards, consultant teams, etc. This permitted learning and proactive approaches to help prevent errors from reoccurring.

Figure 4 shows the financial cost savings made by CPs’ clinical activities over the various weeks of the year. Again the graph could be converted to a run chart by utilising a common denominator. This graph is useful to give an indication of the total cost savings made by clinical pharmacists through their interventions. This type of data can assist HCP management to justify staff salaries and/or resources for the department. This is particularly of value especially to non-clinical staff, as it helps to illustrate the overall performance of the department in monetary terms. The costs were calculated based on the following:

• Assumption 1) Cost of preventable adverse event in NZ is NZ$5388 [3]. These rates are comparable to specific medication adverse event costs as reported in the literature [36-38]
• Assumption 2) Adverse event occurrence rate in NZ specifically for preventable drug and therapy related problems is 1.6% (expressed in another way, the number needed to treat to prevent one adverse drug event is 63 patients) [39]. This correlates with international findings (range from 1.1-12.5 per 100) [1,37,40-45]

Using the assumptions above and adapting a model proposed by Westergard [46], for every 63 patients admitted into hospital there would be one ADE costing at least $5,388. If a pharmacist intervenes and prevents that adverse drug event then that $5388 could be saved (i.e. the costs of treatment of ADE would have been averted). However, since pharmacists also often make interventions and recommend on drug therapy which may not have resulted in an ADE, the average cost of an ADE (i.e. $5,388) is divided by the number of interventions made. This equates to an approximate NZ$85 per pharmacist intervention. Interventions that changed medication therapy were given this cost saving attribution.

It is appreciated that this is a very basic approximation. Arguments may include the fact that interventions may not necessary prevent an ADE nor actually give cost savings. Conversely cost savings may not be a true representation since, many clinical activities are not necessarily easily attributable to costs but are extremely important in improvement of medication safety and quality of medicines use. In addition, there were no adjustments for inflation (ADE cost studies back in 2002) and the utilisation of the lowest cost of an ADE which did not result in harm (average was NZ$10,264 [3]). Hence, it is likely that total cost savings are an underestimation of the true cost savings made by clinical pharmacists. However, until there is further study and evidence in the NZ literature with more specific costing of individual intervention type, this financial model is simply an approximation and an attempt to simplify CPs contribution to patient care in monetary terms. Using a standard model permitted a base measure allowing subsequent identification of cost saving trends.

4.2.    Issues encountered and unintended consequences (e.g. time, subjective comments)
As with implementation of most IT systems, several unintended consequences and issues were encountered. There were confusions initially amongst the staff with regards to data input due to the nature of the data categorisation. There were also some fields which were not filled in. These issues have resulted in incomplete and inappropriately categorised data. To mitigate these risks, an accreditation program and training were instigated by the Quantifi© project co-ordinator and team leaders to ensure explicit understanding of the different categories amongst the staff and appropriate categorisation at the time of data entry. Moreover, the team leaders were able to oversee data input from the staff to allow for correction if necessary. Certain fields were also made compulsory to ensure complete data entry. Reporting times were roughly 27 minutes/day for data input which was within the initial projected time of 30mins. Despite these problems, staff reported the system easy and intuitive to use.

Another major issue was the performance speed of the system. Quantifi© is a web based system and the performance speed was primarily dependent upon the speed of the internet. Subsequently, the performance speed during peak hours of hospital internet use slowed considerably. For example, if the system was used at 0600hours when hospital internet use was low, the speed between submission of an intervention and a new page loaded was approximately 10 seconds. In comparison, at approximately 0830hours where hospital internet use was high, the same activity was increased to approximately 20 seconds. Internet load and the hospital’s firewall have been recognised to be the underlying causes of substantial change in performance speed and are still currently being addressed to identify whether these factors could be improved. Furthermore, most pharmacists prefer entering data towards the end of the day after they have conducted their work on the wards which resulted in a lack of computers available for data input. Consequently, efforts were made to roster pharmacists at various times during the day to input data to overcome both of these problems, thereby minimising disruptions.

The issue with the lack of ADT interface meant that CPs had to manually generate a unique identifier different to the NHI which has proved to be time consuming and prone to human error. As mentioned previously, this was still being investigated at the time of printing by the IT department for more suitable and efficient workarounds. By permitting ADT information, this would allow the HCP department to take full use of the DCS and minimise time and errors associated with manual input of unique identifiers.

5.    Discussion
The Quantifi® system was successfully implemented at Middlemore Hospital. The results attained from the DCS have allowed pharmacy managers to utilise the data to guide resource allocation decisions, performance manage staff, improve administrative and workforce planning and identify areas for improvement.  Since pharmacists often intervene to prevent medication errors and adverse drug events, data collected from the DCS system also allowed staff to identify trends and patterns of medicines use and errors in a proactive manner. On a personal staff note, pharmacists were able to observe the interventions and recommendations that they’ve made which could then be peer reviewed and allowed self assessment to better improve the quality of care provided.
It appeared that staff generally found the system easy and intuitive to use. However, there were problems with internet speed and the need to issue a unique identifier. The DCS system became the central HCP system which consolidated piece-meal data capture databases and excel sheets on specific areas of care. Various methods have been described for the implementation of an electronic clinical pharmacist activity and workload data capture systems [47-51]. This case was unique in that it is one of the first known health pharmacy IT systems where medical related information was held by an off shore provider in New Zealand. Furthermore, this is one of the first systems for Hospital Clinical Pharmacy in NZ which attempts to attribute cost savings to interventions made by clinical pharmacists.  There is literature which demonstrates clinical pharmacist’s contribution to patient care [4-6] but such studies are lacking in New Zealand. The author of this article has also recently conducted a study looking at key stakeholders’ perception of relevancy and measurability (e.g. chief medical officer, director of nursing, chief pharmacist, executive administration team and quality and risk manager) on currently recommended KPIs by major pharmacy and non-pharmacy organisations which purport to demonstrate clinical pharmacy’s contribution to patient care [52]. This study is currently awaiting publication, and together with research presented in this article potentially guides the NZ clinical pharmacy profession to have standardised performance measures and data capture at a national level.

As with all case studies reports limitations of this article include the view from only one perspective so there hasn’t been an examination from the users and other participants. This opens the study to bias with recollection and memory by the article’s author. The data is not generalisable to other settings and is primarily a descriptive account of the processes involved in implementing such a program hence cause-and-effect relationships cannot be drawn. Nonetheless, this case report was set out to give a description of what issues were encountered throughout the implementation of a DCP system and provides an insight to some potential issues that may also be experienced by others undergoing such processes. Future studies may include conducting similar measures and systems at other hospital sites, an evaluation of the perception of key stakeholders (both at a local, national and international level) on the data collected by the DCP system on its relevancy, evaluating the sustainability and validation of the financial modelling used. In short, there are many potential studies that are yet to be completed.

Key lessons learnt for a successful implementation required a multifaceted approach which needed to encompass technological and human factors. Privacy, legal and security issues also need to be overcome. In light of the ever changing health system with different service deliveries and literature it is recognised that the definitions, categorisation and modelling used in the DCS system will need to continually to evolve to ensure sustainability and relevance of the system.

6.    Conclusion
A data collection system which could measure CPs’ workload activity and contribution to patient care at Middlemore Hospital was implemented. Information required for both pharmacy administrative and medication safety purposes based on pharmacy clinical activity and workflow were provided.


7.    Acknowledgements
The author wishes to thank the Hospital Clinical Pharmacy department at Middlemore Hospital for all the contributions, belief and help during the implementation of the program who tirelessly contributed which ensured that the project was a success. The author would like to especially thank Sarah Wang for her support, editing and various other components for this project. The author also wishes to thank contributions from all the legal, technical, management, support staff and the Quality Improvement Unit which all helped to make this project a success.

8.    References
[1]            Bates D, Cullen D, Laird N, al e. Incidence of adverse drug events and potential adverse drug events: implications for prevention. JAMA. 1995;274:29-34.
[2]            Davis P, Lay-Yee R, Briant R, al e. Adverse events in New Zealand public hospitals I: occurrence and impact. Journal of the New Zealand Medical Association. 2002;115(1167 ).
[3]            Brown P, McArthur C, Newby L, al e. Cost of medical injury in New Zealand: a retrospective cohort study. J Health Serv Res Policy. 2002;7(1):S1:29-34.
[4]              Kaboli P, Hoth A, McClimon B, al e. Clinical pharmacists and inpatient medical care: a systematic review. Arch Intern Med. 2006;166:955-64.
[5]            Schumock G, Butler M, Meek P, Vermeulen L, al e. American College of Clinical Pharmacy; Evidence of the economic benefit of clinical pharmacy services: 1996-2000. Pharmacotherapy. 2003;23:113-32.
[6]            Bond C, Raehl C. Clinical Pharmacy services, pharmacy staffing and hospital mortality rates. Pharmacotherapy. 2007;27:481-93.
[7]             Hepler C, Strand L. Opportunities and responsibilities in pharmaceutical care. American Journal of Hospital Pharmacy. 1990;47(3):533-43.
[8]            SHPA. Society of Hospital Pharmacists of Australia; Standards of Practice for Clinical Pharmacy J Pharm Pract Res. 2005;35(2):122-46
[9]            NZHPA. NZ Healthcare Pharmacists Association: Clinical Pharmacy Guidelines. NZ: NZHPA; 2007.
[10]    Stowasser D, Allinson Y, O’Leary K. Understanding the medicines management pathway. . Journal of Pharmacy Practice & Research. 2004;34:293-96.
[11]    Scally G, Donaldson L. Clinical governance and the drive for quality improvement in the new NHS in England. BMJ. 1998;317:61-5.
[12]    Fowler A, Campbell D. Benchmarking and performance management in clinical pharmacy. International Journal of Operations and Productions management. 2001;21(3).
[13]    Overhage J, Lukes A. Practical, reliable, comprehensive method for characterizing pharmacists’ clinical activities. Am J Health-Syst Pharm. 1999;56:2444-50.
[14]    McDaniel M, DeJong D. Using pharmacists’ documentation of clinical activities to reclaim employees and reposition the department. Am J Health-Syst Pharm. 1996;55:285-8.
[15]    MacCabe T. Pharmacists to assess the value of services. Med Herald. 1997 21 August;Sect. 21.
[16]    Yee W, Norton L, Catania H, Shen S. Pharmacists' clinical interventions as a tool to improve patient outcomes through continuous quality improvement. ASHP Midyear Clinical Meeting. 1995;30(Dec):P-330(D).
[17]    Buell L, Sergeant K, Lazarus M, Galvan E. Implementing and tracking a clinical intervention program: effect on performance evaluation. ASHP Midyear Clinical Meeting. 1993;28(Dec)(P-480(D)).
[18]    Smith P. Performance measurement in Health Care: History, Challenges and Prospects. Public Money & Management. 2005;25(4):213-20.
[19]    HIPC. Health Information Privacy Code.  2008  [cited 2009 22/07/09]; Available from: http://www.privacy.org.nz/health-information-privacy-code/
[20]    PharmacyOneSource. PharmacyOneSource homepage.  2009  [cited 2009 22/07/09]; Available from: http://www.pharmacyonesource.com/
[21]    HealthProLink. Quantifi Security Architecture Bellevue: HealthProLink- 12505 Bel-Red Road Suite 208, WA 98005 2005.
[22]    Interdepartmental-Committee-on-Security. Security in the Government Sector.  2004  [cited 2009 25/07/09]; Available from: http://www.security.govt.nz/
[23]    GCSB. Government Communications Security Bureau (GCSB).  2009  [cited 2009 25/07/09]; Available from: http://www.gcsb.govt.nz/
[24]    Crown. Overseas hosting risk Analysis.  2009  [cited 2009 25/07/09]; Available from: http://www.e.govt.nz/policy/trust-security/overseas-hosting.html
[25]    Crown. Trust and Security - Home.  2009  [cited 2009 25/07/09]; Available from: http://www.e.govt.nz/policy/trust-security/
[26]    Hackathorn R. Data warehousing energises your enterprise. Datamation. 1995;41(2):38-42.
[27]    Shore B. Failure rates in global IS projects and the leadership challenge. J Global Inform Manag. 2005;8(3):1-5.
[28]    Quaddus M, Intrapairot A. Management policies and the diffusion of data warehouse: a case study using system dynamics-based decision support system. Decision Support Systems. 2001;31:223-40.
[29]    Mantere S, Martinsuo M. Adopting and questioning strategy: exploring the roles of cynicism and dissent.  17th EGOS - European Group for Organisation Studies Colloquium; 2001 July 5-7; Lyon, France; 2001.
[30]    Reichers A, Wanous J, Austin J. Understanding and Managing Cynicism About Organizational Change. Academy of Management Executive 1997;11(1):48-59.
[31]    Day K, Norris T. Leadership in times of crisis during change due to Health IT projects. Health Informatics and Review online. 2006.
[32]    Thamhain H. Leading technology-based project teams. Eng Manag J. 2004;16(2):35-42.
[33]    Heeks R, Mundy D, Salazar A. Why healthcare information systems suceed or fail. Information systems for public sector management: Institute for Development Policy Management; 1999.
[34]    Mintzberg H. The structuring of organisations. Englewood Cliffs: Prentice-Hall 1979.
[35]    Millar T, Sandilya R, Tordoff J, Ferguson R. Document pharmacist's clinical interventions in New Zealand Hospitals. Pharm World Sci. 2008;30:99-106.
[36]    Bates D, Spell N, Cullen D, al e. The costs of adverse drug events in hospitalized patients: Adverse Drug Events Prevention Study Group. . JAMA. 1997;277:307-11.
[37]    Senst B, Achusim L, Genest R, Cosentino L, al e. Practical approach to determining costs and frequency of adverse drug events in a health care network. Am J Health Syst Pharm. 2001;58(12):1126-32.
[38]    Classen D, Pestotnik S, Evans R, al e. Adverse Drug Events in Hospitalized Patients. JAMA. 1997;277(4):301-6.
[39]    Davis P, Lay-Yee R, Briant R, al e. Adverse events in New Zealand public hospitals II: preventability and clinical context. Journal of the New Zealand Medical Association. 2003;116(1183 ).
[40]    Bates D, Miller E, Cullen D, Burdick L, al e. Patient risk factors for adverse drug events in hospitalized patients. ADE Prevention Study Group. Arch Intern Med. 1999;159(21):2553-60.
[41]    Classen D, Pestotnik S, Evans R, Burke J. Computerized Surveillance of Adverse Drug Events in Hosptial Patients. JAMA. 1991;266(20):2847-51.
[42]    Jha A, Kuperman G, Teich J, Leape L, al e. Identifying Adverse Drug Events: Developing of a Computer-based Monitor and comparison with Chart Review and Stimulated Voluntary Report. J Am Med Inform Assoc. 1998;5(3):305-14.
[43]    Forster A, Murff H, Peterson J, Gandhi T, Bates D. The Incidence and Severity of Adverse Events Affecting Patients after Discharge from the Hospital. 138. 2003;3(Annals of Internal Medicine  ):161-7.
[44]    Honigman B, Lee J, Rothschild J, Light P, al e. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc. 2001;8(3):254-66.
[45]    Kaushal R, Bates D, Landrigan C, al e. Medication Errors and Adverse Drug Events in Pediatric Inpatients. JAMA. 2001;285:2114-20.
[46]    Westergard C. Standard Intervention Cost Avoidance Value Justification Worksheet. 26/2/2009 ed: PharmacyOneSource 2009.
[47]    Zimmerman C, Smolarek R, Stevenson J. A computerized system to improve documentation and reporting of pharmacists' clinical interventions, cost savings, and workload activities. Pharmacotherapy. 1995;15:220-7.
[48]    Mason R, Pugh C, Boyer S, Stiening K. Computerized documentation of pharmacists' interventions. American Journal of Hospital Pharmacy. 1994;51(17):2131-8.
[49]    Simonian A. Documenting pharmacist interventions on an intranet. Am J Health Syst Pharm. 2003;60:151-5.
[50]    Huntress J, Possidente C, Harry D. Documenting pharmacists' interventions on a hospital's mainframe computer system. Am J Health Syst Pharm. 1990;47(12):2711-5.
[51]    Fox B, Felkey B, Berger B, al e. Use of personal digital assistants for documentation of pharmacists' interventions: A literature review. Am J Health Syst Pharm. 2007;64:1516-25.
[52]    Ng J. Hospital clinical pharmacy key performance indicators (KPI) –What matters and what doesn't? A New Zealand perspective - Awaiting publication. Auckland: University of Auckland School of Pharmacy 2009.