.jpg){kind=logo}
This article is also available as a PDF file.
Contents
Abstract
Background
Methods
Results
Discussion
Acknowledgements
References
Appendix 1. Data collection form
Appendix 2. Process for PREDICT use by health practitioners
Abstract
Computerised clinical decision support systems require health data to be captured in an explicit, structured way. However, traditional patient medical records contain data that is recorded in multiple ways using coding systems, free text, medical jargon and idiosyncratic abbreviations. To be meaningful, data transferred either automatically or manually from medical records to a clinical decision support program must accurately reflect data held in the patient medical record.
Aims
To assess the accuracy of health data captured routinely in primary care practice by PREDICT-CVD, a clinical decision support program for supporting the assessment and management of cardiovascular disease risk.
Methods
Data saved in the PREDICT clinical decision support system were audited against the same patients’ data held within an electronic patient management system. The audit was conducted in three general practices in Auckland, New Zealand. Within each practice the sample included all Maori patient records and a random sample of non-Maori patient records that made the total up to 100 per practice (n=300).
Results
We found good agreement between the data stored within PREDICT and that held within the patient management system. For 12 of the 27 variables examined there was perfect agreement. The most common disagreements, in order of frequency, were for smoking, diabetes and ethnicity recordings. Overall, there were 70 observations where data were recorded in PREDICT (but not in the patient management system), compared to 21 occurrences where data were recorded in the patient management system (but not in PREDICT).
Conclusions
Health data captured routinely in general practice by the clinical decision support system PREDICT were found to be highly consistent with data held in electronic patient records. However, the use of PREDICT-CVD improved the completeness of cardiovascular risk factor documentation.
Background
The development of PREDICT, a web-based clinical decision support program has been described elsewhere.[1,2] A key function of clinical decision support programs is to systematise and support patient care. PREDICT-CVD enables health practitioners (mostly general practitioners (GPs) and practice nurses) to assess a patient’s cardiovascular disease (CVD) risk and obtain patient-specific evidence-based recommendations for their ongoing clinical management at the moment of care. PREDICT-CVD is based on New Zealand guidelines for the assessment and management of CVD risk.[3] The PREDICT program has been integrated with the patient management system MedTech which has the largest market share for practice management software in New Zealand.[4] The integrated version of PREDICT has been implemented by the majority of general practices in ProCare Health Ltd a large primary care organisation in Auckland, New Zealand serving about 650, 000 patients.
The PREDICT program depends on high quality health data being collected and maintained throughout the clinical decision support system. A potential point for error is that data collected by PREDICT may differ from data held in the patient management system. The rationale for this study was to determine the degree to which error occurred at this point in the system; be it health practitioner, data entry, transcription, coding or mapping error.
Methods
To be eligible for this study, practices needed to be members of the ProCare primary health care organisation and use PREDICT integrated with the MedTech patient management system. We invited three practices (total of 12 GPs) that had performed the highest number of PREDICT assessments on Maori patients. All GPs who participated in the study provided written consent.
Between June and September 2005 the practices were visited and the audit was conducted. First, researchers ran an electronic query that produced a list of patients who had undergone a PREDICT assessment. From this list all Maori patient records were identified and selected. A random sample of non-Maori patient records was then produced to make up a total of 100 records in each of the three practices. The random number function in Excel was used for this purpose. A researcher extracted data onto a paper audit form (Appendix 1) for each patient. Only non-identifiable patient information was recorded of which most (272/300 records) was collected by one researcher (TR). The patient management system records were retrospectively examined for a maximum of five years to find documented CVD risk assessment and risk factor information. Data from all three practices were aggregated for the purpose of this study and were entered, and analysed, in Excel.
The process followed by practitioners when using PREDICT-CVD is described in Appendix 2. For audit purpose, ethnicity was categorised into two groups: Maori and non-Maori (combining European and other, Pacific, Indian and other Asian peoples). The audit data were counted and categorised as a data agreement or disagreement; or, as data that were recorded in PREDICT but not in the patient management system, or vice versa.
Results
Of the 300 patient records in this study, 99 (33%) were classified as Maori and 201 (67%) as non-Maori. Not one of the 300 records that were audited had a 2nd or 3rd ethnicity recorded, although this is possible in the patient management system. All 300 patient records had a PREDICT risk assessment template completed; 126 of these also had a risk management template completed. Definitions for the risk factor data can be found in the paper by Bannink et al.[2] The table presents the audit figures. For 12 of the 27 variables there was perfect agreement. In total, 27 data disagreements were observed. In order of frequency these occurred for smoking, diabetes and ethnicity recordings. The ten data disagreements for smoking status were six recordings of smoker in PREDICT versus non-smoker in MedTech; and, four recordings of past smoker in PREDICT compared to non-smoker in MedTech. For diabetes, PREDICT has eight data entries for Type 2 diabetes versus six Type unknown and two gestational diabetes entries in MedTech. The seven ethnicity disagreements were five recordings of a patient being Maori in PREDICT versus European in MedTech; and, two as Pacific Islander versus European or Asian in MedTech. Additionally, there were 70 observations where data were recorded in PREDICT (but not in the patient management system), compared to 21 occurrences where data were recorded in the patient management system (but not in PREDICT). Compared to the patient management system, PREDICT data were more complete for smoking status, height, weight and family history of ischaemic heart disease (IHD). In contrast, data for aspirin were more likely to be recorded in the patient management system than in PREDICT.
Discussion
Most adults in New Zealand visit their GP annually[5] and have health data collected and held within an electronic patient management system. Over the last decade the electronic information systems for primary health care have become increasingly sophisticated making the patient management system a rich source of data encompassing many facets of patient care. Moreover, it is possible to link clinical decision support systems to electronic patient management systems. A key issue for the introduction of clinical decision support programs into general practices is the successful integration with the existing patient management system.[6] A recent rigorous systematic review identified that automatic provision of decision support as part of clinical workflow was the most significant predictor of improved clinical practice.[7] With the introduction of the PREDICT program, integrated with the MedTech patient management system, it was important for researchers to assess the consistency of PREDICT data to that held within MedTech. This study has reassured us that data held in these two systems is highly consistent. Of note, PREDICT data on CVD risk factors were found to be more completely recorded compared to data held in the patient management system where it was often unstructured, more difficult to access and retrieve, and therefore to use for CVD risk estimation.
Where data discrepancies occurred, two patterns were noted. First, disagreement between the datasets occurred particularly for the smoking status, diabetes and ethnicity variables. In these cases, it was likely that PREDICT contained the more accurate data since all three variables were compulsory for the calculation of a patient’s five-year absolute CVD risk. Smoking and diabetes are established risk factors for CVD that require health practitioner attention and awareness with or without medical intervention. For those of Maori and Pacific Island ethnicity, the New Zealand guidelines recommend a 5% upward adjustment to their calculated five-year CVD risk[3]. This adjustment can only be made where the consulting practitioner is aware of the ethnic group with which a patient self-identifies and effectively risk stratifies Maori and Pacific Islander patients for more aggressive risk factor modification, treatment and preventive strategies. The systematic collection of data by the PREDICT clinical decision support system will assist health practitioners to identify those people most at risk and ensure they are appropriately managed.
Second, data for smoking status, height, weight and family history of IHD provided most occurences where variables were recorded in PREDICT but not in the patient management system. In these instances, the practitioner had the ability to confirm, or measure, data during the patient consultation and enter it directly into the PREDICT templates. Although the PREDICT input templates and decision support advice could be saved as one entry, the version of PREDICT used in this study was not configured to code single data variables (such as history of ischaemic stroke) back into the patient management system. The only PREDICT information that was able to be separately coded into the patient management system was the patient’s absolute risk estimation. Although this prevented potentially incorrect patient data being stored in the patient management system it limited its use. The updated version of PREDICT is able to code data, for example ischaemic stroke with the appropriate Read code, back into the patient management system. Compared to PREDICT the patient electronic record more frequently had data for aspirin. This result was not unexpected as GPs had requested that the PREDICT system treat such data as optional and default to "No" if left unfilled. In the updated version of PREDICT appropriate CVD and diabetes drugs from the long-term medication list will be identified and inserted into the risk management template. Even so, aspirin may remain under recorded given that many patients purchase their supplies over the counter.
This study had limitations. First, we only examined the consistency or reproducibility of data. The study method did not assess the external validity of information recorded, ie, whether the patient data was true. An example would be to validate the ethnicity of patients by comparing the electronic (or paper) record with a self-identified patient questionnaire, which is being conducted independently of this audit. Second, a possible source of bias was that the researchers were not "blinded" during the data extraction phase. When data had been collected from the PREDICT templates it was systematically searched for within the patient management system. Once found, the search ceased. This method of data collection was implemented in order to mimic what occurs within general practices where the systematic documentation and collection of CVD risk assessment information is a challenge. Another issue was the generalisability of findings, in particular the representativeness of the practices using PREDICT and of the patients risk assessed. Because misclassification of ethnicity within New Zealand primary care data is problematic, and New Zealand CVD guidelines specify earlier CVD risk assessment for Maori, ethnicity data was a priority in the audit. Therefore, practices were selected because they had performed the greatest number of PREDICT assessments on Maori patients.
We found no other published information specifically comparing data in clinical decision support systems with data from electronic medical records. Most studies we reviewed assessed the validity of the patient management system databases only.[8-11] One study that assessed the validity of a database set up specifically for epidemiological and health research found good comparability with a general practice research database.[12] Healthcare providers are increasingly using clinical decision support systems to address deficiencies in the delivery of optimal evidence based care to patients across a range of conditions and settings. [13] Such systems have been shown to improve adherence to recommended care standards[14] and the delivery of preventive care,[15] and to reduce medication and prescribing practice errors.[16,17] A decision support system is only as effective as its underlying knowledge base and data systems. Ensuring evidence updating and data consistency are essential steps in the quality assurance of clinical decision support systems and more research into these areas is required.
In summary, this study has shown that the data held by the clinical decision support PREDICT-CVD is consistent with data in the patient management system MedTech. This is reassuring for primary health care users of the system and allows health researchers to meaningfully collate and analyse CVD risk factor data at the population level. The time has come to put significant resources into building clinical decision support systems that support patient care, primary health care and public health.
Acknowledgements
The authors would like to thank the GPs who participated in this study for allowing us to work in their practices and obtain the data we required. We are also grateful to Pritibha Singh and Roger Marshall from The University of Auckland for their assistance with analysis.
PREDICT was developed by The University of Auckland and Enigma Publishing Ltd in collaboration with ProCare Health Ltd, Counties Manukau District Health Board, the Ministry of Health, the National Heart Foundation, the New Zealand Guidelines Group and MedTech Global Ltd.
This study was supported by funds from the Health Research Council of New Zealand. SW is the recipient of a National Heart Foundation Research Fellowship.
The PREDICT project was approved by the Auckland Ethics Committee (AKY/03/12/314).
The authors state that no conflicts of interest exist.
References
1. Wells S, Jackson R. 2005. Online management of cardiovascular risk in New Zealand with PREDICT - Getting evidence to the "Moment of care". HCIRO. March 2005.
2. Bannink L, Wells S, Broad J, Riddell T, Jackson R. 2006. Web-based assessment of cardiovascular disease risk in routine primary care practice in New Zealand: the first 18,000 patients (PREDICT CVD-1). N Z Med J. 2006 Nov 17;119(1245).
3. New Zealand Guidelines Group. Best Practice Evidence-based Guideline. The Assessment and Management of Cardiovascular Risk. NZGG; 2003.
4. MediMedia PMS General Practice survey. MediMedia; 2001.
5. Ministry of Health. Taking the Pulse - the 1996/97 New Zealand Health Survey. Wellington: Ministry of Health; 1999.
6. Linnarsson R. Decision support for drug prescription integrated with computer-based patient record in primary care. Med Inform (Lond). 1993 Apr-Jun;18(2):131-42.
7. Kawamoto K, Caitlin H, Balas E, Lobach D. Improving clincal practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ, doi:10.1136/bmj.38398.500764.8F; 2005
8. Pringle M, Ward P, Chilvers C. Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer. British Journal of General Practice. 1995;45:537-41.
9. Jordan K, Porcheret M, Croft P. Quality of morbidity coding in general practice computerised medical records: a systematic review. Fam Pract. 2004 Aug;21(4):396-412 .
10. Thiru K, Hassey A, Sullivan R. Systematic review of scope and quality of electronic patient record data in primary care. BMJ. 2003 May 17;326(7398):1070.
11. Hassey A, Gerrett D. Wilson A. A survey of validity and utility of electronic patient records in a general practice. BMJ. 2001 Jun 9;322(7299):1401-5.
12. Carey I, Cook D, De Wilde S, Bremner S, Richards N, Caine S, Strachan D, Hilton S. 2004. Developing a large electronic primary care database (Doctor’s Independent Network) for research. Int J Med Inform. 2004 Jun 15;73(5):443-53.
13. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002 Oct 26;325:941.
14. Hunt D, Haynes R, Hanna S, Smith K. Effects of computer based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998:280:1339-46.
15. Shea S, DuMouchel W, Bahamode L. A meta-analysis of 16 randomised controlled trials to evaluate computer based clinical reminders systems for preventive care in the ambulatory setting. J Am Med Inform Assoc 1996;3:399-409.
16. Bennett J, Glasziou P. Computerised remiders and feedback in medication management: a systematic review of randomised controlled trials. Med J Aust 2003;178:217-22.
17. Kaushal R, Shojana K, Bates D. Effects of computerised physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163:1409-16.
Appendix 2. Process for PREDICT use by health practitioners
When health practitioners use PREDICT, they are presented initially with a template for CVD risk assessment (Figure 1 “Basic” tab); and optionally, a template for CVD risk management (Figure 2 “Advanced” tab). Much of the risk assessment template is automatically transferred from the patient management system to PREDICT (known as pre-population) if the data are available in a format that allowed mapping between the two systems. When pre-population cannot or does not occur, the practitioner is required to complete all blank compulsory fields in this template. He or she can also overwrite fields that had been pre-populated from the patient management system. Data are submitted via a secure connection to the PREDICT server where data from the risk assessment template is used to calculate a patient’s absolute CVD risk and data from the risk management template is used to produce patient-specific recommendations and education information. Using a broadband connection, both the risk assessment and risk management results are generated within seconds during the consultation. The completed templates and computer generated advice are saved together as a complete set in the patient medical record. The PREDICT server also stores each anonymised patient profile. As of July 2006, over 23 000 baseline PREDICT assessments had been undertaken in a large scale pilot within ProCare-affiliated general practices in Auckland.
