Patterns of adoption and use of a web-based decision support system in primary care (PREDICT CVD 7)

• Abstract
• 1. Introduction
• 2. Methods
  • 2.1 Data collection
    • 2.1.1 ProCare PHO Clinical Registry
    • 2.1.2 PREDICT usage data
  • 2.2 Main outcome measures
  • 2.3 Data analysis
  • 2.4 Ethical Approval
• 3. Results
  • 3.1 Description of adopters and non-adopters of PREDICT
  • 3.2 Adoption patterns of the decision support system by doctors and nurses
  • 3.3 Patterns of use over time, all users
    • 3.3.1 Patterns of use over time among the most frequent users
• 4. Discussion
• 5. Conclusions
• 6. Acknowledgements
  • 6.1 Funding
• 7. References

Abstract
Objective: To describe the patterns of adoption and use of PREDICT-CVD, a web-based decision support system for CVD risk assessment and management.
Setting: General practices affiliated with ProCare Health Ltd, a network of three Auckland-based Primary Health Organisations.
Population: Approximately 500 general practitioners (GPs) and 450 practice nurses looking after a total enrolled population of around 660,000.
Design: Cross-sectional survey linking ProCare clinical registry data augmented by professional medical and nursing councils’ data with PREDICT usage data from August 2002 to January 2007.
Results: Over this 53-month period, 45,437 risk assessments were conducted on 25,705 patients by 416 GPs and 117 nurses who had incrementally adopted PREDICT-CVD. GPs who graduated over 30 years ago and those without vocational registration were less likely to adopt the program, but uptake did not differ by gender or country of medical degree. On average, 485 patients were assessed per month with a marked reduction over the December-February holiday period. GPs conducted 92 percent of the risk assessments and there was a large variation in the frequency of use. Four distinct user groups were identified; 31 percent completed a risk assessment on less than 5 patients and were labelled as non-users, 23 percent were infrequent users (5-20 patients), 25 percent were frequent (20-89 patients) and 21 percent were the very frequent users (90 or more patients). Infrequent or non-users were more likely to be less than 10 years since graduation. There was no difference in frequency of use by gender or by country of medical degree. An incentive payment scheme made available for those who assessed at least 90 patients appeared to have had very little impact on usage patterns: 80 percent never reached the target and only 2.4 percent of total GP users completed 90 or more risk assessments in an "all then nothing" pattern.
Conclusions: There were four relatively distinct patterns of use that may inform interventions to improve uptake of electronic decision support systems. Differences were found in uptake and frequency of usage of PREDICT by vocational registration status and year of graduation but may be due to part-time practice. No differences were found by gender or country of medical degree. Adoption of the programme was responsive to promotional activities via GP cell groups but a financial incentive had very little impact on use. Determining why almost one-third of GPs who adopted PREDICT assessed fewer than 5 patients over the study period should be a priority for further evaluation.
 

1. Introduction
Since the early 1990s, New Zealand guidelines have advocated CVD risk assessment as a means of identifying those at high risk of cardiovascular disease (CVD) and for targeting subsequent management. However, a national survey of GPs conducted in 1999 showed that while most GPs used CVD risk tools, 70 percent used them about once a month or less.^[1] A web-based clinical decision support system, known as PREDICT-CVD was designed and developed collaboratively to facilitate CVD risk assessment and risk based management whilst also integrating CVD risk prediction research within routine practice.^[2] The collaboration included epidemiologists from The University of Auckland, Information Technology specialists from Enigma Publishing Ltd (a private provider of online health knowledge management) and clinicians and support staff from ProCare Health Limited, Counties Manukau District Health Board, New Zealand Guidelines Group, National Heart Foundation and the Ministry of Health.

PREDICT was implemented as an opportunistic CVD risk assessment and management programme under the name of "Prompt" in ProCare in August 2002. This web-based programme was only compatible with MedTech and subsequently Next Generation, primary care electronic medical record (EMR) software programs. These systems were used by the majority of general practices. PREDICT-CVD opens as a window within the EMR, automatically extracts CVD risk data from the EMR, generates a quantitative five-year CVD risk assessment and provides evidence-based patient-specific decision support according to current New Zealand cardiovascular guidelines. The target patient group was those adults (mainly over the age of 40 years) who met guideline criteria for risk assessment. At the same time, aggregated anonymised risk profile data is stored and with permission from providers can be used for epidemiological research purposes. The roll-out of PREDICT-CVD was supported by educational seminars to general practice continuing medical education groups. Adoption of the program was encouraged but entirely voluntary. GPs were also offered $900 including tax per GP as a one-off incentive payment once they had assessed 90 patients. As few GPs at the time had the high speed internet connection required for PREDICT, this money was provided to cover the cost of installation of secure high speed web access and the user charges for three months. GPs who had eligible patient management systems and who chose to adopt PREDICT-CVD were visited by practice facilitators who installed the software, ensured safe connectivity to the Internet and provided limited training to the primary care team. Whilst GPs were the target of the original implementation plan by the primary care organisation, practice nurses were also subsequently encouraged to be users.

The "acid" test for a decision support system is whether it is perceived to be useful enough to be adopted and, once adopted, whether the ease of use and fit with clinical work flow is enough to change clinical behaviour.^[3] To achieve sustainable usage, the benefits to individual clinicians must outweigh the time and effort to use it. Heeks et al^[4] note that although some heath care information systems succeed, the majority are likely to fail. The greater the personal and organisational change required by an information technology system, the greater the risk of failure. The system needs to fit with the user’s values and change needs to be in small enough steps to be achievable by the majority.^[4]

Evaluations of the use of computer systems have focused at multiple levels from the individual, group, organisation, industry or social sector^[5] and have used research methodologies from various perspectives such as cognitive psychology and other social sciences, management, ergonomics, computer science and clinical epidemiology.^[5-8] We conducted a three-part evaluation using mixed qualitative and quantitative methods of the barriers, challenges and attitudes to CVD risk assessment practice and to the use of PREDICT by a group of New Zealand primary health care doctors and nurses. This sub-study explores the adoption patterns, the characteristics of adopting clinicians and subsequent frequency of use of the PREDICT tool. In particular we wanted to investigate whether: 

• Practitioner characteristics such as gender, vocational registration or years since graduation from medicine or nursing made any difference to adoption or subsequent usage; and
• the financial incentive influenced the usage of PREDICT CVD.
  
  
   

2. Methods

 

2.1 Data Collection
 

2.1.1 ProCare PHO Clinical Registry
One co-author (JB), a ProCare GP working with members of ProCare Health management team (PR and KM), was given permission to access the Primary Health Care Organisation (PHO) registries and other administrative sources to create a dataset of practices, location, PHO (ie, ProCare Network Manukau, Auckland or North) and GPs and practice nurses working within each practice. Data were also available on whether practices were Interim, Access or Very Low Access funded. This represents the funding formulae for capitation payments received by a practice. These formulae are based on the proportion of an enrolled primary care population meeting Ministry of Health criteria for being "high needs" (Maori or Pacific ethnicity or living in NZ Deprivation deciles 9/10). These data were then augmented from New Zealand Medical and Nursing Councils’ registries regarding year of registration, country of training, vocational registration (for doctors) and year of registration (for nurses).

Following collation of the clinical registry, a new dataset stripped of personal and practice names and identifying clinicians only by their New Zealand Medical Council (NZMC) registration number or New Zealand Nursing Council (NZNC) registration number was generated and made available for analysis by the research team.
 

2.1.2 PREDICT usage data
When a clinician uses PREDICT for CVD risk assessment and management of patients their professional registration number (NZMC or NZNC), time and date of usage is recorded on the PREDICT server along with anonymised patient risk profile data. This usage data was extracted from the PREDICT server with permission from ProCare Health Ltd and linked to the de-identified clinical registry data via NZMC or NZNC.
 

2.2 Main outcome measures 

• Differences between adopters and non-adopters of PREDICT. 
• Uptake and usage patterns of PREDICT by doctors and nurses over time. 
• Differences between infrequent compared to frequent users of PREDICT. 
• Patterns of usage by the most frequent users.
  
  
   

2.3 Data Analysis
Univariate analyses were conducted and differences in dichotomous outcomes assessed using the Chi Square statistic. All analyses were conducted using SAS statistical software Version 9.1 with usage distributions over time plotted using Excel spreadsheet functions.
 

2.4 Ethical Approval
This study was approved by the Northern Y Regional Ethics Committee (NTY/07/01/004) in March 2007.
 

3. Results

 

3.1 Description of adopters and non-adopters of PREDICT
An adopter is defined as a general practitioner (or nurse) who chose to acquire high speed web access, have PREDICT installed on his/her patient management system (MedTech or NextGen) and receive training in the use of the program. Three-quarters of those who chose not to adopt the program had compatible patient management systems. Table 1 compares clinician and practice characteristics for PREDICT adopters and those who did not have PREDICT implemented in their practice. For 23 GPs, the NZMC stored on the PREDICT system was unable to be matched to the New Zealand Medical Council Register, possibly due to incorrect data entry within the patient management system. For GPs, differences were found for adoption of PREDICT by having vocational registration (?2 = 10.19, p-value 0.0014) and year of graduation (?2 = 14.13, p-value 0.0069) with younger GPs less than 20 years from graduation being more likely to adopt. No differences were found by gender or country of medical degree. About 20 percent of GPs that could be matched had incomplete data particularly with regard to practice characteristics (funding, size of practice, patient management system). Some of the reasons were: doctors changing practices within ProCare; practices becoming affiliated with another PHO; doctors retiring and practices being sold; or being a locum and not connected to one practice. Data was incomplete for more than a quarter of the nurses and differences in all nurse characteristics and practice variables (funding, location and size of practice, patient management system) for doctors were not assessed due to missing data. 

 

3.2 Adoption patterns of the decision support system by doctors and nurses
PREDICT-CVD was implemented from August 2002 in planned increments by geographic location (south, central, west and north Auckland). Whilst GPs were the target of the original implementation plan by the primary care organisation, it was subsequently also promoted to practice nurses. Nurse uptake lagged by 20 months and had a slower trajectory. The cumulative rate of PREDICT adoption by doctors and nurses is shown in Figure 1. Figure 2 plots the same data displayed as monthly rates of first time users, showing annual peaks in adoption following implementation evenings conducted via GP cell groups.

Figure1. Adoption of PREDICT by doctors and nurse; cumulative count of first-time users

Figure 2. Adoption of PREDICT by doctors and nurses; monthly count of first-time users

3.3 Patterns of use over time, all users
Between August 2002 and January 2007, 45,437 CVD risk assessments were conducted on 25,705 patients by 416 GPs and 117 nurses. The program was often used multiple times on individual patients within one consultation mainly for the purposes of demonstrating benefits of lifestyle changes (eg, stopping smoking) or drug treatment as well as used for subsequent follow-up of patients. Figure 3 shows assessment patterns over this time period. On average 485 patients were assessed per month (ranging from 43 in the first month to a peak of 1120 in June 2004). Usage declined from this point dropping to 2002 levels by the end of 2006. A key secular change that occurred during this time was the December 2003 publication and subsequent implementation of national guidelines for The Assessment and Management of CVD risk9 and The Management of Type 2 diabetes.^[10] Subsequently, a new PREDICT module, PREDICT CVD-Diabetes was developed and released for licence in December 2005. ProCare Health started implementation of this updated program in October 2006. There was a seasonal variation in usage with CVD risk assessments more likely to be conducted in autumn, winter and spring than summer months (?2 = 1273.93, p-value < 0.0001).

Figure 3. Total number of patients assessed per month by GP or practice nurse

There was large variation in the use of PREDICT by providers. The majority of assessments (92.2 percent) were conducted by doctors. The mean (sd) number of risk assessments completed by each GP over all the time period was 57 (94.2) and 17 (17.0) by each nurse. The median number of assessments was 15 and 3 respectively indicating that some GPs were very frequent users. The maximum number of patients risk assessed by provider category was 621 by a GP and 161 by a nurse. However, 31 percent of GP adopters and 56 percent of nurse adopters completed less than 5 risk assessments.

PREDICT GP adopters were then categorised by number of patients assessed using the tool (Table 2); a non-user being classified as completing risk assessments on less than 5 patients; an infrequent user completing risk assessments on 5–20 patients, a frequent user, 21–89 patients and most frequent user assessing 90 or more patients. We aggregated frequent and most frequent users and compared them to infrequent and non-users. There were no statistically significant differences by gender or country of medical degree. Infrequent and non-users were more likely to be less than 10yrs since graduation (?2=17.28, p-value 0.0017). The older doctors (over 30 years from graduation) were as likely to be in either group. A higher percentage of frequent and most frequent users had vocational registration compared to infrequent or non users (?2=22.60, p-value<.0001). We were unable to adjust for full-time equivalent status (FTE) which is highly likely to influence frequency of use. 

 
 

3.3.1 Patterns of use over time among the most frequent users
The most frequent users were categorised as having used the PREDICT for CVD risk assessment on 90 or more patients (ie, the criterion for receiving a one-off incentive payment of $900 plus GST). There were 88 GPs in this category (21 percent of GP users). Their patterns of use over time could be classified into four types. Individual GP examples of these four types are given in Figure 3A-D; A, start slowly, build up then decline; B, start with a rush then slowly decline; C, a fairly constant pattern overtime with assessment rates at the start similar to assessment rates at the end of the programme; and D, an all then nothing pattern defined as conducting at least 70 percent usage activity in one time block (3–6months).

Figure 3A. Build up then decline 36/88 (41%)           

Figure 3B. Start high then decline 29/88 (33%)
Figure 3C. Fairly constant over time 13/88 (15%)     

Figure 3D. All then nothing 10/88 (11%)

Of those GPs who met the criterion to receive the one-off incentive payment, 89 percent demonstrated sustained usage patterns over the 3–4 years following adoption, whilst only 11 percent of the most frequent users (2.4 percent of total GP users) appeared to be directly influenced by the payment target, resulting in a one-off burst of activity then virtual cessation.
 

4. Discussion
This paper describes the lifecycle of version 1 of a CVD risk assessment and management decision support software from adoption by 416 GPs and 117 nurses to obsolescence (ie, when it was replaced by an updated version that also included diabetes management). For GPs, there were differences in adoption of PREDICT by year of graduation and by vocational registration status but not by gender or country of medical degree. Adoption patterns had distinct peaks following annual renewed promotion efforts. Between August 2002 and January 2007, 45,437 risk assessments were conducted on 25,705 patients. On average, 400 patients were assessed per month with a marked reduction over summer-time, a time when most GPs have holidays and often take turns covering the practice whilst partners are away. GPs conducted 92 percent of the risk assessments and there was a large variation in frequency of use. It is possible that some nurses used a doctor’s NZMC number when using practice computers and we therefore underestimated nurse usage.

When GPs were classified by frequency of use, a higher percentage of frequent and most frequent users had vocational registration compared to infrequent or non-users. However this finding could be confounded by part-time work. Infrequent and non-users were more likely to be less than 10 years since graduation. These younger doctors are often practising as locums and therefore spend less time in a practice but we were unable to adjust for this. There was no difference by gender or by country of medical training.

Of note, just under a third of doctors who had the program and broadband installed and who received training and practice support did not subsequently use the program. A qualitative evaluation examining the barriers, challenges and attitudes to CVD risk assessment practice and decision support is underway and may provide some insight into why this occurred and help identify strategies to increase future usage.

The incentive payment scheme may have been suboptimal. Firstly, payment was in retrospect, so that most GPs needed to fund initial broadband installation. Secondly, there were no usage reports to give practices feedback on their progress towards the one-off payment. GPs who had reached between 80 and 89 risk assessments were contacted and told that they were very close to their target, but other GPs were not contacted. It is possible that a re-designed financial incentive could have been a more powerful driver of uptake and usage.

In terms of sustainability of usage, while there appeared to be a steady usage state between mid-2004 and late 2005 with just under 800 assessments per month occurring, the rapid decline in usage in 2006 is likely to mirror its loss of clinical currency with the development and subsequent implementation of a new guideline via an updated version of PREDICT. Other possible explanations include practice staff turnover and the need to renew or re-promote programs to sustain their use. Early indications of usage of the new CVD-Diabetes program suggest a rapid increase back to the previous high level.

The clinical registry data was derived from multiple primary care administrative datasets linked with data held on medical and nursing council registries. Often information was either lacking or out of date. Identifying nurses on the public register was difficult as there often were nurses with the same names and many nurses used their middle names as their first name. There was also no region or specialty information recorded comparable to the medical registry data. These problems resulted in a higher rate of unmatched nursing data. Moreover, the denominator population of GPs and practice nurses in Procare Health is by nature constantly changing, with GPs and nurses joining or leaving practices, joining or leaving the primary care organisation (eg, moved out of Auckland or changed to another primary care organisation). Therefore interpretation of differences between adopters and non-adopters and categories of users needs to be treated with caution, particularly for nurses.

For both GPs and nurses, no date of birth was available. Years since registration were therefore used as a proxy measurement for this variable. For overseas trained doctors who have only recently immigrated to New Zealand, year of registration may not accurately reflect their age. ProCare Health is currently instigating a new web-based practice database which can be updated regularly by practice staff and will be able to provide a more accurate record.

Despite these weaknesses, valuable information on the users of an electronic decision support system could be generated and linked to usage in a de-identified way.

Comparison with other studies Studies of physician adoption of information systems (IS) or other innovations^[11-13] often draw from technology diffusion theory^[14] with adoption related to: relative advantage (perceived benefit of an innovation over current practice); compatibility (perceived consistency with values, past experiences and needs); complexity (perceived ease of use); trialability ( extent to which innovation can be experimented on); and observability (degree to which results of innovation are visible to others). Other models of IS adoption in small businesses have also found decision maker characteristics (in this case GP owner’s innovativeness and IS knowledge) and organisational characteristics (business size and level of employee’s IS knowledge) to be important.^[15]

Specific studies on adoption of expert systems and computerised physician order entry systems have noted distinct user adoption groups similar to this study.^[16-18] Determinants of variation of usage have been found to be associated with physician attitude towards its effect on time-efficiency, perceived disruption to normal work practices, perceived ease of use and impact on quality of care^[3,13,17] but not with sex, or years in practice at the study institution^[15] or years since graduation.^[16,19] Some studies cite prior computer use or experience, level of training and limited IT skills as barriers,^[20] rural compared to urban practices,^[21] whilst others have not.^[16,17,22] Practice size has also been strongly correlated with electronic health record adoption^[23,24] but we were unable to assess this due to missing data.

External incentives such as financial compensation for capital outlay or payment for quality have been cited as facilitators for adopting best evidence into practice.^[12] It is possible that the financial incentive offered for the use of PREDICT may have lowered resistance to adoption. However, we found little evidence that it influenced actual usage. Several studies of primary care guideline implementation via decision support systems^[25-27] have demonstrated that willingness to adopt various computerised interventions does not necessarily translate to actual use due to the reality of busy clinical practice and fit with work processes.
 

5. Conclusions
Differences were found for adoption and frequency of use of PREDICT and were associated with having vocational registration and year of graduation but this may reflect time spent working within a practice. No differences were found by gender or country of medical degree. Three key findings are relevant to future implementation of ECDS in primary care. First, adoption of the program appeared to be responsive to annual promotional activities via GP cell groups. Secondly, while a financial incentive may have helped to facilitate adoption, it had very little impact on the ongoing usage of the tool in routine clinical practice. Thirdly, a substantial number of practitioners who had the program installed did not subsequently use PREDICT and identifying the reasons should be a high priority for future research.
 

6. Acknowledgements
We wish to thank ProCare Health Ltd staff and affiliated practices GPs and practice nurses for supporting and participating in this programme. Our thanks to Joanna Broad for her help with analyses.
 

6.1 Funding
PREDICT-CVD development was funded by ProCare Health Ltd, Counties Manukau District Health Board and the Ministry of Health. The PREDICT research project is supported by a research grant from the Health Research Council. Susan Wells was the recipient of a National Heart Foundation Research Fellowship. Janine Bycroft received a training endowment from the New Zealand Population Health Charitable Trust.
 

7. References
 

1. Arroll B, Goodyear-Smith F, Kerse N, Lloyd T. Four clinical guidelines – their use and usefulness to GPs. N Z Fam Physician. 2002;29:177–83.
2. Wells S, Jackson R. Online management of cardiovascular risk in New Zealand with PREDICT-Getting evidence to the "moment of care". HCIRO. 2005;March:1–6.
3. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. Management Information Systems Quarterly. 1989;13(3):319–40.
4. Heeks R. Health information systems: failure, success and improvisation. Int J Med Inform. 2006 Feb;75(2):125–37.
5. Olson GM, Olson JS. Human-computer interaction: psychological aspects of the human use of computing. Annu Rev Psychol. 2003;54:491–516.
6. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review.[see comment]. JAMA. 2005 Mar 9;293(10):1223–38.
7. Kaplan B. Evaluating informatics applications – -some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform. 2001 Nov;64(1):39–56.
8. Kaplan B. Evaluating informatics applications – clinical decision support systems literature review. Int J Med Inform. 2001 Nov;64(1):15–37.
9. New Zealand Guideline Group. The assessment and management of cardiovascular risk. Wellington, New Zealand; 2003.
10. New Zealand Guideline Group. The management of type 2 diabetes. Wellington, New Zealand; 2003.
11. Ford EW, Menachemi N, Phillips MT. Predicting the adoption of electronic health records by physicians: when will health care be paperless J Am Med Inform Assoc. 2006 Jan¬-Feb;13(1):106–12.
12. Grol R, Wensing, M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust. 1995 Mar;180(6 suppl):S57–S60.
13. Williams RB. Successful computerized physician order entry system implementation. Tools to support physician-driven design and adoption. Healthc Leadersh Manag Rep. 2002 Oct;10(10):1–13.
14. Rogers EM. Diffusion of innovations. 4th Edition ed. New York: The Free Press; 1995.
15. Thong TYL. An integrated model of information systems adoption in small businesses. Journal of Management Information Systems. 1999;15(4):187–214.
16. Lindenauer PK, Ling D, Pekow PS, Crawford A, Naglieri-Prescod D, Hoople N, et al. Physician characteristics, attitudes, and use of computerized order entry.[see comment]. J Hosp Med.2006 Jul;1(4):221–30.
17. Schectman JM, Schorling JB, Nadkarni MM, Voss JD. Determinants of physician use of an ambulatory prescription expert system. Int J Med Inform. 2005 Sep;74(9):711–7.
18. Zheng K, Padman R, Johnson MP, Diamond HS. Understanding technology adoption in clinical care: clinician adoption behavior of a point-of-care reminder system. Int J Med Inform. 2005 Aug;74(7-8):535–43.
19. Clayton PD, Pulver GE, Hill CL. Physician use of computers: is age or value the predominant factor? Proceedings The Annual Symposium on Computer Applications in Medical Care. 1993:301–5.
20. Short D, Frischer M, Bashford J. Barriers to the adoption of computerised decision support systems in general practice consultations: a qualitative study of GPs’ perspectives. Int J Med Inform. 2004 May;73(4):357–62.
21. Menachemi N. Barriers to ambulatory EHR: who are ’imminent adopters’ and how do they differ from other physicians? Inform Prim Care. 2006;14(2):101-8.
22. White C, Sheedy V, Lawrence N. Patterns of computer usage among medical practitioners in rural and remote Queensland. Aust J Rural Health. 2002 Jun;10(3):137–46.
23. Menachemi N, Ford EW, Chukmaitov A, Brooks RG. Managed care penetration and other factors affecting computerized physician order entry in the ambulatory setting Am J Manag Care. 2006 Dec;12(12):738–44.
24. Simon JS, Rundall TG, Shortell SM. Adoption of order entry with decision support for chronic care by physician organizations. J Am Med Inform Assoc. 2007 Jul–Aug;14(4):432–9.
25. Bonnevie L, Thomsen T, Jorgensen T. The use of computerized decision support systems in preventive cardiology - Principal results from the national PRECARD survey in Denmark. Eur J Cardiovasc Prev Rehabil. 2005;12(1):52–5.
26. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.[see comment]. BMJ. 2002 Oct 26;325(7370):941.
27. Hetlevik I, Holmen J, Kruger O, Kristensen P, Iversen H. Implementing clinical guidelines in the treatment of hypertension in general practice. Blood Press. 1998;7(5-6):270–6.