Comparative study of electronic pilot and paper data collection methods in a survey of general practice consultations

• Abstract
• Introduction
• Methods
• The NatMedCa survey 2001/2002
• The electronic data collection sub-study
• Data collection methods
• Results
• Discussion
• References
• Footnotes

Abstract
Aims
The aim of this paper is to report on a comparison of data collection by electronic methods with paper collection methods in the context of a large, nationally representative, survey of general practice consultations in New Zealand.

Methods
The National Primary Medical Care Survey (NatMedCa), carried out over 2001/2002, was a nationally representative, multistage, probability sample of general practitioners (GPs) and patient visits. The primary purpose of the survey was to collect data on the content of patient visits. In a pilot sub-study of data collection methods, data were captured in the course of consultations using practice management software enhanced with supplementary electronic forms. These data were compared with data from the main NatMedCa survey, which were collected using paper collection methods. This analysis focuses on a subset of the sub-study data comprising data from 10 community-governed, non-profit practices.

Results
The patient visits data from the four practices in the electronic data collection arm of the study provide evidence for differences in practitioner data reporting, when compared with data from the six practices employing paper collection methods, despite similar patient characteristics. Reasons-for-visit and problems-per-visit data from the electronic arm of the study appeared to be less complete than data from the paper arm. By way of contrast, practitioners using electronic data collection methods had comparatively high rates of recording prescription items per 100 problems. The very low number of reasons-for-visit in the electronic arm indicates a high likelihood of systematic bias in the practices employing electronic data collection methods, as every visit should have had at least one reason for the encounter.

Conclusions
The findings of the comparison of electronic and paper survey data collection methods are important for researchers intending to carry out general practice-based surveys. Survey data generated routinely via practice management systems may differ considerably from survey data collected using tailored paper collection instruments.

Introduction
The use of computers in a primary health care setting has increased markedly over the past decade in, for example, New Zealand,^[1] Australia,^[2] the UK^[3] and the Netherlands.^[4] Computers are used for a variety of purposes, including business management, electronic patient records, population health management and patient recall.

The potential of computer-based information systems to provide a tool for epidemiological studies,^{[5, 6]} morbidity studies^[7] and needs assessment^[8] is well recognised.

Surveys based in the general practice setting are a common and important research modality, particularly so for clinical and health services research. A number of such surveys have been conducted in New Zealand, yielding numerous publications and a wealth of information relevant to clinical practice, management and organisation, and to primary health care policy.

Three surveys conducted over the past two decades are the CoMedCa (1979/1980),^[9-13] WaiMedCa (1991/1992)^[14] and NatMedCa (National Primary Medical Care Survey) (2001/2002)^[15] surveys. While differing somewhat in their depth and breadth of coverage, each of these surveys sought to collect information on the content and organisational characteristics of a sample of New Zealand general practices. Each survey was conducted using paper collection methods consisting of pre-printed pads kept on the desks of participating general practitioners (GPs) and nurses.

Given the revolutionary change in the way data are collected, stored and analysed in primary health care settings, researchers in New Zealand are naturally interested in the potential for survey-based research data to be generated "automatically" via routinely collected patient and administration data collections within electronic records, rather than via paper collection methods which require extra time on the part of participating clinicians.

Several proprietary software products are commonly used in general practices in New Zealand, all serving the same basic set of core tasks involving the management of: clinical data; recall and follow-up data; prescribing and laboratory test data; office and accounting data; patient register data; and disease management. In 2001/2002 close to 100% of practices in New Zealand used some components of an electronic information system.^[1]

The aim of this paper is to report on a comparison of electronic data collection methods with paper collection methods in the context of the large nationally representative survey of general practice consultations (NatMedCa). This analysis focuses on a sample of community-governed, non-profit, primary health care organisations (for example union clinics and other community-based providers). A more detailed description of the community-governed non-profit arm of the NatMedCa study is provided elsewhere.^[16]

Methods

The NatMedCa survey 2001/2002
The National Primary Medical Care Survey (NatMedCa), carried out over 2001/2002, was a nationally representative, multistage, probability sample of GPs and patient visits. The primary purpose of the survey was to collect data on the content of patient visits. For two periods, both of one week, each selected GP completed a questionnaire for a 25% systematically selected sample of patient visits. The questionnaire was adapted from that used by the annual US National Ambulatory Medical Care Survey (NAMCS).^[17] Practices in the study were categorised according to their ownership status—private for-profit and community-governed private non-profit (criteria are listed elsewhere^[18] ).

A list of community-governed non-profit organisations was obtained from the umbrella organisation Health Care Aotearoa (HCA), and to this were added other organisations that fulfilled the above criteria. The number of practitioners working in community-governed non-profit organisations is relatively small, and these clinics are of particular interest given their dedication to poorly served populations. All clinics were approached and all practitioners and all nurses were asked to participate – although four of these practices would only participate in the electronic collection method.

The total visit sample for NatMedCa consisted of 10,506 records gathered from 246 GPs, 48 (19.5%) of whom worked in non-profit practices and 198 (80.5%) of whom worked in for-profit practices. The overall GP response rate was 71.7% (70.7% in the for-profits and 72.7% in the non-profits).

The electronic data collection sub-study
The NatMedCa survey contained a pilot sub-study to trial the feasibility of electronic data collection methods that included four community-governed non-profit practices (all Auckland-based). In total, 65 practitioners from 22 practices participated in this electronic data collection sub-study. Data from the four community-governed non-profit practices were compared in detail with six community-governed non-profit practices (mostly Wellington-based) where data collection had been by paper based methods. Key results from the comparisons are reported below. More complete results and background information are available elsewhere.^[16] In contrast to the paper data collection method of two one-week periods, electronic sub-study participants collected data for one period of two weeks.

The response rate for the entire electronic sub-study was 83.3% (100% for the non-profits).

Data collection methods
In the sub-study of electronic data collection methods, data were captured using practice management software enhanced with supplementary electronic forms in the course of consultations. In collaboration with the software supplier, an extra screen displaying survey questions, collecting data that were not routinely captured by the electronic information system or by primary health care practices, was purposively designed following the paper questionnaire structure and content (figure 1). This was incorporated and displayed ("popped-up") at the end of every fourth patient consultation (in keeping with the 25% systematic sampling of the paper collection), albeit only when the practitioner selected the option enabling this pop-up screen feature. At the completion of a practitioner’s data collection period, that practice’s manager/administrator then selected the collection period and practitioner participating in the survey, created one text file per participating practitioner encompassing both routinely captured patient consultation data (eg, diagnosis, prescriptions, etc) with the additional information collected. This text file was then encrypted and transferred via a secure virtual private network to the NatMedCa data server, where data were receipted and processed for analysis.

Data for the main NatMedCa survey were collected using paper collection methods consisting of pre-printed pads on the desks of participating GPs and nurses.

Figure 1: Screen shot of the NatMedCa screen that appeared once in every four patient consultations

Results
Overall, in the total NatMedCa sample of 192 practices, 96% of practices used computer age–sex registers, 70.1% used computerised patient records, 96.7% used computer-based recall systems and 80.4% used computerised disease registers.

The following tables show comparative results from a subset of the complete NatMedCa sample that includes four practices from which data were collected electronically (nine doctors and four nurses) and six practices from which data were collected using paper-based methods (24 doctors and 20 nurses). These 10 practices in the subset were all community-governed non-profit practices.

Table 1 compares patient characteristics across electronic and paper data collection practices. The two groups of patients were similar in terms of gender and age group distributions, but differed in ethnic group distribution of doctor visits, with doctors at paper collection practices having a higher proportion of visits by New Zealand European patients (31.0% as compared to 10.6% in electronic collection practices), and a somewhat lower proportion of Pacific patients (28.0% in paper collection practices compared to 36.2% in electronic collection practices). The ethnic group distribution of nurse visits did not differ to the same degree, with similar proportions of visits by New Zealand European and Pacific patients (20.9% vs 20.3% for New Zealand European and 47.4% vs 45.1% for Pacific patients respectively in electronic and paper collection practices).


Table 1: Percentage distribution of visits, by patient gender, age and ethnicity^[a]

* â€œPaper” – ethnicity was self-reported with multiple categories allowed; one ethnic category was then assigned per patient according to prioritisation of Mäori and Pacific people; 32 patients had missing data.

† â€œElectronic” – ethnicity was obtained from the practice management database, which allowed for a single category per patient; “European” has been included in “New Zealand European”, and “Other European” assigned to “Other”; 16 “electronic” patients had missing data.

Note: all tables have rounding errors

Table 2 shows the number of reasons-for-visit to nurses and doctors in electronic and paper data collection practices. Reasons-for-visit were the patients’ stated reasons for visiting, as recorded by the GP or nurse. Up to four reasons-for-visit could be recorded for each visit. Problems per visit (table 3) were the diagnoses/problems for each visit (up to four per visit) listed by the GP or nurse. For example, if the patient’s reason-for-visit was "sore throat", the GP or nurse may have recorded "bacterial pharyngitis" as the problem.

The most important finding, in terms of the capture of comparable data, concerns the reasons-for-visit and problems per visit for consultations (tables 2 and 3). In the electronic arm the number of reasons-for-visit for GP consultations averaged 0.29 and the number of problems per visit averaged 1.29; in contrast the paper collection practices averaged 1.76 and 2.13 respectively. For nurse consultations the number of electronically captured reasons-for-visit averaged 0.44 and the number of problems per visit averaged 1.06; in contrast the paper collection practices averaged 1.58 and 1.75 respectively.

Table 2 Percentage distribution, and mean number of reasons per visit^[b]

* Up to four reasons per visit could be recorded.



Table 3 Percentage distribution, and mean number of problems per visit

* Up to four problems per visit could be recorded.


Table 4 shows the proportion of visits where any test or investigation was ordered, a prescription or other treatment recorded, or where follow-up or referral was organised for the patient. Referrals were to medical specialists or other health professionals.

The most important difference between the electronic and paper practices was referral rates. From GPs, 7.1% of patients in the electronic practices and 23.3% in the paper practices were referred, and from nurses, 3.9% of patients in the electronic practices and 19.5% in the paper practices were referred (table 4). There were also differences in the prescribing and "other treatment"’ rates between nurses in electronic and paper collection practices.

Table 4 Rate per 100 visits where there was any test/investigation, prescription, other treatment, follow-up or referral

Table 5 records prescriptions and treatments per 100 visits. There were important differences for GPs and nurses between electronic and paper practices. For example, "all treatment items per 100 problems" was 213 for GPs and 207 for nurses in electronic practices in contrast to 157 for GPs and 154 for nurses in paper practices. Prescription items per 100 problems were 121 for GPs and 124 for nurses in electronic practices in contrast to 71 for GPs and 54 for nurses in paper practices.

Table 5 Number of treatment items per 100 visits, and per 100 problems

* All treatment items = All prescription items + All other treatment items.

Discussion
The patient visits data from the four practices in the electronic data collection arm of the study provide evidence for differences in practitioner data reporting, as compared with paper collection methods, despite similar (but not identical) patient populations. Data from the electronic arm of the study appeared to be less complete than data from the paper arm. The comparatively low number of reasons-for-visit and problems per visit for both doctors and nurses in the electronic arm indicates a high likelihood of systematic bias in the practices employing electronic data collection methods, as every visit should have had at least one reason for the encounter (in the patient’s words). By way of contrast, the electronic practices had comparatively high rates of recording prescription items per 100 problems for both doctors and nurses. As a result of these differences, it is likely that if grouped with paper collection practices data from electronic data collection practices would have introduced bias.

A likely reason for under-reporting of reasons-for-visit and problems per visit is that routinely used practice management software packages do not prompt (make visually available) additional reporting of survey information, as is possible with pen and paper data collection, but instead have practitioners collect information in their normal, more limited and utilitarian, manner. For example, in routine consultations not all the reasons for the visit are necessarily recorded but perhaps only the main one that the practitioner has decided will aid their care of the patient or that resulted in an action such as prescribing. This thesis is supported by the study findings which show that electronic data collecting practitioners were much less likely to record the patient’s reasons for visit and secondary problems/diagnoses than the primary, or single, problem/diagnosis and corresponding practitioner actions (investigation, prescription, etc). Furthermore, the paper method prompted the completion of a comprehensive survey questionnaire rather than a few auxiliary questions that perhaps seemed unimportant and disconnected from the main consultation. Practice management software systems do not readily encourage retrospective completion of the patient notes (and thus the study instrument), as the normal process is to proceed to the next fields and then exit, with billing details being automated. Therefore, without appreciation of these limitations, the analysis of electronically collected primary health care data on reasons for visit, diagnoses/problems and actions taken is likely to result in significantly different findings compared with data collected via paper questionnaires.

The higher rate of recorded prescription items in electronic practices most probably reflects the fact that in the vast majority of practices prescriptions are routinely generated electronically and therefore prescription data capture is very thorough (in contrast to paper collection methods used specifically for the purposes of the survey).

It must be noted that training of participants in the paper-based collection method was more rigorous than that of electronic collection clinicians. Participants were visited by a clinical director who gave instruction on the importance of the survey and what was required, and received detailed written instructions on the completion of survey forms. In comparison, electronic collecting participants were visited by the clinical director and/or project manager who gave instruction on the importance of completion of the additional pop-up screen/supplementary electronic form; detailed instructions were provided on the method to enable practitioners to complete the supplementary form, and the date ranges to do so were also given. The other visit fields, that were routinely collected, for example, reasons for visit, diagnoses, etc, did not receive full detailed instructions on their completion. The practice manager was responsible for creating data files for individual practice participants in the electronic pilot. Thus, it is expected that training and further instruction may have reduced the under-reporting seen with the electronic data collection method. This reinforces the need for intensive instruction in surveys of this kind in future.

The most important limitation of this comparative study is the small number of practices included: four electronic data collection practices and six paper data collection practices. In addition, practice location may account for variation in results – the practices in the electronic data collection arm were all Auckland-based while practices using paper collection methods were mostly Wellington-based. Some of the observed differences between these two groups of practices may represent real differences and some may be due to chance variations.

The differences between the data collected using the two data collection methods had considerable clinical/research significance in the context of the NatMedCa survey. However, due to rapid changes and improvements in electronic information systems, and the increasing familiarity and expertise of practitioners using them, it would be highly desirable to repeat this comparative study now using a wider variety of organisational types of general practice.

The findings of this comparison of data collection methods between a pilot electronic survey and a paper-based survey are important for researchers intending to carry out general practice-based surveys. Survey data generated routinely via practice management systems may differ considerably from survey data collected using tailored paper collection instruments.

References

1. Crampton P, Davis P, Lay-Yee R, Raymont A, Forrest C, Starfield B. Does community-governed non-profit primary care improve access to services? Cross-sectional survey of practice characteristics. Int J Health Serv 2005;35:465-478.
2. Henderson J, Britt H, Miller G. Extent and utilisation of computerisation in Australian general practice. Med J Aust;185(2):84-87.
3. Hippisley-cox J, Pringle M, Cater R, Wynn A, Hammersley V, Coupland C, et al. The electronic patient record in primary care – regression or progression? A cross sectional study. Br Med J 2003;326(7404):1439–1443.
4. Knottnerus J. Role of the electronic patient record in the development of general practice in the Netherlands. Methods Inf Med 1999;38:350–345.
5. Hansell A, Hollowell J, Nichols T, McNiece R, Strachen D. Use of the general practice research database (GPRD) for respiratory epidemiology: a comparison with the 4th Morbidity Survey in General Practice (MSGP4). Thorax 1999;54(5):413–419.
6. Nilsson G, Ahlfeldt H, Strender L. Computerisation, coding, data retrieval and related attitudes among Swedish general practitioners – a survey of necessary conditions for a database of diseases and health problems. Int J Med Inform 2002;65(2):135–143.
7. Fleming D. Morbidity registration and the fourth general practice morbidity survey in England and Wales. Scand J Prim Health Care 1993;Supplement 2:37-41.
8. Pearson N, O’Brien J, Thomas H, Ewings P, Gallier L, Bussey A. Collecting morbidity data in general practice: the Somerset morbidity project. Br Med J 1996;312:1517–1520.
9. Davis P. Office encounters in general practice in the Hamilton Health District I: social class patterns among employed males, 15-64. N Z Med J 1985;98:789–792.
10. Davis P. Office encounters in general practice in the Hamilton Health District II: ethnic group patterns among employed males, 15–64. N Z Med J 1986;99:265–268.
11. Davis P. Office encounters in general practice in the Hamilton Health District III: social class patterns among females, 15–64. N Z Med J 1986;99: 573–576.
12. Davis P. Office encounters in general practice in the Hamilton Health District V: social class and ethnic group patterns among children, 0–14. N Z Med J 1987;100:615–617.
13. Davis P. Office encounters in general practice in the Hamilton Health District IV: ethnic group patterns among females, 15–64. N Z Med J 1987;100:127–130.
14. McAvoy B, Davis P, Raymont A, Gribben B. The Waikato medical care (WaiMedCa) survey 1991–1992. N Z Med J 1994;107 Supplement Part 2:387–433.
15. Raymont A, Lay-Yee R, Davis P, Scott A. Family doctors: methodology and description of the activity of private GPs; The National Primary Medical Care Survey (NatMedCa): 2001/02. Report 1. Wellington: Ministry of Health; 2004.
16. Crampton P, Lay Lee R, Davis P. The National Primary Medical Care Survey (NatMedCa): 2001/02 Report 2: Primary Care in Community-governed Non-profits: The work of doctors and nurses. Wellington: Ministry of Health; 2004.
17. NAMCS. Accessed 19 March 2003. http://www.cdc.gov/nchs/about/major/ahcd/ahcd1.htm
18. Crampton P, Davis P, Lay-Yee R, Raymont A, Forrest C, Starfield B. Comparison of private for-profit with private community-governed not-for-profit primary care services in New Zealand. J Health Serv Res Policy 2004;9 Suppl 2:S2: 17–22.

Footnotes

b Note: rounding errors of up to 0.1 may apply