The Development of a Data Quality Framework and Strategy for the New Zealand Ministry of Health - Part 2

Click here for Part 2 of this paper.

• Methodology
• Internal and External Piloting
• Research Results
• Future Work

Methodology
The Canadian Institute for Health Information (CIHI), similar in function to the NZHIS, has developed a framework based on Statistics Canada guidelines and methods, information quality literature and the principle of Continuous Quality Improvement. Eppler and Wittig’s 2000 research^{[ 14 ]} on the Wang and Strong’s 1996 study^{[ 7 ]} noted above further informed the Canadian framework’s development.

Following an assessment of the literature on data quality frameworks, and assessment against Eppler and Wittig’s^{[ 14 ]} evaluation criteria, the CIHI Framework was found to be robust. The researcher met with the developers of the CIHI Framework to discuss the feasibility of adjusting the CIHI Framework to a New Zealand health environment. The researcher attended the 2003 Massachusetts Institute of Technology Information Quality Conference in Boston and the Massachusetts Institute of Technology Information Quality Course, (Information Quality Certification Programme, Course One) at the Sloan School of Management.

The CIHI Framework was assessed for completeness and relevance against current Ministry information technology and information management strategy. These include the Information Systems Strategic Plan (ISSP) and the WAVE Report (Working to Add Value through E-information). Compliance with New Zealand legislation was also considered.

Proposed additions and changes to the CIHI Framework that take into account the above policies, along with the assessment of the existing framework, were included for discussion at two focus groups with internal Ministry staff. Focus groups were used in an effort to bring together business units who appeared to have similar issues with data quality, but no formal infrastructure was in place to co ordinate quality initiatives. The "Ministry Data Quality Team" was formed to specifically look at ways of improving quality in a consistent way across the organisation. The terms of reference for the group state the objectives as being to:

• Educate and create awareness of the advantages of using quality data in decision-making
• Coordinate data quality improvement initiatives across the Ministry
• Assist in the development of a data quality framework for the Ministry of Health
• Assist in the development of an organisation-wide data quality strategy.

Membership of the group was selected from across the Ministry and its separate business units to ensure wide representation. The proposed framework was sent to all participants of the group. A presentation to the group was made prior to the focus groups meeting, to ensure all participants had a common understanding of the purpose of the framework and the outcome goals of the focus groups. The group participated in two focus groups of two hours each. A member of the strategic Information and Technology team (the researcher) led the focus groups and an administrator was present to make audio recordings and to later transcribe the recordings, noting also the interaction between group members on discussion points.

Internal and External Piloting
This research is still in progress with piloting of the framework and its user manual on all data collections held by the Ministry. One of the main considerations of the pilot study is the clarity and ambiguity of the language used in the framework and its manual. It is important the framework is used consistently across collections and misinterpreted meanings will affect this.

The draft framework is currently being piloted on national health collections within NZHIS on the Mortality Database. Data for this database is provided to NZHIS by the Department of Internal Affairs from Births, Deaths and Marriages, by Practitioners and Coroners, and from existing Ministry of Health data collections. This means that much of the collection process is not under the influence of NZHIS.

An assessment of the HealthPAC Capitation Based Funding (CBF) system is also taking place. The Capitation Based Funding System’s primary function is to allocate funding on a population basis according to funding formulae.

External assessment, on a health related data collection managed outside the Ministry of Health, is being undertaken on a collection held by the A+ Network Centre for Best Patient Outcomes. The aims of the Centre are to assist clinicians to improve patient outcomes through the development of a generic tool to help manage care delivery.

A template has been developed to assist data managers to assess the effectiveness of the framework, its user manual and the proposed "Data Quality Documentation Folder" for each collection and to document their findings. The template was used in the pilot assessment of the framework to assess its validity. Areas formally assessed through semi-structured interviews and a formal questionnaire include:

• The language used in the framework
• The language and examples provided in the user manual
• The length of time required to complete the assessment using the framework
• The value of the information provided from using the framework, as found by various users of the data
• The applicability of the dimensions, characteristics and criteria for the collection being assessed
• The contents of the data quality folder.

Research Results
Although changes were made to the content of the CIHI framework these in effect were minimal. The most significant change to the content was to add two further dimensions - Privacy and Security. The CIHI state that Privacy and Security are implicit requirements that are embedded in all their data management processes. Whilst this could also be said of the Ministry of Health, the pervading culture in New Zealand requires that Privacy and Security of information, and in particular of health information is paramount. Therefore, the Data Quality Team felt there was a requirement for explicit and transparent consideration of these quality dimensions. The characteristics for these dimensions were developed by the Senior Advisors in strategic roles in Health Sector Privacy and Security to ensure alignment with new Privacy and Security Policies.

An assessment of the framework on a yet-to-be-implemented collection, the Mental Health Workforce System, will be undertaken. The framework will be used as checklist for ensuring data quality in instilled in the collection processes prior to implementation and an assessment made of frameworks applicability to this type of use.

Findings from the two Ministry data collection assessments show that the Information Analyst group requires the most detailed information on how the assessment was made for each criterion, whereas management required summary information. Some further changes were made to language, ensuring better local "ownership" of the framework. The time taken to undertake assessment would be a minimum of four hours if all documentation about a collection were available. In reality, the assessments took far longer as the available documentation was held in disparate locations by different staff. Subsequent assessments of the same collections are likely to be completed much more efficiently, as much of the information could remain the same or merely need updating. Concern was expressed, however, around the time taken to complete the framework by already busy staff. Overall, the framework was found to provide useful data quality information by collection users and mangers and to provide sufficient information to make at least preliminary prioritised lists of essential data quality improvement projects. Further work has been required to ensure assessors use the framework consistently and that it is a practical and easy tool to use.

Particular attention to the language used in the accompanying User Manual is required as the CIHI wording was found to be too simplistic for the intended audience. Those using the framework are likely to be systems administrators, data quality advisors, and members of the business intelligence team but the language implied the need for little underlying understanding of data and systems. The manual can also be shortened with less background information on data quality (this may be produced separately to show the underlying theory used to develop the Data Quality Strategy for those taking part in education programmes). Therefore, extensive changes to the CIHI User Manual are required to make it useful to the New Zealand health environment.

Assessment of the framework using the hospital clinical data collection shows that a data quality framework is an invaluable tool that helps to guide developers to produce robust and valid clinical databases. Also, the majority of the Ministry of Health DQF criteria could be applied to external clinical databases, as shown in table 2 below. This table outlines 52 criteria, out of a possible 69 in the framework, that conform to the data quality requirements of the clinical database held at the hospital level.

Table 2. Applicability of the Ministry of Health DQF with a Hospital Clinical Data Collection

The framework assessment process also proved valuable to the hospital submitting the clinical data set. It was suggested by the Data Analyst that some formal, sector-wide criteria based on the framework, together with a certification process, would help to ensure that clinical databases are valid and reliable.

The findings and consequent recommendations of the assessment of the Data Quality Framework using Eppler and Wittig’s (1996)^{[ 14 ]} criteria are outlined below in table 3.

Table 3: Assessment of the New Zealand Data Quality Framework Using Meta Criteria Defined by Eppler and Wittig (1996)^{[ 14 ]}.

The summary information gained from assessments of all collections will be collated to form a prioritised list of data quality improvement initiatives across the Ministry. Ongoing assessment using the framework will provide information on the success of data quality improvement initiatives

Future Work
The development of the framework is an iterative approach. The pilot study will provide valuable practical information, as noted by the Canadians following the implementation of their first framework. Change management is required to ensure those working on data quality accept the ethos that prevention is better than rework and the improvement of data quality is everyone’s job across the organisation.

Using the Ministry Data Quality Team to assess the usefulness of the framework has provided an internal user perspective on the aspects of data quality that are important to all types of users. Further assessment of the framework by external data users, such as researchers at the Centre for Best Patient Outcomes, will be helpful in improving the assessment of the framework further.

A programme of work will be undertaken to improve the objective metrics used within the framework. Currently, many of the metrics associated with each criterion are subjective assessments, made by those who manage the collections. While this is a valid form of measurement, the robustness of the framework will be improved through the addition of relevant objective metrics. The metrics will be based on current literature on data quality metrics, trend analysis of historical data, current key performance indicators for data suppliers as outlined in their contracts with the Ministry of Health, and on legislative requirements.

The summary information gained from assessments of all collections will be collated to form a prioritised list of data quality improvement initiatives across the Ministry. Ongoing assessment using the framework will provide information on the success of initiatives.

The framework is a tool that will be used as part of a data quality strategy that covers the entire organisation. The data quality strategy development will be informed, in part, by the outcomes of the framework assessment of collections, highlighting areas of need. The strategy will follow the guidelines found in the Ministry of Health Information Systems Strategic Plan, which include tasking data quality to those at a strategic level in the organisation.

While the Ministry values data for their own purposes and for the sector, there is further potential for the use of this data. Building "trust" in the data throughout the health sector will ensure the data are used to the highest possible benefit. Extensive use of data for a variety of reasons results in improved data quality. Through extensive data mining, combining currently disparate collections will provide far more granular information, knowledge and wisdom on the state of our nation’s health.