Data Quality Maturity in the New Zealand Health Sector

• Abstract
• Introduction
• Methods
• Results
• Current State Analysis of New Zealand Health Care
• Capability Maturity of New Zealand Health Sector Organisations
• Maturity Comparisons
• Discussion
• Conclusion
• References
• Footnotes

Abstract
This article details the results of a “current state analysis” of data quality practices undertaken within the New Zealand health care sector. The practices were compared with the practices of companies based overseas and outside the health care sector in order to understand the relative data quality maturity within New Zealand health sector organisations. The impetus for undertaking this research was the increasing complexity of data management through improved information technology and telecommunications, allowing for the movement, integration and subsequent mining of data. Following a review of the literature, the researchers undertook a survey of data quality work in organisations both within and outside New Zealand. The survey tool was designed to provide a comparison with the New Zealand health care organisations’ data quality maturity or capability. The researchers then undertook an extensive review of current data quality practices within the New Zealand health sector using stakeholder interviews. The key findings to emerge from the research provide insight into the data quality maturity of New Zealand health care organisations. Interviews with key stakeholders in the New Zealand health sector provided further local information, spanning both primary and secondary care providers. The many disparate sources of data in health care lead to considerable complexity in the management of the data. Currently, there tends to be a reactive approach to data quality management, contributing to a lack of trust throughout the health sector in data, as users become aware of data quality problems for the first time when they try to use data. Health care organisations have similar data quality issues to other types of organisations. The research highlights that those organisations with an established data quality team are more mature in their management of data quality.

Introduction
Without adequate information it is difficult to determine the success or failure of health policies.^[1] The quality of policy decisions is partly reliant on the quality and nature of the data held in national health data collections. There is a significant trend towards evidence-based medicine in New Zealand health care, which forms an underlying philosophy that is applied in policy decisions. Evidence-based policy requires an investment in health information for the ongoing maintenance of existing data collections, the development of new collections to address the significant areas of incomplete data, and the development of the skills of information analysts and researchers to glean the best information from that data.^[1]

Organisations are beginning to realise the importance of data as an asset, the potential loss of profits through poor customer relationship management and the high cost of having to fix poor data once it is in the information systems.

A review of current data quality literature reveals that data quality is now emerging as an academic discipline in its own right, with specific research programmes underway within Universities. The most significant being that of the Sloan School of Management Information Quality Programme at the Massachusetts Institute of Technology (MIT)^[a] Klein and Rossin^[2] note there is no single definition of data quality accepted by researchers and academics working in the discipline. Data quality is viewed from a consumer perspective (consumers being people or groups who have experience in using organisational data to make business decisions) that quality data are “data that are fit for use”.^[3,4,5] Data quality is “contextual”; the user defines what is good data quality for each proposed use of the data, within its context of use.^[6,7] Therefore:

Data are of high quality if they are fit for their intended uses in operations, decision-making, and planning. Data are fit for use if they are free of defects and possess desired features.^[4]

Often the same data are used several times across the organisation for different purposes, using different presentations. Therefore, data quality needs to be a multidimensional concept^[2] as data themselves are multidimensional.^{[8, 9]}

The field encompasses the well-established Quality Discipline, drawing on the work of Deming,^[10] with the adaptation of Crosby’s “plan, do, check, act” cycle,^[11] through the notion that “quality is free” because of the cost of doing things wrongly. It also draws on the work of Juran^[9] via the utilisation of Six Sigma and Total Quality Management (TQM) adapted to Total Data Quality Management (TDQM), and the management of information as a product.^[12] This is achieved through the adaptation and application of TQM processes and the implementation of TDQM.

Some organisations have implemented TQM programmes with considerable success, ie, increased efficiency and profits. There has been less success in companies where there has been incomplete buy in to the TQM philosophy, particularly at management level.^[9] The health care sector is now beginning to implement quality management programmes to improve care processes in the light of poor safety records for patient care^[13] following recognition that more and better quality information is required to manage health care effectively.^[14] Research is now developing ways to combine TDQM into the strategic direction of an organisation, aligning the data quality requirements with the overall goals of the organisation. At present, there is little research published that develops the more theoretical principles into practical systems although some organisations do have data quality programmes with some strategic alignment to their business requirements.

The research thus far has provided the discipline with the theoretical underpinnings required to develop practical structured programmes to address data quality from a holistic perspective, whereby all aspects of data management are addressed an a high priority assigned to improvements that meet the needs of customers, as defined by those customers. The roles of customer, collector and custodian of data have been defined and research has noted the differing data quality needs and perspectives of those in each of these roles.

The aim of this review of data quality “maturity” within the New Zealand health sector was to elicit the potential and readiness for change, which would provide further guidance for the development of a national data quality improvement strategy that was both applicable and feasible to implement. The assessment of data quality maturity provides information to management that is a quantifiable score of how well data operations manage the quality of their information. Davis^[15] utilises a data quality management maturity grid and notes that other reasons to understand the data quality maturity capability in an organisation include:

• Identifying where on the data quality maturity continuum an organisation currently falls provides managers with a benchmark, and a direction in which to improve.
• Scores will reflect whether organisations are further ahead than or behind others.
• Managers will understand their organisation’s behaviour – good, bad, or indifferent – towards data quality.
• Managers will have a framework to begin changing an organisation’s attitudes towards management of its information.
• Managers will know what actions to take in order to further improve data quality as the maturity level increases, which will vary depending on level.
  

Methods
The researchers undertook a survey of data quality work in organisations both within and outside New Zealand. The survey tool was designed to provide a comparison with the New Zealand health care organisations’ data quality maturity. Fifteen different organisations provided responses comprising free-text answers to a one-page questionnaire. Responses came from organisations in New Zealand, Australia, the US, Germany and Brazil.

The survey elicited information on the data quality practices of the respondent organisations, particularly in relation to strategies and programmes in place. The survey questionnaire was tested for the applicability of responses by the 12 attendees of the MIT IQ1 foundation course held at Boston, Massachusetts, in November 2003. Most attendees were employed full time as data quality practitioners in large organisations and had several years’ experience in the field and extensive knowledge of the data quality programmes in place within their organisation.

The questionnaire was revised in light of this exercise and then used to survey the attendees at a New Zealand data quality course and also at an Australian Business Intelligence conference. All of these respondents were employed in data quality related roles, such as Data Quality Managers or Data Quality Analysts. Some worked in the area of business intelligence, where data quality management was one of their many areas of responsibility. There were some respondents who worked solely on data quality in very large organisations, where they had developed central requirements for their organisation’s branches internationally.

The 15 free-text responses from these surveys were analysed using grounded theory methodology, by coding content into categories and, eventually, themes. Content analysis produced a summary of the themes elicited from the responses.

The researchers then undertook an extensive review of current data quality practices within the New Zealand health sector using stakeholder interviews. The results of the current state analysis of data quality in the New Zealand health sector and the international organisations surveyed provided information that could be used to assess the data quality maturity of both types of organisation. These data were then used to compare the two types of organisation to ascertain the level of data quality maturity in the New Zealand health sector benchmarked against other organisations.

The researchers used a data quality management maturity grid to assess the maturity of organisations in managing data quality. This tool was originally developed by Crosby^[11] and modified by English^[16] for use in TDQM practices to provide a “gap analysis” for data quality practitioners planning improvement programmes. The tool provides a grid that can be used to assess the maturity of organisations in relation to:

• Management understanding and attitude
• Data quality organisational status
• Data quality problem handling
• Cost of data quality as percent of revenue
• Data quality improvement actions
• Summation of organisational data quality maturity.

Results in each of these sections are then rated from one to five:

1. Uncertainty (ad hoc)
2. Awakening (repeatable)
3. Enlightenment (defined)
4. Wisdom (managed)
5. Certainty (optimising).

Results

Current State Analysis of New Zealand Health Care
There is almost no formal accountability for data quality at management level. Some data collectors and departmental managers have defined responsibilities in their job descriptions, however this is rare. There are many roles within organisations that undertake data quality initiatives and this has led to some disparate silos of work. This issue is being addressed by the formation of organisation-wide data quality teams in at least three health care organisations. The data quality programmes are limited to minimal formal assessment of completeness and formatting of data by means of automated audits and looking at the raw data. Data quality requirements are defined informally and are inadequate in terms of meeting users’ needs.

The New Zealand health sector organisations studied are beginning to move towards preventative initiatives for data quality. Loshin^[17] finds a mature data quality programme determines where the risks are, the objective metrics for determining levels and impact of data quality compliance, and best approaches to ensure high levels of quality. Lack of accountability and role definition within New Zealand health care organisations appears to be the most significant hindrance to effective data quality management, along with the lack of a formal, regularly utilised assessment tool.

The level of support for managing data quality from management in the New Zealand health sector varied, and this was found not to have any correlation with the size of the organisation. Black et al,^[18] when studying data quality in clinical databases, found that levels of data quality were not affected by the size of the organisation. A study of accounting organisations in Australia^[19] has also found that the size of an organisation did not have any impact on the perceived importance of data quality. Whilst the organisations in this study vary considerably in size and data quality maturity, the level of awareness of data quality issues was reasonably consistent. However, the size of the organisation may impact on the successful implementation of a data quality strategy. Robson^[20] states that the size of an organisation affects the nature of its strategic problems and the resources available to deal with them. The need for a generally agreed strategy is more apparent in large organisations, because of the complexity and diversity of their actions. Smaller organisations in the health sector, such as smaller District Health Boards (DHBs) and Primary Health Organisations (PHOs), may not feel that a data quality strategy is required at their level; the national strategy may be considered sufficient to guide the management of data quality issues for the national health collections.

A 2005 study of information management trends in the US and Canada found that improving data accuracy and integrity is the most important issue in business intelligence systems for 75 percent of respondents.^[21] Managers in the New Zealand health care organisations in this study appeared to have less awareness of data quality than overseas organisations and it is unclear why. It is possible that the lack of formal assessments of data quality means that managers can chose to deny, or may not be aware of, data quality problems in their departments. Those organisations in this study that had organisation-wide data quality teams appear to be considerably more mature in their overall management of data quality, and this is discussed below.

No organisation in this study could identify a management role within that had overall accountability for data quality. It was often found to be the role of the CIO (Chief Information Officer) “by default”, without any performance expectations having been set out in job descriptions. The CIO often does have a very good overview of their organisation, but departments such as Decision Support are data users, and were also found to play a role in data quality management. Three organisations had data quality teams that met regularly to discuss data quality problems across the organisation. The teams were formed and led by the Information Systems department in response to significant and persistent data quality problems that could not be solved by any one department.

Preventative measures centred on education of data collectors, sometimes targeted after auditing of collection practices and data quality errors. Information systems were also designed to prevent the input of erroneous data where possible. There appeared to be considerable understanding of data quality issues and their impacts on decision making, with some organisations displaying a high level of understanding of methods required to improve data quality. This varied considerably from one organisation to the next. The competing priorities in health care mean that requests for funding for improvement projects or extra staff are not always granted. Whilst some organisations have management who were supportive of data quality improvement, data quality work was entirely initiated by lower level staff.

While the findings that have emerged from this research may appear disparate and somewhat unrelated, they do indicate that data quality work pervades many functions of an organisation and has an impact across it. The results highlight the need for a level of awareness of data quality that, thus far, has not been consistent within the New Zealand health sector. There is a need for a sector-wide management programme that provides clarity about the level of data quality that data consumers and managers could expect of data held in the national collections.

Capability Maturity of New Zealand Health Sector Organisations
The key findings to emerge from this research provide insights into the data quality maturity of New Zealand health care organisations. English^[16] has adapted Crosby’s^[11] quality management maturity grid for assessing the capability of an organisation to make changes in the management of data quality. The researchers applied this grid to assess the data maturity of the New Zealand health sector as a whole, using the information provided from the Current State Analysis. The variation of capability between health care organisations is considerable; therefore, the researcher made the assessment on two representative health care organisations – one with little apparent maturity and one with considerable apparent maturity in data quality management – to provide a range of capability maturity for the sector. The measurement categories and stage of maturity of the organisations studied are defined in Table 1.

The “immature” organisation was generally found to be at stage one, called “uncertainty” by English^[16] and “ad hoc” by Crosby.^[11] This is the least mature stage found in the grid, with the organisation at this level being highly reactive to problems as they arise. Little is done to solve long term and/or persistent problems.

The more “mature” organisations were found to be variable in the grid rating scale. In particular, no organisation had formally measured the impact of data quality on the organisation and, therefore, even the most mature organisations are at stage one on the grid. Stage two is the “awakening” or “repeatable” stage, where, due to some kind of incident, the organisation has become aware that there are data quality problems. Data quality roles are in place and teams are set up to deal with problems.

All organisations, including so-called immature organisations, are, at a minimum, fixing obvious data quality problems.

The most mature organisations are at stage three on the grid. Stage three is “enlightenment” or “defined”, where attitudes to data quality are noticeably different, with supportive and educated managers within the organisation providing resources for improvements. Data quality management programmes are in place. Few organisations in this study were found to be at stage three for any category of the capability maturity grid.

Stage four is “wisdom” or “managed”, where the organisation sees significant benefits from its data quality initiatives and is continuing to implement and mature its data quality improvement processes. Stage five is “certainty” or “optimising”, where there is increased customer satisfaction and virtually complete data defect prevention. All errors are analysed for cause and preventative measures taken. None of the organisations surveyed in this study had any stage four or five practices in place.

Maturity Comparisons
Organisations surveyed by the researchers from outside the health sector, some of which were outside New Zealand, were compared with New Zealand health care organisations, with the aim of understanding the context of data quality in New Zealand health care organisations; the results are shown in Table 2.

The researchers found that the surveyed organisations were mostly similar in data quality maturity. Most New Zealand health care organisations do not have defined roles for data quality, however, and many of the international organisations did define at least one role for data quality. It is important to note the sample completing the overseas questionnaire may not be a representative sample. Many were attending a data quality course and may have more data quality maturity than typical organisations.

The most mature of the international organisations studied had implemented Six Sigma methodologies several years ago and their data quality team used this methodology where applicable. This provided for more maturity than the New Zealand organisations had possessed, particularly around the measurement of the cost of data quality. Surprisingly however, the data quality work done in this more mature organisation was still somewhat limited due to an emphasis on improvement and data cleansing rather than the prevention of errors through process management. The management of data quality problems was, in general, handled in a more structured way by the international organisations, with clearer reporting lines to departments responsible for their own data quality. This may have been due to the length of time that data quality initiatives had been in place. In New Zealand, health care organisations are relatively new organisations having been set up in 2001, following health sector restructure, and are still developing institutional knowledge.

Discussion
The grounded analysis and coding of the data collected from interviews with health sector stakeholders enabled evolved data collection, with changes made to the data collection tools as the researcher began to understand more about the data quality capability maturity of New Zealand health care organisations. The questions moved away from a focus on strategic data quality management to examination of current operational data quality work, management attitudes to and the roles actively employed in managing data quality.

At present, data quality management in the New Zealand health sector is entirely initiated through “bottom up” work, in general through the information services or IT teams. Management has not yet taken responsibility for, and indeed does not understand their role in, the process of ensuring high data quality. Data quality work was initiated when staff became frustrated with their inability to use the data for its intended purpose and there was no one else in the organisation responsible for overall data quality management. This bottom up approach has introduced silos of data quality improvement work, introducing more data quality issues through the increasing discrepancy of data from the same source, and producing conflicting reports. This was evident in all types of organisations in the health sector.

An organisation-wide data quality team provides for a higher level of capability maturity in data quality across the whole of an organisation. Data quality is perceived as a “whole of system” problem requiring “whole of system” solutions, with members finding that data quality issues were similar across the organisation. Teams were able, in organisations where these existed, to engage management through education and data quality reports, bringing increased management support for data quality work. Membership of the team assists in ensuring each department is aware of its role, with the CIO playing a pivotal role in communicating with executive management. Roles and responsibilities became more clearly defined as the team members from diverse departments understood the impact of poor data quality in each department and the need for each department to have a role in the management of data quality. Data quality team members were often involved in training other staff as they had considerable knowledge of persistent data quality issues. Funding and planning staff were also members of data quality teams, bringing in expertise about their data needs, which differed from those of clinicians. Their presence reflected the importance of accurate data for reporting and claiming to obtain funding for services. Where possible, data collectors, custodians and users should be members of a data quality team as each has a different view of data quality.^[24]

The analysis of data quality maturity confirms that TDQM and the strategic management of data quality is a relatively new phenomenon, still rarely found in organisations either in New Zealand or overseas. New Zealand health care is certainly not “lagging behind” the international position, and one of the New Zealand Ministry of Health’s mission goals is to be world leading in its knowledge and leadership of health information management. Leading a strategic data quality programme utilising TDQM principles, for the whole of the New Zealand health sector, would constitute world-leading practice. The Ministry of Health currently has the capability to do this, with strong management support, a structured assessment tool with defined data quality dimensions, and the methodology to develop appropriate data quality metrics.

Conclusion
The impact on an organisation of an organisation-wide data quality team is considerable. This research emphasises that those organisations with an established data quality team were more mature in their management of data quality. Their management personnel were more informed and supportive of data quality work than in organisations without such a team, roles and responsibilities relating to data quality were clearer, and individual departments were beginning to understand their responsibilities towards data quality, which in turn brought about improved attitudes towards data quality throughout the organisation. Membership of a data quality team is important and should include those who use data as well as those who manage data.

The data quality capability maturity assessment undertaken in this research assessed both New Zealand health care organisations and international organisations using the data quality management maturity grid. This assessment indicated that health care has similar data quality issues to other organisations. However, the methods required to solve such problems in the health care sector may require a more multidimensional approach due to the complex nature of the systems that are found in health care organisations and health sectors. Atkinson et al,^[22] in investigating health informatics methods, note the “special case” for health informatics with unique demands. For example, governments provide the legislative and regulatory frameworks for health care as well as the delivery of care. The structure of the health care system drives the shape of IS application developments, not the need to enhance profit or market share, where a predominantly public health service exists, as it does in New Zealand. A dominating factor in health informatics is the particular social, professional and cultural context of health care.

Hamel and Prahalad^[23] discuss the need for external benchmarks to understand industry best practice and identify key capability challenges. Benchmarking helps to highlight specific areas where an organisation is not performing to best practice standards, and may provide evidence to management that more could and should be done. The comparison of New Zealand health care organisations with overseas organisations provides this benchmark.

References

1. van Gool K, Lancsar E, Viney R, Hall J, Haywood P. Diagnosis and prognosis of Australia’s health information for evidence-based policy. J Health Serv Res Policy 2002;7(Suppl 1):S1:40-S1:45.
2. Klein B, Rossin DF. Data errors in neural network and linear regression models: An experimental comparison. Data Qual 1999;5(1):25.
3. Wang RY, Strong DM, Guarascio LM. Beyond accuracy: what data quality means to data consumers. J Manage Inf Syst 1996;12(4):5–33.
4. Redman TC. Data quality: the field guide. Boston: Digital Press; 2001.
5. Loshin D. Enterprise knowledge management: the data quality approach. California: Academic Press; 2001.
6. Strong DM, Lee YW, Wang RY. Data quality in context. Communications of the ACM 1997;40(5):103–110.
7. Pringle M, Wilson T, Grol R. Measuring "goodness" in individuals and healthcare systems. Br Med J 2002;325:704–707.
8. Fox CJ, Levitin AV. Data and data quality. Encyclopedia of Library and Information Science 1996;57(Supplement 20):100–122.
9. Juran JM, Godfrey AB. Juran’s quality handbook. 5 ed. New York: McGraw-Hill; 1999.
10. Deming WE. Out of the crisis. Cambridge: Massachusetts Institute of Technology; 1982.
11. Crosby P. Quality is free. New York: Penguin Group; 1980.
12. Wang RY, Lee YW, Pipino LL, Strong DM: manage your information as a product. Sloan Manage Rev 1998(Summer):95-105.
13. Institute of Medicine. To err is human: building a safer health system. Washington: National Academy Press; 2000.
14. Pierce N. Health care Rx: information. The Seattle Times 2004 25 July.
15. Davis F. IQ Maturity, it will be smaller than your shoe size. In: Firstlogic; 2004.
16. English L. Improving data warehouse and business information wuality. New York: John Wiley & Sons; 1999.
17. Loshin D. Issues and opportunities in data quality management coordination. In: Data Manage Rev; 2004.
18. Black N, Barker M, Payne M. Cross sectional survey of multi-centre clinical database in the United Kingdom. Br Med J 2004;328:1478-1482.
19. Nord GD, Nord JH, Xu H. An investigation of the impact of organisation size on data quality issues. J Database Manage 2005;16(3):58–71.
20. Robson W. Strategic management and information systems. Second ed. London: Pitman Publishing; 1997.
21. TechRepublic. Information management trends in 2005. http://i.i.com.com/cnwk.1d/html/itp/InformationMgmtTrends2005_06051.pdf. Accessed 4 September 2006.
22. Atkinson C, Eldabi T, Paul RJ, Pouloudi A. Investigating integrated socio-technical approaches to health informatics. In: 34th Hawaii International Conference on System Sciences; 2001: IEEE; 2001.
23. Hamel G, Prahalad CK. Competing for the future: breakthrough strategies for seizing control of your industry and creating the markets of tomorrow. Boston: Harvard Business School Press; 1994.
24. Lee YW, Strong DM. Knowing why about data processes and data quality. J Manage Inf Syst 2003; 20(3): 13-39.

Footnotes

a http://mitiq.mit.edu