Evaluation Informed Management

• Abstract
• Introduction
• What is "Evaluation-informed Management"?
• How to Practice Evaluation-informed Management
• Conclusion – Developing Evaluation-informed Management
• References
• Appendix: Definitions of Terms

Abstract

Managers and policy-makers can no longer afford not to make greater use of evaluation in making and implementing decisions. Yet doing so is not so easy: the methods are different from those used in evidence-based medicine; evaluation methods and data-bases are less well developed; and the decision-making processes are not like those in clinical practice. This paper defines evaluation-informed management and explains how managers can use evaluation to make more effective decisions and projects. It gives a framework for assessing evidence from management research and proposes that different hierarchies of research design are developed for grading research studies of different management changes and methods. The paper notes the challenges which an evaluation-informed management approach presents to researchers, managers and different institutions.

Key words: Evaluation, research, management, policy, evidence-based medicine.

Introduction

In 1994 Sweden introduced a scheme to allow patients to choose their general practitioner, rather than being given one according to where they lived. General practitioners could choose whether to become independent contractors or continue as salaried employees. By 1996 about half of the 26 Swedish counties had partially implemented the scheme before the reforms were halted by the government. It is generally agreed that the scheme was badly thought through, poorly implemented and has further reduced the low credibility of managers and politicians. Could things have been otherwise? Would time and money have been saved if politicians had used evaluation in their debate before the change? Did they consider research elsewhere or trying a pilot experiment? Would implementation have been more effective if managers had used evaluation to help decide how best to manage the change?

There are those that would argue that things are different in the United Kingdom. But in most countries using evaluation to inform management and policy decisions is the exception rather than the norm. Cynics argue that this will not change. It is true that management and policy research is not always of sufficiently high quality to give a better foundation than educated guesswork or the support of powerful groups. The best research in this field rarely gives evidence of the same degree of certainty as medical research. The political process and implementation has its own logic and will always be influenced by power and values. Yet the information age and a more educated public will put pressures on managers and politicians to become evaluation literate and make use of research, not least because sectional interests will be using the evidence to support their positions.

In an era when health is increasingly politicised, this paper does not propose a rationalist solution, neither does it underestimate the problems of developing management practice. Rather it proposes that policies and management decisions could and should be more informed by evaluation research and concepts. Clinicians are being urged to practice evidence-based management, and more clinicians are contributing to or taking positions in management. Management actions and policies will be increasingly challenged for evidence of effectiveness, and for evidence of whether the results of changes are worth the costs. However it is not so easy to develop “evidence-based management” and, this paper argues, management decisions should not be based on the model of evidence-based medicine.

The paper first defines evaluation-informed management and explains how managers can use evaluation to make more effective decisions and projects. It proposes that there are limitations to evidence-based medicine methods when applied to management decisions. It considers the personal and institutional capabilities which need to be developed to make a greater use of evaluation in management. The paper does not consider in detail the use of research in making health policy decisions as this is a broader subject involving different decision-making processes and types of research. However, it shows how some principles discussed apply to policy decisions, and also how managers can be more effective in implementing policies by using evaluation research and concepts.

What is "Evaluation-informed Management"?

Evaluation-informed management can be defined as making more informed management decisions by using research evidence and evidence from inside the organisation, and making more effective actions and projects by using evaluation concepts to plan management interventions.

There are three elements to this approach to management practice.

The first is managers or their advisers looking for research carried out elsewhere into a change which is similar to one which they are considering. New policies usually leave open many choices about local implementation. If a manager must introduce a quality system, for example, their choice of which system is suitable and how best to implement it can be informed by research into what other people have done elsewhere. Another example would be whether to adopt a new management method or structure, such as a project management system, or decentralised budgets. In addition to using research to help implement an already decided policy or change, managers can also use research to learn about new methods and innovations which might be adopted locally. This “scanning for innovation” and “management technology transfer assessment” can help managers not only to discover new methods and service models but to decide whether they would be effective locally. Some managers already take this research-searching approach – the paper later shows how managers can become better and quicker at searching, assessing and applying the research.

The second element of this approach is a manager carrying out a simple small-scale local evaluation of a change which they are introducing. This gives the manager more objective and systematic evidence to that which they would gain from their usual methods of reports, personal visits and observation. It is evidence which can complement or correct the “feel” which managers get from using monitoring methods. Local evaluations are also useful before introducing a change, for example to assess the capabilities of the organisation, to consult widely, to identify specific problems, and to lay the basis for change. There are a number of feasibility assessment techniques for evaluating proposed changes.  ¹  

Evaluation can help to decide whether to introduce a change as well as how to introduce it. Examples of areas in which evaluation-informed management can be useful include:

• management methods or technologies
• performance appraisal and personnel reward systems, project management systems, budgeting methods, staff recruitment and selection
• organisational changes: decentralising budgets, care programme structures, team or network professional structures, single or joint departmental management, parallel or integrated quality structures, dedicated specialist units
• policies: care management, discharge management systems, patients’ rights or guarantees, patient participation, record systems and recording policies, patient information.

The third element is to use evaluation concepts as a discipline for thinking about or to focus a management decision or project. Managers can improve their decision about whether to make a change or about how to manage a project by asking:

1. What is the intervention? What exactly is the change or action to be taken or being considered?
2. Who or what will be changed? Who or what is the target of the change?
3. What is the intended outcome? What difference do you intend to make, which changes are intended in the target?
4. What results do we not want? What are the unintended outcomes to watch for?
5. How would we know if we have made a difference? Which data or measures would show whether the outcomes had been achieved?
6. How could we know that something else did not cause the outcomes?
7. Why are we making or considering the change/intervention? What values underlie this action?

Asking these questions and using the concepts of ‘intervention’, ‘target’, ‘outcome’, ‘confounders’ (other explanations for any results), and ‘value criteria’ improves a management action, regardless of whether the action is then evaluated or not.  ¹  

This approach to management practice is not so different from what some managers do informally: asking colleagues about their experience with or knowledge of management changes or methods, looking up management journals on the subject, ensuring that implementation is monitored or that actions or projects are well defined and structured. The proposed approach is, however, more systematic and disciplined, and congruent with quality management practice.

Will the extra time and effort result in more cost-effective decisions and programme management? In some cases it will not, in part because of the poor quality of management databases and of some of the research; searching often does not reveal helpful evidence. But for high expenditure or high risk decisions and projects this approach will be necessary. More importantly, learning the skills, concepts and philosophy of evaluation-informed management is likely to improve a manager’s practice and decision-making in other situations when there is not time to gather evidence.

How to Practice Evaluation-informed Management

There are four steps to making greater use of evaluation research in order to help make or implement a management decision: defining the question; searching for research; assessing the research; and deciding the local implications. Managers can delegate some of this work, but they also need the knowledge and skills to be able to do it themselves.

Spending time defining the right question to ask, and making it specific and answerable, is a necessary first step to guiding any search for the relevant research. It also focusses the attention on what exactly the issue is: being able to take a general inquiry or area of interest and to define an answerable question is an important general skill in management. “How do we improve cooperation between primary care and the hospital?” is a general “marker question”: it marks out an area of interest within which work then needs to be done in order to decide the specific questions which will guide the research investigation.

To make the question more specific we can ask:

• which are the most important aspects to improve (for example, of co-operation)?
• which aspects might have been studied elsewhere?
• which aspects might be improved by knowing how successful different actions have been when carried out elsewhere?

A question, such as “what is the most cost-effective way to ensure that general practitioners and nurses are warned about a forthcoming discharge and get details as soon as possible?” can be answered meaningfully.

Sometimes managers need to do no more than work on defining the question – having done so they find that the answer lies at hand or is obvious. A specific question is also necessary to focus the search otherwise the search will discover too many possibly relevant studies which cannot all be collected: without a sufficiently specific question, time will get wasted searching only in order to then have to specify the question. It is always possible to subsequently broaden a question to bring in other possibly relevant research. The skill to be able to take an issue and define the right question is a greatly underestimated one – it may well be true that 50% of the work in finding the right solution is defining the question, and poor question definition can waste much time.

The search for relevant research studies is probably the most time-consuming of the four steps, and one which managers might delegate to assistants or specialists. Unlike the research databases for clinical disciplines, management databases can be poor, difficult to use and costly to access, and there are few good electronic management databases (“HELMIS” at Leeds University, UK, is one exception). Another difficulty is that the design of a study is often not well described in abstracts and there are a great variety of research designs within this field. There are fewer reviews of research and the searcher usually has to go direct to the primary research study and often needs a copy of the full research paper to decide whether it is relevant or not. Time can be saved by asking experts for their advice about whether there are studies relevant to the question, and by following up references listed in the most recent research.

The practical problems of discovering relevant research are equalled by the technical problems in assessing the quality of the research and judging the certainty of the findings. This raises the questions as to what constitutes “evidence” in management research and whether ordinary managers can assess the validity of management research. To do so properly requires an understanding not just of experimental evaluation techniques, but also of social and policy research methods and of qualitative data gathering methods and analysis. For some types of research, managers will need expert researcher advice. However, experienced training managers and others have shown that that they can make a reasonable assessment of most types of study if they use frameworks to guide their assessment. Two examples follow.

The first is a general purpose assessment framework. The approach for assessing an evaluation is to score each of the following (for example on scale 1-5):

1. Description: Is the intervention well described? (Described over time – any changes in the intervention described? The elements of the intervention? Intervention separated from other change/things? Replicable? ).
2. Data: Are the measures or other types of data true or an artefact of the researcher or method? (Valid? Reliable? When? How long for? How often?)
3. Design: Could anything else explain the findings? (Are all confounders considered? Which are controlled for? Does the design exclude other explanations (Were these predicted at the start of the study?). Does the analysis consider and account for other explanations?).

The second approach is to grade the design used in the research study for its potential to produce valid findings about the outcomes of the intervention in question. Those designs which are graded lower in the hierarchy can be excluded and conclusions from those graded as producing acceptable evidence can be drawn by looking for any patterns in the findings. The type of grading system to use depends on the type of intervention which is the subject of the research. A single “hierarchy of evidence” which applies to all management and policy research is not appropriate. The hierarchies used in evidence-based medicine are based on experimentalist criteria, which are used to judge medical research into treatments, where outcomes are studied in a particular way and the assessment is made in terms of how confounding influences are controlled for.   ²  For some types of management research, experimentalist hierarchies can be used. However, in much management research controls of the same type are not practical or ethical, and an experimentalist approach with a narrow focus on one or a few measurable outcomes is not appropriate. Often we are interested in different stakeholders’ perceptions, not just of outcome but also of how the management change was implemented over time, and in the case of services, patients’ experiences during their contact. Controlling for confounding variables prospectively or retrospectively by statistical analysis is not the only way to judge whether other factors apart from the intervention have an influence.

The approach proposed here is that hierarchies of research design are useful for assessing management research but that we will need different hierarchies of design to grade studies of different types of management interventions. Researchers assessing health promotion programmes have put forward alternatives to the evidence-based medicine hierarchies,  ³   and a hierarchy of designs has been proposed for managers to use in assessing quality programmes.  ⁴   The debate has begun within the Cochrane Collaboration community, initially in a consideration of research into practice improvement techniques. Meanwhile the research assessment framework such as that outlined above is a useful general method.

Conclusion – Developing Evaluation-informed Management

Is it “whistling in the wind” to propose that health managers use evaluation in making decisions and managing programmes? There are many challenges to developing evaluation-informed management. Managers are sceptical about whether the time and effort will be worth the knowledge gained: is a research-based approach really more useful than the informal methods which many managers use? Health managers are reactive and action-oriented by temperament or because of the nature of the work. Few have the time or the inclination to define the question, look up the research and assess it, or even to delegate this work. This paper has noted some of the problems in accessing and assessing research, the poor quality or lack of usefulness of some management research and also the general lack of it. Then there are the problems in deciding the local implications, often from unambiguous and uncertain findings.

These challenges, together with the “politicisation of health” suggest that a more rational and considered approach to management may be difficult to develop. However, the paper emphasised an evaluation-informed management not an evidence-based management, recognising these and other challenges as well as the importance of values in making health management decisions. It also noted that it may be necessary to be selective, and concentrate only on some decisions, issues and projects which can most benefit from an evaluation-informed approach, and this may be those where there is a good research base.

Developing such an evaluation-informed management approach will need more and better management research and a better understanding of different research designs and their use for answering different questions. It will also need training for managers and their assistants, as well as research into the effectiveness of evaluation-informed management itself. Learning how to assess the quality of evidence in management research is an effective way to learn how to gather valid evidence from within an organisation about the progress of a management intervention. The paper proposed that such training and the use by managers of evaluation concepts will improve the practice of health management. The skills to define a question and to specify a proposed intervention and what it intends to achieve are skills which can make management more professional and effective. What are the alternatives and the better ways for managers to be spending their time? In the future can we afford not to make use of evaluation methods and research in management and policy-making?

References

1. Ovretveit J. Evaluating health interventions. Milton Keynes: Open University Press; 1998.
2. Fowkes F, Fulton P. Critical appraisal of published research: introductory guidelines. BMJ 1991;302:1136–40.
3. Macdonald G, Veen C, Tones K. Evidence for success in health promotion: suggestions for improvement. Health Educ Res 1996;11(3):367–76.
4. Ovretveit J. Assessing evaluations of hospital quality programmes. Evaluation 1997;3(4):451–68.
5. Rosenberg W, Donald A. Evidence-based medicine: an approach to clinical problem-solving. BMJ 1995;310:1122–26
6. Ovretveit J. Learning from quality improvement in Europe and beyond. J Jt Comm J Accred Healthc Orgs 1997;23(1):7–22.

Appendix: Definitions of Terms

Evidence Based Medicine (EBM)
“the process of systematically finding, appraising and using contemporaneous research findings as the basis for clinical decisions.”  ⁵  

Evaluation-Informed Management (EIM)
Making more informed management decisions by using research evidence and evidence from inside the organisation, and making more effective actions and projects by using evaluation concepts to plan management interventions. Using evaluation research and concepts to make more cost effective projects, policies and management decisions.

Evidence-based Quality Programme (EBQP)
A quality programme planned using research findings about the effectiveness of different quality methods in comparable organisations, as well as evidence from within an organisation about the progress and effects of its own quality activities. It is also one which promotes and supports the practice of evidence-based health care by clinicians by: using or developing evidence-based guidelines; promoting evidence based clinical audit; providing training and information support; showing that the same principles and methods apply in quality improvement as in EBM.

Management Technology
A tested method or system for making or implementing a management decision.

Quality Transfer Technology (QTT)
“A systematic method for seeking out and adapting others solutions to quality problems which we face, and for sensitising us to issues and problems which we might face.”  ⁶  

Transferability Assessment
An assessment of the resources and conditions required to successfully apply in one organisation or area a management method, system or a policy which has been implemented in another organisation or area, including a prediction of the likely consequences.