The case against performance and results measurement in public management is pretty well-known by now and has been made by many critics, including in previous issues of Optimum itself. (See for example: Carroll, 2000; Fountain, 2001; van Thiel and Leeuw, 2002). Christopher Pollitt, one of the wisest and most elegant scholars of comparative public management, has summarized the three main lines of attack against measurement under the categories of: (1) conceptual, (2) motivational and (3) technical. (Pollitt, 2000) 

The “conceptual” problems of measurement are all related to the difficulty of agreeing on what should count as “good.” Goods in public management are almost as contestable as goods in public policy, which accounts, in part, for why performance indicators can never be complete, or completely “objective,” or perhaps even stable over long periods of time. They are debatable, and so they change regularly – much too regularly – according to fashion and power. As a result, comparisons – even over time, let alone between organizations – can become very difficult, sometimes even meaningless.

The “motivational” problems of measurement have to do with “gaming” and other motivationally-based distortions that can undermine measurement systems, and regularly do. The introduction of any measurement system simultaneously introduces an element of “politics” – either “real” politics (in the public and democratic sphere) or bureaucratic politics. Depending on measurement outcomes, people are liable to lose or gain reputation, prestige, resources, performance pay, career prospects, or power itself. So the “politics of performance” lead to distortions such as: the suppression of measures exposing persistent differences in performance; relabeling or redirecting of activities or outputs to conform to measurement categories; over-emphasis on things that are easily measured and the neglect of other equally or more important activities where measurement is more difficult or even unattractive, and so on.

Measurement also encounters persistent “technical” problems including the difficulty of establishing plausible causal linkage between inputs, outputs and desired or actual outcomes; a tendency to over-emphasize process, output, and economy and efficiency measures instead of measures of outcomes, quality, or user satisfaction; the difficulty of establishing adequate times series of indicators; and a tendency for measurement systems to drive toward an increasing comprehensiveness which is ultimately self-defeating and brings abut the collapse, curtailment or abandonment of the measurement system. Measures tend to “wear out,” and this contributes to overall instability.

Having surveyed the criticisms of results measurement, Pollitt’s “‘message’ however is not that the measurement approach should be abandoned.” On the contrary, he suggests it can be helpful “to examine the limits and common problems of measurement as a way of knowing about the character of public services, but not to deny either its fundamental usefulness or the scope for its further development. It should be readily acknowledged that it is usually much harder – if not impossible – to form a reliable judgment as to the quality of public services without measurement” (Pollitt, 2000: 122, 140).

Some wag has said that the role of academic research is to prove that what works in practice cannot work in theory! That used to be a much better joke before I rejoined the ranks of academe. But as far as results measurement and performance management are concerned, it has at least a grain of truth. Despite all of the theoretical hand-wringing, there are some concrete examples where performance measurement has yielded interesting and promising results. Perhaps that’s one of the reasons two other leading scholars of comparative public management, Geert Bouckaert and John Halligan, concluded in a recent study that: “long-term trends now appear to support the ascendancy of performance ideas as a dominant force in public management” (Bouckaert and Halligan, 2008: 1; Halligan, 2008: 3).

Rather than add to the already ample theoretical literature on measurement in public management, I propose in this paper to review four specific cases where results measurement has been used in fruitful ways and which illustrate both its current role and future potential in public management. However I should mention right away – in the interest of full disclosure! – that I had a role in developing three of them. So I am not entirely unbiased about them.

Public sector service delivery

The first example is public sector service delivery in Canada. Twenty or even fifteen years ago, the improvement of service delivery was all over the map. Some public sector managers were pursuing something called service quality, some believed in service standards, and some in other things. But whatever the virtues of these various approaches, they were all means to an end. And no one could tell whether they were actually achieving that end, whatever it might be.

By the mid-1990s, there was a widespread feeling that all of this worthwhile public sector effort lacked rigour and focus. It was not clear that it was actually achieving results. There was no way to know whether all this well-meaning effort was directed to the right things, or was making a real difference. By 1997, it appeared that something else was needed if the Canadian public sector was to achieve a significant improvement in public sector service delivery.

As a result, a kind of intergovernmental “summit” of public sector leaders of service delivery and service policy was held in Ottawa, for two days in early July 1997, involving thirty-five senior service “champions” from across the Canadian public sector, together with knowledgeable academics. The participants from all three levels of government, and most regions of the country, concluded that the public sector in Canada and elsewhere had so far taken a largely “inside-out” approach to service improvement, with little input from citizens as to their service needs, or their priorities for improvement;

1. the public sector had no real idea how satisfied Canadians were with public sector service delivery; and
   
2. that citizen satisfaction should be used as the results measure for service delivery in the public sector – as the end or outcome for which all the other things were just the means or outputs.

I want to highlight five things that occurred over the following decade as a result (Marson and Heintzman, 2009). First, as I already mentioned, was early agreement on a common objective and results measure, an outside-in measure (not an inside-out or process measure, as so often occurs in the public sector), based on citizen satisfaction with public sector service delivery. 

Second, the undertaking of joint “action research” to achieve a deeper, shared understanding of the nature and dimensions of the challenge, as a basis for action, including a series of national surveys, especially Citizens First, which has now been conducted every two years since 1998. 

Third, the development of a common measurements tool (CMT) and related service improvement methodologies to facilitate results measurement, service improvement planning and implementation, and performance benchmarking within and between Canadian governments.

Fourth, the creation of communities of practice within governments and across governments, together with the institutions necessary to support those communities, including intergovernmental councils and an Institute for Citizen-Centred Service (ICCS) to serve, among other things as a research centre and data warehouse for results from the CMT. 

And, fifth, a shared understanding (also resulting from the “action research”) about the distinctive “drivers” of citizen satisfaction with public sector service delivery.

So what has all this achieved for public sector service delivery in Canada, over the past decade? 

Well, for one thing, it enabled the Government of Canada both to set and then to exceed a target of a 10 percent improvement in service results between 2000 and 2005, overtaking the provincial level in the process, and even closing the gap with the municipal level in service reputation. Ten years ago, a federal task force on service delivery proclaimed that public sector service delivery lagged woefully behind private sector service delivery performance, such as the banks (Canada, 1996: 12). Ten years later, the service performance of many Canadian public sector organizations now surpasses comparable private sector results and benchmarks, including the banks. In the process, Canada has become a world leader in public sector service delivery, the only country that has achieved, and can measure and demonstrate, continuous improvement in citizen satisfaction with public sector service delivery over a ten-year period. The international consulting firm, Accenture, has identified Canada as a world leader in public sector service delivery, and other countries are now looking to Canada as a best practice (Marson and Heintzman, 2009).

And what does this public management story show? I think it is a concrete example of how an “outside-in” approach based on the measurement of results and outcomes can contribute to significant improvements in public management (Heintzman, 2007).

People management and leadership in the public sector

My second example is people management and leadership in the public sector. I think it would not be entirely unfair to say that they are in about the same place service delivery was ten years ago. That is to say, all over the map. There is no agreement on priorities or approaches. Many things are being done, but most focus on means or outputs, not on ends or outcomes. So it isn’t possible to measure or compare results, much less to hold public service leaders accountable for them.

This inability has been deplored by many reports over the years, including by the Tait Report over ten years ago, by the sixth report of the Advisory Committee on Senior Level Retention and Compensation in 2003, and by the Working Group on the Disclosure of Wrongdoing in 2004, among others (Canada, 1996, 2000: 50; Canada, 2003: 18-19; Canada, 2004: 31-4).

But gradually something is being done about it. And that something bears a strong resemblance to what has already been accomplished in the area of public sector service delivery. In fact, as early as 2001, the Canadian public sector service delivery community proposed to the public service commissioners of Canada that the service and HR communities should collaborate on exploring the links between people management and service delivery; applying to people management the same results-based approach based on the “drivers” of performance already employed for service improvement; and achieving a strong service culture in the public services of Canada (Heintzman, 2001). At their annual conference in September 2004, Canada’s public service commissioners eventually agreed to develop a common approach to measuring employee engagement in annual employee surveys.

The first step, once again, was achieving agreement on a high level result or outcome, the end for which many of the other activities in people management and leadership are simply means. This end or outcome is often called employee engagement. Employee engagement is composed of at least two related sub-outcomes: job satisfaction and employee commitment to achieving the objectives of the organization (Schmidt, 2004; Heintzman and Marson, 2005: 559-561; Schmidt and Marson, 2007).

Having begun to agree on an outcome, Canadian governments have also begun to measure it, and also to explore the drivers of this outcome, especially in BC, Ontario, Manitoba, New Brunswick, and in some parts of the municipal sector such as the Region of Peel in Ontario (Employee Engagement Interjurisdictional Team, 2006). By analyzing the data from large employee surveys, it has been determined that fewer than ten factors drive the outcomes in public sector employee engagement. These include: management effectiveness; colleagues/work unit; understanding of and support for the organization’s vision, goals, and mandate; career progress and development; quality of supervision; autonomy: having the authority to make needed work-related decisions; and workload (Spears, 2006).

The most impressive work in this area is now being done in the BC government (Herrin, 2008a, 2008b; Adams and Matheson, 2008). The British Columbia public service now fields an annual employee survey, designed and conducted by BC Stats, one of the partners in the inter-jurisdictional employee engagement research. The BC surveys not only measure over-all employee engagement but have also succeeded in identifying the key “drivers” of engagement in the BC public service. Survey reports measure performance on these drivers, and also identify the organizational units requiring improvement, so that BC public service executives know where and on what to focus their improvement efforts each year. 

As a result, BC is now able to hold senior public service executives accountable for improvement, and has made employee engagement scores a factor in its annual performance management program and in calculating performance pay for deputy ministers and assistant deputy ministers. In the second round of measurement, employee engagement scores improved across the BC public service, suggesting that this methodology can indeed be used to improve results in areas of public management beyond service delivery.

But BC Stats has gone even further than this, and has been able to demonstrate a link between measurable results in this second area of people management and leadership, and measurable results in the first results area I discussed: service delivery. These are the first two links in what has been called the “Public Sector Service Value Chain” (Heintzman and Marson, 2005). BC Stats has explored how the engagement of employees contributes to citizen service satisfaction, by comparing employee engagement results and service satisfaction results across 41 work units in the BC public service and in the Region of Peel in Ontario, units that participated in both employee and service surveys between 2005 and 2007. On average, work units with high employee engagement scored 11 points higher in service satisfaction than work units with low engagement (69 versus 80 out of 100 points). In other words, employee engagement is an important driver of public sector service satisfaction. Overall, regression analysis found that engagement scores accounted for 20 percent of the difference in service satisfaction scores achieved by the work units. Thus approximately 20 percent of the variation or differences in service satisfaction scores can be explained by changes in employee engagement scores. In general, service satisfaction scores improved by 1 point when employee engagement increased by approximately 2 points. 

The BC Stats study begins to demonstrate the impact measurable results in one area of public sector management (employee engagement) can have on measurable results in another area (citizen satisfaction with public sector service delivery). It shows the strong linkage between the first two elements of the “Public Sector Service Value Chain” (Matheson and Watters, 2008).

Canadian governments have not yet agreed on a common measurements tool for employee engagement, as they have for service delivery, nor have they yet created a common data warehouse. But they have taken some small steps in the direction of benchmarking by agreeing on a set of common questions to be included in all public sector employee surveys. An employee engagement inter-jurisdictional team (EEIT) has developed a protocol, an employee engagement model, and common survey questions. The initiative has also begun to report on comparative intergovernmental employee engagement results (Employee Engagement Interjurisdictional Initiative, 2008a, 2008b).

Management accountability framework

A third example I will mention is the federal government’s Management Accountability Framework, or MAF, as it is commonly known. This may seem a surprising example of a success story, because MAF has as many detractors as admirers. But I think any balanced assessment should rate it as one of the genuine successes of modern public management.

MAF is a comprehensive framework for good organizational management in the public sector, comprising ten critical aspects of public sector management. Since its introduction in 2003, MAF has had two purposes: 

1. to serve as a guide for public sector leaders and managers, especially deputy ministers, to help them improve the management and performance of their own organizations; and 
2. to serve as an assessment tool employed by the centre of government to rate the organizational performance of departments and agencies. 

Unfortunately, the second role so far seems to have largely squeezed out the first. But this second role has also been remarkably – though still very far from perfectly – successful.

In the five years since its introduction, MAF has been used by the Treasury Board Secretariat as a framework to assess the organizational performance of departments and agencies in the Government of Canada on an annual basis. The assessments, which were rather crude in the first years, have since grown regularly more rigorous and consistent. As a result they can now be used to rank departments and agencies annually in four performance quartiles with an impressive degree of consensus, and have thus become an important input into the annual performance management process for deputy ministers. (Unfortunately there is not yet any published report on the MAF, but information about it and the departmental assessments for 2005 and 2006 can be found at: http://www.tbs-sct.gc.ca/maf-crg/index-eng.asp.)

If one compares the Canadian MAF to other international examples such as the UK’s capability reviews, for example, the Canadian model seems distinctly superior. The UK organizational performance framework is less complete, the assessment process is less thorough and rigorous, and the cycle of assessment is occasional rather than annual, so it cannot feed in, as the Canadian MAF process now does, to other annual cycles or processes, including performance management. (Barwise et al., 2007; United Kingdom, 2009) 

Considered on its own merits, however, it is clear that, for all its genuine achievements, MAF still has many weaknesses, or to use MAF’s own language, “opportunities for improvement.” In the original design for the MAF, there was a tension between two approaches to performance. One approach (emerging from the comptrollership and audit cultures) emphasized management process and infrastructure. The other approach (emerging from the service delivery culture) emphasized management results, such as the kind highlighted in the previous two sections of this paper. In the original 2003 version of the MAF, this tension was reflected in a balanced approach to performance measures, with a reasonable split between process and results measures. Upon implementation, however, the original management results measures were quickly dropped in favour of what now appear to be almost exclusively process and infrastructure measures. Reflecting this change, even the terminology of measures now seems to have been dropped in favour of the more harmless language of “criteria.” 

There is nothing wrong with process and infrastructure measures. They have their place. But they are not measures of actual management performance or results, only the potential or the conditions to achieve such results. So the MAF assessments seem now in reality to be largely “capability reviews,” to use the British language, even if they appear to offer a superior model for such reviews.

A second disadvantage of the current emphasis on process rather than management results is that the current assessments are largely (and necessarily) based on qualitative or value judgments by committees of Treasury Board Secretariat officials rather than on actual result measures. Again, there is nothing wrong with qualitative judgments. They definitely have their place in public management and in performance assessment. But they are inherently more contestable. They are easy to dismiss (if you have a mind to do so) on the grounds that they are based on faulty information or have been made by the wrong people (too junior, too inexperienced, etc.), or show faulty judgment or assumptions, or for some other reason. Much of the current criticism or dismissal of MAF assessments employs these kinds of arguments.

A third disadvantage of process and infrastructure measures (or “criteria”) is that they are almost infinitely extendable. How many are enough? Once you start down this road, it is hard to know where to stop. The original 2003 MAF model included some forty-six “measures,” an average of less than five for each of the ten elements of the MAF framework. The current MAF assessment approach employs literally hundreds of “criteria,” some of which would appear to be of relatively modest importance. To be fair, some of the original MAF “measures” would probably have required sub-measures. So achieving simplicity is by no means a simple matter!

Nevertheless, to build on its remarkable success so far, the future direction for the MAF is relatively clear: back to the future. The MAF needs to return to something closer to its original design. That is to say, it needs to revive the emphasis, where possible, on actual management results or outcomes (such as those highlighted in the first two sections), and reduce the emphasis on process and infrastructure, or “capability.” After all, if an organization can demonstrate high and increasing levels of engagement from its employees together with high and increasing satisfaction from its clients and stakeholders, these are pretty good measures of over-all management performance. At the very least, measures like these would allow central agencies to pay a lot less attention to the processes and infrastructure required to achieve them. This would in turn allow the MAF measurement system to return to something closer to the simplicity of the original model, significantly reducing both the analytic effort required by central agency staff and the reporting burden on departments. 

It would also provide something closer to a measure of management results actually achieved, rather than simply the “capability” or potential to achieve such results.

Ontario municipal benchmarking initiative

Some hints about the way forward for MAF can perhaps be gleaned from a fourth example of intelligent public sector performance measurement I will mention briefly. 

The Ontario Municipal Benchmarking Initiative (OMBI) has produced impressive performance comparisons across Ontario municipalities based on concrete performance measures for municipal activities and services. OMBI has developed standardized methodologies to collect consistent performance information to ensure information or measures of results are as comparable as possible between municipalities, including detailed technical definitions for each performance measure, costing methodologies and a web-based data warehouse to collect and share information. The initiative has also aimed to identify the key factors that can influence results for each measure, or what I referred to as the “drivers” of results (OMBI, 2008).

Building on this initiative, individual municipalities like Toronto have begun producing their own, more detailed reports on performance results and benchmarking. Among other things this takes the form of an impressive report card on specific municipal government performance areas, colour-coded to highlight varying levels of performance. The Toronto report card compares service levels (resources) and efficiency, customer service and community impact/outcomes results against both internal historical trends and against other Ontario municipalities. The colour coding uses four colours for external benchmarking (within a quartiles approach similar to MAF) and three colours to show performance improvement, stability or decline against internal historical benchmarks (Toronto, 2008). 

The Toronto performance report card is easy to read and understand, even for the uninitiated and provides an attractive model for other management performance assessments, such as MAF. Indeed a mock-up of a report card very similar to the Toronto model was actually prepared in the development process for the original MAF.

Lessons learned

So what lessons, if any, can be learned from these four examples?

Lesson 1:

First, that performance measurement can be done intelligently, and it can contribute significantly to management practice and discourse in public organizations. So the glass is half full at the moment. It’s not simply half empty, though it may be that too. Everything depends on the lens – critical or practical – you choose to employ. If anything, we probably suffer from a lack of intelligent measurement at the moment rather than an excess. 

Lesson 2:

A second potential lesson is that results measurement may well produce its best results at the level of management rather than policy, where goods are even more contestable, more value-driven. That, after all, is what politics are for – to contest contestable goods! In most areas of public policy, the causal chains are too long and too complex; the outcomes are too remote from actions taken to influence them: the links may be there, but they can’t easily be demonstrated. Governments just don’t have enough of the levers (nor should they) for us to expect to be able to attribute public policy outcomes directly to government actions. In their outstanding comparative study of public management reform, Christopher Pollitt and Geert Bouckaert could not find a single study that “convincingly links the actions taken with a set of positive and safely attributable final outcomes” (Pollitt and Bouckaert, 2004: 118).

The annual report of the President of the Treasury Board of Canada entitled Canada’s Performance: The Government of Canada’s Contribution is a case in point (Canada, 2009). It is, in many ways, an admirable document. But, because the links between the high level outcomes it reports and the specific actions of governments are so difficult to trace, it is not at all clear what you can actually do with this kind of performance report. Which perhaps explains why it is almost completely ignored.

But some aspects of public administration are much more within our control, especially public management itself: the management and performance of public organizations qua organizations. For these areas, what has already been achieved in public sector service delivery and people management can be an instructive model.

Lesson 3:

So a third possible lesson is that measurement based on outputs or outcomes will be more or less appropriate depending on the level or focus of measurement: that is to say, whether you are measuring results for public policies or for public management. 

The New Zealand public sector reform movement in early 1990s made a celebrated distinction between the production of outputs for which public managers could be held accountable and the more contestable outcomes for which only elected officials, it was then argued, can be held responsible. Because ministers decide on what outputs to produce to achieve their desired outcomes, and because they also decide on the level of resources (inputs) to be allocated for the production of outputs, they alone should be held accountable for outcomes (Boston et al., 1996). In the intervening years, this distinction has become blurred and even non-elected officials, in New Zealand and elsewhere, have gradually been held answerable for “managing to outcomes,” not just for outputs. But this trend runs up against the difficulty of establishing linkage between outputs and outcomes of public policies and programs. It also rums into the ineradicably political character of the debate about the likelihood of particular outputs achieving desired outcomes, or about how precisely outcomes and measures should even be identified in the first place (Gill, 2008). 

So, as far as public policy is concerned, expecting public managers to be answerable for outcomes might now appear unwise. But for public management itself – something over which public managers do exercise greater control – measurement of outputs and process is no longer enough. We will not have genuinely high standards of public management – and we will not reliably know that we have them – unless public managers qua managers work, as far as possible, towards outcome measures rather than output or process measures. Wherever possible (which is not everywhere), good public management should seek to measure the end rather than the means to the end – which may vary from organization to organization.

Lesson 4:

A fourth possible lesson is that, within public management itself, some uses of measurement will involve fewer distortions and less game-playing than others. For example, results measurement has at least two uses for public management: 

1. external control and accountability; and 
2. internal leadership and performance improvement. 

It seems very likely that the purposes of external control and accountability are going to produce more of the gaming and distortions the literature highlights than the internal purposes of learning, discussion and continuous improvement. If you’re sincerely trying to use performance information to improve the performance of your own organization, you’re not as likely to distort the data for external consumption.

At a recent symposium jointly sponsored by the federal Treasury Board Secretary, the Auditor General of Canada and the Canadian Comprehensive Auditing Foundation (November 28, 2008), Michael Wernick, the federal deputy minister of Indian and Northern Affairs, made an insightful distinction between the “fake” and the “authentic” in public management. The “fake” is the stuff you pretend to do and that you feed to the central agencies to get them off your back. The “authentic” is what you really do and how you really manage. Wernick suggested that the challenge for MAF, for example, is to make it authentic rather than fake: to “drive out” the fake, as he put it, in favour of the authentic. That is to say, the challenge is to return to the original conception of MAF as a genuine management framework for senior leaders, one they actually use in their own pursuit of improvement in organizational performance. Not just something with which thy comply for reporting purposes. 

As long as performance measurement is carried out to satisfy some external authority, managers will always be tempted by the fake and by “gaming.” Measurement will have the greatest chance of becoming meaningful and useful when it is employed by organizational leaders themselves, as the indispensable platform for organizational improvement and organizational learning. 

Lesson 5:

A fifth and perhaps surprising lesson is that measurement may offer its greatest promise in some of the traditional “soft” areas of organizational management and performance (like people management and leadership) rather than in the traditional “hard” areas of measurement, such as financial, resource management, or “efficiency” measures. There is perhaps more to be gained from making “soft” areas hard than making so-called “hard” areas even harder!

After all, even these so-called “harder areas” are bedevilled by hidden and contestable premises. “Efficiency” by whose criteria or values, for example? This may be at least one of the reasons the research literature on performance and results management does not yet seem to have turned up a lot of international evidence of results information being used effectively in resource allocation (Halligan, 2008: 6).

If measurement always involves hidden value judgments, it may actually offer its greatest promise where these are no longer – or are somewhat less – hidden. That is to say, where what is being measured is explicitly a question of perceptions and value judgments – where valuing is the whole point of the exercise, as in the measurement of service satisfaction. And that is exactly the nature of the so-called “softer” areas, such as leadership and people management. Success in these areas depends on the perceptions of those who are led, on the degree to which they value the role models they are offered and the workplaces they experience. 

One of the reasons public sector performance has been held back in these “soft” areas, as the reports cited earlier all complained, was the failure to take these perceptions seriously. Many value perceptions added together can become a “hard” measure of success. “While it is true that individuals’ views on (an aspect of public sector organizational performance) might be subjective, even prejudiced,” as Colin Talbot has recently observed, “this does not make data about such views subjective. Assuming that such views have been collected and analysed in a reasonably scientific way, the resulting data is objective – an objective analysis of opinions” (Talbot, 2008: 147). Without such measures we have no way of evaluating performance, assessing competence, or holding people accountable for progress and improvement in public sector management, especially leadership, people management and workplace wellbeing. The BC experience to date shows that the development and use of such measures within a rigorous framework can and has led to exactly those improvements.

Another example of a “soft” area where measurement can and should be used more effectively is public service values and ethics. This might seem like the quintessentially unmeasurable, “soft” dimension of public administration. Yet, perhaps for that very reason, it is actually one of the most promising frontiers for measurement and accountability. Because public service values and ethics are ultimately a question of public service behaviour – and because we have as many potential observers or “witnesses” of this behaviour as there are public servants – values and ethics are actually an area that lends itself well to the kind of measurement I’ve been talking about. Of course there’s another more traditional but still useful kind of measurement: the inventory of management infrastructure and practices, and this certainly needs to be carried out, even for values and ethics. But it will only take you so far, certainly not to behavioural outcomes. In order to measure these effectively, we will need to develop measures of organizational climate, behaviour and culture based on employee feedback, just as we have already done successfully in other areas of public management, such as service delivery and employee engagement (Heintzman, 2006). The U.S. Office of Government Ethics offers various examples of the first kind of measurement and the Ethics Resource Center offers suggestive examples of the second, based on employee feedback (Ethics Resource Center, 2008).

Lesson 6:

A sixth lesson is that, where possible, performance management should work with a limited number of very important measures, rather than a vast array of much less important measures. Of course this is easier to say than to do, because some important high-level measures may need other sub-measures to support or determine them. That’s one of the reasons measurement systems can get out of hand and collapse of their own weight. Nevertheless it remains an important objective to reduce the top-level performance measures to a manageable number so that deputy ministers and other senior leaders can really focus on them, communicate them throughout the organization, and link other key processes – such as planning, reporting, learning and performance appraisal – to them (Heintzman and Marson, 2005: 552).

Lesson 7:

A lesson closely related to the previous one is that, where possible, research should be undertaken to identify what I have called the “drivers” of the most important high-level measures. After all, if you do not have any idea of the “drivers” of the elements of organizational performance you wish to improve, you really have no solid basis for deciding what to do or where to start. The chances are your efforts will be entirely wasted, and will make little difference to the outcome. Without any knowledge of the relevant “drivers,” it may be irresponsible to set an improvement target, because you are simply setting yourself up for failure, or – even worse – for “success” that isn’t really success at all. It is important to identify not only the current level of performance in a give area, but also its relative importance in achieving the over-all performance objective. Unless you know the second, you will not really know where to focus your improvement efforts (Heintzman and Marson, 2005; Heintzman, 2007).

Lesson 8:

An eighth lesson, based on the four examples, is that it will be helpful to identify common measurements and common ways of measuring across governments and across the public sector. And it will be helpful to develop common but independent institutions to store, benchmark and validate the data. 

One of the strengths of the service delivery and OMBI models is their success in achieving both of these objectives. The BC example of measuring employee engagement shows how developing a common measuring system and identifying “drivers” across an entire government can raise the performance of an entire public service. The work of the employee engagement inter-jurisdictional team (EEIT) shows how it could be extended to establish intergovernmental benchmarking, using a common but independent platform like the ICCS or the OMBI.

One of the gravest problems – not just for performance measurement but for public management in general – is the absence of stability or constancy of purpose, one side-effect of which is frequent changes in measurement and measurement systems (not to mention management priorities and approaches). One of the remarkable successes of the MAF has been the federal Treasury Board Secretariat’s admirable constancy of purpose since 2003, within a framework of gradual and continuous improvement. This is relatively and regrettably rare in public management, and it will be interesting to see how long and how stable this approach to management performance can be maintained. 

One of the factors that can help to promote stability and continuous improvement is the identification of an external measurement authority, like the ICCS, OMBI or BC Stats. Establishing an external custodian for the measurement system and for benchmarking data appears to be a potentially important ingredient of success, as the service delivery and OMBI examples show. As fads come and go, or as personalities and political priorities change within organizations or governments, an external body can help to provide stability of purpose and approach, allowing jurisdictions to opt out for a time, but maintaining focus and purpose until they rejoin the improvement effort. Such neutral bodies can also help to maintain the stability and integrity of the measurement system itself (Pollitt, 2000; Marson and Heintzman, 2009).

Conclusion

Are there major problems (conceptual, motivational and technical) with performance measurement? Absolutely! Can these problems ever be fully overcome? Not a chance. Should we give up the effort? Of course not. Abandoning the effort to develop relevant performance and results measures for public management just because they present formidable and perhaps intractable problems would be a bit like human beings giving up the effort toward goodness or social progress just because we recognize humans are inherently flawed and imperfectible. 

Measurement is clearly not a panacea for public management. In fact, as suggested in the previous lesson, it should be used with some considerable care and even circumspection. But can public management dispense with results measurement? No, it can’t. Despite all its drawbacks and weaknesses, we could never get along without performance measurement entirely, and should never wish to do so. In a sense we’re damned if we do, and damned if we don’t. It’s sometimes difficult to live with, but from now on we can’t live without it. 

If anything, as I already suggested, public sector management suffers at the moment from a dearth of intelligent measurement, not an excess (Heintzman, 2007). The relative slowness and reluctance with which we have devised and embraced intelligent measures of management performance – and our preference for process and output measures at the expense of genuine result or outcome measures – have slowed our progress in key areas of organizational performance that are critical to our role in upholding the public good. The best public managers in future will be those who seek, continuously and creatively, for rigorous ways to measure their own outcomes, especially the outcomes of those things over which they can exercise sufficient influence: the leadership, management and performance of their own public organizations qua organizations.

Remember what Christopher Pollitt said: it’s necessary “to examine the limits and common problems of measurement as a way of knowing about the character of public services, but not to deny either its fundamental usefulness or the scope for its further development. It should be readily acknowledged that it is usually much harder – if not impossible – to form a reliable judgment as to the quality of public services without measurement.” 

So our only choice in public management is to do measurement as intelligently as possible, and to use it as intelligently as possible. Public management – like much else in life – can perhaps best be thought about not as an exercise in technocratic manipulation and control but rather as a conversation (Paquet, 1999: 238; 2004: 109; Paquet, 2005a: 149; 2005b: 3, 6). Intelligent measurement intelligently employed can make the conversation richer, better informed, more grounded, more constructive, and more promising.

Ralph Heintzman writes this article from the Graduate School of Public and International Affairs at the University of Ottawa.

---

References

Adams, Sarah and Angela Matheson. 2008. Making the Most of the Model: An Employee Engagement User Guide for the BC Public Service. Victoria, BC: BC Stats. July 31.

Barwise, Patrick, David MacLeod, Sue Richards, Howard Thomas, David Tranfield. 2007. Take-Off or Tail-Off? An evaluation of the Capability Reviews programme. Ascot, U.K.: Sunningdale Institute, National School of Government. November.

Boston, Jonathan, J. Martin, J. Pallot, P. Walsh. 1996. Public Management: The New Zealand Model. Auckland: Oxford University Press.

Bouckaert, Geert and John Halligan. 2008. Managing Performance: International Comparisons. London: Routledge.

Canada. 1996. Report of the Deputy Minister Task Force on Service Delivery. Ottawa: Privy Council Office. 

Canada. 1996, 2000. Task Force on Public Service Values and Ethics. A Strong Foundation. Ottawa: Canadian Centre for Management Development.

Canada. 2003. Advisory Committee on Senior Level Retention and Compensation, Sixth Report. Ottawa: Treasury Board of Canada Secretariat.

Canada. 2004. Public Service Human Resources Management Agency, Working Group on the Disclosure of Wrongdoing. Report. Ottawa: Public Works and Government Services Canada.

Canada. 2009. President of the Treasury Board of Canada. Canada’s Performance: The Government of Canada’s Contribution, 2006-07 Annual Report to Parliament. Treasury Board of Canada Secretariat. http://www.tbs-sct.gc.ca/reports-rapports/cp-rc/2007-2008/cp-rc-eng.pdf.

Carroll, Barbara Wake. 2000. “Some Obstacles to Measuring Results,” Optimum: the Journal of Public Sector Management, Vol. 30, No. 1, March: 41-47.

Employee Engagement Interjurisdictional Initiative. 2008a. Interim Report of Results. January.

Employee Engagement Interjurisdictional Initiative. 2008b. Usage and Data Sharing Protocol. May 2.  EEIT@gov.ab.ca.

Employee Engagement Interjurisdictional Team. 2006. “An interjurisdictional initiative to building stronger public services,” Canadian Government Executive. November. http://www.networkedgovernment.ca/cp.asp?pid=412.

Ethics Resource Center. 2008. National Government Ethics Survey 2007: An Inside View of Public Sector Ethics. Arlington: Ethic Resource Center.

Fountain, Jane E. 2001. “Paradoxes of Public Sector Customer Service,” Governance: An International Journal of Policy and Administration. Vol. 14., No. 1, January: 55-73.

Gill, Derek. 2008. “Managing for Performance in New Zealand: The Search for the “Holy Grail”?” in Holy Grail or Achievable Quest? International Perspectives on Public Sector Performance Management. n.p.: KPMG International: 29-40.

Halligan, John. 2008. “Performance Management in Government: International Perspectives on the Challenges,” in Holy Grail or Achievable Quest? International Perspectives on Public Sector Performance Management. n.p.: KPMG International: 3-10.

Heintzman, Ralph. 2001. The HR Dimension of Citizen-Centred Service Delivery. Presentation to the Canadian Public Service Commissioners’ Conference, Regina, Saskatchewan, September 17.

Heintzman, Ralph. 2006. “Public service values and ethics: From principles to results,” Canadian Government Executive. Vol.12, no. 1. February: 10-13. www.networkedgovernment.ca/PSValuesandEthicsHeintzman.

Heintzman, Ralph. 2007. “Toward a New Moral Contract: Reclaiming Trust in Public Service,” Optimum Online, Vol. 37, Issue 3. Fall: 33-48.

Heintzman, Ralph and Brian Marson. 2005. “People, service and trust: is there a public sector service value chain?” International Review of Administrative Sciences. Vol. 71, No. 4. December: 549-575.

Herrin, Lee. 2008a. Measuring the Service Value Chain. BC Stats. February 4.

Herrin, Lee 2008b. “Employee: Engagement: High ROI Strategies,” Canadian Government Executive. October. http://www.netgov.ca/cp.asp?pid=762.

Marson, Brian and Ralph Heintzman. 2009. From Research to Results: A Decade of Results-Based Service Improvement in Canada. Toronto: Institute of Public Administration of Canada.

Matheson, Angela and Matthew Watters, Public Sector Service Value Chain – Linking Employee Engagement & Customer Satisfaction: A Preliminary Analysis (Victoria, BC: BC Stats, July 2008).

OMBI. 2008. “Ontario Municipal CAOs’ Benchmarking Initiative,” 2007 Performance Benchmarking Report. December. http://www.peelregion.ca/finance/ombi/2007-appendix1.htm.

Schmidt, Faye.2004. Identifying the Drivers of Staff Satisfaction and Commitment in the Public Sector. Report prepared for the Public Service Human Resources Management Agency of Canada. Schmidt Carbol Consulting Group.

Schmidt, Faye and Brian Marson. 2007. “Employee Engagement: A foundation for organizational performance,” Canadian Government Executive. Vol. 13, No 1, February: 12-15. www.networkedgovernment.ca/cp.asp?pid=150.

Spears, George. 2006. Finding Common Ground: Drivers of Employee Satisfaction in the Public Service. Paper prepared for the Treasury Board of Canada Secretariat. March 2006. Erin Research Inc.

Paquet, Gilles. 1999. Governance Through Social Learning. Ottawa: University of Ottawa Press.

Paquet, Gilles. 2004. Pathologies de gouvernance : Essais de technologie sociale. Montreal : Éditions Liber.

Paquet, Gilles. 2005a. Gouvernance: Une invitation à la subversion. Montreal : Éditions Liber.

Paquet, Gilles. 2005b. The New Geo-Governance: A Baroque Approach. Ottawa: University of Ottawa Press.

Pollitt, Christopher. 2000. “How Do We Know How Good Public Services Are?” in B. Guy Peters and Donald J. Savoie, eds., Governance in the Twenty-First Century: Revitalizing the Public Service. Montreal & Kingston: Canadian Centre for Management Development and McGill-Queen’s University Press: 119-152.

Pollitt, Christopher and Geert Bouckaert. 2004. Public Management Reform: A Comparative Analysis. Second Edition. Oxford: Oxford University Press

Talbot, Colin. 2008. “Competing Public Values and Performance,” in Holy Grail or Achievable Quest? International Perspectives on Public Sector Performance Management. n.p.: KPMG International: 141-152.

Toronto. 2008. City of Toronto. 2006 Performance Measurement and Benchmarking Report. Toronto: City Manager’s Office, Executive Management Division. March. http://www.toronto.ca/city_performance/pdf/2006_toronto_measurement_benchmarking.pdf

United Kingdom. 2009. Report by the Comptroller and Auditor General. Assessment of the Capability Review programme. London: National Audit Office, 5 February.

van Thiel, Sandra and Frans L. Leeuw. 2002. “The Performance Paradox in the Public Sector,” Public Performance & Management Review, Vol. 25, No. 3, March: 267-281.