Designing Metrics for Government Agency Performance
By Paul Arveson
"I am the true vine, and my Father is the gardener. He cuts off every branch in me that bears no fruit, while every branch that does bear fruit he trims clean so that it will be even more fruitful." -- John 15:1
In a well-maintained garden, the gardener doesn't have much work to do at any particular time; the work is at a vigilant and consistent steady pace. In a well-managed company, the situation is similar; major management handling is unnecessary, crises are minimized, and the mission of the organization continues to be viable. However, in the current environment of rapid political, technological and economic change, few organizations can remain calm and steady. There are too many forces for change that threaten the viability of the organization -- even if it is an established governmental agency.
Decisions regarding budget priorities are the central decisions of any organization. They are analogous to the decisions made by a gardener. The gardener may either prune and fertilize a plant, cut it down and discard it, or leave it alone. The same approach applies to management decision-making. Exactly what kinds of decisions are usually meant? What things do senior managers decide? We find that the decisions often take the form of triage: they may either increase an activity's budget, reduce it, or maintain it at a consistent level.
In a previous essay, it was argued that metrics for performance evaluations of nonprofit or governmental agencies fall into three broad categories:
- Projected future needs
- Mission effectiveness
- Support work efficiency.
Below we will examine how triage can be applied to each of these categories, which will help us to determine the most relevant metrics needed to guide senior management decision-making.
Before we get into the specifics, a little needs to be said about the role of metrics (as opposed e.g. to reports, stories, judgements or guesswork) in the decision-making process. Consider these four questions:
1. What are you doing?
2. How well are you doing it?
3. How do you know how well you are doing it?
4. How can you demonstrate to others how well you are doing it?
The answer to the first question should be the agency's or department's mission statement, PLUS all the other activities done to support the mission. There are a lot of these, and many may be 'hidden', so they need to be identified.
Answering the other questions requires managers to define metrics, collect data (such as surveys) and analyze the data. The better the documentation, the more visible and convincing the decision-making process becomes.
Stage 1: Strategic Needs Triage
Strategic needs issues address the question: To what extent is the agency's mission needed, now and in the future? Or within an agency: To what extent are the departmental missions needed, now and in the future? These issues are resolved primarily by looking outside the agency to its customer's needs now and related needs anticipated in the future. Other considerations are the replacement cost and time it would take to reconstruct the capability if it were abandoned or outsourced and then required inside the government once again. Often demographics play a role in future projections on both these questions.
"Departmental" missions are here defined as more specific missions (or "sub-missions") that are contained within smaller departments or other units of an agency.
"Future" is the time horizon of the strategic plan, which should be clearly stated in it. For information technology capabilities, a 3-year time horizon is 'long-term', for urban infrastructure or Navy ships, a 10- to 20-year time horizon is 'long-term'.
At the strategic stage, the mission capabilities are defined rather broadly, because decisions about future needs cannot be too precise.
Strategic needs decisions aim to sort out the following three categories of mission capabilities:
- Mission capabilities that are going to be needed increasingly the future, that are not duplicated in the private sector, that have inherently governmental requirements, and that have a high replacement cost and long time to build. EXPAND these capabilities by aggressive hiring, training, construction and other such efforts.
- Mission capabilities that are currently viable and strong within the government, and that meet the other characteristics of #1. Viability (age of employees, size, skills etc.), customer satisfaction (sponsors, users, taxpayers), output quality, outcome success. SUSTAIN these capabilities at current funding levels.
- Mission capabilities that are currently weak, and that will probably have a declining need in the future. REDUCE these capabilities. (Preferably this may be done by employee retraining, attrition and consolidation as opposed to outright reductions in force (RIFs). Also, care should be taken to document and preserve essential knowledge and capital assets of these capabilities in case our assumptions about future needs were incorrect.)
Strategic Needs Metrics
Obviously it is difficult to predict what will be needed in the future. But a decision to remain the same is also a prediction that future needs will be what they are now. Is this the best option? There is no shortcut to this aspect of strategic planning; it requires a lot of reading and study and wise judgement on the part of senior managers.
Mission uniqueness plays a role in the strategic triage decision. This is analogous to assessments of competitive advantage in the private sector. In the current political and economic environment, an agency needs to be aware of any aspects of its work that overlap work in the private sector. If agencies wish to sustain their previous dominance of activities that have moved to the private sector, or expand business in such activities, then they will face political hurdles as well as the economic challenge of competition with profit-oriented companies. A careful determination of what activities are "inherently governmental", including security considerations, should be undertaken to define agency policy in this area. Such a planning effort goes beyond the scope of this paper.
Stage 2: Mission-specific Effectiveness Triage
In stage 2, decision-making focuses on those capabilities that will continue to be needed in the future, as determined in stage 1. At this stage, mission capabilities are broken down into more specific sub-missions so that their effectiveness can be evaluated individually. The meaning of "effectiveness" is defined differently for each sub-mission, and it takes experts in each unit to evaluate this effectiveness. These capabilities are sorted into the following three categories:
- Weak but unique capabilities that are currently struggling, e.g. with a poor prognosis due to demographics, but are unique within the government and are unlikely or undesirable to be supported by the private sector. ENHANCE these capabilities by focused training, hiring and promotions.
- Viable capabilities that are currently strong and sustainable, with good demographics, that are either unique and inherently governmental or competitive in quality and performance with the private sector. SUPPORT these capabilities by appropriate incentives and budgets.
- Weak and not unique capabilities that are currently in decline, and are duplicated or exceeded by the private sector or are not inherently governmental. REDUCE these capabilities by outsourcing or other methods.
Mission Effectiveness Metrics
Metrics for departmental mission effectiveness will be primarily focused on outcomes: the long-term success of the mission's work. However, this is often difficult to evaluate because of its long time scale. Shorter-term metrics can consider viability as a leading indicator of effectiveness: the level of learning and skills in the department, demographics (ratio of younger to older employees with the same skills), employee satisfaction, and the quality of internal teaming, communications, and management. Evaluations based on the Capability Maturity Models appear to focus on these issues of organizational viability and are recommended for this purpose.
Another difficulty with mission effectiveness metrics is that they are defined differently for each departmental mission: they are unique. Hence they cannot be directly compared or benchmarked against each other, as the more generic efficiency metrics can. However, each of them can be compared with themselves -- by plotting their trends across time. This requires an initial baseline measurement to be made, followed by periodic measurements using the same metric. In this way, improvements in any effectiveness metric, no matter how specialized, can be evaluated.
Customer satisfaction -- where the customer is the specific sponsor of each departmental program -- is an exception to the above difficulty. Customer satisfaction can be standardized and scored across departments, as well as across time, irrespective of the specific mission of the department. So customer (sponsor) satisfaction measurements should be a requirement for monitoring the health of any agency or department.
Stage 3: Operational Efficiency Triage
The third stage of triage is to determine the general efficiency and productivity of routine office processes and business practices. The scope of this evaluation varies with the organization; usually each physical campus or physical plant has established networks, procedures and standards and cultures for administration that are similar within this physical area, so this is probably the most appropriate scope. The determination of process efficiency can then be compared with that of other companies or agencies using similar metrics -- that is why it is important to develop standardized operational metrics.
Triage decisions for efficiency take the following form:
- Organizations with "dysfunctional" work processes, e.g. inappropriate or antiquated business processes, excessive paperwork, lack of standards, etc. IMPROVE business processes, reengineer and then add modern automated technology, EDI, etc.
- Organizations with appropriate modern processes and standards. MAINTAIN processes by vigilant management.
- Organizations with antiquated business processes, excessive paperwork, and without sufficient trained leadership to manage significant change. GET OUTSIDE HELP to plan and manage the organization, make outsourcing decisions, etc. This probably involves private consultants, but also increased dependence on the parent agency's support and guidance.
Metrics for generic efficiency include: Office work processing times, electronic/paper ratios, staff/employee ratio, standardization, total cost of ownership, overhead rate, results of customer satisfaction surveys (of sponsors, internal users, or taxpayers), output quality, and office productivity. The precise data collection and sampling procedures, scales, intervals and calculations of these metrics should be standardized at as high a level as possible in the organization in order that individual departments can be compared or benchmarked easily and meaningfully. This evaluation will allow managers to identify best practices to expand, areas of poor efficiency to improve, and how the agency compares with the best-of-breed in the private sector.
Triage decision-making is a key responsibility of senior managers in government agencies. These triage decisions focus on three kinds of questions:
- To what extent is the agency's mission needed, now and in the future?
- How effective are we at doing our overall mission, and each of the sub-missions or core capabilities?
- How efficient are our generic business processes in comparison with the best-of-breed?
Each of these questions result in a triage that determines areas where improvements should be focused, areas that should be left alone, and areas that should be downsized or outsourced. Once these areas have been sorted out, it is vital to link them to budget allocations. Improving or enhancing a capability means increasing the funds used for things like business process reengineering, upgrading information resources, recruiting, hiring, training and pay incentives.
All three kinds of questions need to be continually asked and evaluated. The easiest way to do this is by implementing a measurement system (such as a balanced scorecard) that can provide more or less continuous monitoring of the need, effectiveness and efficiency of each capability. If triage decisions are based on these measurements, and if these decisions are linked to budgets, then the agency has established a process for continuous strategic alignment and continuous improvement of internal operations.
But the linkage to budgets is vital: if senior decision makers are not willing to allocate budgets according to measurement-based triage, then what is their function? They have become not managers, but mere caretakers of processes (see Jac Fiz-Enz, Benchmarking Staff Performance).
This article has aimed at working backwards from the kinds of decisions that senior managers make to determine the kinds of metrics that need to be defined in order to help them make these decisions reliably. This process of design-by-working-backwards has resulted in a rational method of determining what metrics are really needed -- a question that often arises in developing a Balanced Scorecard. Most of the "textbook examples" of scorecards apply to commercial firms that may have little relation to governmental agencies. If these models are taken at face value, it may lead to developing scorecards and data collection processes that are excessive or not clearly focused on the key needs of decision-makers.
We find that the metrics really needed by government agency decision-makers fall into three general categories:
- Strategic needs metrics: ways to assess the future needs related to the agency's general mission and sub-missions based on an analysis of the external situation. (SWOT analysis and gap analysis are often the approaches used here).
- Mission effectiveness metrics: ways to assess the health and viability of those missions that are going to be needed in the future. Detailed definitions of these metrics are required by each unit's capability experts, as well as collection of appropriate data on outputs and outcomes.
- Operational efficiency metrics: ways to assess the quality of support functions in enabling the needed missions to be accomplished for the minimum cost and time. This requires standardized metrics, customer surveys and benchmarking to identify best and worst practices.
©1999 Paul Arveson