Introduction

Training and development refers to a variety of educational and learning-based activities that are used to acquire knowledge or skills needed by an employee to perform effectively on the present job. Training activities can vary in complexity - a classroom seminar, computer-based learning, an in-house executive course, individual coaching and mentoring.

Training and development has become an essential component of most organizations for a number of reasons:

• From an organizational perspective, training provides the means to maintain the organization's competitive advantage by attracting and retaining its employees. For example, the Gartner Group has discovered that those organizations spending over four percent of their salary budget on training and development for informatics staff tend to retain twice as many employees as those who spend less. The competition for staff is so stiff in certain areas that the willingness of an organization to provide the commitment to develop and train its staff provides a powerful incentive to remain. Training and development is an indispensable strategy that helps facilitate the achievement of an organization's mission or to successfully complete any large-scale change.
• The locus of organizational functioning has changed from the traditional rules-based approach to a more decentralized and empowered mode of operation. Employees must use their skills, abilities and creativity to solve problems, work in teams and resolve conflicts. With training and development, employees can improve the quality of their work, make fewer mistakes, evolve toward more progressive responsibilities and serve their clients more effectively. In the future, levels of training and development will likely increase as change accelerates with new personal productivity tools and different ways of communicating and coordinating activities.

A new approach to measuring training and development

Measuring the effectiveness of training and development is often an informal and unsophisticated practice, despite the large amounts of the salary budget allocated to it. Often assessment consists of completing a one or two page evaluation form at the culmination of a training course. The quality of this information to make effective strategic decisions regarding training and development is of minimal value.

Without such information, budgetary decisions to do not ensure an adequate return on the investment of training and development funds. The full potential for training and development to support critical organizational change and to promote improved employee growth/job satisfaction is minimized.

The course management index (CMI) is a new and innovative instrument used to measure the effectiveness of training events from a lifecycle point of view. The lifecycle approach ensures that measurement tracks the progress of a training event from the initial perceptions of participants (baseline measures), through their reaction to its content and delivery (design/delivery measures). Further, it assesses the degree to which participants acquired the desired knowledge and/or skill (knowledge acquisition/skill development measures) and the degree to which the participant applied these abilities on the job (learning application measures). The CMI also tracks the cost of developing and delivering the training event (financial measures).

By using an index approach, the CMI allows the combination of qualitative and quantitative information yet provides the ability to quantify both so that an overall score can be obtained for each training event. Quantitative measures include financial measures. Qualitative measures include design and delivery measures, knowledge acquisition and learning application measures. The use of an index allows these measures to be easily adapted to the unique requirements of each training environment. These dimensions of the CMI can be expanded or decreased in response to changing conditions/experience or the complexity of information required about each training event or particular training environment.

Here is a graphical illustration of the course management index.

Course management index

CMI measures

The CMI lifecycle measures include baseline information, financial indicators, design and delivery measures, knowledge acquisition/skill development measures and learning application measures for each training event.

Each of these CMI measures will be described briefly.

Baseline information

Baseline information is used to provide an assessment of the learning environment, the requirements, and the motivation before an individual attends a training event. By determining the context of the training, it provides a baseline against which application of learning on the job can be assessed. The specific baseline questions relate to the work environment that required the acquisition of these new abilities and skills. Sample questions include the need that motivated participation in this course; the skills, knowledge, values and/or expertise expected from the training event; the frequency with which this new ability will be used; and the knowledge of whether the skill development or behavioural outcomes have occurred.

Financial measures

Financial measures provide a means to determining the most cost effective method of delivering a high-quality training event that will meet the needs of the staff and organization. This involves a consideration of the development, operational, fixed and overhead costs. From such an analysis, it can be decided whether it is more feasible to develop a tailored course offering or obtain it from a commercial vendor. These figures can also be used to track the cost of each training event and compare it with the costs of other course offerings. For those organizations providing training on a cost recovery or profit basis, additional measures provide the means to determine whether each training event is revenue neutral or providing a profit or loss.

Sample measures include actual versus planned development, operational, fixed and overhead costs. Cost recovery or profit/loss measures include course demand increasing, decreasing or steady; actual versus planned number of students per course; number of cancellations (declining or escalating); actual versus planned revenues from course fees; and the cost/revenue ratio per course.

 

Design/delivery measures

Course design and delivery measures provide an outline of participant reaction to the training event in terms of its content, difficulty, amount of work required, and technical detail. It also provides a rating of the instructor and teaching assistant in terms of organization, preparation, knowledge of the subject area, ability to communicate, and ability to provide remedial assistance. Design and delivery measures also provide an assessment of the course overall in terms of whether or not the course should be recommended or whether another should be taken from the same source.

Knowledge acquisition/skill development measures

The learning evaluation measures determine the degree of acquisition of skills or knowledge by the student on a learning event-by-event basis. It provides a means of assessing how well the participant internalized the learning provided by the training event. It can be applied to a certification program such as those inherent in academic programs. It can also be applied to non-credit courses focusing on skill development. Sample measures include graduation/failure rate; dropout rate; average grade on course assignments of participants; average final grade level of participants; and the number of remedial training episodes.

Learning application (behaviour/skill application) measures

The learning application measures determine the degree of use of knowledge or skill on the job. By comparing baseline information with current usage measures, a fuller picture can be obtained as to the degree of application of the behaviour or skills. Sample learning application measures include the frequency of usage; behaviours/skills most/least used; degree of satisfaction of the business need that initiated the request for training; and the degree of organizational support for the usage of new behaviour/skill.

Implementation strategy

In order to use the course management index effectively, an implementation strategy is needed with the following elements:

• roles and responsibilities of management, training administrators and training participants;
• methods of data collection for performance measures previously described;
• methods of analyzing the results to determine accomplishments and issues that influence the effective use of training;
• interpretation of information to determine what is a normative level of training performance for the organization; and
• methods for communicating the results of the CMI.

Each will be discussed briefly.

Roles and responsibilities

There are three major sets of roles and responsibilities with regard to the implementation and maintenance of the CMI.

• The management of the organization is responsible for providing the leadership necessary to develop, implement and maintain the new approach to measuring the effectiveness of training and development. Management must allocate the resources necessary to carry out training and development within the organization and use the results from CMI to improve its management practices.
• The administrators of training and development are responsible for developing, implementing and maintaining the new training evaluation approach using the CMI. This includes customizing CMI measures; setting up or amending business processes to collect the CMI data; analyzing CMI information; interpreting and reporting the results. More information is provided upon these functions in upcoming sections.
• The training event participants provide the facts required for baseline information, design/delivery, and knowledge application. Instructors provide information on knowledge acquisition/skill development.

 

Data collection strategies

The data collection strategies for the CMI will vary for each of the training event measures as follows:

• Baseline information can be collected through the course registration form or using a questionnaire at the beginning of each training event.
• Financial measures can be collected from the financial or training information systems or course registration records.
• Design/delivery measures are collected from student evaluation forms completed at the end of each course. Participants' opinions are given on a 5-point scale with 1 being the lowest and 5 the highest.
• Knowledge acquisition/skill development measures are collected from course completion records for such elements as graduation/failure rates, dropout rate and overall final mark.
• Learning application measures are collected through a questionnaire sent to a sample of training event participants three to six months after its completion. Participants' opinions are given on a 5-point scale with 1 being the lowest and 5 the highest.

Analysis of the training event results

An analysis of the CMI performance information provides the means to identify accomplishments and issues for each training event and across the entire training inventory as follows:

For each training event, the analysis of the CMI dimension of performance provides the information to improve each offering by pinpointing the specific areas of improvement necessary to better satisfy participant needs. The issues tend to deal with the design/delivery of the training, knowledge/skill acquisition, or the degree of its application.

For the entire inventory of training event offerings, such analysis permits the identification of training issues by participant groups, by region or over time. It helps determine whether there are gaps between participant expectations and the quality of training event offerings. It provides a means to determine the most cost effective method of delivery and any necessary changes in training philosophy or strategy.

Interpretation

The performance information presented in the CMI outlines accomplishments and issues related to the achievement of the full benefits of training and development across the entire inventory of training events from a financial, design/delivery, knowledge acquisition and application view points. Cumulatively, this information provides a causal link between achieving the continuous learning goals/mission of the organization and the strategic, tactical and operational issues interfering with this goal. Yet, given this information, the basic question remains of how to interpret it to take the necessary corrective actions. In other words "What is normal for my training and development environment?" The interpretation of CMI information can be accomplished using the following three methods.

 

Rating systems

Each of the CMI training and development measurement perspectives is captured in a training event status report. To ascertain the effectiveness of each training event on a lifecycle basis, the CMI uses a three option summary indicator:

• green rating: The green rating indicates everything is functioning according to plan. There are no major concerns with the training event.
• yellow rating: A yellow rating is assigned where there are issues that need to be addressed in order to deliver the training event according to participant/organizational expectations.
• red rating: A red rating indicates that there are major weaknesses or concerns with a training event that need to be addressed.

Each training event is given a red, yellow or green rating for each of the CMI measures: financial, design/delivery, knowledge acquisition, and learning application measures. If one of these CMI measures is red, the training event is rated red. If most of the indicators are yellow or green without a red rating, then the training event is given either a yellow or green rating.

Then, an overall training score is given for each training event across the entire inventory of courses. These ratings, then, make it fairly easy to concentrate on those training events that are rated red while ignoring those in the inventory which are rated green. Management of the training becomes focused on attaining excellence in each training event while dealing with those few exceptions that require attention to bring them up to a higher standard. It is management by excellence and exception.

Use of baseline information

The availability of baseline information provides a historical perspective on the training performance dimensions permitting the analysis of trends over time. This is the most powerful method of interpretation and is required to understand any changes in the training and development organization. It is also the most difficult to obtain. Financial information is likely the only type of CMI measure available during the first year of operation. In subsequent years, year over year comparisons can be made across all performance measurement dimensions of the CMI.

Use of benchmarks

Benchmarking allows comparisons with training and development functions in organizations of similar size and complexity in order to provide a comparative reference point. For example, spending three to four percent of a salary budget on training would seem very high in most organizations. Yet, this is the norm in most information technology organizations. Without this benchmark information, the interpretation of financial training performance indicator would have been faulty. However, useful benchmarking information is typically very difficult to obtain.

 

 

Communicating

Communicating the training and development accomplishments and issues to both senior management and staff is imperative. The delivery of training events that are affordable, accessible and responsive to the needs of its participants can only be achieved if management is able to obtain effective feedback provided through all the measurement dimensions embodied in the CMI. Such feedback provides the information needed to steer the training and development in support of the mission of the organization in general, and towards its continuous learning objectives in particular.

A composite training and development report, summary training event status report (STESR), should be published quarterly. It is used to provide feedback in the form of accomplishments and issues for financial, design/delivery, knowledge acquisition/skill development and learning application measures across the entire training development program.

Conclusion

CMI can be applied to the management of training and development in large public sector branches or departments as a whole. It can be used by organizations specializing in providing training and development services on a certification basis such as colleges or universities or for personal development only. It can range in complexity from the full suite of CMI measures to only those needed in a particular environment. Whatever the level of complexity chosen, several best practices should be used to assist in applying CMI to the unique circumstances of each organization:

• The development and implementation of CMI is both an art and a science. The science of developing and implementing the CMI involves the use of a basic set of steps, previously described, to provide similar philosophy, structure and process so that comprehensive information can be provided across all the complete inventory of training events. The art of developing and implementing the CMI involves tailoring it to the unique requirements of each organization in terms of specific measures, timing, sequence of activities and knowledge transfer.
• The use of a pilot provides evidence of the utility of the CMI performance measurement approach. It gradually builds acceptance/support for this approach to assessing training and development. It also provides the experience needed to tailor the CMI to the unique requirements of the organization while developing the in-house expertise necessary to maintain it.
• The development and implementation of the CMI must begin from the top of the organization and be implemented progressively downwards. It requires the full commitment and support of management in terms of providing a supportive environment and communicating the necessity and importance of this approach. It also requires a financial investment to develop, implement and maintain this operation.
• Adopt a "just do it" approach to the development and implementation of the CMI. Don't be tied to expensive data gathering and implementation approaches. Use "work around" strategies to find simple but effective solutions. Where progress is delayed on development and implementing the measurement of one CMI perspective, focus more attention on others. Since the whole process is self-improving and self-correcting, the key is to go with whatever information, processes and interpretation methods you have developed.
• Where there are large numbers of participants in training events, sampling techniques can be used to reduce the amount of time and effort associated with the gathering, analysis and interpretation of CMI training information.

Unless the information generated from the CMI is used to take corrective actions to steer the training and development, the implementation effort is wasted. This CMI performance information must be used to move the training and development forward progressively. Where it is demonstrated that CMI performance information is used to improve the functioning of the training and development, then the CMI will become fully self-sustaining.

Bryan Shane is a senior partner with BPC Management Consultants. For more than 20 years, he has provided services in strategic management, information technology and performance measurement to a wide variety of public and private sector organizations. Patricia Lafferty is also a partner with BPC Management Consultants. She possesses extensive experience as a senior professional manager, serving both the public and private sectors, in her areas of specialty: strategic change management, facilitation, implementation planning and performance consulting.

---