Kirkpatrick's
Evaluating Training Programs
About the Author of Evaluating Training
Programs
Donald L
Kirkpatrick's training evaluation model - the four
levels of learning evaluation - is widely used in business models. This
model can easily be adapted to be used in developing and managing a
figure skating program by rink management and the board of directors of
the local figure skating club associated with the rink. A
figure skating club is non-profit, but
it must have a
business plan that carefully controls its expenses to conduct test
sessions and host skating competition, ice carnivals, shows, and other
activities such as workshops and seminars.
Professor
Emeritus Donald L Kirkpatrick achieved his BBA, MBA and PhD at the
University Of
Wisconsin where he first published his
ideas in 1959, in a series of articles in the Journal of American
Society of Training Directors. The articles were subsequently
included
in Kirkpatrick's book Evaluating Training
Programs (originally
published in 1994; now in its 3rd edition - Berrett-Koehler
Publishers).
He also was president of the
American
Society for
Training and Development (ASTD) in 1975. Kirkpatrick has written
several other significant books about training and evaluation, more
recently with his similarly inclined son James, and has consulted with
some of the world's largest corporations.
The 1994 book Evaluating
Training Programs
defined his originally published ideas of 1959 and his theory has now
become one of
the more widely used and popular model for the evaluation of
training and learning. Kirkpatrick's four-level model is now considered
an industry standard across the HR and training communities.
More recently Don Kirkpatrick
formed his own
company, Kirkpatrick
Partners, whose website provides information about their services
and methods, etc.
Kirkpatrick's four levels of
evaluation model -
The four levels of Kirkpatrick's evaluation
model
essentially
measure:
- Reaction of student - what they thought and felt about
the training
- Learning - the resulting increase in knowledge or
capability
- Behavior - extent of behavior and capability
improvement and implementation/application
- Results - the effects on the business or environment
resulting from the trainee's performance
All these measures are recommended
for full
and meaningful
evaluation of learning in organizations, although their application
broadly increases in complexity, and usually cost, through the levels
from level 1-4.
Kirkpatrick's
four levels of training evaluation -
This grid
illustrates the basic
Kirkpatrick structure at a glance. The second grid, beneath this one,
is the same thing with more detail.
Level |
Evaluation
Type
(what is measured) |
Evaluation
Description and Characteristics |
Examples of
Evaluation Tools and Methods |
Relevance and
Practicability |
1
|
Reaction
|
Reaction evaluation is
how the delegates felt about the training or
learning experience.
|
'Happy sheets', feedback
forms.
Verbal reaction, post-training
surveys or questionnaires.
|
Quick and very easy to obtain.
Not expensive to gather or to
analyze.
|
2
|
Learning
|
Learning evaluation
is the measurement of the increase in knowledge - before and
after.
|
Typically assessments or
tests before and after the training.
Interview or observation can
also be used.
|
Relatively simple to set up;
clear-cut for quantifiable skills.
Less easy for complex learning.
|
3
|
Behavior
|
Behavior evaluation is
the extent of applied learning back on the job - implementation.
|
Observation and interview over
time are required to assess change, relevance of change, and
sustainability of change.
|
Measurement of behavior
change typically requires cooperation and skill of line-managers.
|
4
|
Results
|
Results evaluation is
the effect on the business or environment by the trainee.
|
Measures are already in place
via normal management systems and reporting - the challenge is to
relate to the trainee.
|
Individually not difficult;
unlike whole organization.
Process must attribute clear
accountabilities.
|
Kirkpatrick's four levels of
training evaluation in detail
This grid illustrates the Kirkpatrick's
structure
detail, and
particularly the modern-day interpretation of the Kirkpatrick learning
evaluation model, usage, implications, and examples of tools and
methods. This diagram is the same format as the one above but with more
detail and explanation:
Evaluation Level and Type |
Evaluation Description and
Characteristics |
Examples of Evaluation Tools and
Methods |
Relevance and Practicability |
1. Reaction
|
Reaction evaluation is
how the delegates felt, and their personal
reactions to the training or learning experience, for example:
Did the trainees like and
enjoy the training?
Did they consider the training
relevant?
Was it a good use of their
time?
Did they like the venue, the
style, timing, domestics, etc.?
Level of participation.
Ease and comfort of experience.
Level of effort required to
make the most of the learning.
Perceived practicability and
potential for applying the learning.
|
Typically 'happy sheets'.
Feedback forms based on
subjective personal reaction to the training experience.
Verbal reaction which can be
noted and analyzed.
Post-training surveys or
questionnaires.
Online evaluation or grading
by delegates.
Subsequent verbal or written
reports given by delegates to managers back at their jobs.
|
Can be done immediately the
training ends.
Very easy to obtain reaction
feedback
Feedback is not expensive to
gather or to analyze for groups.
Important to know that people
were not upset or disappointed.
Important that people give a
positive impression when relating their experience to others who might
be deciding whether to experience same.
|
2. Learning
|
Learning evaluation is
the measurement of the increase in knowledge or intellectual
capability from before to after the learning experience:
Did the trainees learn what
what intended to be taught?
Did the trainee experience
what was intended for them to experience?
What is the extent of
advancement or change in the trainees after the training, in the
direction or area that was intended?
|
Typically assessments or tests
before and after the training.
Interview or observation can
be used before and after although this is time-consuming and can be
inconsistent.
Methods of assessment need to
be closely related to the aims of the learning.
Measurement and analysis is
possible and easy on a group scale.
Reliable, clear scoring and
measurements need to be established, so as to limit the risk of
inconsistent assessment.
Hard-copy, electronic, online
or interview style assessments are all possible.
|
Relatively simple to set up,
but more investment and thought required than reaction evaluation.
Highly relevant and clear-cut
for certain training such as quantifiable or technical skills.
Less easy for more complex
learning such as attitudinal development, which is famously difficult
to assess.
Cost escalates if systems are
poorly designed, which increases work required to measure and analyze.
|
3. Behavior
|
Behavior evaluation is
the extent to which the trainees applied the learning and changed
their behavior, and this can be immediately and several months
after the training, depending on the situation:
Did the trainees put their
learning into effect when back on the job?
Were the relevant skills and
knowledge used
Was there noticeable and
measurable change in the activity and performance of the trainees when
back in their roles?
Was the change in behavior
and new level of knowledge sustained?
Would the trainee be able to
transfer their learning to another person?
Is the trainee aware of their
change in behavior, knowledge, skill level?
|
Observation and interview over
time are required to assess change, relevance of change, and
sustainability of change.
Arbitrary snapshot assessments
are not reliable because people change in different ways at different
times.
Assessments need to be subtle
and ongoing, and then transferred to a suitable analysis tool.
Assessments need to be
designed to reduce subjective judgment of the observer or interviewer,
which is a variable factor that can affect reliability and consistency
of measurements.
The opinion of the trainee,
which is a relevant indicator, is also subjective and unreliable, and
so needs to be measured in a consistent defined way.
360-degree feedback is useful
method and need not be used before training, because respondents can
make a judgment as to change after training, and this can be analyzed
for groups of respondents and trainees.
Assessments can be designed
around relevant performance scenarios, and specific key performance
indicators or criteria.
Online and electronic
assessments are more difficult to incorporate - assessments tend to be
more successful when integrated within existing management and coaching
protocols.
Self-assessment can be useful,
using carefully designed criteria and measurements.
|
Measurement of behavior
change is less easy to quantify and interpret than reaction and
learning evaluation.
Simple quick response systems
unlikely to be adequate.
Cooperation and skill of
observers, typically line-managers, are important factors, and
difficult to control.
Management and analysis of
ongoing subtle assessments are difficult, and virtually impossible
without a well-designed system from the beginning.
Evaluation of implementation
and application is an extremely important assessment - there is little
point in a good reaction and good increase in capability if nothing
changes back in the job, therefore evaluation in this area is vital,
albeit challenging.
Behavior change evaluation is
possible given good support and involvement from line managers or
trainees, so it is helpful to involve them from the start, and to
identify benefits for them, which links to the level 4 evaluation below.
|
4. Results
|
Results evaluation is
the effect on the business or environment resulting from the
improved performance of the trainee - it is the acid test.
Measures would typically be
business or organizational key performance indicators, such as:
Volumes, values, percentages,
time scales, return on investment, and other quantifiable aspects of
organizational performance, for instance; numbers of complaints, staff
turnover, attrition, failures, wastage, non-compliance, quality
ratings, achievement of standards and accreditations, growth,
retention, etc.
|
It is possible that many of
these measures are already in place via normal management systems and
reporting.
The challenge is to identify
which and how relate to to the trainee's input and influence.
Therefore it is important to
identify and agree accountability and relevance with the trainee at the
start of the training, so they understand what is to be measured.
This process overlays normal
good management practice - it simply needs linking to the training
input.
Failure to link to training
input type and timing will greatly reduce the ease by which results can
be attributed to the training.
For senior people
particularly, annual appraisals and ongoing agreement of key business
objectives are integral to measuring business results derived from
training.
|
Individually, results
evaluation is not particularly difficult; across an entire organization
it becomes very much more challenging, not least because of the
reliance on line-management, and the frequency and scale of changing
structures, responsibilities and roles, which complicates the process
of attributing clear accountability.
Also, external factors greatly
affect organizational and business performance, which cloud the true
cause of good or poor results.
|
Since Kirkpatrick established his
original
model, other
theorists (for example Jack Phillips), and indeed Kirkpatrick himself,
have referred to a possible fifth level, namely ROI (Return On
Investment). ROI can easily be included in Kirkpatrick's
original fourth level 'Results'. The inclusion and relevance of a fifth
level is therefore arguably only relevant if the assessment of Return
On Investment might otherwise be ignored or forgotten when referring
simply to the 'Results' level.
Learning evaluation is a
widely
researched area. This
is understandable since the subject is fundamental to the existence and
performance of education around the world, not least universities,
which of course contain most of the researchers and writers.
While Kirkpatrick's model is not the
only one of
its type, for
most industrial and commercial applications it suffices; indeed most
organizations would be absolutely thrilled if their training and
learning evaluation, and thereby their ongoing people development, were
planned and managed according to Kirkpatrick's model.
Bibliography:
- Jack Phillips' Five Level ROI
Model
- Daniel Stufflebeam's CIPP Model
(Context, Input, Process,
Product)
- Robert Stake's Responsive
Evaluation Model
- Robert Stake's
Congruence-Contingency Model
- Kaufman's Five Levels of
Evaluation
- CIRO (Context, Input, Reaction,
Outcome)
- PERT (Program Evaluation and
Review Technique)
- Alkins' UCLA Model
- Michael Scriven's Goal-Free
Evaluation Approach
- Provus's Discrepancy Model
- Eisner's Connoisseurship
Evaluation Models
- Illuminative Evaluation Model
- Portraiture Model
- American
Evaluation Association
- Leslie Rae's Excellent
Training Evaluation and Tools available on this
site, which, given Leslie's experience and knowledge, will save you the
job of researching and designing your own tools.
- Howard
Gardner and multiple intelligences theories
- Bloom's
Taxonomy of Learning Domains
- 360
degree appraisals tips
- employment
termination, dismissal, redundancy, letters templates and style
- exit
interviews, questions examples, tips
- grievance
procedures letters samples for employees
- group
selection recruitment method
- induction
training checklist, template and tips
- job
interviews - tips, techniques, questions, answers
- job
descriptions, writing templates and examples
- performance
appraisals - process and appraisals form template
- team
briefing process
- training
evaluation processes
- training
and developing people - how to
References:
Resources:
The following internet
links have been
gleaned from personal communications
combined
with
information from
public institutions and athletic
organizations/
associations
that
have a web presence with information concerning team
and
individual
sports
programs:
All
materials are copy protected.
The limited use of the
materials for education purposes is allowed providing
credit is given
for the source of the materials.
|