The measurement of effectiveness of online training is a hot topic right now. In this article, I outline how you can use the Kirkpatrick’s model of training evaluation to measure training effectiveness, its impact, and the ROI of eLearning.
What Is Training ROI?
Return on Investment (ROI) in training is a critical metric that measures the effectiveness and financial return of educational programs within an organization. It’s not just about the monetary gains; it also encompasses the qualitative improvements in skills, productivity, and organizational culture that result from training. Understanding training ROI is essential for businesses to evaluate how well their investments in employee development are paying off.
Key Components of Training ROI
- Costs versus Benefits: Training ROI compares the cost of training (like materials, instructors, technology) against the benefits (improved performance, higher productivity, reduced errors).
- Quantitative Measures: These include direct financial gains, such as increased sales or reduced operational costs due to improved skills and efficiencies.
- Qualitative Measures: Often overlooked, these refer to intangible benefits like improved employee morale, better teamwork, and enhanced corporate culture.
- Time Frame: ROI is not just a post-training measure. It’s assessed over a period, from immediate impacts to long-term changes in performance and growth.
Why Measure Training ROI?
Measuring the Return on Investment (ROI) of training programs is crucial for businesses to understand the effectiveness and value of their educational initiatives. It goes beyond mere financial assessment, offering insights into various aspects of organizational growth and employee development.
Key Reasons for Measuring Training ROI
- Financial Accountability: Ensures that the funds allocated for training are being used effectively, leading to tangible business benefits.
- Improved Decision-Making: Provides data-driven insights for future training investments and strategies.
- Aligning Training with Business Goals: Helps in aligning educational programs with the overall objectives of the organization.
- Identifying Effective Training Methods: Differentiates between the most and least effective training methods, allowing for optimization of resources.
- Enhancing Employee Engagement: Demonstrates to employees that their development is valued, leading to increased motivation and job satisfaction.
- Benchmarking and Continuous Improvement: Sets benchmarks for training effectiveness and guides continuous improvement in training methodologies.
- Competitive Advantage: In the long run, effectively trained employees can provide a significant competitive edge in the market.
Challenges of Measuring Training ROI
Measuring the Return on Investment (ROI) of training programs is not always straightforward. Several challenges can arise, posing challenges that make it challenging to accurately assess the true impact of training initiatives.
Key Challenges in Measuring Training ROI
- Attributing Results to Training: Isolating the impact of training from other variables affecting performance can be difficult.
- Long-Term versus Short-Term Impact: Some training benefits take time to manifest and may not be immediately apparent.
- Quantifying Qualitative Benefits: Measuring intangible benefits such as improved morale or stronger leadership skills poses a significant challenge.
- Data Collection and Analysis: Gathering relevant data and analyzing it effectively can be resource-intensive.
- Balancing Costs and Benefits: Accurately assessing all costs associated with training, including indirect costs, is crucial but often challenging.
- Changing Business Environments: Rapid changes in business environments can make it difficult to assess long-term training ROI.
- Stakeholder Expectations: Aligning the ROI measurement with the diverse expectations of stakeholders can be complex.
How to Measure Training ROI
Measuring the Return on Investment (ROI) for training involves a systematic approach that combines both financial and non-financial assessments.
- Identify Training Costs: This includes direct costs like materials and instructor fees, as well as indirect costs such as time spent by employees in training.
- Define Objectives and Outcomes: Clearly outline what the training aims to achieve, including both skill acquisition and business impacts.
- Gather Data Pre- and Post-Training: Collect relevant data before and after the training to measure changes in performance, productivity, or other relevant metrics.
- Calculate the ROI: Use the formula ROI = (Monetary Benefits – Training Costs) / Training Costs × 100%. This calculation gives a percentage representing the financial return on the training investment.
Effectively measuring training ROI often involves the use of Kirkpatrick’s Model of Training Evaluation, a widely recognized and respected framework. It consists of four levels:
What Is Kirkpatrick’s Model of Training Evaluation?
Kirkpatrick’s model of training evaluation is one of the popular models used to evaluate the effectiveness of training. It was created in 1959 and has undergone revisions in 1975 and 1994. The fact that it has survived for over half a century affirms the value it continues to offer.
It features the following four levels:
- Level 1: Reaction – Evaluate how participants reacted to the training.
- Level 2: Learning – Measure the increase in knowledge or capabilities after the training.
- Level 3: Behavior – Observe changes in behavior and application of skills on the job.
- Level 4: Results – Assess the training’s impact on business results and objectives.
Initially, the model was viewed as a pyramid with each level building up from the previous one.
Increasingly, these same 4 levels are viewed as a “chain of evidence,” reflecting a more relevant connection between each level.
How to Determine Training ROI?
The measurement of ROI of eLearning needs an integrated approach that should begin during the Training Needs Analysis or TNA phase and should successively build up right up to the determination of its impact on business.
How Do You Begin the Exercise to Measure the Effectiveness of Online Training?
At EI, we use the approach shown here. Essentially, we focus on each stage from TNA to ROI calculation as the right action at each stage will impact the ROI of eLearning positively.
- Training Needs Analysis (TNA)
This is the baseline and will be eventually used to measure the desired gain for the learners (acquisition of new skill or fixing a gap) and if this gain resulted in the required impact the business had sought. Capturing the needs of the learners and having clarity on how you will measure their progress (that is, the expected gain) is a key component of the exercise. - Determining the right training format (online, blended, or ILT)
The next step is identifying the right format of training that aligns best with the TNA (Sometimes, training may not be the answer, and you need to identify supporting measures like coaching or mentoring). The selection of the right format is crucial in encouraging the learners to pursue it and also in ensuring that they connect with it, complete it and apply the learning on the job. - Identifying the learning strategy
The selection of the right learning strategy format is vital in engaging the learners. As we know, only an effective learning strategy can create a sticky learning experience. However, it is equally important to note that having formal training alone may not be enough. We also need to provide room for application of knowledge and practice or nudges to mastery. Hence, the learning strategy must have a combination of formal training and performance support intervention to successfully manage your mandate. - Determining the gain by the learners
An effective assessment strategy is the main approach to determine this. However, it is important to ensure that the measurement of this gainalso factors for the validation of the application of the acquired learning and is not limited to the validation of knowledge acquisition. - Assessing the gain for business
We can look back at the parameters identified during the TNA stage and assess the required gain that has occurred and if the business saw the required impact. - ROI determination is the final outcome of this exercise
Once the gain is monetized and compared with costs, we will arrive at the ROI of your eLearning. Often, this outcome may require revisiting the TNA and reassessing or adjusting the way forward.
To show how these aspects are interconnected and are part of a cycle is captured in the diagram here.
How Can You Practically Use the Kirkpatrick Model of Training Evaluation to Determine Training ROI?
To use the Kirkpatrick model of training evaluation, we need to identify two aspects at each level namely:
- What are we measuring?
- How will we use this outcome (to improve training effectiveness and increase its impact)?
Level 1: Reaction
Objective: At this level, the focus is to determine the learner’s reaction to the training. Today, we have wide-ranging options through Learner Analytics to identify if the learners liked the training if they found it useful and if they would be able to apply the learning.
From an evaluation perspective, this feedback enables L&D teams to assess if they are on track or if any further changes are required.
Level 2: Learning
Objective: At this level, the focus is to determine what was learned or gained (this should be attributable directly to the training).
From an evaluation perspective, this feedback enables L&D teams to assess if they met their mandate (captured during TNA) that could include:
- Knowledge gain.
- Acquisition of a new skill.
- Further proficiency gain on an existing skill.
- Behavioral change.
The pointers from this stage of evaluation would point to:
- The need for further training.
- The need to supplement formal training with other measures that could include performance support intervention or mentoring/coaching.
Level 3: Behavior
Objective: At this level, the focus is to determine if the learner behavior changed (again, this should be attributable directly to the training).
From an evaluation perspective, this feedback enables L&D teams to assess if there was a demonstrable change in the learner’s behavior.
Often, this is can be tricky. Although, learners had successfully cleared the assessment, yet there is no demonstrable change.
This may need re-assessment to determine why this is not happening.
- Sometimes, it could be because learners have no opportunity to demonstrate what they learned, and often, it may point to the need for reinforcement.
- There may be a need to have refresher programs to be offered over an extended period of time till the required gain is observed.
Level 4: Impact
Objective: At this level, the focus is to determine if the business saw the gain and if the required impact was created on account of the training.
From an evaluation perspective, this feedback enables L&D teams to review if the expected impact identified during the TNA phase indeed happened.
How Can You Use Kirkpatrick’s Model Of Training Evaluation To Measure ROI?
ROI determination (or Level 5): This is an add-on to the initial model (that has 4 levels) and is referred to as the Kirkpatrick-Phillips Evaluation Model of training.
Simply put, a positive ROI of eLearning is achieved when the demonstrable gain from the training exceeds the costs incurred to create and deliver the training.
I hope this article gives you practical insights into how you can enhance the impact of each stage from TNA to the evaluation of its impact and see a positive ROI of eLearning.
