Calculating Measurable Outcomes of Educational Interventions

Once you have completed the first four steps of the Evidence-Based Practice (EBP) model, you can move on to Step 5, evaluating the outcomes of the changes you implemented.

In a webinar hosted by CHA in January 2019, Debra Liebig, MLA, BSN, RN-BC, program manager in the accreditation and readiness department at Children’s Mercy Hospital Kansas City, and Cathleen Opperman, DNP, RN, NEA-BC, CPN, nursing professional development specialist at Nationwide Children’s Hospital, shared how to calculate the economic impact and return on investment (ROI) of your efforts.

Understand cost avoidance

Cost avoidance is the benefits that result when expenditures can be avoided. To identify cost avoidance, you must first gather data. As you implement the changes you wish to measure the impact of, tally the expenditures/costs of the equipment, devices, and supplies needed. Also track the same information for the expenditures/costs that you stop using.

“Tally the costs of teaching and the learners’ time,” Liebig said. “No matter how big or how small.” Cost avoidance is the benefit seen from not spending. The expenditures/costs that you stop using, and avoid incurring in the future, represent this.

Gather data for the formulas 

Develop a clinical question to guide your search. The data you will need in order to calculate the economic impact of your efforts falls into two categories: costs/expenses and benefits. Accurate, complete data is crucial for quality results, so the more detail you can gather on the costs and benefits of your activity, the truer and more accurate your calculations will be.

Costs/Expenses: This is what you spend/lose as you put your intervention into action. Costs include average hourly pay for staff, direct expenses, and devices and equipment purchased to implement. Challenges you may face in this area include getting the average wage for various employees from the human resources department (if not, you will need to use estimates) and taking the time to track all expenses as you move through the process.

“Using a worksheet and placing copies of purchases in a file or a spreadsheet will save you time,” Liebig said. “Digging these things up later is time-consuming.”

Benefits: The counterbalance to the costs of your EBP project are the benefits gained. There can be numerous contributing factors and results from each of your interventions, so consider monitoring multiple outcomes.

Outcomes may include:

• Absenteeism
• Attrition
• Competence and self-confidence
• Turnover intent
• Turnover
• Retention
• Vacant full-time equivalents
• Lost workdays
• Modified workdays
• Expense for agency nurses
• Injury rates
• Job satisfaction
• Reduced time for orientation
• Number of infusion pump alerts
• Number of pressure ulcers
• Patient acceptance
• Ventilator-associated pneumonia
• Reported errors
• Safety climate
• Use of behaviors from program

Get out the calculator

Once you have the necessary pieces above, you can begin to plug the results of your education efforts into the following formulas. Opperman offered the following examples:

Cost Effectiveness: The simplest of calculations, according to Opperman, is Cost Effectiveness. Take the total cost of your program (development cost plus cost of time spent in training) and divide it by the number of participants in the training.

Total cost of program = Cost per participant

Number of participants

For example, if an online self-study cost $450 to create, learners took 15 minutes to complete it, and there were 300 learners with an average hourly wage of $30, you would calculate:

Total cost of program = development cost ($450) + cost of time spent in training (300 learners x ¼ hourly rate)

$450 + (300 X $7.50)   =
       300 learners

$2,700 = $9 per participant

So, the cost of the training was $9 per learner.

Cost analysis

Cost analysis follows the same formula as cost effectiveness but involves comparing two possible interventions.

For example, instead of the online self-study shown above, another option would be a live in-service training. If this took four trainers offering multiple 20-minute sessions each (adding up to 8 hours total), content development cost of $250, 1 hour of training for the instructors ($36 average wage), and 300 learners with an average hourly wage of $30, you would calculate:

Total cost of program = development cost + cost of training the trainers + cost of trainers' time teaching + cost of learners' time spent in training

$250 + ($36 X 4 hr ) +($36 X 8 hr) + ($10 X 300) =

                          300 learners

$3,682 = $12.27 per participant

For the cost analysis, compare the $12.27 cost per participant to the online learning cost of $9.00 per participant. This shows that the online learning would cost less per learner, but Opperman recommends looking beyond the numbers.

“The numbers alone should never be the only basis for a decision,” Opperman said. “Always analyze the advantages and disadvantages of each option for implementation.”

For this scenario, Opperman suggested considering the following factors.

Pros:

• Online learners needed less time to complete.
• Live training offers the opportunity for questions and feedback.
• Online training will be enduring.

Cons:

• If learners miss all offerings, what is the make-up option?
• Online offers no opportunity for interaction.

When choosing between two interventions, also consider the efficacy of both. If one will create a better outcome than the other, the financial perspective may not be the most important consideration.

Benefit-cost ratio

The result of this formula is aimed at representing the overall value gained from money spent on an initiative. Calculated with the following:

Total Benefits = Benefit-Cost Ratio (BCR)

The higher the number, the better the benefit. If the result is less than one, the intervention would have a negative impact. If it is higher than one, then there would be a positive impact.

For example, if an improved preceptor program contributed to reduced turnover by two nurses in the first year, the average cost of turnover is the total benefit. The cost was $22,000 for development, teacher and learner time for 37 participants. The known cost table indicates $36,567 is the average cost per RN leaving. You would find the benefit-cost ratio for this situation as follows:

Total Benefit (Cost Avoidance): $36,567 X 2 = $73,134

Total Cost: $22,000 is given as the total cost

BCR: $73,134 = 3.32 BCR $22,000

Because the BCR is greater than one, this result indicates that providing the program would be advantageous. As with the previous calculations, non-quantifiable advantages and disadvantages should also be considered.

Return on investment (ROI)

The results of this calculation show the value of money put into an initiative. The result of the formula is represented as a percentage and should be greater than one percent to show a favorable outcome. To find ROI, plug your data into this formula:

Total Benefit – Total Costs X 100 = % ROI

Using the scenario with the preceptor program from the previous section:

$73,134 - $22,000 X 100 = 232.4% ROI

Since the final ROI value is well over 1%, this also reflects positively on the value of this program.

While evaluation methods and measures should be considered at the beginning of the EBP process, the time to use these calculations comes toward the end of the process with the Evaluation and Dissemination steps.