APPEL evaluates the effectiveness of all of its courses to confirm their value to NASA’s workforce. Here, participants in the course Project Management for non-Technical Managers, held in September 2016 at NASA Headquarters, explore the tools and techniques that are critical to planning and managing any project successfully.
Credit: NASA/Dan Daly
APPEL supports NASA by providing development and training for the agency’s technical workforce. To reinforce the value of its offerings, the Academy recently brought home the gold.
APPEL takes the design, development, and delivery of courses extremely seriously, as they are critical components of training. Just as important is confirming the value of the training. The Academy accomplishes this by evaluating the effectiveness of its offerings. Evaluation allows APPEL to assess whether its programs meet learner needs and objectives, whether knowledge and skills are imparted as intended, whether desired changes in performance are achieved, and what organizational benefits result from the training.
To ensure the validity of its course evaluation measures, the Academy’s evaluation strategy is aligned with the Kirkpatrick Model: the foremost model in the world for analyzing and evaluating the results of training and educational programs. As part of this effort, members of the APPEL team—including the Academy’s entire Curriculum team—are Kirkpatrick Certified Professionals – Gold Level.
The Kirkpatrick Model measures training effectiveness across four levels:
Level 1: Reaction
- Evaluations at this level assess the degree to which participants find the training favorable, engaging, and relevant to their jobs. For example: Did the participants find the training relevant? Did they like and enjoy the training?
Level 2: Learning
- Level 2 evaluations focus on the degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation in the training. For example: Did the participant’s knowledge increase as a result of the course?
Level 3: Behavior
- Evaluations at the third level measure the degree to which participants apply what they learned in the training after returning to their jobs. APPEL assesses the application of learnings from the course 45 days after training completion.
Level 4: Results
- Level 4 evaluations examine the degree to which targeted outcomes occur as a result of the training and subsequent support and reinforcement. For a business or organization, key performance indicators might be quantifiable aspects of organizational performance, such as a reduction in number of complaints or increase in quality ratings. For NASA, APPEL looks at the collective contribution and impact of specific courses on the agency’s ultimate goal: mission success.
“APPEL is doing quality work that is in alignment with the recommendations of the U.S. Office of Personnel Management’s Training Evaluation Field Guide. Through the efforts of organizations like APPEL, the U.S. government is lighting the way for all organizations with their leadership position in creating and demonstrating program value,” said Dr. Jim Kirkpatrick, co-owner of Kirkpatrick Partners.
Course participants emphasize the value of APPEL courses to their professional development and ability to support the NASA mission. According to one individual who attended APPEL’s Assertiveness Training for Technical Professionals, “The assertiveness course follows me on an almost daily basis. First, it helped me feel better that I’m not alone in terms of being reluctant to speak up. I remind myself to be more assertive every day by asking myself, “What’s the worst that could happen if I speak up?” It’s having an impact on my personal life and my work life—and that’s particularly important because a great deal of my sense of self-worth comes from my work. Because I’m asking more questions at work, we are finding more problems and bringing them to light earlier.”
Data reinforces the feedback from participants, confirming that APPEL courses have a positive impact on employee job performance. For example, at the start of an APPEL course, on average 57% of attendees can perform key course objectives with assistance. By the end of the course, that number increased to 91%. Additionally, when surveyed 45 days after completing an APPEL course, more than 75% of respondents report they are successfully applying the critical behaviors learned in the course.
“By implementing the Kirkpatrick enhanced evaluation approach, we are able to demonstrate the impact of APPEL courses on job performance and illustrate how APPEL contributes to mission success at NASA,” said APPEL Director and NASA Chief Knowledge Officer Roger Forsgren. Furthermore, aligning with the Kirkpatrick Model ensures the Academy retains elite status among industry peers.
APPEL will continue to implement the Kirkpatrick evaluation model throughout fiscal year (FY) 2018. Level 1 and 2 evaluations will be applied to all courses, while Level 3 and 4 evaluations will be conducted across 11 courses in APPEL’s new Building and Leading NASA Teams curriculum category.
Visit the APPEL Catalog to learn more about the Academy’s courses for NASA project managers and systems engineers.
