Many of us watched in disbelief as the Space Shuttle Challenger broke apart 73 seconds into its January 1986 flight, killing all seven crew members, including school teacher Christa McAuliffe. The cause of the disaster was an O-ring seal that failed due to cold temperatures. Media extensively covered the accident. And many have used the Challenger disaster as a case study for discussions of workplace ethics and engineering safety.
NASA responded to the disaster with numerous initiatives, at least one of which was training related. This response included the precursor to the current Academy of Program/Project and Engineering Leadership Knowledge Services (APPEL KS) program, which focused on a curriculum of formal classroom training related to the fundamentals of project management. The figure on page 67 shows the learning and performance path that has been specifically designed to contribute to future NASA mission safety and success.
Thirty years later, the program is still meeting NASA’s technical workforce needs by using a multifaceted approach to promote learning through training, a systems engineering leadership development program, and knowledge management. The curriculum supports the development of program and project managers, systems engineers, discipline engineers, and other project personnel. Along with the cirriculum, the Systems Engineering Leadership Development Program (SELDP)—designed specifically to accelerate the development of core engineering competencies and leadership skills— ensures that well-rounded engineers are equipped to lead complex project teams.
The organization also recently integrated its curriculum and knowledge services arm to help emphasize the dissemination of knowledge resources and lessons learned throughout NASA. All these efforts work toward improving project team performance, which is essential to mission success.
Roger Forsgren, agency chief knowledge officer of APPEL KS, says, “We are doubly blessed because we not only have a workforce that enjoys learning new ideas and concepts, but we also have an upper management that understands the need for training and enhancing the skills of their employees. Our NASA stakeholders—both our course attendees as well as our management—are very demanding as far as the quality and content of their training, and that’s a good thing because we work together to enhance on-the-job performance and contribute to mission accomplishment.”
Tracking Success
The APPEL KS training program routinely collects evaluation data from more than 130 courses it delivers each year across NASA’s centers. However, a recent training impact study of the curriculum was the first time that NASA designed and implemented a structured, four-level evaluation plan. Data points were collected in a blended fashion from multiple sources, including written surveys, interviews, and focus groups. The study yielded positive surprises, including the fact that program participants were excited to provide high-quality quantitative data, as well as the qualitative context to bring it to life. The cumulative data tell a strong story of NASA mission support. This was also the first time the organization followed up with course participants back at work using a 45-day delayed evaluation to collect data on the degree to which they applied what they learned in the training program. The new approach of collecting additional data through interviews, surveys, and focus groups helped validate the coordinated approach and supported leading indicators to success.
For example, one of the participants in the Assertiveness Training course, which was part of the curriculum, reported, “Because I’m asking more questions at work, we are finding more problems and bringing them to light earlier. We are planning some technical testing right now, and I’m not afraid to ask … ‘Why are you doing it this way?’ Before the course, I would have called those stupid questions. But now I see my questions are uncovering unexamined behaviors and assumptions and may be keeping us out of trouble down the road.”
Almost 80 percent of the respondents in the training evaluation impact study indicated they successfully applied to their job critical behaviors that they learned in the course. Critical behaviors are the few key behaviors that employees will have to consistently perform on the job to bring about targeted outcomes. More than half of the respondents shared that the course itself contributed to their success.
SELDP also included the first four-level blended evaluation plan for NASA. As a requirement of this leadership program, participants take part in a ninemonth rotational assignment away from their home center that enables them to learn engineering competencies and practice new leadership skills with these teams before returning to their work site. Upon return, others often view them differently because they have increased confidence, raised awareness of their own strengths and weaknesses, and a better understanding of how to leverage differences among project team members to advance team performance.
Interviews with mentors, confirmation of assignment work products, and self-evaluation all contributed to the positive impact. All participants in the most recent SELDP class took on either a more complex project or were assigned a high-level position within the NASA center upon their return. The belief is that much of the success is due in part to the structured approach requiring continuous touchpoints, monitoring, and reinforcement during program assignments.
Necessities for Success
Program success depends greatly on having several necessities for success in place up front. Necessities for success are defined as prerequisite items, events, conditions, and communications that help leverage success or head off problems before they reduce the initiative’s impact. NASA’s identified necessities for success included:
- stakeholder education and engagement
- a sound communication plan
- senior management approval
- participant career development resources for planning and goal setting
- prepared data collection methods and templates
- identified sponsors, mentors, and coaches
The training program can only impart knowledge and skill; what is most important is how participants apply or modify behaviors learned in training when they get back to work. As such, NASA incorporated critical behaviors for the training curriculum and SELDP. A system of support and accountability is needed to help with the transfer of learning and the application of behaviors. For this reason, NASA put in place required drivers—or processes and systems that monitor, reinforce, encourage, and reward performance of these behaviors when participants returned to their jobs—to enhance the training impact (see sidebar). The first four-level evaluation plan implementation at NASA is ongoing and sets the stage for future success. NASA’s additional future plans include putting mentors and advocates in place for SELDP who closely monitor progress and help the participants apply what they have learned. Further, the SELDP coach remains a valuable resource to promote growth and assist NASA engineers with navigating in changed environments.
The training team will enhance its efforts and increase follow-up with supervisors and project managers by sending surveys and conducting interviews to solicit input regarding how much they’ve seen the employees they manage applying critical behaviors. Currently, supervisors and managers receive a list of tactics on how to provide support in the form of encouragement and monitoring. A few examples of these tactics include:
- asking for an update on employees’ action plans developed during the course
- having discussions about their team members’ key takeaways from the course
- coaching team members on how they are performing what they learned in the class now that they are back on the job
- taking time during meetings to recognize team members for their professional development efforts.
The training website will continue to evolve to meet the changing needs of NASA’s technical workforce with inclusion of guidance on what to do before, during, and after a course; sample scripts for conversations with supervisors; and recommendations for what supervisors can do to enhance team performance.
NASA will implement a broader approach to collecting postcourse feedback from participants and sharing success stories on the website. Another effort for continued success will be to expand the required drivers to include clearer, more tangible documented steps as a reminder model of what participants learned. In several recent pilot courses, many participants shared this request.
Every program or project NASA embarks on involves a technical team. The Building and Leading NASA Teams curriculum category includes 11 key courses that enhance skill areas such as communication, team membership, team leadership, and others relative to improving project team performance. NASA will deliver 46 of these courses in fiscal year 2018 with an opportunity to influence the learning impact of possibly 1,000 learners.
SELDP, which already has many of the driver components in place because of the program’s structure, seeks to move toward a somewhat different type of accomplishment. Going forward, a planned goal for this program is to better determine a more direct correlation between SELDP project leaders and a reduced number of required resolutions that are identified during a project review.
Training is a Process
Any training professional embarking on a similarly critical program can learn valuable lessons from NASA. First, upfront stakeholder communication is vital to the success of any important initiative. Find a program sponsor and get that person to publicly support your program. You also will need boots on the ground in the form of managers and supervisors who advocate for successful implementation. Even with this support, plan numerous touchpoints along the journey. Don’t think, “They know it, so they will do it.” Remain involved after the training course, and be of service and support during implementation on the job.
Finally, remember that human factors are just as meaningful as the numbers. What do you remember most about the program results reported in this article—the numbers or the testimonial of the engineer who now feels confident to meaningfully question the course of action to ensure that quality considerations are taken into account?
View the role of training as a process before, during, and after formal training. Be a partner in execution and performance, and do all you can to create results. Consider building a similar path up your proverbial mountain that, when executed properly, will lead to increased likelihood of business or mission success and will show business counterparts that you are not just an L&D cost center but a true strategic partner.
This article was originally published in the Association for Talent Development’s quarterly publication TD Magazine.
