Back to Top
There’s No Shame in Not Knowing: Lessons Learned while Producing NASA’s Virtual PM Challenge

Space Shuttle Endeavour on Launch Pad 39A at NASA’s Kennedy Space Flight Center in Florida, for its final flight, the STS-134 mission to the International Space Station.

Photo Credit: NASA

Since 2013, the NASA Virtual Project Management Challenge (VPMC) has examined trends in program and program management and shared lessons learned with NASA civil servants, contractors, grantees, and the general public.

The VPMC shares lessons via periodic, interactive live webcasts that are recorded, captioned, and made available on demand from the NASA VPMC website. In May 2015, NASA’s Academy for Program/Project and Engineering Leadership (APPEL) was presented with an opportunity to become the producer of the VPMC. APPEL’s mission is to promote learning among NASA program managers, project managers, and systems engineers. The APPEL team carefully examined this opportunity (see sidebar), and determined that it would accept the role of VPMC producer.

As expected, there was a steep learning curve. Even though APPEL had more than 28 years of experience producing learning events for NASA’s technical workforce, that previous experience was almost entirely in delivering face-to-face courses. Presenting content live to a virtual audience would require APPEL to demonstrate competency in new areas. The APPEL team accounted for this capacity building in their strategic plan for the initiative. What the team did not expect was the need to plan for re-planning. Further, the team identified a critical meta-lesson learned about the inherent risks in sharing lessons learned live in front of a virtual public audience.

Three valuable lessons learned while producing the VPMC

In July 2015, the APPEL team began to enact the strategic plan for producing the VPMC. The first VPMC session that APPEL produced was in September 2015. By the time this article goes to press, APPEL will have produced three VPMC sessions (one in fiscal year 2015 and two in fiscal year 2016) and will be on track to produce an additional three sessions in fiscal year 2016. Along the way, the APPEL team has identified a number of lessons learned that the team was able to immediately apply while continuing to identify additional lessons.

Fitting content to format using pre-session interviews
APPEL’s first VPMC session featured Bo Bejmuk, a retired NASA contractor from Rockwell and Boeing who worked on the Space Shuttle program among other projects. The session was titled “Building Your Systems Mentality: Using Systems Engineering & Integration to Solve Project Challenges” and was held on September 22, 2015. In previous briefings for NASA leadership, Bejmuk had identified 35 distinct lessons learned from the Space Shuttle—more than could be shared in a single VPMC session. The APPEL team had to develop a means to reduce Bejmuk’s content to fit into an appropriate VPMC format. Additionally, Bejmuk had presented versions of his Shuttle lessons learned briefing more than 25 times for various NASA audiences. Thus, the APPEL team had to ensure the content presented at the VPMC was fresh for Bejmuk and relevant for the VPMC audience.

The team learned that it was critical to have face-to-face pre-VPMC session interviews with presenters. These interviews accomplished several tasks. First, they started developing the collaborative relationship needed to support good on-camera rapport during a live web cast. Second, they allowed the presenter and moderator to identify the critical insights around which to build the VPMC session content. In Bejmuk’s case, each theme captured aspects of multiple lessons learned he had previously identified. Third, the pre-session interviews supported the development of the VPMC content flow – a tool that outlined the semi-structured interview that would take place on camera between Bejmuk and the moderator. Having this content flow developed in advance allowed the moderator to more effectively facilitate the conversation with Bejmuk and provided content that the APPEL strategic communications group could use in pre-session and during-session messaging on social media.

Resisting the endemic use of PowerPoint
The APPEL team applied these lessons learned during the second VPMC they produced. This session was titled “Why Don’t They Just Give Us Money? Project Cost Estimating and Cost Reporting” and took place on November 17, 2015, featuring Doug Comstock and Mary Beth Zimmerman from NASA headquarters. Lessons learned during the Bejmuk session proved useful. However, the team identified additional lessons learned around the use of slides.

VPMC sessions are web cast live via a user interface that allows the viewer to see the presenters and the presenters’ slide decks. Historically, VPMC sessions utilized a large number of slides as a part of panelists’ presentations. A critical lesson learned from this second APPEL-produced VPMC session was that in terms of supporting the learner experience, slide content is secondary to the quality of the on-air conversation among presenters and moderators in terms of supporting the learner experience. Though a reliance on slide decks had become a part of the NASA and VPMC culture, there was substantial evidence that over-reliance on slide decks reduces the quality of knowledge sharing and subsequent decision making. The APPEL team was cognizant of findings from the Columbia Accident Investigation Board report that indicated the “endemic use of PowerPoint briefing slides” (CAIB report, p. 191) was a serious problem for NASA, hamstringing the agency’s ability to analyze data and make decisions (i.e., identify and apply lessons learned). For most of this session, the team utilized a nominal PowerPoint format that contained only the question/topic being currently discussed during the live web cast. The lesson learned was that focusing on the content of the conversation—instead of the construction, revision, coordination, and required approvals for slide decks—was a more effective use of the team’s resources and better supported the VPMC’s mission.

Guarding against “magical thinking”
The team producing the VPMC recognized that this new-to-APPEL undertaking was similar to the work NASA teams engage in while embarking on a new project. The team, however, failed to also recognize the need to be wary about a critical threat common to NASA projects: the optimistic bias. NASA employees frequently describe how they are involved in “one-of-a-kind missions.” This is completely true given the many dimensions of NASA missions including, among other things, the science driving the mission, the technology employed in the pursuit of the science, and the environment in which the technology must operate (e.g., low earth orbit, lunar, Lagrange points).

Though each NASA mission is undoubtedly unique, they are comprised of activities and products with which NASA has considerable experience. The same applied here for the APPEL team. As noted earlier, though APPEL had never produced a live and interactive lessons learned web cast before, the program had more than 28 years of experience producing learning events. The team leveraged that expertise to develop processes and a schedule of activities. However, when APPEL’s production of the VPMC started, the team realized their schedule of activities did not completely capture the necessary effort.

Given the aggressive nature of the VPMC production schedule (i.e., produce a session every other month), the team had to engage in frequent re-planning to account for efforts that took longer than expected or that were not predicted. This is not unusual in project management. Neither is the optimistic bias that tends to occur during re-planning efforts when team members routinely overestimate their expected productivity and/or underestimate the challenge of the tasks being re-planned. The result is what is sometimes referred to as “magical thinking” often expressed in language such as:

  • “We’ll make it work”
  • “…and then a miracle will happen, and then we’ll…”
  • “No problem. We can do that”

As a result of not effectively controlling the optimistic bias during the re/scheduling of activities for the first three APPEL-produced VPMC sessions, the APPEL team resolved to devote time to analyzing and re-estimating the durations of all VPMC-related activities. The end product will be a revised integrated process map and schedule for VPMC session production. The team expects to complete that work by May 2016.

On the VPMC set with Bo Bejmuk (left) and Ramien Pierre (right). Photo Credit: NASA

An unexpected meta-lesson learned: The risks of sharing lessons learned

The APPEL team identified an additional lesson learned that merits specific attention because it speaks to the very nature of the NASA Virtual Project Management Challenge, specifically all lessons learned activities involve some degree of personal and professional risk for the sharer.

The APPEL team routinely facilitates knowledge sharing on topics that are non-controversial: communicating technical issues, project risk management, scheduling and cost control, etc. What the team did not account for in the planning phase was the degree to which insights shared during the VPMC would present some level of personal or professional risk for VPMC participants. Examples include:

  • Topics identified during pre-session interviews being dropped from the final content flow in order to avoid implying other project managers or organizations were not effective in executing their duties
  • “Minders” reviewing questions asked during VPMC sessions or pre-recorded interview format in order to ensure that nothing too “political” was asked or said

In an article on what successful project managers do, Alexander Laufer, Ed Hoffman, Jeffrey Russell and W. Scott Cameron made a relevant observation:

“When upper management fosters an organizational climate that embraces problems as an inherent part of the project’s progression, project managers are able to detect and resolve problems more successfully.”

Based on our experience, a friendly insertion (in italics) would be “…project managers are able to detect, resolve, and share problems more successfully.” Sharing lessons learned requires identifying where things went wrong on either your watch or your predecessor’s watch. One thing the APPEL team discovered is that the ability of VPMC presenters to candidly engage in live webcast conversations can be undermined by presenters’ impressions of degree to which their upper management values and embraces the public sharing of lessons learned. The APPEL team is currently identifying ways to support presenters’ ability to share lessons learned, as such is critical to the very mission of the NASA Virtual PM Challenge.

Conclusion

APPEL has learned many valuable lessons since taking on the production of the VPMC. We have recognized the importance of: providing opportunities to establish a good rapport between the moderator and presenters; having engaging delivery methods focused on the presenters and their message as opposed to their PowerPoint slides; valuing the significance of preplanning for unplanned activities and challenges; and identifying concerns regarding the public sharing of lessons learned. Although many of the processes for each VPMC were repetitive, we learned new lessons with each delivery. We not only focused on what went wrong, but also captured our successes, which helped motivate the team. Additionally, we found that correctly documenting lessons learned during and at the end of a project makes all the difference. We are looking forward to continued learning and successful productions of future VPMCs for our audience.

Resources:

NASA Academy for Program Project and Engineering Leadership: /

NASA Virtual PM Challenge web site: https://www.nasa.gov/offices/oce/pmchallenge

NASA Lessons Learned information system: http://llis.nasa.gov/

Ramien Pierre is a member of the APPEL Curriculum Team and moderator of the NASA Virtual PM Challenge.

Donna Wilson is the APPEL Curriculum Manager and has been involved in the APPEL program since 2005.


APPEL Develops a Strategic Plan to Produce the NASA Virtual PM Challenge

From May 2015 to July 2015, the APPEL team analyzed the opportunity with which they were presented by the NASA Office of the Chief Engineer: take on the role of producer for the NASA Virtual Project Management Challenge (VPMC). The results of the analysis would determine whether the team would take advantage of this opportunity. The APPEL team’s analysis was guided by four questions:

  • What is the goal of the NASA Virtual PM Challenge?
  • What does it take to be successful at that goal?
  • Does APPEL have what it takes to be successful at that goal?
  • Based on this analysis, should we undertake this initiative?

What is the goal of the NASA Virtual PM Challenge?

The APPEL team quickly confirmed that the VPMC’s goal would not change under its watch: provide useful and timely insights primarily to NASA program and project managers.

What does it take to be successful at the goal?

The APPEL team facilitated a number of meetings with members of the VPMC production team, NASA Office of the Chief Engineer, and NASA Engineering Safety Center to identify things that would facilitate the VPMC successfully achieving its goal. Additionally, the team engaged those stakeholders in discussions designed to identify risks and obstacles that would need to be mitigated or overcome in order to successfully meet the goal. The team defined the following as critical capabilities:

  • A system for generating/curating content for the VPMC
  • Ability to maintain a consistent presence as perceived by audience/key stakeholders
  • A learner-focused format that would be used to structure VPMC sessions
  • The means to expand the distribution of VPMC sessions

Does APPEL have what it takes to be successful?

The APPEL team recognized that they did not have all of the critical capabilities to successfully produce the VPMC. The team would need to develop some additional capacities and tools, as well as adapt some of its existing capabilities. This was to be expected: APPEL was embarking on an initiative that was related to its historical work but differed in a number of key ways. The APPEL team confirmed that they would need to create a system by which potential VPMC topics could be gathered, vetted, researched, and finalized. The team recognized that APPEL already had an analogous system for developing new APPEL courses. That system was designed to develop new courses or revise existing courses over a period of months. To produce VPMC sessions, the system would need to be adapted to produce content in a matter of weeks. An accelerated development schedule would support the second key to success: maintaining a consistent presence as perceived by audience/key stakeholders. Based on its research, the team set the goal of holding VPMC sessions every other month in order to produce six sessions in fiscal year 2016.

Identifying an appropriate learner-centered format for the VPMC presented a particular challenge for the APPEL team. Historically, VPMC sessions were structured similarly to conference panel sessions. A moderator would introduce multiple presenters who would give discrete but related presentations and then facilitate an audience question and answer session. The APPEL team determined this format was an efficient way to deliver content to a virtual audience, but it might not always be effective in terms of supporting audience learning. Leveraging their experience in delivering learner-centered face-to-face courses, the APPEL team identified the essential characteristics of a learner-centered experience but could not identify the one VPMC session format that would embody those characteristics for all potential VPMC session content. At that point, the team determined the best option was not to have one standard VPMC session format (e.g., panel presentation) but to have multiple formats (e.g., panel presentation, interview, news magazine), which would be selected and deployed depending on each session’s content. The team predicted an additional benefit of the multiple-format approach, which would be greater audience engagement resulting from the variety in delivery styles.

Additionally, the APPEL team proposed that the VPMC have a consistent moderator, as such would address a number of issues identified during conversations the team had with key stakeholders during the planning phase. Historically, the presenters for each VPMC session would provide their own moderator. The moderator would facilitate the live web cast while also serving as the liaison between the VPMC technical crew and the subject matter experts who would be presenting during the session. This meant there was a learning curve that had to be navigated with each and every VPMC session since the production team would be different each time. Moreover, it meant that there was not a consistent “face” to the VPMC sessions since the people on camera would be completely different from session to the next. Thus, a consistent moderator would flatten the technical learning curve that had to be navigated from one session to the next. Over time, a consistent moderator would provide a recognizable face for the VPMC audience. Finally, a consistent moderator could add value by identifying content connections across VPMC sessions during the live web casts.

At the point when APPEL was undertaking production of the VPMC, it already had a robust communications function capability of developing and distributing content to NASA audiences via email and web. The VPMC reboot plan called for the close support of the APPEL Strategic Communications team, including developing VPMC-related content before sessions, to be presented during the sessions, and to be distributed after VPMC sessions.

Based on this analysis, should we undertake this initiative?

As a result of their analysis, the APPEL team decided to pursue the opportunity of producing NASA VPMC sessions. Though the team recognized they would need to navigate a learning curve to be successful, they were confident they had sufficient capability to do so. Further, supporting the VPMC was aligned with the APPEL mission of developing NASA’s technical workforce.

About the Author

Share With Your Colleagues