By Scott Pace
NASA Administrator Michael Griffin established the Office of Program Analysis and Evaluation (PA&E) to supply independent studies and assessments of NASA programs for the Office of the Administrator. To ensure its objectivity, the office has no operational line authority or direct responsibility for any mission areas—it is a staff function. We don’t say “go” or “no go;” we come up with observations and options that help those who do make decisions make them on the basis of the best available information and “full disclosure” of alternatives.
PA&E consists of five divisions: studies and analysis, strategic investments (which focuses on budget planning and programming), cost analysis, an independent program assessment office, and mission support that administers resources for PA&E. We are not an auditing organization—that is, our aim is not to find fault or mistakes but to offer clear, objective analysis that includes alternative solutions to potential problems. For instance, if we have doubts about a program having sufficient reserves of money and time to achieve a critical technical objective, we may identify possible responses such as conducting tests to demonstrate technical readiness, adding reserves, or accepting the identified risks.
To do their jobs effectively, PA&E staffers need multidisciplinary expertise. They need to see the “big picture” and understand areas such as economics and statistics as well as engineering. They regularly supplement their own technical knowledge with the expertise of NASA and industry engineers, including those working on the programs being evaluated, since they are the ones likely to be closest to the problems. Throughout the analytical process, we have to manage the tension between independence and collaboration. We work closely with program teams to understand their work and help them succeed, but we must simultaneously maintain enough distance to avoid the blind spots and optimism that often accompany deep engagement. Optimism is good—it is one source of NASA’s success—but it can lead to a failure to recognize and adequately allocate resources to areas of risk. Cost and schedule seem especially susceptible to optimistic thinking, so it is important for PA&E to provide reality checks wherever possible. As Michael Griffin has said, “You shouldn’t grade your own homework.”
Recent Examples
Some recent decisions based on PA&E’s work have helped avoid unnecessary expense and delay. The office studied the question of whether RS-68 rocket engines could be used for the new exploration mission, or whether it would be preferable to use Space Shuttle Main Engines. At first, technical experts were skeptical about using the RS-68, but careful analysis showed that, given some launch vehicle modifications, it would be able to provide the necessary performance. The decision to go with the RS-68 will save several billion dollars over several decades. PA&E also examines infrastructure needs. The groups developing the James Webb Space Telescope and the Crew Exploration Vehicle were originally planning to use the same thermal vacuum chamber at Johnson Space Center. A PA&E team, working with the centers and mission directorates, analyzed the potential impact of common needs for the same testing facility and worked to develop new plans to avoid conflicts that could have delayed or compromised the technical performance of both programs.
PA&E, 7120.5D, and New Missions
Mark Saunders, the PA&E director of the Independent Program Assessment Office, has been directly involved in developing the new 7120.5D guidance on program and project management. The new clarity the document provides regarding the review process, defining and standardizing when major reviews occur and what must be accomplished before a project or program can move on to the next stage, also clarifies when and how PA&E will carry out its analyses and make recommendations. But 7120.5D leaves room for the flexibility that different kinds of programs require. A research program shouldn’t be treated the same way as a flight project, for example. The aim of 7120.5D is mission success, not making everything fit the same template.
The projects and programs that make up NASA’s new space exploration mission are extremely complex and interrelated, with each element building on the one before and laying the foundation for those that follow. Choices—some of them very difficult choices—will have to be made with the overarching challenges and aims of the mission in mind, so the level of analysis needed to make informed decisions will be especially great. Making the best possible decisions based on the best possible information will be doubly important as NASA prepares to send human beings away from the Earth for long periods of time.
New missions of exploration to the Moon, Mars, and beyond will make great demands on all involved projects and programs and will undoubtedly create tensions in the management of requirements, resources, and schedules. We need to base our decisions on clear, consistent priorities. Not everyone will be happy with every decision, but if the supporting logic is fully and clearly evident, I believe those decisions will be respected by the NASA community, our partners, and stakeholders. We are mindful that the Columbia Accident Investigation Board complained that NASA has been too much a “PowerPoint culture;” the limitations of that tool sometimes obscured information needed for decisions. PowerPoint has legitimate uses, but it is no substitute for detailed analysis that explains not only what was decided but why that choice was made. PA&E’s job is to contribute to mission success by helping the senior leadership of the Agency make better decisions through analysis and thereby enhance the credibility, trust, and cooperation we need to explore new worlds.
About the Author
 |
Scott Pace is the Associate Administrator for Program Analysis and Evaluation at NASA. In this capacity, he is responsible for providing objective studies and analyses in support of policy, program, and budget decisions by the NASA Administrator. |
