Nobody’s Perfect: The Benefits of Independent Review

By Mark Saunders and James Ortiz

During a 1995 independent review of the development of Mars Pathfinder, Dr. Mike Griffin, a member of the review team, asked the project team how the spacecraft’s radar would determine the distance of the spacecraft from Mars’s surface while swinging back and forth below the parachute. Discussion revealed that the team did not have an adequate test to prove that the radar would work as needed. As a result, the project developed a special test program that may have prevented a catastrophic failure.

In other words, the smartest people can miss things. NASA has sent men to the moon, built and launched telescopes that can see billions of years into the past, discovered water on Mars, and sent spacecraft beyond our solar system. Dedication, technical excellence, and “can-do” optimism have made these dreams come true. We have also unfortunately experienced the agony of spacecraft failures and loss of life.

Dedication and technical excellence can’t overcome the fact that we are just not perfect, and sometimes our optimism threatens mission success. Having a fresh set of eyes look at our work can help us see what our own blinders and mental filters may hide. This is the essence of the independent life-cycle reviews specified in NASA’s procedural policies. As former NASA Administrator Mike Griffin said, “You cannot grade your own homework.” Independent experts review program and project “homework” with team members to find, and help them correct, weaknesses that could turn into problems or disasters later on.

The Evolution of Independent Review

Independent review has existed at NASA for decades, in many different guises. Centers have used it to ensure technical designs and products will perform as expected. After the Mars Polar Lander and Mars Climate Orbiter failures in 1999, the Jet Propulsion Laboratory revamped its internal independent review process to improve the chances of catching the types of flaws that led to those losses. Mission directorates commissioned their own review teams to independently verify what they were hearing from centers working on their projects. In the mid-nineties, the NASA Administrator directed that the Independent Program Assessment Office be established so NASA could confidently promise its stakeholders we could deliver our missions on cost and on time. Sometimes the membership of these multiple review teams outnumbered the project management staff, and the multiplicity and variety of the reviews took up an inordinate amount of the project’s attention.

The Gravity Recovery and Interior Laboratory mission relies on twin spacecraft flying in formation above the lunar surface to investigate the moon’s gravity field in unprecedented detail.
Image Credit: NASA/JPL

After the 2003 Columbia tragedy, NASA revised its governance structure, improving the checks and balances between organizational authorities, and rewrote its program and project management policies in part to ensure that technical and managerial concerns about potential problems would be heard and adequately evaluated. Among the changes incorporated in NASA’s new procedural requirement NPR 7120.5D (which we both worked on) was a new and, we hoped, more effective independent review process.

The new requirements eliminated the earlier multiple review boards and called instead for a single standing review board (SRB) that would evaluate programs and projects at all their life-cycle milestones. We also strove to ensure that the reviews would be collaborative and constructive, rather than adversarial. The fact that program and project teams can suggest members whose expertise they respect contributes to the collaborative character of the reviews. (They do not have approval authority, since the SRB is independent.) Also, the core board membership stays the same throughout the mission, fostering trust and good communication. Typically, the program or project participates in the SRB kickoff meetings where the rules of engagement for all life-cycle reviews are set—and the project and review teams have an opportunity to establish a good working rapport.

Another change was that the review process, mainly used for robotic missions previously, would be applied to human spaceflight missions as well. Typically, SRBs for large, category 1 and 2 spaceflight projects have a chairperson, a NASA review manager, and approximately thirteen experts covering the basic disciplines required to execute the project (for instance, propulsion and systems engineering, as well as cost and schedule analysis expertise). Smaller, category 3 projects may not require a formal review manager and may be about half the size.

Once the independence of each member is verified, the SRB chair, with support from the review manager, organizes the board and submits the names of proposed members to the convening authorities, who include NASA’s associate administrator, the mission directorate associate administrator, the chief engineer, and the Program Analysis and Evaluation Office associate administrator. Their approval is based in part on a detailed review of the nominees’ qualifications.

The overall purpose of these independent reviews is to accomplish the following:

• Provide the program/project with a credible, objective assessment
• Supply NASA senior management with an independent view of program/project performance and identify whether externally imposed impediments to the program/project’s success are being removed
• Present a credible basis for a decision to proceed to the next phase

Benefits to Programs and Projects

Developing complex systems is iterative and recursive. Teams regularly circle back and revisit earlier work as new information becomes available. This iterative process happens throughout missions, but programs and projects separate design and development activities into logical stages punctuated by major milestone reviews, such as a preliminary design review, that are designed to answer a couple of basic questions:

• Have we done our work sufficiently well and completely in the previous stage to justify continuing on our current path?
• Are our proposed actions, plans, and resources sufficient to complete the next stage as well as the overall development, launch, and operations?

In preparation for these independent reviews, the program or project team conducts its own internal reviews to examine its plans, technical approaches, and programmatic commitments. The team assesses major technical and programmatic requirements along with the system design and other implementation plans and compares technical and programmatic performance with earlier predictions. This preparation for the milestone review gives the team an opportunity to step back from the narrower focus of its daily work and examine its progress holistically. The development team has a chance to examine the assumptions and analyses that support the conclusion that they are, in fact, at the required level of maturity and are ready to proceed.

The SRB’s role is assessment; it does not have authority over any program or project. Its review provides expert assessment of the technical and programmatic approach, risk, and progress against the program or project baseline and readiness against criteria in NPR 7120.5D and NPR 7123.1A. The depth of an SRB review is the board’s responsibility and must be sufficient to permit the board to understand whether the design holds together adequately and whether the analyses, development work, systems engineering, and programmatic plans support the design and key decisions that were made.

The review objectively assesses the following:

• Adequacy and credibility of the technical approach (requirements, architecture, design)
• Schedule
• Resources
• Cost
• Risk
• Management approach
• Compliance with agency policy (NPR 7120.5D, NPR 7123.1A)
• Readiness to proceed to the next phase

Recently, the agency has begun using more probabilistic techniques for budgeting programs and projects, including budgeting to a target joint confidence level for cost and schedule. Review team programmatic analysts are shifting from performing purely independent estimates of cost and schedule to working in parallel with the program or project control offices, using the same cost-and schedule-estimating methodologies and tools. This approach provides a more efficient, and less adversarial, evaluation of budget risk.

Individual SRB members usually offer recommendations to improve performance or reduce risk. These recommendations and the SRB findings are collected in the board’s report. The board chair and the program or project team ensure that all the facts are correct by vetting the report with the program or project manager. Once that process is complete, the team from the program or project under review determines which of the board’s findings and recommendations to accept, modify, and implement, and presents its response and action plans to senior management, up to and including the decision authority, at the same time they receive the SRB’s reports.

Experienced project team members know that identifying risks and problems early makes it easier and less expensive to deal with them. The SRB meetings and reports are key to finding and fixing those issues as early as possible. As in the Mars Pathfinder case, review boards have caught problems that could have turned into major difficulties later. A member of the Gravity Recovery and Interior Laboratory (GRAIL) SRB, for example, identified a range safety launch issue—insufficient “inhibits” between the pressurant and the propellant. Although a waiver would have been granted in the past, he knew that the standards had gotten tougher and the range safety folks would not approve the design.

The project decided to add a valve to the propulsion system to fix the problem. This may have prevented GRAIL from having a launch slip or from spending an enormous amount of money to fix it. Similarly, the independent review team for the Magnetospheric Multiscale science mission at Goddard identified the late selection of the launch vehicle (scheduled after the critical design review) as a major risk to the spacecraft design. As a result, the launch vehicle selection was moved to precede the preliminary design review.

Benefits to Management and Stakeholders

The independent review process is a collaborative effort between agency senior management, center management, technical authorities, and program or project management. Each entity plays a key role in establishing, conducting, and reporting independent life-cycle reviews. The head of each of these organizations approves SRB members and the SRB charter (called Terms of Reference). This collaboration ensures that the needs of each organization will be met.

This image is a digital combination of panoramic pictures taken by Pathfinder on Mars and a picture of a lander scale model back on Earth. Sojourner itself is visible inspecting a rock nicknamed Yogi.
Photo Credit: NASA/JPL/IMP Team

The independent life-cycle review process culminates at key decision points when the results of the reviews are presented to the decision authority and his or her management council. At these meetings, the program or project manager presents the findings of the SRB and an approach to resolving the issues identified. The SRB chair and the various levels of management all participate in the open dialogue, offering their views and recommendations about the way forward. Success criteria for all life-cycle milestones are considered, and the decision authority determines if the program or project should proceed, proceed with specific actions to resolve outstanding issues, or not proceed until critical actions are resolved. In some cases, they may direct the program or project team to put additional recommendations into practice. This process ensures that all decisions on how the mission is to be carried out reside with the appropriate authority.

A critical part of this process is assuring NASA senior management that we have the cost and schedule resources required to deliver what we’ve promised. When presenting the results to the decision authority, both the project estimates and the SRB assessment of the estimates help ensure realistic commitments to our stakeholders. Independent estimates now typically fall within 5 percent of the final outcome. In the long run, more realistic commitments will result in increased credibility of the agency with the Office of Management and Budget, Congress, and other stakeholders.

We are using our SRB experiences to continue to improve the review process itself. We periodically meet with the mission directorates, programs, projects, and centers to discuss what we have learned about the process and how to make it better. We have, for instance, developed standard terms of reference for particular classes of reviews to avoid spending time and effort negotiating nearly identical terms for each review. We have also reduced reporting time from several months to thirty days. We expect to continue to learn how to make the standing review board process serve the best interests of NASA missions and all those who benefit from their success.

About the Author

Mark Saunders is the former director of the Independent Program Assessment Office, part of NASA’s Office of Program Analysis and Evaluation. He was responsible for evaluating the agency’s major programs and projects to ensure they were on paths for mission success and ready to roceed through key decision points. He retired in December 2008 and is now consulting part time with NASA.

James Ortiz has served as section head for International Space Station systems training and chief of the Advanced Projects Office in the Mission Operations Directorate, as well as manager of the Johnson Space Center Systems Management Office. He is the deputy director (and current acting director) of the Independent Program Assessment Office within the Office of Program Assessment and Evaluation at NASA Headquarters.