Independent mechanical walkdowns of spacecraft are sometimes credited with discovering discrepancies that could have interfered with mission success.
The Mars Science Laboratory (MSL) spacecraft is an extremely complex system that was designed and built by NASA’s Jet Propulsion Laboratory (JPL). Prior to stacking MSL’s Curiosity Rover, Descent Stage, Cruise Stage, Backshell, and Heatshield for launch at NASA’s Kennedy Space Center, a final mechanical walkdown of each vehicle was performed in August 2011 to identify workmanship discrepancies that would not be discernible after the vehicle was stacked. This included cabling, multi-layer insulation, structural items, close clearances, obstacles to spacecraft separation, and paperwork closeout and vehicle readiness.
The objective of the detailed examination of the spaceflight hardware in its final configuration was to assess the overall readiness to proceed with the MSL launch from an “as-built” hardware perspective. The intent was to include walkdown participants – a combination of generalists and specialists — who could provide a fresh look at the hardware in the Assembly, Test, and Launch Operations (ATLO) configuration. The final mechanical walkdown for MSL took two days to complete and discovered discrepancies such as sharp edges and close clearances.
Prior to MSL, JPL relied only on interim inspections and walkthroughs conducted during ATLO and did not conduct formal final walkdowns of in-house builds. For out-of-house builds performed for JPL, such as the Nuclear Spectroscopic Telescope Array (NuSTAR) Project, the system contractor is responsible for vehicle readiness and paperwork closeout. Based on experience gained with the MSL walkdown process, walkdowns of the NuSTAR spacecraft were performed by JPL during final integration. The independent process produced significant findings and observations. To gain greater project visibility into contractor preparations for spacecraft integration, an additional independent assessment step may be useful.
Lesson Number: 6636
Lesson Date: April 10, 2012
Submitting Organization: Jet Propulsion Laboratory
HIGHLIGHTS
LESSONS LEARNED
- A final mechanical walkdown of an in-house build (MSL) — conducted formally prior to spacecraft integration with the launch vehicle — discovered, tracked, and corrected discrepancies in the as-built configuration.
- For an out-of-house build (NuSTAR), a less comprehensive final mechanical walkdown of the as-built configuration was performed by JPL independent of the contractor and produced and documented findings and observations.
RECOMMENDATIONS
- For all in-house spacecraft builds, conduct a final mechanical walkdown of the spacecraft prior to stacking or encapsulation for launch.
- For out-of-house builds by a spacecraft system contractor, conduct an independent final walkdown and assessment at the contractor facility and the launch vehicle integration site that includes thorough inspections of the spacecraft exterior surfaces and, at least, randomly selected spot reviews of closeout procedures and processes.
Consult the lesson learned for complete lists.
Ben Thoma
Credit: NASA
NASA Jet Propulsion Laboratory Mars Sample Return, Lander Mechanical Product Delivery Manager Ben Thoma on the importance of this lesson learned:
I think it’s important to accept the need for independent reviewers. It’s not a criticism or reflection of a poor performing team if issues are found. When the same team is looking at the same hardware day in and day out for weeks or months, it’s easy for them not to spot potential concerns that are right in front of them.
An independent mechanical walkdown reduces the chances for issues — from minor performance degradation to complete mission failure. It also gives the integration team all the way up to project management additional confidence that their hardware is ready to proceed into the next phase of build, test, or launch.
It’s easy to put it off or completely skip the final walkdown due to schedule pressures or feelings that ‘We’ve got this. We have a good team, and we don’t need another review.’ I have yet to participate in a mechanical walkdown — and I’ve participated in dozens on both the receiving and giving sides — where the integration team was not thankful and appreciative of the observations and in agreement that it was value added and worth the time it took to prepare, execute, and respond.
You want an independent and ‘fresh’ set of eyes on the hardware. Typically, you are looking both for people who are completely independent and do not have any knowledge of the intended design as well as folks who participated during the design and know what the as-built configuration should look like. The walkdown participants are not part of the integration and assembly team, which is what gives the ‘fresh’ set of eyes. Similar to when you have a home construction project, not only do you want the foreman to inspect the different teams’ work, but you have an independent inspector who makes sure everything looks appropriate.
Most of the observations that come from the final mechanical walkdown might be considered minor, but it’s obviously better to err on the side of caution given the cost and effort that go into our projects and missions. It only takes one finding that could have prevented complete mission failure to make the walkdown process worthwhile.
Typically, the areas that will have the most observations or findings are the ones that are hard to model in the CAD system, making it more difficult to reflect the exact design intent. This typically includes soft goods like cabling or blankets. Do the cables go over any sharp edges? Are the blankets adequately secured such that any billowing that occurs during launch depressurization won’t obstruct any field of view or hinder any separations or deployments?
Two examples come to mind from my experience. I was the ATLO Mechanical Lead on the Mars Science Laboratory (Curiosity) project. One significant finding occurred at Kennedy Space Center during our final ‘pre-stacking’ of the spacecraft walkdown. A walkdown member observed that we had some cable harness in relative proximity to our reaction control thrusters, which get extremely hot while they’re firing during our entry phase at Mars. It was conceivable that heating could have caused the cabling to melt and no longer function, which could have been mission-ending. We didn’t have time to run the analysis, and because it was identified during the walkdown, we were able to quickly respond and put in some thermal protective material over that area of cables.
The second example was from a subcontracted project where I led the mechanical walkdown. We discovered that there was a Remove Before Flight (aka ‘red tag’) item — a shorting ‘jumper’ bar that went across terminals of a thermal battery (in place to prevent accidental discharge during ground operations) — that was not identified on their official Remove Before Flight list. While it’s very likely that the team would have remembered to remove that shorting bar, if they had not, it would have been mission-ending.
It’s important to note that these walkdowns are not intended to be reviewing or questioning the design. Rather, they are looking at the “as-built” configuration, notably workmanship areas, to help provide additional assurance that the vehicle is ready to proceed to its next phase of build, test, or launch. The walkdown team provides observations — not actions. It is up to the integration team to determine if and how they respond to any of the observations.
Related Resources
Mars Science Laboratory – Curiosity
Spotlight on Lessons Learned is a monthly series of articles featuring a valuable lesson along with perspective from a NASA technical expert on why the lesson is important. The full lessons are publicly available in NASA’s Lessons Learned Information System (LLIS).
If you have a favorite NASA lesson learned that belongs in the spotlight, please contact us and be sure to include the LLIS Lesson Number.
