Hard Lessons and Lean Engineering

By Kerry Ellis

Future human space exploration will mean getting beyond low-Earth orbit—and returning safely. Several projects across NASA are working on the challenges that goal presents, among them propulsion alternatives and guidance, navigation, and control. Three years ago, Project Morpheus and the Autonomous Landing and Hazard Avoidance Technology project, or ALHAT, began collaborating on advances in these areas.

Morpheus is a rapid-prototype vertical lander concept testing many ideas at once. It is known as a vertical test bed: a lander that can be adjusted, scaled, and reconfigured to test different design ideas. This makes it a great platform on which to test ALHAT’s sensors and software, meant to detect hazards in real time and adjust flight trajectories to avoid them without human intervention.

Together, they can provide a template for future planetary landers, one that can scale down for asteroid missions or potentially scale up for human spaceflight to Mars.

Lean Engineering

To provide quick technology demonstrations, both projects have focused on keeping their management and engineering approaches lean and mean. For the project managers, this has meant finding ways to document processes and lessons learned with enough rigor to satisfy requirements and benefit current and future projects but without a mountain of paperwork that might prevent rapid design and development. On the engineering side, the philosophy has focused on a test early, test often approach using low-cost materials that can be found commercially or quickly modified.

At the core of these approaches has been collaboration and communication. The Morpheus and ALHAT teams have looked to their peers at other centers to provide their expertise and knowledge to the projects, and to industry partners with experience in operating rapid-prototype projects.

According to Jon Olansen, Morpheus project manager, Morpheus is about 90 percent in-house collaboration spanning several NASA centers, including Johnson Space Center, Goddard Space Flight Center, Stennis Space Center, Kennedy Space Center, and Marshall Space Flight Center. “By keeping the work predominantly in house, our civil servants have learned a tremendous amount,” he said. The team includes experienced personnel as well as those new to the agency and students. “We have all learned so much doing this hands-on work, and I think the benefit to the agency is astronomical. It’s been a great training ground.”

One lesson Morpheus has taken from its commercial partners is finding the right level of documentation. In one instance, the team worked closely with Armadillo Aerospace to see how a small development team operated. They learned how to improve processes for lean development and were able to pass on some NASA knowledge to improve Armadillo’s safety and process measures.

“We decided to pick and choose from procedural requirement 7120.5 and the agency’s project management policies to determine what was applicable to Morpheus. Those policies primarily exist for larger projects and programs, but they’re a great information source for project management if appropriately tailored to your project,” explained Olansen. “We don’t produce a bunch of documents, and we only produce a handful for written signatures, such as range-safety documents.” Everything else is kept online to ensure the project has enough rigor regarding safe operations and capturing lessons learned.

“Engineers like to do things, not write documents,” added Chirold Epp, project manager for ALHAT, “so as a project manager, I have to work a bit to make sure we document what we’ve done, and people can pick it up and understand what we did right and what we did wrong. Our effort has been to document all data whenever we do a field test and ensure it’s readable; otherwise, you can spend way too much time writing documents. We need to do the work. And for good technology development, we believe that’s the right way to go.”

The lean development approach also applies to the engineering itself, often relying on “good enough” solutions that will allow for safe testing and progressive learning in the moment.

On Morpheus, for example, engineers needed to figure out how propellant would slosh within their fuel tanks. Usually, this requires a lot of time creating and analyzing models before development takes place. Instead, one of the engineers went to a hardware store and spent $80 on wood, attachment fittings, four light globes, and food coloring. They put together a simple model of Morpheus’s four-tank structure, filled the globes with colored water, hung it from a single point, and induced oscillations to see how the fluid would slosh within the globes.

“We could see that if we induced an oscillation in one direction, eventually the water would swirl in the globes instead of slosh back and forth, due to the tank configuration. It got us 80 percent of the answer,” explained Olansen. “It didn’t give us every detail, but it gave us plenty of information to design baffles we could put into the tanks to reduce slosh to the point where it’s not an impact to the way we fly. It’s great for a prototype, but it would require more work if we were going to fly in space with a follow-on vehicle. But we now have an 80-percent solution, and it cost us $80 to get it there.”

“Because you don’t always have the money to buy the most expensive and best parts, you’ve got to build something that works, then go out and test,” added Epp. “You just proceed in that fashion and move ahead.”

Both teams follow a build-test-build philosophy. “When you do that kind of testing, things don’t always work how you expect. But you learn, then you go back and do it again,” said Epp.

Crucial to that learning is good communication—across the team and up the chain of management. Since the combined teams include seven NASA centers and a few commercial partners spread out across the nation, much of the communication happens in teleconferences and e-mail, but Olansen and Epp are co-located at Johnson and get folks face to face when needed.

“Whenever we felt it was necessary to get the group together face to face, we would do that. You can do a lot with telecommunications, but sometimes you still need to get together and talk,” said Epp. Early on, the ALHAT team got together four times a year for a few days to review what they were learning and how to proceed. “This year we moved everything initially to Langley Research Center and tested there with the whole team: Jet Propulsion Laboratory, Langley, Johnson, and Draper Laboratories. Then we came to Johnson and brought everyone here to work on Morpheus.”

Crash Landing

After several successful tethered tests at Johnson—where the Morpheus lander was held aloft by crane and its thrusters fired for continuous periods—both teams were anticipating the first free-flight attempt. They began at Kennedy on August 3 with another tethered test to ensure all systems were working as expected. Everything checked out. No shipping-related issues were found.

On August 7, Morpheus made its first free-flight attempt. The vehicle successfully rose a couple feet off the ground but, shortly after liftoff, sensors onboard the vehicle falsely detected an engine burn-through. The rest of the system reacted as programmed: it initiated a soft abort, descended to the launch-pad, and shut off its engines.

“The test lasted probably a total of 7 seconds,” said Olansen. “We brought the vehicle back to the hangar, and we knew immediately it was a false indication, which we fixed.” During that review, the team discovered the lander’s footpads had melted slightly from being in the engine plume. They reached out to the thermal-protection experts at Kennedy for advice. “They came up with a design using excess shuttle materials, implemented it, and built new thermally protected footpads for us in about four hours.”

Two days later, they were ready to try again. Loaded with mass simulators to represent the ALHAT payload—the actual sensors would be used once free-flight tests completed successfully—Morpheus again fired up its engines and began to ascend. Just 0.6 seconds after liftoff, the lander experienced data loss from its inertial measurement unit (IMU), the prime navigation sensor that tells the vehicle where it’s headed.

“Without that data, the vehicle had no way to control itself,” Olansen explained. “It continued to try to respond to the last piece of data it had, which was a slight correction in attitude. As a result, it continually corrected for that pitch error and never received information it was corrected, which resulted in a parabolic flight trajectory.”

Morpheus crashed—and the crash was streamed live on the Internet. The response, both within the agency and from external news media, was immediate. Calls and e-mails came pouring in. Those from the agency, including from Administrator Charles Bolden, were supportive and reassuring. Upper management let the team know immediately the project would continue, and they should work to recover, learn, and improve the next build.

For Olansen, the toughest part of being a manager during the time immediately following the crash was ensuring his attention wasn’t pulled away from his team. Because everything was streamed publicly, there was a lot of attention that required his response. “Instead of responding to those things right away, the first thing I did was ensure the emergency procedures and recovery activities were occurring properly. Take care of the important things first and make sure the team, the hardware, and everything else was safe,” he said.

As the team picked up the pieces from the crash site, Olansen paused to gather everyone in the middle of the field and let them know their efforts were not over; Morpheus wasn’t canceled; this was a chance to learn and make the next lander better.

The failure investigation never escalated to a full, formal mishap investigation largely because the team’s communication and documentation had been robust, even with its scaled-back customization.

The team worked to “pre-declare” expected test outcomes, a process introduced for rapid-prototype projects at NASA. Gerry Schumann, the mishap investigator program manager at Kennedy, sat down with the project managers and safety personnel to define the potential risks. “Tests are just that: tests,” he said. “If we pre-declare what might go wrong through fault analysis and perform engineering analysis afterward, then we don’t need a full-blown mishap investigation.

“Appropriately notifying everyone when the crash happened was also important,” Schumann added. “Terry [Wilcutt, NASA’s chief of safety and mission assurance] knew right away it was not a mishap because I notified him that this outcome was identified in a pre-declare.”

“When Terry got the initial notification, his quick response was this does not rise to the threshold for NASA mishap,” said Mike Ryschewitsch, NASA’s chief engineer. “From his perspective and my perspective, they had pre-identified that loss of the hardware was one of the possible outcomes and had done a very thorough job of safety planning to protect against the worst-case incident, which was what actually did happen, to be sure that no one would get hurt. If either one of those had not been true … then it would have been a different slice.”

Morpheus’s deputy project manager, Stephen Munday, led the failure-investigation meetings that followed, sitting down with Olansen to discuss findings and next directions, which were communicated to the team, who were simultaneously working on design improvements. Since much of the evidence had burned in the crash, a definitive root cause could not be determined. But knowing the IMU failure contributed to the crash and analyzing the flight data they could recover, the team deduced that heavy vibration likely led to connectors from the IMU losing contact.

“We were able to recover vibration data all the way through the crash, and we could evaluate and assess the vibro-acoustic environment, which we believe was a significant player in the cause of the crash,” said Olansen. “We know there was a failure between the IMU and the computer that was receiving the data, but the computer itself and the software were working fine. It was in the transmission from the IMU to the computer where the problem occurred. There are cable connectors, bus couplers, and the IMU itself—any of those components could have been the failure and would have provided the signature we saw.”

To reduce the chance of recurrence, they are adding a second IMU and will isolate both units from vibration (which was not done initially because isolation could affect the vehicle’s ability to meet ALHAT’s stringent pointing requirements). In addition, they plan to upgrade the cable connectors and bus couplers with military-grade hardware as well as create a flame trench on their launch-pad to reduce the vibration.

Confident they will have a lander on which to test their payload, the ALHAT team has proceeded with testing and improvements to their sensors.

“We realized there was going to be a lull, so we quickly set out to run a helicopter test and fly trajectories toward the hazard field using our sensors exactly the way we would fly them on Morpheus,” said Epp. “And that has turned out to be extremely valuable. It’s going to help us get a big head start on success once Morpheus flies again. Our sensors are being updated and improved based on that helicopter test. And that test has gotten our team excited.”

“We didn’t stand the team down while we did a failure investigation,” said Olansen. “A couple of us focused on the failure investigation, but the rest of the team focused on the redesign effort, the improvements we needed, and the rebuild. We still put rigor into the failure investigation, but we didn’t have the whole team stand down to do that. I think giving them something to look forward to and work toward, which was driving them the couple years prior, was a key component to getting back on the horse.”

Epp added, “The impact to us wasn’t quite as bad as it was for Morpheus, but one of the things I always try to seize on is opportunity. Failure frequently opens up opportunity. Suddenly there was opportunity for us to make our system better. The whole idea of moving on and finding ways to do it better became a pretty good rallying point.”

Future Flight

Since last summer, the Morpheus and ALHAT teams have become a single team, though not much has changed in the way they work together. The camaraderie and trust that existed before continue today.

“I’ve been with NASA for a while, and NASA culture rallies around accidents and failure,” said Epp. “NASA has a culture that says pick your feet up, figure out what went wrong, and do it better. I’ve seen that over and over again, and I think this was another beautiful illustration of that.”

Related Links

• Project Morpheus Mission Site
• ALHAT Mission Site
• Project Morpheus Facebook
• Project Morpheus Twitter
• Project Morpheus Flickr
• Project Morpheus YouTube

More Articles by Kerry Ellis

• Radiation-Ready with a Little Rock ‘n’ Roll (ASK 49)
• What I’ve Learned from NASA (ASK 49)
• Kepler: The Long Road to Other Worlds (ASK 47)
• WIRE: Learning from Failure (ASK 45)
• International Life Support (ASK 44)