The Trouble with Success
By John Brunson
In preparation for the February 1996 re-flight of the Tethered Satellite System (TSS) payload, the Marshall integration and test team traveled to Kennedy Space Center to support the Interface Verification Test (IVT) between the satellite and tether connector. The test, which was run in the summer of 1995, proved to be hotter than the Florida sun and caused the team to sweat just as much.
We were feeling the heat because TSS hardware was failing to pass the IVT. A great deal of our frustration was caused by the fact that this system had flown before and had successfully passed the same test. The Marshall and Kennedy test team, many of whom had been involved during the first mission, pulled together to try and understand the cause of this failure.
On the surface there was no reason for this simple but critical test to be failing. Every precaution had been taken between missions to safely stow the hardware. Inspections were made prior to connecting the two halves. The same procedure successfully used during the first mission had been followed, and we had made no modifications to the hardware.
As the integration and test team lead, I had to make the call back to Marshall and alert the Program Manager as to our status. We were eight months away from launch and a solution was needed quickly to keep us on schedule. All eyes, including Headquarters, were focused on us identifying and correcting the problem.
Start with the Obvious
We were fortunate to have good people from Marshall, Kennedy, and the contractor community as members of the team. We also pulled in expert help from outside as needed. You’ve got to remember that success occurs due to the “people” on the team and their commitment to solving “the team’s” problem. Everyone on the team understood the urgency of the problem.
It’s hard to describe exactly the energy that comes from working on a crack team in a pressure situation like this. Say nothing of the fact that all the while everyone knew our actions were being watched throughout the agency. We all were doing the best job we could anyway, but with this “little” bit of added pressure, it was an awesome motivating force. Situations like this are when the true character of the individuals and their contributions to the team surface. When you actually experience something like this at a crux moment in a project, it’s almost like you are operating in a totally new space, and you yourself are transformed, knowing that the energy you are getting from your teammates is bringing out the absolute best in you.
Hours were spent reviewing procedures and drawings. We considered all the contingencies that might be contributing to the problem. Additional testing and analysis was conducted and evaluated. We spent hours gathered around the conference table, throwing ideas out and putting them up on a white board. The pros and cons of each one were explored, and the proponents of their theories argued vigorously why one was superior to another.
Finally, we selected an option to implement — what came to be known as the “360-Degrees Test” — and were hopeful it would support our assumptions, verify the problem and, if successful, lead us to correcting the problem, re-running the IVT, and verifying our fix.
You look for the obvious and try to work your way back. We believed the connector on the tether side was manufactured improperly and was actually cocked off its normal perpendicular path and recessed by several thousandths of an inch back into the connector body. The 360-Degrees Test allowed the team to connect, disconnect, rotate, and reconnect the bottom half in 15-degree increments. This test was designed to find the region around the connector that got connectivity.
It was obvious to us when we got the data back that there was a manufacturing problem within the Tether Connector. The vendor acknowledged that it was a manufacturer’s defect. There was a great sigh of relief all around because we knew the problem could be fixed quite easily. X-rays and other data helped to verify this. Once we were satisfied with all the test results, we set off to replace the connector and ultimately passed the IVT.
You May Never Know How Close You Come
The moral of this story is, “The trouble with success is you may never know how close to failure you came.” As I said at the start, this mission was a re-flight. There was actually no change between the first time we did the integration and the second. The procedures we used were exactly the same. Probably the first test team got lucky and nailed the connection just right.
We have known risks in every program, and we have unknown risks because it’s the nature of the beast. The problem is our past successes drive the schedule that we create for re-flight missions. We try to plan for the best we can, but until the vehicle is up in the air in an environment it was built for, doing what it is supposed to do, you have a lot of restless nights. Plan and hope for success, pray for luck, but be ready to address failure.
- If results do not meet expectations, for better or worse, we have little choice but to see this as an opportunity for learning.
- Teamwork is of the utmost importance during crisis situations.
To what extent does luck play a role in project success?