Test What You Fly?

By Donald Margolies

It was the first time on any NASA project I know of that all the instruments on an observatory came off for rework or calibration after the full range of environmental tests, and then were reintegrated at the launch center without the benefit of an observatory environmental retest.

Perhaps you’ve heard the expression, “Test what you fly, fly what you test”? In theory, it’s hard to argue with that. In this case, I was willing to take the risk of not testing what I flew. As the project manager for the Advanced Composition Explorer (ACE) mission, I was the one who ultimately decided what risks to take, just as it was my responsibility to get buy-in from the stakeholders.

Logo for the Advanced Composition Explorer (ACE) Mission, which was launched in August 1997. ACE is studying low-energy particles of solar energy and high-energy galactic particles, and continues to collect data that scientists use today to understand our universe.

The ACE observatory had a suite of nine instruments and an electronics box that interfaced with a number of the instruments. We had planned from the outset of the project to remove three of the instruments to have microchannel plates replaced and be re-calibrated. Our implementation plans took this into account, and we developed retest programs for the individual instruments. Subsequent developments changed this plan significantly, when I authorized the removal of all instruments and the electronics interface box.

Several people on the project thought I was crazy. Why do this? We had gone through our environmental test programs successfully, and everything seemed to be working okay. We had previously stated that the only reasons for removing an instrument after environmental testing would be either because it was one of the three mentioned earlier, or because something had broken and needed to be fixed.

Normally, I would have taken a pragmatic approach: “You’re on the observatory, your instruments are working, and good enough is good enough.” On the other hand, we had more than adequate slack in the schedule, and we were coming in $30 million under budget — amazing, I know. The question then was, what could we do to make the science better? Given that we had the time, given that we had the money, one answer to this question was: better calibration. For those who just do a marginal job of calibration prior to testing, the alternative is to calibrate again in orbit. Calibration in orbit takes a long time to do, and it’s not as precise as it is on the ground. So there really was a net benefit to be gained from the science of doing this. If our scientists had the opportunity to tweak and calibrate their instruments on the ground, they would likely get better science in space. In order to provide a proper return on our $100M NASA investment, ACE — an Office of Space Science mission — had to perform on all cylinders, so to speak.

Dealing with My Stakeholders
How do you evaluate the risk of putting an instrument back on without retesting it under vibration? That was the question my management put to me. When I approached them about this, they thought I was crazy. “Don, you are crazy,” they told me, in fact — but I had known I was going to hear this, and was prepared to explain.

The way that the spacecraft was designed made the job of removing the instruments and reintegrating them very simple. The instruments were mounted on the outside of the spacecraft and easily accessible. You basically disconnected the connectors — and there weren’t that many — removed the mounting bolts, and lifted off the boxes. When you re-bolted them down, you made sure of their mechanical integrity, and you did functional tests on each of the instruments to verify that they were working okay. The solar panels were off anyway at this point, so if we had had to get into the guts of the spacecraft that would not have posed a problem.

Now this may all sound well and good on paper, but you don’t persuade your management to do something it usually does not want to do just by sounding logical. When you want to do something this radically different, you have to be cool and clear in how you present the issue. The last thing you want to seem is impulsive. Upper management wants to hear that things have been checked out according to protocol. They sleep better when they know that protocol has been followed (“Test what you fly, fly what you test”), and consequently a project manager sleeps better when he knows his management isn’t tossing and turning about a decision he’s made.

Engineers at the Applied Physics Laboratory test for true perpendicular solar array deployment of the Advanced Composition Explorer.

“Yes,” I admitted, “when we reintegrate everything we will not have the same degree of certainty as we did before the testing. Without another vibration test, no one would ever be able to say categorically that these instruments will hold up under the stress of launch.”

Ultimately, I was able to get management to buy off on the decision, but not without first making an independent review of our plans. The review board agreed with me that, based on the design of the spacecraft, reintegration would be “less complex” than it could be. The fact that all instruments would be environmentally tested before they were returned to the spacecraft also helped. But there were other stakeholders, including those at headquarters, whom I also had to convince.

The Folks Who Were Doing the Work
The person I was most concerned about was the project manager of the organization that was building the spacecraft, the Johns Hopkins Applied Physics Laboratory (APL). It was APL’s responsibility to reintegrate the instruments, and I knew the APL project manager, Mary Chiu, had hardly been delighted when I told her what I wanted to do. In fact, it was she who reminded me before anybody else that you should “Test what you fly, fly what you test.”

Mary never yelled or screamed or jumped up and down and said “no, we can’t do this”; and I don’t believe she ever said, “Don, you are crazy,” (at least, not to my face like my management did); but she did voice her displeasure in writing, and this was not something I took lightly.

We talked about it exhaustively, especially in terms of what the impact on her team would be. Mary was definitely a key part of the planning process because it was going to be up to her people to reintegrate the instruments, as well as to do all the other things we needed to do as the launch date approached. Now, I wouldn’t have changed the plans had I thought the APL team felt I was running an impossible risk. Getting Mary’s buy-in, albeit a reluctant buy-in, was a major precondition for going through with this.

I left it up to Mary to decide when she needed all the instruments on site. The way the instruments’ schedules were laid out, they were going to arrive in a waterfall fashion so there would be adequate time to integrate them. If they all showed up at the door the same day, Mary’s team would really have to hustle, and I didn’t want that. The APL team was working very hard as it was. I had to make sure they were taken care of, so Mary’s schedule dates were what the scientists committed to.

And Then There Were the Scientists
In order to even consider this risk in the first place, I had to have complete faith in the scientists I was working with. More than any of the others, they were the group who would determine whether or not we could accept such a risk. How did I guarantee their full cooperation? The scientists were told that if they did not get their instruments back in time, they might not fly. They understood that, and more importantly, they believed it. I know they did because the Principal Investigator, Dr. Stone, stood behind me on this. I insisted that each of the co-investigators write a letter to Dr. Stone promising that they would return their instrument no later than the date given them. I don’t know whether or not they would have written such a letter to me, but based on what I had seen so far, I knew that once they promised Dr. Stone, no matter what condition the instrument was in — whether it was fully calibrated or not — it would arrive by the day they had agreed it would.

Our Science Team knew that our primary objective was to launch on schedule. They also knew that because of some science overlap in the instruments, we would be willing to leave an instrument on the ground if we had to. The truth of the matter was that despite what we had said outwardly, our goal was always to fly a complete science suite, with all instruments functioning as well as they could, and with all the appropriate calibrations.

And so it all worked out in the end. The orderly return of the instruments didn’t happen exactly as we had planned, but due to the skill and dedication of the APL team we reintegrated the instruments at the launch site, and launched ACE within four days of the date we had specified three and a half years earlier. This happened because people were willing to work with one another to make it happen. The proof, I believe, is that the instruments have been working for almost five years now and are providing wonderful scientific results.

Lessons

• Project principles should be adjusted to suit particular circumstances. However, before you break a rule, consider the ramifications and understand the importance of getting buy-in from all those who have a stake in making something happen.
• Lead by influence rather than authority, by seeking cooperation rather than using solicitation and you are much more likely to get buy-in from your team.

Question
When do you know it’s okay to break a rule?

Read more about the ACE mission:
Implementation Reviews

About the Author

Donald Margolies was Project Manager for the Advanced Composition Explorer (ACE) mission, launched in 1997 and still operating successfully. He received the NASA Medal for Outstanding Leadership for his work on ACE and a NASA Exceptional Service Medal for the Active Magnetospheric Particle Tracer Explorers (AMPTE) mission. Don and his wife Joan have been married for 41 years. They have two children and five grandchildren.