Testing Positive

By Ron Zellar

As a line manager overseeing the software development team for the Mercury Laser Altimeter (MLA) on the satellite MESSENGER (MErcury Surface, Space ENvironment, GEochemistry and Ranging), which launched successfully on August 3, 2004, I was charged with the responsibility of making sure the software delivery was complete and on time. Because the spacecraft is going to Mercury, its launch date is constrained by planetary alignment. And since the launch was constrained, everything that came before it had to go according to plan.

Near the end of 2002, and after months on Goddard’s MLA project, I became concerned that the software test effort had not made as much progress as was needed.

The development effort was nearing completion, but the testing effort seemed to be struggling. We had created a loose schedule when the team was formed, but now it wasn’t serving our needs. It was difficult to tell how much work was left in the test effort, and instrument integration was just a few months away. By early 2003, it was clear that action had to be taken to clearly quantify the magnitude of the remaining work.

Another manager and I made some inquiries about why the test effort was struggling. We found that the test procedure environment wasn’t as easy to use as we’d hoped, and that delays in production of the flight software were translating into delays for the test effort.

I asked the software Test Lead to come up with a detailed schedule that showed delivery of all the test procedures by June 30th. I asked that it include all the empty staff positions required to pull it off. That way, we could see exactly how much work was left to do and how many people were needed to do it.

I knew the prevailing opinion of adding newcomers late in a project was that it didn’t work. Putting more people on a job supposedly causes greater delays, since it distracts the already established team from making progress. Still, we needed more manpower to get the job done. And after looking at the Test Lead’s schedule, I realized just how big our problem was. Realistically, in order to develop the test procedures required to test all of the functionality in the flight software, our test team would have to double or even triple in size.

The Recovery Plan
We decided that we needed to add five or six full-time people to the project. Although we needed the help desperately, we still knew the importance of getting the right people. We needed people who had a proven track record of success, and a background in software script languages. A history with software testing would have been a bonus.

The other branches in our division really came through for us. They realized that this mission was an important one for Goddard, and that on-time completion was critical to its success. They took people off their current projects part-time and delayed the schedules on those projects. In the end we managed to get eight part-timers, who were essentially the equivalent of five or six full-time people.

As people were identified to us, we approached them individually. Some were concerned about moving to the project so late in the game, and we wanted to ease their transition. We negotiated a “contract” with them. “If you can help us out,” we told them, “then we will make a commitment to you that we will not involve you past June 30th.” This was the key for getting people to agree to work with us.

By late January and early February, new people began joining the team. In March the full team was in place, and we scheduled a training session so the existing team members had an opportunity to share their knowledge about how the system worked. We were trying to promote a unified team atmosphere. It gave the senior team the opportunity to say, “Here, let me take you under my wing,” instead of saying, “You’re an outsider. Stay away.”

The Mercury Laser Altimeter (MLA) software and development team from left to right: Anne Koslosky, Brett Mathews, Samira Ghazi-Tehrani, Dan Berry,Maureen Armbruster, Dave McComas, Lisa Hoge, and Ron Zellar.

The Best-Laid Plans…
The first thing we did as a newly organized team was to plan a new schedule. We planned it in a lot of detail, essentially week-by-week, using an earned value system. This allowed us to keep on top of whether we were making our plan or falling short of it. It also allowed us to keep everyone informed about our progress.

Things were going well for a few weeks, but then we started to fall behind schedule again in late April. This time, we saw the slip right away.We met with the entire team to address the issue. I asked them, “What can we do to recover? What can we do to get our results to match our plan?” The meeting resulted in many recommendations and some greater insight into the team’s challenges.

For example, one recommendation was to acquire another test bed. I talked to the Project Manager and was able to get time on a second test bed and have it moved into the software development area. This way the team wouldn’t be constrained by limited resources. It went a long way toward keeping the team’s momentum going.

I also reminded the team members that comp time was available, and they made regular use of it. During this time, though it was difficult to do, I always tried never to refuse a request for leave. We all tried to create an atmosphere in which it was clear that we needed to get a lot of work done in a short amount of time, but in which the team as a whole would fill in the gaps when necessary.

It was suggested that weekend work might be a good idea for those team members who could do it. A lot of people had children and other commitments, so I had to be respectful of those who couldn’t come in. It was definitely a “request” rather than a “demand,” and I left it to the team leads to talk with their staff.

For those people who showed up on weekends, I tried to always show my appreciation for their time and efforts. I made sure that I was there on those extra days as well. I usually found one or both of the team leads there along with a few other staff members. I would call ahead and ask what their lunch request was. Then I’d stop on my way in to pick up sandwiches for anyone in the lab. I made sure that everyone’s supervisor knew they had worked on a weekend. My manager would even make a special trip to an individual’s office just to thank them for working extra hours.

The Mercury Laser Altimeter (MLA) CPU board that operates the MLA software.

The attitude seemed to spread. The Test Lead brought in candy for everyone and music to listen to as we worked. People were making a real effort to stay positive. In response to that, others were more willing to work additional hours without even being asked. The workplace became a more “fun” place to be, but at the same time there was an acknowledgement of my high expectations leading to a more aggressive schedule. People were giving their all, and for the most part they were giving it with a smile. This demonstrated a real dedication to success — the dedication to succeed as a team and not as individuals — and created an atmosphere that was mutually supportive.

Against the Odds
By late June we had actually made up for lost time. The test procedures were substantially ready. There were a few exceptions, but these were well-noted and understood. We began our formal acceptance testing of the flight software. Some of the staff members even decided to help us as best they could after their committed deadline of June 30th had passed.

In the end there were a number of factors that contributed to our success: a sense of commitment, more attention to schedule, Branch Managers willing to reassign their best people, and Division Managers that championed our work. We did it despite the common belief that it doesn’t work to add extra people in the middle of a development cycle. The Product and Test Leads continually exceeded my expectations and visibly grew in their leadership capacity. As a testament to their group effort, many members of the development and test teams received awards and recognition from our division for their outstanding efforts. Most feel, however, their greatest reward will be the day MLA returns its first science data from Mercury.

Lessons

• Fostering a positive atmosphere and an attitude of team spirit may make all the difference in the success of your project. People will be more willing to step up when they feel like appreciated members of the team.
• High levels of team energy and enthusiasm are sustained by both high expectations and an environment where each member can effectively work at the peak of their capabilities.

Question
When adding people to your project team, how do you know you are getting the right people?

About the Author

Ron Zellar graduated from Virginia Polytechnic Institute in 1991 with a B.S. degree in Physics and a concentration in Astronomy. “When I got out of school, I was working with scientists,” he says. “I learned software as an aside, as a means of supporting them.” His “aside” soon turned into his career.He made the jump to flight software, becoming a contractor for the Naval Research Laboratory where he worked on a project called the Interim Control Module (ICM) for the International Space Station. After leaving the Naval Research Laboratory, he landed his first management position as an Associate Branch Head at Goddard Space Flight Center. There he serves in a branch that develops and maintains flight software for Goddard’s missions. He oversees and reviews the implementation of computer-based subsystems, instruments, and sensor suites that enable the operation of Goddard’s scientific spaceborne systems. One of his first assignments was to oversee the development of the MLA software, an instrument on the MESSENGER Spacecraft set to orbit Mercury in 2011.His motto: “Build cool stuff. Launch it into space.”