By Dan Holtshouse
My first job was on the Apollo program. When I left Ohio State University with a graduate degree in electrical engineering, I went to work for AC Electronics in Milwaukee, Wis., then a division of General Motors. This division was the prime contractor for the Apollo guidance and navigation (G&N) system that was responsible for guiding the Apollo spacecraft to the moon and back. There was an air of excitement at AC, and working on the Apollo program satisfied two of my long-time interests—aeronautics and computers. (Aeronautical engineering was my second engineering choice, and computers my main focus in electrical engineering.)
A complete greenhorn at work, I was put on a team supporting the integration of the Apollo Guidance Computer (AGC) with the gyro-based inertial navigation platform, which AC Electronics supplied. The computer was designed by MIT’s Draper Lab and manufactured by Raytheon. Integration activities included testing all the components together as a system before the complete system was shipped to NASA for integration with the launch vehicle. There were two G&N systems on board: one on the command module (CM) and one on the lunar excursion module (LEM), each requiring its own unique systems integration and testing process.
Lunar excursion module at the Lunar Landing Research Facility.
Photo Credit: Langley Research Center
As with any new complex system that has elements supplied by various contractors from different parts of the country, collaboration, coordination, and communication (the 3 Cs) were absolutely critical. We accomplished the 3 Cs primarily through face-to-face meetings in Boston and Milwaukee and multiple, daily phone conferences to address problems and action items. (In the mid-sixties, we had no e-mail or videoconferencing.) Because of the physical distances between vendors, we also established a program office at Raytheon with our people on site to keep up with progress and handle problems. This seemed to work well, and we ended up with on-site personnel at several other contractor venues throughout the program to reduce miscommunication and ensure successful integration with the other subsystems. Being there matters.
Final integration testing was done in clean rooms constructed especially for the Apollo program. We donned white booties, smocks, and caps before entering the test area through an air lock. We tested around the clock to meet delivery deadlines, and this led to a lot of 3:00 a.m. phone calls from the test crew that required one of us from the AGC group to go in and diagnose the problem. It took us several months to figure out that more than half the system test problems were due to human operators making mistakes in test procedures that then put the whole system under a cloud. We finally realized that we could shadow and record the operator’s entries from a downlink connection, and we designed and built a monitor system (Telmons) that used a then-state-of-the-art asynchronous tape drive to record the test procedures. This eliminated a lot of the late-night calls (once they were being recorded, the operators were more careful) and relieved us of having to write lengthy reports documenting our analysis of why a system problem that could not be repeated was due to operator error. (The experience did make me a much better writer and required me to learn the workings of the rest of the G&N system.)
Before any of the system components went into final test in the clean rooms, they were tested at length at the individual component level. Here we were learning on the fly. Functional testing to see if the components were doing what they were supposed to do was fairly straightforward for the AGC and inertial platform. We knew, of course, that the equipment needed to withstand launch vibration and work in the vacuum of space, but we had no idea about the number of ways that components that worked fine on terra firma would fail in space at zero gravity. We learned, for example, that a single circuit connection out of hundreds in the computer might flex enough to cause an intermittent error during the process of “pulling a space vacuum” in the large vacuum chamber, only to reconnect at full vacuum and not want to repeat itself. To solve this problem, we introduced a series of sawtooth vacuum test profiles—increasing and decreasing vacuum in a sawtooth pattern to flex all the components more—on all our tests to ensure we stressed the equipment enough to confirm it was defect free.
The Apollo Display Keyboard Assembly (DSKY) used on the lunar module.
Photo Credit: NASA/Dennis Taylor
We were also concerned about “floaters,” that is, small pieces of contamination from manufacture that might lie dormant and undetected in all ground tests but become airborne in zero space gravity, after being shaken loose during launch, and cause a problem during flight. This turned out to be a continuing issue for the Display Keyboard Assembly (DSKY), which used mechanical relays in those days before the advent of mature solid-state switch technology. The DSKY was the crucial keyboard interface to the G&N system that the astronauts used to key in data and instructions. We learned that vacuum tests were not enough to surface all contaminates when a floater caused a failure after successful vacuum tests. We were wondering if there was only one floater, or more. We ended up developing a procedure to vibrate and shake every module, while powered up, on three different axes, to certify that they were free from defects and spaceworthy.
Soon after coming on board the Apollo program, I decided that I needed to thoroughly learn how the computer system worked. I began to study the schematics supplied by MIT but found them hard to follow, and I thought they would not be much use to the integration software programmers who needed to understand the logical operations of the computer system. So across several months we reverse-engineered the schematics to create a set of logic diagrams that filled an 11 x 17–inch book that was an inch and a half thick. Some said, “Why on earth would you ever want to do that? You must not have anything else to do.” But we found during the course of the program that “the book” was instrumental in helping us debug many integration problems. It also helped unravel a dramatic system error during flight.
As many people know, a computer alarm on the DSKY went off during the LEM’s descent, one that basically signified, “I’m too busy to do everything you want me to do.” Actually, the computer never lost control, having been designed to be fail safe and with extra capacity, but since the LEM seemed to be heading for an undesirable landing spot, Armstrong took over control for a manual landing—and history was made. Meanwhile, back at the office, we scrambled to help find out what had happened. We used our computer logic statement book to see that unintended radar signals were being sent at too great a rate and that the AGC operated correctly after all. It was said that the Apollo G&N system worked so well that it guided Apollo 11 all the way to the moon and back and landed the crew closer to the splashdown target than the recovery ship that never left Earth’s surface.
Looking back, one thing that strikes me about the program was how much focus there was on contingency planning. There were redundant systems designed in, there were alternative paths identified for performing critical functions if something failed, and a lot of software efforts came from NASA Houston to test for flight programming weak spots and “what ifs.” We all knew the stakes were high if a problem developed after liftoff, so a “sustained level of worry” ran throughout our part of the program, causing us to test and retest for potential “left-field” problems that might occur. We did not want to end up blaming ourselves for having missed a potential problem that might have been discovered beforehand by thinking and working harder. The stress of that responsibility created some worry casualties, however. In retrospect, I think we, as coworkers, should have been more aware and offered help to those who were not handling the stress well.
I was at the Cape for that awesome Apollo 11 liftoff. Even at the observation stand a long way from the launchpad, the sound of the Saturn V going up was a physical force that pounded me in the chest like a one-two punch. It was the last Apollo launch that I saw and was the culmination of intense focus for me and the others I worked with on the program. Afterward, I was in need of a change, so I decided to go back to school for another round of study.
Working on Apollo was one of the most exciting times of my life. The goal of going to the moon created such positive force—a powerful draft of energy that aligned and focused the efforts of all those many contractors and people working at locations from coast to coast. Like some other national initiatives over the decades, the Apollo program continues to be a lasting benchmark and example of how to mobilize great collective efforts in achieving a challenging goal and vision. It shows what can be accomplished when everyone works together with common purpose and commitment. It was a great ride!
About the Author
 |
Dan Holtshouse is Executive in Residence at George Washington University and retired director of corporate strategy at Xerox Corporation. |
