By Matthew Kohut
The opportunity to build a new launch vehicle that can loft humans into space does not come along often. The Ares family of launch vehicles, conceived in response to the Vision for Space Exploration, presented the first chance for NASA engineers to get hands-on experience developing human space flight hardware since the development of the Space Shuttle thirty years ago.
In 2005, NASA Headquarters solicited proposals from integrated product teams, or IPTs, for different segments of the Ares I-X test flight vehicle. The objectives focused on first stage flight dynamics, controllability, and separation of the first and upper stages. The launch vehicle would comprise a functional booster stage and an upper-stage mass simulator, which would have the same mass as an actual upper stage but none of the functionality.
A team at Glenn Research Center prepared to bid for the job of building the Ares I-X Upper-Stage Simulator (USS). The first challenge in bringing the project to Glenn was assembling a core team with the right skills to develop a winning proposal. “Even before we had gotten authority to proceed but were doing concept studies, and cost and schedule estimations, I needed a good systems engineer to look across this conceptual simulator that we were coming up with and help us identify if we were missing any functions,” said Vince Bilardo, who headed the proposal team and would eventually become the project manager. “We needed a good systems engineer to help us create a draft functional allocation.”
Workers unwrap segments of the upper-stage simulator at Kennedy Space Center. The segments were built at Glenn Research Center and shipped to Kennedy for assembly.
Photo Credit: NASA/Jim Grossman
As the proposal development period for the upper-stage work progressed, Bilardo drafted Bill Foster to serve as his lead systems engineer. Foster began attending systems engineering technical interchange meetings while Bilardo ran concept teams that drew up a series of designs ranging from high fidelity and expensive to low fidelity and inexpensive. “[Vince] had different teams laying out concepts, and that’s where the first ‘tuna can’— the design we actually ended up with—came up. He ran those concept teams over a three-day period, and that kind of kicked everything off,” Foster said.
The Glenn team continued to define its concepts and cost estimates as the Constellation program developed the requirements for the test vehicle. “The requirements were pointing us toward a higher-fidelity simulator. So some of our concepts started to fall to the wayside while the higher-fidelity one—the expensive one—was really the only one that was going to pay off,” said Foster. “When we rolled the cost all up and Constellation was figuring out their budget, they said, ‘We’re not doing high fidelity because it’s way too expensive.’”
A few weeks later, Glenn came back with a trimmed-down version of its low-fidelity proposal. “This low-fidelity launch did a few things. One, it gave us good flight data about whether we could launch this long, skinny rocket. Second, it was fairly inexpensive. Third, it was going to be an early launch [2009] to get this early data, whereas the high-fidelity version pushed out into 2011,” said Foster. “That’s what got us turned on [approved]. At that point we started ramping up people.”
An In-House Development
In May 2006, the Glenn team received provisional authority to proceed with the USS as an in-house project, meaning the Glenn team would design, develop, and build the hardware in its own facilities using its own technical workforce, rather than contracting the job out to private industry. After a probationary period, the project got full authority to proceed in August.
The selected design required manufacturing eleven segments of half-inch-thick steel that stretched 18 ft. in diameter and 9.5 ft. tall—the tuna-can shapes that gave the simulator its nickname. The job would incorporate all the basic hardware development functions: cutting, rolling, welding, inspecting, sandblasting, painting, drilling, and tapping for instrumentation.
Since the project team was beginning with no in-house expertise in large-scale fabrication or manufacturing, it required an entirely new set of procedures that documented each step of the building and assembly process in detail. Bilardo called on Dan Kocka, a recently minted engineer who had spent most of his career as a technician, to serve as the production-planning lead. Kocka had never assumed these duties before. “I said to Vince, ‘Are you sure I’m the right guy for this?’” Kocka recalled. Bilardo was confident that Kocka’s unique background would be an asset that would outweigh his relative lack of experience. Once the project got under way, Kocka’s doubts dissipated; he found that his ability to think like an engineer and a technician served him well. “I really was in a good position to have both of these things going on at the same time in my mind,” he said.
An additional challenge that fell heavily on Kocka concerned demonstrating compliance with AS 9100, an aerospace manufacturing quality standard. Glenn’s management team was making a centerwide effort to achieve AS 9100 certification. For the USS, this meant putting in place rigorously documented procedures that met with the approval of both the Safety and Mission Assurance organization and the technicians doing the work. The AS 9100 standard added another level of rigor to the process of designing and building space flight hardware.
Bob Peters, of Kendel Welding and Fabrication, welds part of an internal access support for the Ares I-X upper-stage simulator at Glenn Research Center in Cleveland.
Photo Credit: NASA/Quentin Schwinn
(RS Information Systems, Inc.)
Preparing for a fabrication job of this size and scope also demanded a wholesale renovation of a facility: new cranes, new assembly platforms, and a new sheet-metal roller. This meant retrofitting an older manufacturing shop floor that was large enough to accommodate the hardware. The facility modification had to be done quickly—in about three or four months—so the project could begin work as scheduled.
“From a project management perspective, I needed somebody who could handle all that facilities work and go work with the facilities organizations and the facilities directorate at Glenn to start planning, designing, and implementing the overhauls that we needed to accomplish in a very short period of time,” said Bilardo. He found Jack Lekan, an experienced project manager who was finishing up another job at the time. “Jack was a longtime Glenn guy who had excellent contacts across the center and a skill set that was very much oriented to team building and cooperation and working well with the various performing organizations. [He was] perfect for the job.”
Ramping Up
Managing the USS project required constant interaction with the other three NASA field centers responsible for Ares I-X: Langley Research Center (systems engineering and integration office), Marshall Space Flight Center (first stage, avionics, and roll control system IPTs), and Kennedy Space Center (integration and test functions as well as the launch itself). Bilardo spent a significant amount of time traveling or otherwise coordinating with his counterparts at these centers, so he needed a deputy project manager who could handle the “down-and-in” details of running the project on a daily basis.
He turned to Foster, his lead systems engineer, who had project management experience from his years on microgravity science projects where he’d served as both the project manager and systems engineer. With Foster moving over to project management, the team needed a new lead systems engineer. They brought in Tom Doehne, who was just finishing up a trade study for the upper-stage thrust vector control system of the Ares I vehicle.
Doehne’s primary focus was on managing the design integration of the simulator hardware, documenting the design in the Design Definition Memorandum, and developing the project requirements. The design evolved and the requirements database kept expanding as the larger Ares I-X management team kept adding requirements for the USS. Doehne realized he needed more systems engineers to support the project. “Initially, I was the only systems engineer as we were developing this task, and we had a lot of work up front that we were trying to do in the early July–November [2006] time frame,” he said.
As the systems engineering workload increased leading up to a Systems Requirement Review, Doehne had trouble finding qualified systems engineers since the new Orion and Ares I projects at Glenn had been ramping up during the past year. Eventually, he was able to transition two civil servants who were in the Space Mission Excellence Program as well as some experienced contractor support. “We took qualified engineers from other areas of the center who were in training as systems engineers. They received real project experience, and we were able to complete the large volume of work that was in front of us,” Doehne said.
In addition to knowledge and experience, Doehne valued team members who could remain engaged and flexible on a project with an aggressive schedule and a rapidly changing context. “Team dynamics is also a very important key to building a successful project team and shouldn’t be mistaken for something that isn’t needed,” he said. “In today’s projects with limited budgets and aggressive schedules, we need to work as a cohesive team unit and have the ability to adapt to a dynamic work environment to achieve our common goal.”
“Welding Is Not Easy”
The scale of the USS demanded a manufacturing capability that didn’t exist at Glenn. The recent focus of the center’s manufacturing efforts had been on microgravity payloads that called for highly intricate machining of sophisticated instruments, not on the rough fabrication skills needed to roll, weld, and attach large segments of a launch vehicle. This reorientation toward heavy manufacturing posed challenges both in terms of the workforce and the organization.
Glenn had several highly skilled machinists among its civil service workforce, but it had few fabricators and a critical shortage of welders. Since the center no longer had enough work to fully utilize the majority of its machinists, the project management team, in consultation with Glenn’s upper management, set out to retrain a cadre of about twenty machinists as welders.
Ares I-X construction work area at Glenn Research Center.
Photo Credit: NASA Glenn Research Center
The effort was well-intentioned, but it did not work as planned. “It turns out welding is not easy,” said Foster, who had advocated for hiring outside welders. Even with training, it took years of practice as a welder to achieve the level of proficiency that the job demanded. Flight-quality welds must pass a litany of tests, including radiographic and ultrasonic inspections by certified weld inspectors. “Unless you are welding day in and day out for a living, it’s really difficult to maintain the level of skills required to execute flawless welds that are going to fly on a flight test for NASA,” said Bilardo.
The next step was to hire welders on contract. The project reached out to some local non-union shops, which began sending over welders for qualifying tests. Again, the necessary skill level proved to be a formidable barrier. “They were probably washing out at a 60 percent rate,” said Foster. The project retained the services of only one of these shops, and they still needed more welders. A call went out for union welders. “Even then, with top-notch welders, we were getting about a 25 percent washout rate,” said Foster.
The drawn-out hiring process cost the project time that it hadn’t built into its schedule. Having found enough qualified welders, the project then had to align the number of welders on the shop floor with the work flow. “We needed a lot of welders at the beginning, but then we cut back because we were not able to keep them busy,” said Foster. Then the pendulum swung too far. “We cut back too far because things were going well and we hadn’t gotten into the complicated segments. When we got into the complicated ones and the welding went back up, all of a sudden we needed welders again.”
The juxtaposition of union and non-union welders in the manufacturing facility created other issues. Some union welders did not want to work alongside their non-union counterparts; in one instance, union welders walked off the job. “We ended up deploying them [union and non-union personnel] on different segments so they didn’t have to rub elbows on the same build stand,” said Bilardo.
Getting Smarter
There was probably no way to foresee the welding problems; the team learned by doing. As the USS entered the home stretch of fabrication before preparing to ship segments to Kennedy Space Center for integration and testing, Bilardo reflected on the difficulties the project had encountered: “About two-thirds of our [cost] growth is due to requirements and scope growth and events outside our IPT, and about one-third has been within our control and really attributable to what I would call ‘maturing estimates.’ There are things that you now put in your budget that you couldn’t have guessed that you needed. Or you did guess at it, and you guessed low, because you weren’t smart enough. You just get smarter over the course of the project.”
