The Toothbrush Hack

By Haley Stephenson

 

Collaborative problem solving, a jumper lead, and a toothbrush turned around an unsuccessful late-August spacewalk.

On Wednesday, August 30, 2012, NASA Astronaut Sunita Williams and Japan Aerospace Exploration Agency Astronaut Akihiko Hoshide were tasked with replacing a malfunctioning main-bus switching unit (MBSU) on the center segment of the International Space Station (ISS), called the Starboard-0 truss. The MBSU is one of four 220-lb. boxes responsible for routing power from the ISS solar panels to the U.S. and Russian segments. The uninstallation went smoothly, but the replacement did not. One of the two bolts required to secure the new unit properly got stuck.

During several attempts to drive the bolt down, the crew observed debris inside the bolt receptacle. They attempted to clean it, but the bolt still refused to cooperate. After eight hours and seventeen minutes, Williams and Hoshide temporarily secured the MBSU and returned to the station’s interior, completing what became the third-longest extravehicular activity (EVA) in history.

The unsuccessful MBSU installation meant the ISS was running at 75-percent power, prompting NASA to initiate power-mitigation plans, which were robust enough to maintain ISS operations. However, the urgency to fix the problem stemmed from Canadarm operator and NASA Astronaut Joe Acaba’s scheduled return to Earth. With the desire to complete the MBSU’s installation with the current crewmembers, who had direct experience with the task and EVA configuration, NASA wanted to resolve the issue sooner rather than later.

The “Big 12” and Team 4

The story of the MBSU began in October 2011, when it stopped communicating with the computers onboard station. While the unit was still functional and routing power as it should, it would need to be replaced. The MBSU is one of twelve items whose failure would render the space station “zero fault tolerant.” In other words, if one of the “Big 12” failed and then another item failed, maintaining research operations would become complicated. In the event the second failure happened to be another Big-12 item, the U.S. segment could be lost completely.

ISS Flight Director Ed Van Cise is part of the team dedicated to developing contingency EVAs for items on the Big12 list and worked on the plans for the MBSU replacement. In August 2012, Van Cise served as the lead flight director on EVA 18, during which the crew would remove and replace the unit. “That, of course, did not go very well,” Van Cise said.

“The EVA was approximately six and a half hours in and my management was already pulling together the groundwork for what we call a ‘Team 4 Effort,’” said Van Cise. In mission control, there are three teams, each covering an eight-hour shift. Team 4 is called in when extra support is needed. “Pretty much everybody needs to drop what they’re doing and we throw every resource we need at the problem,” he said. “That was Thursday night. My role after that was lead for developing our response to the problem.”

There were three main challenges to address, Van Cise explained. First, how would the team fit another EVA into an already-packed expedition schedule? Second, once they find time in the schedule, how do they install the MBSU properly? Third, what is the next worst failure if they are unable to install the MBSU? In other words, what else might go wrong and how should they prepare for it?

One group was assigned to find time in the expedition schedule for the additional EVA; the rest of the team tackled the last two issues. After several briefings, the likely problem on the MBSU emerged: the bolt was getting stuck on the metal shavings and debris the crew reported. To fix it, the crew needed to clean and lube the bolt threads.

“The first place you start is, OK, if I had this problem at home, what would I do?” said Van Cise. “My response to things like this is just go get my big torque wrench and torque it real hard, and if it breaks I go get a new one. The problem is that in space we can’t run to the hardware store and buy a new nut if it breaks.

“When you throw out options like that, then you start thinking up alternatives. We needed to [lube] the threads, and we needed to clean them out. How would we do that?” he continued. Specific tools for these jobs didn’t exist on station, so they had to be invented.

Tools for the Job

Inside Building 9 at Johnson Space Center, Victor Badillo, an operations support officer and sixteen-year NASA veteran, was part of the team responsible for figuring out how to clean the bolt receptacle. “In situations like this, part of our job is to see what tools we have and try to imagine different uses for them,” said Badillo. “We were wondering if we could make our own brush: a wire brush to clean out the bolt receptacle.”

Badillo and another team member started making prototypes. Badillo grabbed a 4-gauge jumper lead identical to one found on ISS, removed the insulation from one end, and frayed the wires inside, creating a brush. The team liked what Badillo developed and improved upon the tool by using EVA tape to make the flimsy lead firm and finding ways to ensure the crew could grab and maneuver the tool with their oven-mitt-like gloves.

Another team worked the lubrication challenge. After surveying the inventory on station, the team settled on using a toothbrush to lubricate the bolt threads. A quick trip to the drug store and the team was ready to start testing the two types of toothbrushes onboard ISS: name-brand and bargain-brand. The name-brand brush had a bend in the neck that interfered with it accessing the threading. The bargain-brand fit, but the full-length brush bent too easily, so the team had to cut it down to make it stiffer and less likely to snap.

The tools also had to be EVA-ready, meaning they needed to be tethered and able to survive the extreme cold. Once members of the astronaut office and the safety office tested the wire brush and the toothbrush, performing evaluations such as a 30-lb. pull-test to ensure the toothbrush wouldn’t come off its EVA handle, they declared them ready for use.

But their work wasn’t done. “We’re not only trying to put together the tools, but also thinking about the techniques for using them and writing concise yet detailed procedures of what the crew needs to do in order to build these things and also use them,” explained Jeff Stone, an operations support officer who worked the toothbrush challenge. For the toothbrush, Stone and his team pared down the procedure to two pages.

By Saturday morning, the team had ready for the crew their procedures and tools: a wire brush made out of a jumper lead and a modified bargain-brand toothbrush.

 

The Next Worst Failure, Almost

Even with the tools, everyone had to consider the next worst failure. If Williams and Hoshide couldn’t remove the bolt or drive it all the way down, the team would have to resort to removing the MBSU at its cooling plate, bringing everything inside station, and trying to fix the box.

“That would have been a big deal,” said Stone. “[The unit] is designed to be changed out, but we don’t have a spare cold plate on orbit. We could send one on a subsequent cargo mission, but in order to do that there would be a whole lot more logistics, planning, and training.”

Fortunately, this did not come to pass. On Wednesday, September 5, Williams and Hoshide conducted another spacewalk. They cleaned and lubricated the receptacle and threads, drove down the bolt, and successfully installed the MBSU.

From the beginning, the goal was to successfully assist the crew. There was no finger pointing and no turf wars, explained Van Cise. Just find a solution and implement it. “It was neat to be at the head of pulling that whole team together and watch them do all their great work. It was fun. Now that I’ve slept, it was fun,” laughed Van Cise.

“I think there’s a good parallel to a hack-a-thon-type event,” said Stone. “You just get everybody who might have an interest in a particular challenge together so they can talk about it, brainstorm, have this common experience, and implement this creative, innovative approach to solving a problem.”

“It’s not always one person who works one particular task,” added Badillo. “We have a lot of talented people here, but when you put them all together on one problem, it’s amazing what we accomplish.”

Hindsight

Almost immediately after the second EVA, team members approached Van Cise to ask when their lessons learned meeting was to take place. “One of the things we work really hard on is going and asking what surprised us, why, and how can we prevent it from happening again,” said Van Cise.

One lesson that came to light pertained to a specialized technique for installing hardware like the MBSU, Van Cise explained. On previous spacewalks, crews used an installation technique called “dithering” to replace ISS batteries, which have a similar two-bolt installation configuration to the MBSU and require a crewmember to be on the end of the fully outstretched robot arm on station. Dithering involves carefully wiggling the unsecured end of a piece of hardware while drilling down a bolt on the other end. The technique relieves any structural loads from building up between the bolt threads and receptacle, and mitigates the movement generated by the astronaut from travelling down the outstretched arm.

For EVA 18, “We didn’t anticipate having any issues installing the box since we were not going to be in the arm configuration that called for the special dithering technique,” Van Cise said. “We learned afterward that dithering is needed for installing this type of hardware regardless of being on the arm or not.”

After realizing the technique was needed for the MBSU, it is likely that other similarly configured hardware on the ISS boxes to see if they will require similar installation techniques or if there are problems we should anticipate when replacing this sort of hardware in the future,” Van Cise added.

Where the Know-How Lives

When asked how NASA knows how to solve problems like this, Van Cise replied, “We train hard not only on failure recognition and resolution, but also on how to think ahead to the implications of the next possible failure and how to protect for it. We have a culture of developing creative solutions, bred out of the mind-set that failure is not an option. For a flight controller, when you’re not on console and hear that something failed, you feel bad for the team that is dealing with the issue. That said, on the inside you’re wishing you could be there, helping to resolve it. This is a key tenet to our training and what it means to be a part of mission operations—not just for ISS but for any program we support.”

Part of the Johnson Mission Operations Directorate’s mantra is “plan, train, fly,” explained Van Cise. Lessons learned are documented and folded back into planning and operations, but also training for crews and flight controllers. They are built into everything. “This way, we don’t just have to rely on someone like Kieth Johnson, lead EVA officer, remembering the horror stories and telling them to the other people on his team,” said Van Cise.

The other parts live in IT systems like databases. “The operations team, the engineering team, and the [operations support officers] all have their own databases,” he said. With this setup, one possible concern is that the solution to a problem could live in an operations database where an engineer might not think to find it. “How do we put things in a visible place where everybody can find the information?” asked Van Cise. Fortunately, a robust, cross-database search capability mitigates this risk.

Stone, who joined NASA in 1989 and has worked events like this on programs such as shuttle throughout his career, explained that this is just business as usual. NASA trains for situations like this. “We prepare people to have the right mind-set. You try to train so much that the initial reaction is instinctive on how to start handling a problem,” said Stone. “That was another neat thing to see. We hadn’t seen this problem before, but we kind of already knew what we needed to do to make it better.

“We try to remind ourselves that at any given time our actions or our decisions could have the ultimate consequences,” he said. “Keeping that in the front of our minds, trying to analyze a situation, and actively learn the lessons and apply them in the future is just part of the way we operate.”

Related Links

• International Space Station Mission Site
• Williams, Hoshide Complete MBSU Installation