EPISODE 79: ENGINEERING BEST PRACTICES – PART 2

Transcript

Steve Hirshorn: The ‘leading people’ part of it, the ability to find solutions, the ability to work with a diversity of different people — that is a significant part of being a successful chief engineer and it’s really just as important as being able to talk in the language of chief engineers and understanding life cycle reviews and so forth.

Deana Nunley (Host): Welcome back to Small Steps, Giant Leaps, a NASA APPEL Knowledge Services podcast where we tap into project experiences to share best practices, lessons learned and novel ideas.

I’m Deana Nunley.

We’re continuing our series on engineering best practices, taking a closer look at how excellence in engineering helps drive mission success. Our guest today is NASA Aeronautics Research Mission Directorate Chief Engineer Steve Hirshorn.

Steve, thanks for joining us.

Hirshorn: Well, thank you for having me. It’s a pleasure to be here. Thanks for the opportunity.

Host: Absolutely. What are the key ingredients of being a chief engineer at NASA?

Hirshorn: A few years ago, I authored a book that was basically about how to be a chief engineer at NASA. The book is published by NASA. It’s available free as a download. And so, if you search on the title, which is ‘Three Sigma Leadership,’ you’ll come up with a webpage where you could download an EPUB version of the book or something that you could use on an e-reader, that sort of thing. It’s publicly available and free to anybody, inside NASA or outside of NASA.

When I wrote that book, I organized it by 24 different skills and traits, but after I put those down, I recognized that all 24 of those largely falls under three different categories. The first category is the obvious one, is technical acumen. It’s being able to speak the language, the technical language. It’s understanding the vernacular of margins and dispersions and knockdown factors and all the things that we do during the design and operations process. A second part of that is really managing yourself, and that is continuous learning and just the ways in which we present ourselves. But a very significant portion of the book is about leading others, and is about people skills. Those sorts of things many times don’t come intuitively to those of us who are chief engineers that just want to stay purely in the technical. But the leading people aspects of this, which includes teambuilding and negotiating and even having and utilizing emotional intelligence, to me, is just as critical in the roles of being a chief engineer because we do have to understand the technical landscape, but we also have to be able to lead people.

So, being a chief engineer is a combination of all those things, and I won’t wage a guess on proportions. But I would offer that the ‘leading people’ part of it, the ability to find solutions, the ability to work with a diversity of different people, people from different centers, different national cultures, different perspectives and different backgrounds, and so forth. That aspect of leading people, which can be ineffable, it’s very difficult to define, but for me anyway I’ve found that that is a significant part of being a successful chief engineer and it’s really just as important as being able to talk in the language of chief engineers and understanding life cycle reviews and so forth.

Host: When you think about leadership skills in general, what’s different or unique about engineering leadership?

Hirshorn: One unique characteristic I think of engineering leadership is in the realm of being a technical authority. Tech Authority was created and was inserted in the agency governance coming out of the Space Shuttle Columbia accident back in 2003, and Tech Authority was inserted into agency governance about two years after that. Tech Authority, the way it’s characterized, is a healthy tension check and balance between the programmatic authority, and that’s all the people that run programs and projects and have to be concerned with budgets and schedules and so forth. The programmatic authority, and the institutional authority, which also includes our centers and our workforce and our facilities. Tech Authority was that third leg of the stool, and it was there to ensure technical excellence in the work that we do as an agency but also it ensured that we’re aligning with technical processes and expectations.

It’s very easy when a project manager is encumbered by budgets, by cost overruns, let’s say, or schedule constraints or what have you, to take a path that is, let’s say, more expeditious to completing a system. Tech Authority is there to ensure that the appropriate amount of thought and oversight is given to make sure that the project is ending up at an acceptable level of risk for mission success, which is really what it’s about. It’s about making sure that everybody is on the same page in terms of the amount of risk that the project is taking.

Project managers can be sometimes focused on the cost risk or the schedule risk. Tech Authority is there, in part, to ensure that technical risk, and on the safety and mission assurance side that obviously includes safety and quality assurance. That technical risk and finding that right level of acceptability is part of the equation. Many other leadership models out there in business and in industry and in sports and other things don’t include that unique aspect of Tech Authority that I think sets engineering leadership aside.

Host: What do you see as the challenges and rewards of being a chief engineer?

Hirshorn: The first thing that I would offer is just the reward that you get from a successful mission, and I’d put it in a context of the very first position that I had here at NASA, which will probably be one of the best. One of my favorite positions was a flight controller in Mission Control in Houston working space shuttle flights. For about 11 years, I supported 55 space shuttle missions all the way through the planning and training process, the preparation process. But then you get on console during the pre-launch shift and you can work through the launch. You can work through the duration of the mission on orbit, and you can work it all the way to what colloquially we call ‘wheel stop,’ so the vehicle has returned back to Earth and is standing there on the runway and the mission is over.

There is this very special feel that I got at the end of each mission at wheel stop because you work for months and months, you trained incessantly. And then there’s the drama of the mission during launch and during all the operations on orbit and on entry, the drama of danger. But the drama of making sure that you’re doing everything to succeed in the mission, you have a successful mission. One of the greatest rewards that I’ll ever have, I think, is when you come to the end of the mission and you’ve been successful.

For me, I characterize it in terms of the space shuttle, but it’s just as applicable for folks that have been working on a spacecraft under development and you’ve been putting every day in your blood, sweat and tears for five years or 10 years into it, and it launches, and it fulfills a mission’s requirement and it’s successful in terms of its mission. There’s this incalculable feeling that you get from that. The James Webb Space Telescope, I think, is a great recent example that all the work and millions of person hours that went into making sure that all the deployment mechanisms occurred correctly and so forth. And we launched and they did, and all those million hours of work really, really paid off. There’s a very tangible feeling that you get from that.

Challenges. I think that you mentioned rewards and challenges. Right now we’re dealing with a timeframe that is good and bad from a standpoint that the agency budget has increased pretty significantly over just the last five years, which means that we have more missions that we want to accomplish. From an engineering standpoint, however, our engineering workforce has been reasonably flat, so we have more things to do, more responsibilities, more projects to cover with the same workforce. I think that is going to be a challenge of how to navigate between those pressures and a growing amount of tasks for the engineering workforce. But it is an issue that is being discussed across all the centers and the engineering leadership about how do we handle basically more things on our plate when we have the same number of people.

Host: What are your thoughts on the importance of lessons learned in sharing engineering knowledge?

Hirshorn: I’ve spent, let’s see, 32 years in the agency and counting. I’m not done yet, but I got 32 years of runway behind me. One of the things that I’ve always valued is the knowledge captured by those who have been here before. And I actually have lamented that it’s common for folks to be very, very deeply embedded in a project they’re working towards, and when the project comes to an end they’re very quickly ready to move on to the next project. I feel as an agency we don’t always capture that experience that can only benefit future generations.

I worked on space shuttle, and we spent 30 years of operations in space shuttle. Some of that operational experience is captured in documentation and it works its way into policies, but an awful lot of that operational experience left when people retired or moved on, or in the case of contractors, were laid off. There is huge amounts of operational space flight experience, including processing of the vehicles, that we learned through probably two generations of engineers. And some of it, a small portion, was captured but an awful lot of it left with the people.

We spent 10 years constructing the International Space Station on orbit that had over a hundred spacewalks. There are lifetimes of experience, not just from those that perform the spacewalks, the astronauts that perform the spacewalks, but all the thousands of engineers and planners that planned those spacewalks, that worked issues real time, that did all these things that over the course of a hundred forays into the vacuum of space, after 10 years we had an operational space station. That, I think, has always been an opportunity for the agency to capture that experience somehow. The challenge in doing that is, as I mentioned, knowledge capture and things like that sometimes are considered over and above what you need for mission success. So you keep your head down and you develop your system, you launch it, you operate it, and at the end you’re successful.

I always feel that somehow capturing the knowledge that we gained and in the context of being able to apply that to future generations that don’t have to live through the same mistakes, let’s say, that we made, that type of stuff is gold. You know, it’s incalculable, the benefit that we could get from that as an agency. It’s difficult to do, but folks need to set aside a bit of time to be able to do that. Some of that information, we have carried through the generations by documenting it in policy and process, but I’ve always felt, again, that there are just lifetimes of experience that transcend just simple shall statements in a policy that are of tremendous benefit for folks coming down the pike, for new engineers that are just joining the agency, and the benefit that we could get from being able to figure out a way to pass that on is just incalculable.

Host: When people don’t share information, what are the potential consequences for the organization?

Hirshorn: I think the simple answer to that is that there’s a risk that you could repeat the same mistakes that people have made in the past. As engineers, we always talk about scars and scar tissue of things that we’ve learned, of missions that didn’t go the way that we wanted it to, or design processes and choices that we made that didn’t pay off or work out. Those are the scars and the scar tissue that we carry throughout our career. Once you have that scar tissue, it’s very unlikely that you’ll make the same mistake again. You learn from those experiences.

The typical process is a new engineer comes in, joins the agency, and builds their own scar tissues and makes those mistakes, or as part of missions in which mistakes are made. The ideal to this is being able to carry that information forward, to provide that knowledge capture such that future engineers and future project managers and so forth can avoid having to go down those paths before they can benefit from mistakes that have been made in the past. To some extent, we try to do that with documenting some of this in policies and practices. Some of our centers have extremely good and rigorous engineering best practices, golden rules, and so forth. But I think we as individuals collectively carry this lifetime of experience, and there’s a lot that formulates our wisdom as we accumulate this that doesn’t get documented.

By the way, knowledge capture doesn’t have to be pure ‘just’ documentation. It can be in the form of mentoring. It can be in the form of doing things like this podcast. There are many, many ways to convey information and to convey lessons learned and experience and wisdom, hopefully. It only, in my mind, can benefit us as an agency to avail ourselves of the opportunity to find ways to continue to perpetuate that wisdom such that we can be even more successful in the future.

Host: In passing this information along you talked about mistakes that people have made and how important it is to share information. What if people just don’t speak up because of a mistake or a failure? What impact does that have?

Hirshorn: In late January, we, as an agency, commemorate what we call A Day of Remembrance which acknowledges the loss that we have sustained as an agency, three different times with the loss of astronaut crews. And those three events, by the way were Apollo One fire in 1967, the Challenger space shuttle in 1986, and the Space Shuttle Columbia breaking up on entry in 2003. It was pointed out that if you count the time duration between Apollo One and Challenger, it was 19 years. If you look at the duration between Challenger and Columbia was 17 years, and right now in 2022, we are sitting on right about 19 years since Columbia. And so in some contexts you could say we’re ripe to repeat those sorts of mistakes that we’ve made in the past.

I lived through Columbia. I was deeply involved in the accident investigation and return-to-flight activities. I know every single person that was on console in Mission Control during Columbia STS-107 entry, know them personally. That event for me, as I expect it was for the NASA employees that lived through Challenger and Apollo One, was emotionally traumatic. You don’t forget when you experience something like that, and it stays with you for the rest of your life. But there will come a day when I retire and there will come a day when many of the folks that are still part of the agency that lived through the Columbia mishap will retire and move on. Same thing happened with the Apollo One and same thing happened with Challenger. It’s almost a generational sort of thing.

So, there are elements of organizational silence in all three of those events. They were not the root cause. You could track the hardware-related root causes in all three of those accidents to very specific things, and there’s no commonality amongst those. The commonality across all three tends to be in the organizational shortcomings that we had. The term that has become common is “normalization of deviance,” where you get very, very used to an out-of-certification condition as being normal just because it’s happened a bunch of times before. There were instances of organizational silence in each of those three events where people had recognized that we were probably tripping the boundary of acceptable risk, but we were still continuing to press on. But there were forces within the organization that prevented people from being able to speak up, from feeling free to speak up, from having the latitude and the liberty to raise and elevate these concerns.

All that sounds really, really bad but I will say that I look for the ability of us as an agency to learn. And I think where we are now in 2022, and part of it is a hope, but I really do think that there’s evidence out there that we have finally learned as an agency some of these lessons and how things like organizational silence can be cancerous to us as an agency. Just in the last few years, I’ve seen evidence on things like mandatory training in organizational silence, mandatory training that everybody, as far as I know, across the agency has to take, SATERN courses on those three events, and not just the hardware-related root causes, but the organizational root causes as well. The key to not making mistakes in that vein are to keep not just the memory of the astronauts who perished, but the realization of the impacts of some of these behaviors. Keep it alive and fresh. You can’t count on that being instilled in our culture simply because people live through it, because eventually folks retire and they move on.

I do feel like we are trying to inculcate into our NASA culture those things that in part led us down the paths of Apollo One, Challenger and Columbia and instill that institutionally, such that folks that are coming out of college right now, they realize the importance of how organizational silence can be poisonous to the agency, and they don’t have to live through it themselves and carry those emotional scars in order to understand the importance to the agency. I actually think that we’re in a good place by maintaining that as part of our corporate identity, our corporate knowledge. And the thing that we have to be careful about is maintaining the vigilance to maintain, to keep it there.

So, when I retire and when agency leadership that is instilling this right now for mandatory training, when they move on, that future generations of NASA leadership still feel that this is an important thing to maintain as part of our culture and instill that vigilance to make sure that we do. I think we’re in a good place where we’re keeping that fresh in our engineers’ and our managers’ and our technicians’ minds. We just have to make sure that it stays there.

Host: Steve, how has engineering changed during your career, especially say over the last 10 years?

Hirshorn: There are two things that stand out in my mind, probably if I thought about it, there would be more. But there are two things that I think stand out in my mind. First off, one thing is the utilization of model-based systems engineering. The use of MBSE in our design processes throughout all of our design, development, test, and evaluation is becoming, or probably has become, a standard and is commonplace in industry more so than it has within NASA processes. I see MBSE, I see pockets of MBSE, and there certainly is a rich amount of pathfinding going on within the agency to help us figure out what works and what doesn’t work and so forth.

But that ability to model our processes, that ability to create these digital models, for lack of a better word, of the systems that we’re developing and utilize that to assess the impacts of changes, the impacts and potential hazards that we hadn’t recognized before when we do make changes and things like that. 10 years ago, that was not part of our normal processes at all. Today, it is becoming more and more standard. Again, in my experience, I think industry is out ahead of us in this case, but we are working and striving for more digital transformation within the agency. It still is uncomfortable, I think, to many because it’s not standard, because it’s not the old school way of doing things. But I didn’t see that 10 years ago to the extent that I’m seeing it now.

The other thing is a more difficult area to really quantify, but I think it is changing the way that we’re doing engineering to some extent, or at least the way that we oversee our engineering developments, and that’s in the last 10 years we’ve seen a lot more projects go down the path of public-private partnerships to where NASA is not procuring a system that we develop but is procuring a service.

The obvious example to that is Commercial Crew. We’re paying SpaceX and Boeing to carry astronauts, to transport astronauts, to and from the International Space Station. NASA does not own those systems. We don’t own those spacecraft. Our industry partners maintain ownership of the spacecraft. The typical NASA model to where we could apply and mandate our engineering standards, our engineering processes, and our project management standards and processes, and well, that’s changing to where before we maintained very, very close insight and oversight of what the contractors are doing, now our engineering relationship tends to be more collaborative than just pure insight and oversight. I think we found models that work fairly well on commercial crew. I think we’re still learning the ins and outs of how to do that well, but we’re learning and I think we’ve been reasonably successful.

The Human Landing System, HLS project, in human spaceflight to carry astronauts to the surface of the Moon and back is going to be done under the same model, and that model is also being translated to areas of NASA aeronautics in some of our X-Plane developments as well. That relationship and that role of moving from pure oversight and insight to a collaboration with industry partners where we never get handed the keys to the car, that certainly wasn’t there 10 years ago. And I think we made a lot of progress in figuring out how to do that effectively, but we’re probably still learning on that account.

Host: What changes do you think we may see over the next 10 years?

Hirshorn: Going back to the MBSE, model-based systems engineering is one example of how technology is affecting our design processes. I have a feeling, and it’s largely a feeling. It might be a slightly educated feeling, but it’s still a feeling, that our very design processes are going to be changing over the next 10, 20 years as computer technologies get more and more advanced and capable. Technology matures and computer technologies continue to advance and what capabilities they will have 10 years from now, we probably can’t even dream of. My guess is that our engineering design processes and practices will continue to evolve. There is a transition from the very 20th Century way of doing design drawings via vellum and paper drawings and so on and so forth to computer-aided design, and that happened in the ‘80s and the ‘90s, and that was a very, very large transition in our design processes because computer technology was advancing. I have a feeling that advancement is going to continue and it’s going to manifest in ways that we probably can’t even predict today.

I do think that, again, in the public-private partnership, that is going to be more and more common and not just in terms of what NASA does, but I think that we’re going to have to get comfortable in an environment where there are lots of other people doing what we do. I hired in 1990 and at that point, I worked with a whole swath of Apollo veterans that were just retiring from the agency. Well, even in the 1990s, there were very few entities on the surface of the planet that were sending astronauts into space. Today, it’s becoming more and more common. I think SpaceX had announced about private missions to very, very high orbits, 800 kilometer-high orbits, well above low-Earth orbit, and actually doing things like spacewalks.

There will be a transition, I think, between being the only people on the planet that can do something, to being one of many people on the planet that could do something. And what that means for NASA though, is that we’ll probably at some point hand off the responsibility of doing things that industry and private entities are now doing for themselves so that we can move on to the next area that nobody has done before. That, I think, is largely going to manifest in permanent habitations on the Moon and interplanetary travel. The most common destination is Mars right now, but eventually, hopefully not too much longer, sending people out to the outer solar system and asteroids and so on and so forth.

I definitely feel that there’s a role for the agency to play in that. I don’t think that private industry is going to be that invested because it’s just far too expensive and far too risky. But again, that transition is going to have an effect, I think, on engineering. But it’s a challenge, but it’s a good challenge to have. It’s the sort of thing that I think most people join NASA to be able to do, to do things that other people aren’t doing, that are too complex or too risky, but to do it well and to do it methodically and to do it safely and successfully.

So, I would expect in the next 10 years that there will continue to be some evolution of the sorts of missions, particularly in human spaceflight, but it’s not limited to human spaceflight at all that NASA does. And our engineering practices will continue to evolve to meet those challenges of systems that will be far more complex than we can even conceive today.

Host: Many thanks to Steve for sharing his insights and experiences. You can download a free copy of Steve’s NASA-published book, Three Sigma Leadership Or, the Way of the Chief Engineer, via a link on our website at APPEL.NASA.gov/podcast. Links to related APPEL courses, Steve’s bio and a show transcript are also available.

Up next in the Engineering Best Practices series is a conversation with Space Launch System Chief Engineer John Blevins, and that episode is set for release March 23.

Do you have ideas for future guests and topics? It would be fantastic if you’d let us know on Twitter at NASA _APPEL – that’s APP-EL – and use the hashtag Small Steps Giant Leaps.

As always, thanks for listening.