EPISODE 130: INGENUITY’S FINAL FLIGHT: LESSONS LEARNED

Transcript

Andrés Almeida (Host): Welcome to Small Steps, Giant Leaps, the NASA APPEL Knowledge Services podcast. I’m Andrés Almeida, your new host moving forward. In each episode, we focus on the role NASA’s technical workforce plays in advancing the agency’s mission through the development and sharing of knowledge. Today’s guest is Teddy Tzanetos, project manager for the Ingenuity Mars helicopter, a record setting demonstration mission, which soared into history as the first spacecraft to perform powered, controlled flight on another planet. Originally designed for up to five flights, Ingenuity would go on to perform 72 flights, with its final one occurring on January 18, 2024, roughly three years after its initial hop. 
 
As project manager Teddy supervised the mission. He is also manager of the NASA Jet Propulsion Laboratory’s sample recovery helicopters that are expected to be used as part of the Mars sample return mission. Before joining the Jet Propulsion Laboratory in 2017, he was the head of engineering at the Drone Racing League, a New York City-based startup where he managed a team of developers and engineers to build drone brace infrastructure, embedded avionics, and an online first-person view simulator, played by tens of thousands of users worldwide. Let’s dive in.   

Host: Hey, Teddy, thank you for joining us today.  

Tzanetos: How’s it going, Andrés? Thanks for having me. 

Host: Tell us about your role with Ingenuity. 

Tzanetos: It’s been a wild ride. I joined the team back in 2017. And at that point, the original project manager MiMi Aung, and the chief engineer Bob Balaram. They were well on their way they had their engineering model at the beginning of its development phase. And I begged to try and hop on and, you know, any job that was available, I was happy to take. And Bob gave me a shot with some, some maintenance and clearing up of some cables in the closet, and things kind of snowballed from there. My first role was electrical ground support equipment. And what that translates to is, effectively, life support for the robot.

Any time NASA builds a spacecraft, there’s typically a rack, some big box, sitting next to it in the cleanrooms in the test chambers. And that acts as life support, giving power, control signals, data conditioning, things like that. And the helicopter needed that. So, my background in embedded systems, electromagnetism, low physics, and in low level computer science, I said, “Sign me up,” and I held on as tightly as I could. Things lead to the next that became a test conductor that led to at-low delivery. Simply test launch operations. That’s kind of the varsity team within NASA, that’s responsible for putting together the final spacecraft. And at JPL it’s a special set of hands, the blessed hands that are trusted to touch those things.

So I was part of the team that handed things off to that varsity team. Finally, launch operations and two-and-a-half years, three years later, after, after our first flight, you know, now we’re at the end of it and looking back and it’s been a remarkable journey for our baby on Mars. 

Host: Can you describe some of the technical challenges of operating a rotorcraft on Mars?  

 Tzanetos: Yeah. First off, helicopters here on Earth are miracles to begin with, right? The fact that they operate at all and they don’t explode every time they start spinning up really is a mechanical marvel. I encourage listeners to go online and look at videos search for the term “squash plate” and you’ll find some videos of the helicopter mechanism moving in the middle of flight. So that’s tough on its own. But we figured that out here at Earth. It should stand to reason that that same mechanism should work on other planets, right? But what makes flying at Mars particularly hard are three things.

One is the atmosphere is exceptionally thin. It’s about one percent of the density of Earth’s atmosphere here at sea level. When you move your arms around here on Earth, the hairs in the back of your hand will actually feel, you know, the counter pressure. You’ll feel your hand pushing some air. You wouldn’t feel anything at Mars. Alright, that’s how thin it is. It’s the equivalent of flying around 100,000 feet. And there’s a reason you don’t see helicopters flying at 100,000 feet here on Earth. It’s the first challenge.

The second challenge is that Mars is incredibly cold. So any aircraft that you build to fly at Mars needs to withstand the bitter drops in temperature that happen every single night. In the summer times, it can get down to negative, you know, 30,40, 50. In the winter times, down to negative 90. And I’m talking about Celsius her, okay, so, bitter cold.

And the final, you know, lineup here in terms of the big challenges, is the fact that it’s far away, right? We know how to operate robots at Mars, which is very far away for rovers. And that works out well because rovers move pretty slowly. So you can joystick it and actually wait 15 minutes to see what happens. But that sort of lag, you can imagine if you’re playing a video game or flying a drone in your backyard, you’ve got a 15-minute stick lag from when you told the drone to go left and then it went left, it wouldn’t be a good day for you. So that means everything needs to be autonomous on board. The baby needs to be able to see where she’s flying, navigate accordingly and figure it out on her own. within — we’re talking about a two-to-three-minute flight time, right? So, a photon traveling from Mars to Earth just to indicate that the flight has started — by the time that photon reaches Earth, the flight’s already over, okay? Those are the three big challenges. And thankfully, the team did a fantastic job. 

Host: It is a fantastic job, for 72 flights when it was designed for up to five.   

Tzanetos: Exactly.

Host: And it came in for a hard landing off flight 72. What have you learned about what happened?  

Tzanetos: Yeah. A lot. And we’re still learning. The team, actually later on this afternoon, there’ll be some meetings that I’m going to attend, where our GNC [Guidance, Navigation, and Control] team and our mechanical team are still digging through the data to try and better our understanding. We will never know, likely, the full, you know, cause analysis, root cause analysis rather, of what occurred. But here are some of the things that we do know: The area of Mars that Ingenuity was flying over towards the end of her mission — which we’ve now affectionately called the place Valinor Hills for nerdy “Lord of the Rings” reasons — was some of the most featureless terrain that Ingenuity had ever experienced. Because of that, it’s kind of a great victory story for us in that we finally found the edge, right? The whole journey over the last three years is, if you imagine the envelope as a sphere, our flight envelope, we’ve been pushing on in every direction between velocity, how high we can fly above ground level, the automatic landing hazard avoidance. In each vector, we’ve been expanding our envelope, and we now found the edge.

That boundary had to do, we believe the leading theory here is, that it had to do with how featureless the terrain was. That’s important because the way Ingenuity flies is she has a downward looking camera, an altimeter, and a set of inertial sensors like an accelerometer and a gyroscope. Very similar to drones here on Earth. Even if you were to take a drone here on Earth, and you were to put it in a white, pure white room, it wouldn’t be able to visually orient itself and figure out from moment to moment where the features are moving in the field of view. And it uses that information to determine and close the loop of where it actually thinks it is in space.

So going back to Mars here, that terrain near Valinor Hills was very sandy, very fine in terms of the texture of the sand. There weren’t a lot of boulders sticking out. And we saw on the logs in flight 70 and in 71 that the number of features that the baby was able to resolve was dropping and going lower, and she was reporting, “Hey, guys, this is getting very challenging for me to, you know, navigate it.” And there was no other path for us to go. We had to go to the northwest because that’s where the river was headed. So the best thing to do was to keep trying to tweak some parameters if we could, and learn as much as we could and, and we succeeded in that regard. Unfortunately, had the rough landing, we weren’t able to keep flying. But she surprised us still.

And as far as the helicopter, you know, computer system is concerned, it’s healthy as a fiddle. She doesn’t know that the rotor is broken. And she’s going to keep on waking up every single day recording data. We have about 20 years’ worth of storage on board and we’re going to be taking a picture every day for future explorers to you know, get a really great time lapse from. 

Host: You refer to Ingenuity as “the baby.” That means the team had an affinity for it.   

Tzanetos: Oh, yeah.  

Host: So what was the atmosphere like in the room during that final flight or after that final flight?  

Tzanetos: During that downlink, there was a lot of, of course, focus. There was the downlink of the flight and then there was our final downlink, which happened many weeks later and that was the official you know, final downlink. But the downlink of that flight, it was a focused and solemn affair. Because you’ve got to imagine: the whole team over two and a half, three years, some of whom, you know, weren’t there in the beginning, you know, joined up in the middle of operations, some of whom had been there since the beginning. They all prepared, or most of them had prepared, for a sprint. One month, that’s all they were going to invest into this and then move on to some other project.

And it had transformed into a marathon. And now you’re looking at your laptop screen, at an image of a shadow of a broken blade, and you’re thinking to yourself, Okay, this may be the end. Myself, personally, I was in disbelief. At first, I thought it was just some visual tricks being played. In my mind, I said, “No, no, the blade’s fine. It’s just a shadow. There’s probably a hill or the weird angle that the sun’s coming in on, she’s fine. We’ll spin up the blades. And I don’t think I’m seeing what I’m actually seeing.” I remember, very clearly, that was my initial reaction. We were, “Go. Let’s do it again.” And then about 45 seconds later, I’m staring a little harder at the screen. And the neurons, you know, start refreshing in the back of your mind, and you realize the patterns start forming. And you recognize, “Wait a second, that isn’t normal. That’s a little different,” right. And that’s when I think, you know, most of the team started coming around to the realization that, you know, Ingenuity had finally flown her last.

We wanted to do a couple of more experiments after that, and we did, where we tried wiggling the blades. We tried just doing a little kick to the blades to rotate them 180 degrees, so that the downward looking camera could capture the shadow as the other blade came around in rotation. There were even some thoughts of, well, you know, let’s try it anyway, you know, what can we learn if we tried spinning up, I wasn’t a fan of that idea. There was a lot of debate throughout the team, these blades are designed to spin, the rotor system is designed to spin it at close to 2,600 revolutions per minute. That’s one thing to keep in mind. The other thing is that a circle as it rotates, the fastest moving bits are on the outside, right? That means that that’s also where most of your lift is generated, the fastest part of your blade. So if now you’ve lost the outer one-third of your rotor system, you’ve also lost most of the lift generating surfaces. But on top of that, God forbid there’s any imbalance, the whole thing is going to shake itself apart. So after careful analysis, you know, we really did go through and debate a lot of the pros and cons. The team, as a group decided, you know, yep, we’re going to, after our little wiggle tests and our rotational tests, we would stop the actuations there, and then transition to what can we do in terms of a final flight software upgrade I mentioned earlier, about the 20 years worth of storage. And, you know, still have one trick up our sleeve as a team.  

Host: Can you share any key lessons learned from Ingenuity?

Tzanetos: Oh, yeah, a ton. A lot. We’ve written down a lot in the papers that have been published, and a lot of conferences. We’re still learning some. The freshest lessons learned are the ones that we really took to heart for the next set of helicopters that we’ve been building over the last two, three years, as part of the Mars Sample Return architecture, which is currently undergoing a re-architecture phase at the moment. But for the last three years, I’ve been leading the team called the sample recovery helicopters. And our challenge, sidebar for a second: Ingenuity was a technology demonstrator. There’s classes at NASA for risk tolerance. Class A, it must work. Class B must work. But there’s some things that you know, lesser quality, class C, Class D. And then below class, these type two. Ingenuity was all the way to the side on type two meaning high risk, high reward, low budget. Even if the thing never flew or never survived it was okay when it comes to risk tolerance.

From our sample return, it needed to be Class B, okay, meaning no single point failures. So they asked, “Could you take Ingenuity’s design and come up with a Class B answer to how, as a backup, you would pick up sample tubes?” We thought about it long and hard. And we said, “Yeah, we can do that.” So we scaled up Ingenuity from the 1.2-meter coaxial helicopter we scaled up to around 1.4 meter. We added an arm, we added a set of wheels. And we made it all the way through our preliminary design review, which got glowing reviews from our board and confirm that, yeah, the physics holds. We can scale up the aircraft. We can pick up one tube at a time.

And going back to your question of the lessons learned, it really goes to the ability to trust commercial off-the-shelf parts, like a cell phone processor or like a cell phone camera. The tried-and-true heritage methods of designing a lightweight structure and a lightweight aircraft. So much of the SRH (sample recovery helicopter’s) design was borrowed directly from Ingenuity. The underlying principles of our flight software system, we now know works and we knew works. F Prime is a fantastic system for that. The guidance, navigation, and control system, we know works. And we had a lot of ground truth data thanks to 72 flights in two-and-a-half years to know that, yeah, these systems are robust, these systems can provide reliability and, hey, if you’ve got a pair of them, then you know, you can avoid any single point failure there.

We’re also now carrying those lessons forward into even larger endeavors, where we’re designing multicopters that we call the chopper team, we call it, and the goal there is to design a multicopter, you can imagine a hexacopter, so six rotors arranged in a ring. And in the center, you have your fuselage. This thing is huge. It’s about the size bigger actually, than Perseverance itself. This would be the only thing in the rocket if we were to send this to Mars. And the whole point of it is is that you can now access the entire planet. You can fly to the poles, you can fly the equatorial regions down to Valles Marineris, down lava tubes, up cliff walls, and you can bring in this 50-kilo 40-kilos – still in design here – you can bring in this 40-50-kilo aircraft, about five kilos of science payload, which doesn’t sound like a lot compared to a rover. But it’s the access to bring a lightweight payload anywhere.

And you could fly, you know, many kilometers per Sol as opposed to hundreds of meters per Sol, right? So it’s a tradeoff to be sure it’s, you know, different tools for different applications. But we think there are a lot of use cases to be able to tell the science community and future human astronauts, that yeah, we can build aircraft that can fly kilometers per Sol and go anywhere on the planet that you need.

Host: And tradeoffs are important since you said Mars is hard. 

Tzanetos: Yes, there will always be trade offs. There is no free lunch. 

Host: What do we understand about this technology now? And what is still left to learn? Do you think? 

Tzanetos: Yeah, we understand a lot, particularly when it comes to the core physics, of what made Ingenuity hard. I mentioned earlier at the beginning of our chat, the low air density, thermal environment, and the autonomy, right. And we have all three of those now in spades locked up, we know how to deal with the low air density. In partnership with NASA Ames Research Center, our friends at AeroVironment. We know how to build lightweight aircraft, we know how to design hyper-optimized rotor systems that can cut through that thin air and still eke out just enough downward thrust to lift yourself up.

On the thermal side, I haven’t really mentioned this, most of Ingenuity’s energy did not go into flight. It went into overnight survival, close to two-thirds of the battery pack, which is not that different from the batteries inside of your power drill at home, just simple 18-650 lithium ion cells, but two-thirds of that energy, every single Sol just went into powering some strip heaters on the battery to keep things from freezing.

Three years’ worth of up-and-down cycles on your data plots is a ton of data to correlate your models and correlate your theories as to how much energy do you lose to the outside environment from the surfaces? What’s your thermal conductivity paths? What is your efficiency? Every single Sol, how much you’re getting in your solar panel? How much dust is deposited on the panel? How much dust gets cleared per flight, right? So that second problem, again, we have learned an immense amount and are extremely well equipped now to do it again.

And that third one, you know the autonomy I think 72 flights speaks for itself. Clearly, we have a robust flight control system, autonomy system; and not just the software, but the team and the operations that were developed to transform that 30-Sol sprint into the marathon. It’s a well-oiled machine here, ready and willing to keep flying in the skies of Mars.

Host: On episode 88 of this podcast Ingenuity’s chief pilot Håvard Grip mentioned his favorite moments of the demonstration mission. What are some of yours? 

Tzanetos: My most memorable moment you might think would be flight one, and that is special for all of us, but for me, the most nerve wracking actually was on drop-and-drive day. Ingenuity was a stowaway on the rover Perseverance. We were tucked away in the belly and there was a careful ballet of operations between the helicopter operations team, the Perseverance operations team, the robot operators, and a stack of procedures, you know, almost, you know, a foot thick that delved into how do you remove this gentle surgical, lightweight equipment, which was our helicopter out from underneath the massive tank that is Perseverance.

The challenge – there’s a lot of mechanical challenges which we don’t have time to go into – but the energy challenge, just to cut straight to the point here, is that the rover sits, is driving around on the surface looking for a nice flat parking lot to drop us off. And once, once they had found that parking lot, and our team gave a thumbs up, it dropped the carbon fiber debris shield, drove forward and then started unfolding the helicopter. There were a couple of actuators that fired to rotate us vertically, deployed our legs, and then we were ready for the final one. Now that final one was just holding us from the tippy top right above the solar panel, it was a bolt that we get snapped. And the do-you-or-don’t-you situation here is that you have the rover making its own decisions. And then you have the mechanism which could fail or not fail; or not fail, but could work or not work on that Sol. And then you have the humans 15 minutes away, likewise, on Earth, all right? If the rover drops the helicopter, but does not drive off the helicopter, we would die. We later found out that we could survive freezing, but solely in the mission we had never intended to freeze. The threat was death. If the rover thought the helicopter was dropped, but it was not in fact dropped and then began to drive. It could maybe hit a rock and scrape us off like a barnacle, or cause questionable forces now, because you have this pendulum thing hanging on originally underneath and the rover, you know driving over the surface of Mars, which wouldn’t be great.

What we wanted to happen was that the helicopter was actually dropped, the rover thought the helicopter was dropped, and drove off so that we could start getting photons on the solar panels so that we can start charging ourselves and begin our mission. To do all this, I mentioned that 15-minute window, controllers on Earth had about a 15-minute window to hit the button to override what the rover thought it should do.

So we had, we had a copper path going through our umbilical and it would be shorted if the helicopter were connected. But if the helicopter dropped, that circuit would become an open and we were able to see that indication on the rover side. So the rover was sending back telemetry. And, and we said, “Okay, we’re going to plan the sequence out such that the rover will drop us, then the rover will report back the status of that signal bit. And then we’re going to interrogate that here on Earth.” Now the rover will have its intended path in case there’s a solar flare, in case we lose DSN [Deep Space Network] coverage or whatever. But if we still have comms to Mars, we would like the ability to override so that if the rover guesses wrong, or somehow gets confused, human controllers could either tell it to stop and not drive or force it to drive.

Thankfully, on the day of the rover got it right, the human controllers did not have to intervene. But I remember standing over the shoulder of one of our flight software leads at the time. And we were staring at his terminal screen updating, updating, updating, waiting for the first indications of life from Ingenuity after that drop and driving maneuver. And I remember, I will always remember it was two hours and 15 minutes, precisely from the moment of drop, where the rover drove off and two hours and 15 minutes later, a timer in the helicopter triggered, booted up our cell phone processor, and began communicating wirelessly over our cots ZigBee style radio link to the rover base station. And two hours and 15 minutes later, we were staring at a screen refreshing, refreshing, refreshing, waiting for the first bits and signs of life to come back to our controllers here on Earth. And they showed up. The data packets arrived. We realized that could only mean one thing. We unpacked the data packets. The baby was alive and well.  

 I think that that is the most, that is the strongest memory that is seared into my mind and will always be because it was the most crucial from a start-of-our-mission standpoint. It was also the one that it was impossible to test nearly as much as we had tested flying. We had done flying the chamber dozens and dozens of times here on Earth. We are actually where we hold the lead for being the longest tenants of the 25-foot space simulator at JPL in this massive 25-foot in diameter by six-story steel tube and no other spacecraft has stayed in there longer testing than the Ingenuity team had. So the flying, we had a pretty good idea of what that was going to feel like. But the dropping from Perseverance we’d only done, I believe two or three times before launch. So it was, it was a very stressful couple of moments there. But thankfully everything went great.

Host: It’s remarkable. What do you know now that you wish you’d known before Ingenuity launched?

Tzanetos: Part of it’s a personal or psychological aspect for myself and maybe the team. There’s a different way you approach, there’s a different way you might think of how to deal with a marathon versus how you might think to deal with a sprint. And it’s, maybe I wouldn’t change it in hindsight, but I do know, it would change maybe how we had planned for things, or how maybe we would have staffed things, knowing, “Hey, we need people for three years instead of 30 assaults,” right? And we need to build tools that will survive for three years, right, instead of just one time use pieces of code that you know, you’re just going to use for a couple of weeks and then move on with your lives, right. There would have been optimizations to make about that, for sure. Do I think we would have done anything differently as a team in those first 30 Sols aside from maybe having, you know, some more thought-out tools and procedures? No, I think we probably would have done the same thing, just would have been more emotionally prepared for maybe planning vacations out a little more intelligently out on the 31st day, when everyone realized that we needed them back. So I’m incredibly proud. And I will always be incredibly proud of this small team of people.  
 
We just, you know, in the weeks prior had our final downlink, officially, and just had a party in the past week, officially ending the mission at JPL. And not every mission’s like this. Not every team is like this. When you work on large rover missions, there’s hundreds and hundreds of people, there’s a lot of processes results, because it’s tough to know, everybody. We were a small team of about 20, 25, 30 people. And because of that we all knew everyone very closely. We all called each other up at 1 a.m. And there was never, you know, for good reasons, right, you know, say, “Hey, there’s an emergency, we need help.” “Hey, you know, I need to double check the numbers.” And one of the things I’m most grateful for is not once in the last eight years that it’s been an honor to work on this team with – over the last eight years not once did anyone ever respond to that 1 a.m. phone call with frustration. Or, “Do you know what time it is?” Everyone understood. The reason everyone understood the dream that we all shared. And there was no cost that was too high. There was no weekend that was sacred. There was no late night call that could not be allowed. There was a special allowance afforded for the Ingenuity dream, and I hope to be lucky enough and be privileged enough to have a chance to work with those people again, and on such a special mission.

Host: That’s wonderful. Teddy, do you have any closing thoughts?

Tzanetos: Yes. I hope that the legacy of Ingenuity does not just stop at a flight log, a set of statistics, scoreboard of flights and kilometers flown and things like that. But I hope and I’m the team is working very hard to ensure that Ingenuity’s legacy carries beyond that, and has an impact when it comes to the development of future projects. For current project managers and future project managers that may be listening, I ask that you are willing to take a look at some of the things that the Ingenuity team was able to accomplish when it comes to taking risks, and trying out new processes that may not be the tried and true methods from the ‘70s, the ‘80s, the ‘90s, and there will be risk. And risk equates to cost and cost is a huge motivator, especially these days when it comes to trying to deliver a project on time and under budget. But risk can also have benefits and when managed with a small team that you trust, those risks can be worth it. And I hope that Ingenuity’s legacy of pushing that technological envelope is carried forward in as many endeavors as possible in the in the listenership that we’re lucky to have here today. So don’t be afraid to take risks. Don’t be afraid to push the envelope. There is value on the other side of that on the other side of that risk trade. And it could be the value that the team needs to get the job done. 

Host: Great advice Teddy, thank you for joining us today. Thank you for your time.

Tzanetos: My pleasure. Thank you for having me. It’s been a journey of a lifetime and couldn’t be more happy to share the message of Ingenuity. 

Host: That’s it for this episode of Small Steps, Giant leaps. For a transcript of this show and more about Teddy Tzanetos or the topics we discussed today, visit our resources page at appel.nasa.gov/podcast. And don’t forget to check out our other podcasts like “Houston, We Have a Podcast” or “Curious Universe.” Thanks for listening.