Joint Confidence Levels Explained

What are joint confidence levels, and what do they tell us about the state of a project?

August’s Virtual Project Management Challenge session featured Charles Hunt, a senior cost analyst at NASA Headquarters in the Office of Evaluation, Cost Analysis Division, on the topic of joint confidence levels or JCLs.

At NASA, all flight projects with costs above $250 million are required to perform a JCL. A JCL identifies the probability that a given project or program’s cost will be equal to or less than the targeted cost and the schedule will be equal or less than the targeted schedule date. As Hunt explained, it combines components such as cost, schedule, and risk into an assessment of a project’s risk posture. Hunt shared the history and evolution of JCLs at NASA and walked through a sample JCL, highlighting critical points in the process.

Throughout the robust question-and-answer session, Hunt dispelled ten myths about JCLs. (Editor’s note: The following transcription has been edited for clarity.)

Myth #1: It’s just a number.

Hunt: It is a number, but it’s not just a number. A JCL is a way of communicating a project’s programmatic risk posture. So, if I look at the evening news tonight and they say [there is] an 80 percent chance of rain tomorrow, the 80 percent, .8, it’s just a number. [But] what that number is representing to me, to you, to decision makers, is, “I really need to bring an umbrella tomorrow.” Those are the kinds of things that we need to be able to communicate to our decision makers and also externally.

Myth #2: It will solve all of your problems.

Hunt: A JCL is a tool. It’s a tool in your tool kit. Tools do different things. A JCL is looking at one aspect of those things. It’s not one-size-fits-all. It’s not going to solve all of your problems. Again, it’s a way of communicating externally and within the agency what our risk posture is.

Myth #3: It is too rigorous to be gamed.

Hunt: It’s definitely not too rigorous to be gamed. What we’re trying to do is move the dial up….The more rigorous something is, the more it can be gamed. Numbers are a wonderful thing. We’re putting more rigor into it so we can have a better conversation with the projects and the decision makers.

Myth #4: You just have to survive the JCL process one time.

Hunt: We are requiring projects to do it one time at [key decision point] KDP-C, or when they get re-baselined. From a policy standpoint that’s all that’s required from an agency perspective. From a JCL advocate [perspective] there’s a lot of great information that you can get from the JCL process. Though from a policy perspective it’s a one-and-done thing, I would highly encourage projects as they’re going through their JCL analysis, make it part of what you’re doing is just good project management perspective. We’re seeing that from projects. They may not be re-running a JCL, but they’re keeping a lot of the artifacts that they’re doing with a JCL and moving forward.

Myth #5: JCL is hard.    

Hunt: I don’t know if ‘hard’ is the right term. I think a lot of times it becomes hard, and the reason it becomes hard is because as we were talking about it earlier, we’re taking products that the project is already doing—schedule, resource management, risk management—and we’re putting them together. The health of those products really depends on how hard a JCL is. If I’m doing all of my project management stuff, 101 stuff, really well, a JCL is not as hard [as] when I have underlying products that still need work.

Myth #6: The JCL process requires me to produce lots of new products.

Hunt: It’s not requiring you to produce a lot of new products. From a cost perspective, what it’s doing is making you map the cost-characteristics….There [are] a lot of benefits associated with that….The other aspect that’s more work from the risk perspective is this whole risk realization or risk recovery. At the end of the day we think we know what it costs to mitigate certain aspects, we know what the consequence is, but how do we say what the consequence is? We’re kind of hit-or-miss on that right now, and that’s where the JCL adds on top of what’s already being done. Those are two big things that JCLs require projects to do that they may not be doing already 

Myth #7: JCL causes projects to get cancelled.

Hunt: No project has ever been cancelled because of a JCL. Period.

Myth #8: JCL is just black magic.

Hunt: At the end of the day, a JCL is just an integration of products that are already there. If it’s black magic, it’s because a) [the] cost-schedule community is not explaining it up to the decision makers or b) there are issues with the products. If anything, we have as much statistical rigor in JCL analysis as anything, and we’re constantly trying to work on how do you make the analysis less [like] black magic. How do you make it more data driven? How do you make it more based off of history? Or based off of what we think the real risks are going to be? 

Myth #9: JCL requires expertise that doesn’t exist within the program and costs a lot of money.

Hunt: It does not cost a lot of money. It does take resources and time. Most centers have the capability to support projects in terms of implementing JCLs. Because we’re relying on the artifacts the projects already generate, of course we want to have the conversations with the schedulers, risk managers, and those are going to take time, but it doesn’t cost a lot of money. And the potential savings for understanding your risk posture—we’re talking about programs that are hundreds of millions of dollars. The amount of time it takes for an FTE or a WYE to crank out a JCL to understand your risk posture is really not comparable [to the understanding of risk it provides].

Myth #10: JCL is just a fad. It’ll go away if we just wait it out.

Hunt: I wouldn’t bank on it. The momentum is growing, if anything. We have GAO talking JCLs. It’s in people’s vocabulary…. If there’s any momentum at all, it’s for doing them better instead of downshifting to not do JCLs.

For projects looking to learn more about performing a JCL (e.g., how to acquire the proper software tools to run the analysis or how to get a waiver) the Cost Analysis Division is the place to start. Hunt’s slides are available on the PM Challenge website.

Watch the full video.

Visit the Virtual PM Challenge website to view archived presentations.

Learn more about NASA’s Office of Evaluation, Cost Analysis Division.

Learn more about the 2013 NASA Cost Analysis Symposium taking place August 27-29.