Academy Case Study: The Deepwater Horizon Accident Lessons for NASA

Sunlight illuminated the lingering oil slick off the Mississippi Delta on May 24, 2010.

Photo Credit: NASA

May 10, 2011 Vol. 4, Issue 3

The events leading up to the Deepwater Horizon accident offers several cautionary lessons for NASA.

Offshore oil drilling is complex work that employs state-of-the art technology in an extremely dynamic environment. It involves using a drilling rig to penetrate the ocean floor, and installing and cementing pipe to secure the wellbore (well hole). Drilling mud is used to lubricate the drill bit and control pressure exerted by hydrocarbons unearthed during drilling. The rig drills progressively deeper until it reaches a depth where returns from the well can be maximized. At that point, drilling stops, the well is cemented and capped, and the rig is removed. The well owner returns later (typically with a less expensive production rig) to collect the oil.

On April 20, 2010, the Deepwater Horizon rig was finishing up a drilling job at the Macondo lease site, a plot in the Gulf of Mexico 49 miles off the coast of Louisiana. At the time, the job was 43 days over schedule and $21 million over budget due to additional leasing fees. At 9:49 p.m., the rig exploded, leading to 11 deaths and the worst oil spill in U.S. history.

The lessons from this tragedy are potent reminders of the pitfalls that can plague complex programs and projects in any industry, even (perhaps especially) those with long track records of success. Prior to the accident, Deepwater Horizon was one of the best-performing deepwater rigs in BP’s fleet. In September 2009, it had drilled to a world-record total depth of 35,055 feet. As of April 2010, it had not had a single “lost-time incident” in seven years of drilling. The deficiencies that set the stage for this tragedy—government oversight, disregard for data, testing, changes to processes and procedures, safety culture, and communications—are common to other high-stakes, high-visibility accidents and failures.

The text below includes edited excerpts from “Deep Water: The Gulf Oil Disaster and the Future of Offshore Drilling,” the final report to President Obama by the National Commission on the BP Deepwater Horizon Oil Spill and Offshore Drilling. (The report is in the public domain.)

View a decision timeline for the events leading up to the explosion.

Government Oversight

• The Minerals Management Service (MMS) faced personnel constraints in the Gulf of Mexico region that limited its ability to exercise effective oversight. The Safety Oversight Board found that “the Pacific Region employs 5 inspectors to inspect 23 production facilities—a ratio of 1 inspector for every 5 facilities. By contrast, the [Gulf of Mexico Region] employs 55 inspectors to inspect about 3,000 facilities—a ratio of 1 inspector for every 54 facilities.”
• At the time of the blowout, MMS had not published a rule mandating that all oil rig operators have plans to manage safety and environmental risks—more than 20 years after a rule was first proposed. The agency’s efforts to adopt a more rigorous and effective risk-based safety regulatory regime were repeatedly revisited, refined, delayed, and blocked alternatively by industry or skeptical agency political appointees. MMS thus never achieved the reform of its regulatory oversight of drilling safety consonant with practices that most other countries had embraced decades earlier.
• Other MMS regulatory initiatives critical to safety faced strong and effective opposition. In 2003, the White House stiffly opposed MMSs efforts to update its requirements for the reporting of key risk indicators.
• As MMS’s resources lagged behind the industry’s expansion into deepwater drilling—with its larger-scale and more demanding technology, greater pressures, and increasing distance from shore-based infrastructure and environmental and safety resources—the agency’s ability to do its job was seriously compromised. Of particular concern, MMS was unable to maintain up-to-date technical drilling-safety requirements to keep up with industry’s rapidly evolving deepwater technology. As drilling technology evolved, many aspects of drilling lacked corresponding safety regulations. The regulations increasingly lagged behind industry and what was happening in the field.
• When industry contended that blowout-preventer stacks—the critical last line of defense in maintaining control over a well—were more reliable than the regulations recognized, warranting less frequent pressure testing, MMS conceded and halved the mandated frequency of tests. Soon afterward, a series of third-party technical studies raised the possibility of high failure rates for the blowout preventers’ control systems, annular rams, and blind-shear rams under certain deepwater conditions and due to changes in the configuration and strength of drill pipe used by industry. Two studies commissioned by MMS found that many rig operators, by not testing blowout preventers, were basing their representations that the tool would work “on information not necessarily consistent with the equipment in use.” Yet, MMS never revised its blowout-preventer regulations nor added verification as an independent inspection item in light of this new information.
• The regulations did not mandate that MMS regulators inquire into the specifics of “rupture disks,” “long string” well designs, cementing process, the use of centralizers, lockdown sleeves, or the temporary abandonment procedures. The MMS personnel responsible for deciding whether the necessary drilling permits were granted lacked the expertise that would have been necessary in any event to determine the relative safety of the well based on any of these factors.
• MMS performed no meaningful National Environmental Policy Act (NEPA) review (e.g., an environmental impact assessment) of the potentially significant adverse environmental consequences associated with its permitting for drilling of BP’s Macondo well.
• Notwithstanding the enormously important role cementing plays in well construction—especially in the high-pressure conditions often present in deepwater drilling—there were no meaningful regulations governing the requirements for cementing a well and testing the cement used. Nor were there regulations governing negative-pressure testing of the well’s integrity—a fundamental check against dangerous hydrocarbon incursions into an underbalanced well. On many of these critical matters, the federal regulations either failed to account for the particular challenges of deepwater drilling or were silent altogether.

Disregard for Data

• Well design. BP’s design team originally had planned to use a “long string” production casing—a single continuous wall of steel between the wellhead on the seafloor, and the oil and gas zone at the bottom of the well. But after encountering cracking in the rock formation on the ocean floor on April 9, which limited the depth to which the rig would be able to drill, they were forced to reconsider. As another option, they evaluated a “liner”— a shorter string of casing hung lower in the well and anchored to the next higher string. A liner would result in a more complex—and theoretically more leak-prone—system over the life of the well. But it would be easier to cement into place at Macondo.On April 14 and 15, BP’s engineers, working with a Halliburton engineer, used sophisticated computer programs to model the likely outcome of the cementing process. When early results suggested the long string could not be cemented reliably, BP’s design team switched to a liner. But that shift met resistance within BP. The engineers were encouraged to engage an in-house BP cementing expert to review Halliburton’s recommendations. That BP expert determined that certain inputs should be corrected. Calculations with the new inputs showed that a long string could be cemented properly. The BP engineers accordingly decided that installing a long string was “again the primary option.”
• Centralizers.As the crew gradually assembled and lowered the casing, they paused several times to install centralizers at predetermined points along the casing string. Centralizers are critical components in ensuring a good cement job. When a casing string hangs in the center of the wellbore, cement pumped down the casing will flow evenly back up the annulus, displacing any mud and debris that were previously in that space and leaving a clean column of cement. If the casing is not centered, the cement will flow preferentially up the path of least resistance—the larger spaces in the annulus—and slowly or not at all in the narrower annular space. That can leave behind channels of drilling mud that can severely compromise a primary cement job by creating paths and gaps through which pressurized hydrocarbons can flow.BP’s original designs had called for 16 or more centralizers to be placed along the long string. But on April 1, team member Brian Morel learned that BP’s supplier (Weatherford) had in stock only six “subs”—centralizers designed to screw securely into place between sections of casing. The alternative was to use “slip-on” centralizers—devices that slide onto the exterior of a piece of casing where they are normally secured in place by mechanical “stop collars” on either side. These collars can either be welded directly to the centralizers or supplied as separate pieces. The BP team—and Wells Team Leader John Guide in particular—distrusted slip-on centralizers with separate stop collars because the pieces can slide out of position or, worse, catch on other equipment as the casing is lowered.Shortly after the BP team decided on the long string, Halliburton engineer Jesse Gagliano ran computer simulations using proprietary software called OptiCem, in part to predict whether mud channeling would occur. OptiCem calculates the likely outcome of a cement job based on a number of variables, including the geometry of the wellbore and casing, the size and location of centralizers, the rate at which cement will be pumped, and the relative weight and viscosity of the cement compared to the mud it displaces. Gagliano’s calculations suggested that the Macondo production casing would need more than six centralizers.

  Gagliano told BP engineers Mark Hafle and Brett Cocales about the problem on the afternoon of April 15. With de facto leader John Guide out of the office, Gregory Walz, the BP Drilling Engineering Team Leader, obtained permission from senior manager David Sims to order 15 additional slip-on centralizers—the most BP could transport immediately in a helicopter. That evening, Gagliano reran his simulations and found that channeling due to gas flow would be less severe with 21 centralizers in place. Late that night, Walz sent an e-mail to Guide explaining that he and Sims felt that BP needed to “honor the [OptiCem] modeling to be consistent with our previous decisions to go with the long string.”

  When Guide learned the next day of the decision to add more centralizers, he initially deferred, but then challenged the decision. Walz had earlier assured Guide that the 15 additional centralizers would be custom-designed one-piece units that BP had used on a prior well and would limit the potential for centralizer “hang up.” But when the centralizers arrived, BP engineer Brian Morel, who happened to be out on the rig, reported that the centralizers were of conventional design with separate stop collars. Morel e-mailed BP drilling engineer Brett Cocales to question the need for additional centralizers. Cocales responded that the team would “probably be fine” even without the additional centralizers and that “Guide is right on the risk/reward equation.”

  Guide pointed out to Walz that the new centralizers were not custom-made as specified. “Also,” he noted, “it will take 10 hrs to install them.” He complained that the “last minute addition” of centralizers would add 45 pieces of equipment to the casing that could come off during installation, and concluded by saying that he was “very concerned.” In the end, Guide’s view prevailed; BP installed only the six centralizer subs on the Macondo production casing.

Testing

• • Cement testing.Halliburton prepared cement for the Macondo well that had repeatedly failed Halliburton’s own laboratory tests. Despite those test results, Halliburton managers onshore let its crew and those of Transocean and BP on the Deepwater Horizon continue with the cement job, apparently without first ensuring good stability results.On February 10, soon after the Deepwater Horizon began work on the well, Jesse Gagliano asked Halliburton laboratory personnel to run a series of “pilot tests” on the cement blend stored on the Deepwater Horizon that Halliburton planned to use at Macondo. They tested the slurry and reported the results to Gagliano. He sent the laboratory report to BP on March 8 as an attachment to an e-mail in which he discussed his recommended plan for cementing an earlier Macondo casing string.The reported data that Gagliano sent to BP on March 8 included the results of a single foam stability test. To the trained eye, that test showed that the February foam slurry design was unstable. Gagliano did not comment on the evidence of the cement slurry’s instability, and there is no evidence that BP examined the foam stability data in the report at all.

    Documents identified after the blowout reveal that Halliburton personnel had also conducted another foam stability test earlier in February. The earlier test had been conducted under slightly different conditions than the later one and had failed more severely. It appears that Halliburton never reported the results of the earlier February test to BP.

    Halliburton conducted another round of tests in mid-April, just before pumping the final cement job. By then, the BP team had given Halliburton more accurate information about the temperatures and pressures at the bottom of the Macondo well, and Halliburton had progressed further with its cementing plan. Using this information, the laboratory personnel conducted several tests, including a foam stability test, starting on approximately April 13. The first test Halliburton conducted showed once again that the cement slurry would be unstable. The Commission concluded that Halliburton did not report this information to BP. Instead, it appeared that Halliburton personnel subsequently ran a second foam stability test, this time doubling the pre-test “conditioning time” to three hours.

    The evidence suggested to the Commission that Halliburton began the second test at approximately 2:00 a.m. on April 18. That test would normally take 48 hours. Halliburton finished pumping the cement job just before 48 hours would have elapsed. Although the second test at least arguably suggests the foam cement design used at Macondo would be stable, it is unclear whether Halliburton had results from that test in hand before it pumped the job. Halliburton did not send the results of the final test to BP until April 26, six days after the blowout.

  • Negative pressure test. The negative-pressure test checks not only the integrity of the casing, like the positive pressure test, but also the integrity of the bottomhole cement job. At the Macondo well, the negative pressure test was the only test performed that would have checked the integrity of the bottomhole cement job. Instead of pumping pressure into the wellbore to see if fluids leak out, the crew removes pressure from inside the well to see if fluids, such as hydrocarbons, leak in, past or through the bottomhole cement job.The crew ran the drill pipe down to approximately 8,367 feet below sea level and then pumped a “spacer”—a liquid mixture that serves to separate the heavy drilling mud from the seawater—followed by seawater down the drill pipe to push (displace) 3,300 feet of mud from below the mud line to above the blowout preventer (BOP).BP had directed mud engineers from M-I SWACO on the rig to create a spacer out of two different lost-circulation materials left over on the rig—the heavy, viscous drilling fluids used to patch fractures in the formation when the crew experiences lost returns. M-I SWACO had previously mixed two different unused batches, or “pills,” of lost-circulation materials in case there were further lost returns. BP wanted to use these materials as spacer in order to avoid having to dispose of them onshore as hazardous waste pursuant to the Resource and Conservation Recovery Act, exploiting an exception that allows companies to dump water-based “drilling fluids” overboard if they have been circulated down through a well. At BP’s direction, M-I SWACO combined the materials to create an unusually large volume of spacer that had never previously been used by anyone on the rig or by BP as a spacer, nor been thoroughly tested for that purpose.Once the crew had displaced the mud to above the BOP, they shut an annular preventer in the BOP, isolating the well from the downward pressure exerted by the heavy mud and spacer in the riser. The crew could now perform the negative-pressure test using the drill pipe: it would open the top of the drill pipe on the rig, bleed the drill pipe pressure to zero, and then watch for flow. The crew opened the drill pipe at the rig to bleed off any pressure that had built up in the well during the mud-displacement process. The crew tried to bleed the pressure down to zero, but could not get it below 266 psi. When the drill pipe was closed, the pressure jumped back up to 1,262 psi.

    The crew had noticed that the fluid level inside the riser was dropping, suggesting that spacer was leaking down past the annular preventer, out of the riser, and into the well. A manager from Transocean ordered the annular preventer closed more tightly to stop the leak.

    With that problem solved, the crew refilled the riser and once again opened up the drill pipe and attempted a second time to bleed the pressure down to 0 psi. This time, they were able to do so. But when they shut the drill pipe in again, the pressure built back up to at least 773 psi. The crew then attempted a third time to bleed off the pressure from the drill pipe, and was again able to get it down to 0 psi. When the crew shut the well back in, however, the pressure increased to 1,400 psi. At this point, the crew had bled the drill-pipe pressure down three times, but each time it had built back up. For a successful negative-pressure test, the pressure must remain at 0 psi when the pipe is closed after the pressure is bled off.

    The Transocean crew and BP Well Site Leaders met on the rig floor to discuss the readings. According to post-incident statements from both Well Site Leaders, one of the technicians suggested that the 1,400 psi pressure on the drill pipe was being caused by a phenomenon called the “bladder effect”— heavy mud in the riser was exerting pressure on the annular preventer, which in turn transmitted pressure to the drill pipe.

    According to the Transocean manager, after a lengthy discussion, BP Well Site Leader Vidrine then insisted on running a second negative-pressure test, this time monitoring pressure and flow on the kill line rather than the drill pipe. (The kill line is one of three pipes, each approximately 3 inches in diameter, that run from the rig to the BOP to allow the crew to circulate fluids into and out of the well at the sea floor.) The pressure on the kill line during the negative- pressure test should have been identical to the pressure on the drill pipe, as both flow paths went to the same place (and both should have been filled with seawater). Vidrine apparently insisted the negative test be repeated on the kill line because BP had specified that the test would be performed on the kill line in a permit application it submitted earlier to MMS.

    For the second test, the crew opened the kill line and bled the pressure down to 0 psi. A small amount of fluid flowed, and then stopped. Rig personnel left the kill line open for 30 minutes but did not observe any flow from it. The test on the kill line thus satisfied the criteria for a successful negative pressure test—no flow or pressure buildup for a sustained period of time. But the pressure on the drill pipe remained at 1,400 psi throughout. The Well Site Leaders and crew never appear to have reconciled the two different pressure readings. The “bladder effect” may have been proposed as an explanation for the anomaly—but based on available information, the 1,400 psi reading on the drill pipe could only have been caused by a leak into the well. Nevertheless, at 8 p.m., BP Well Site Leaders, in consultation with the crew, made a key error and mistakenly concluded the second negative test procedure had confirmed the well’s integrity. They declared the test a success and moved on to the next step in preparing to abandon the well.

Changes to Processes and Procedures

• BP did not have adequate controls in place to ensure that key decisions in the months leading up to the blowout were safe or sound from an engineering perspective. While initial well design decisions undergo a serious peer review process and changes to well design are subsequently subject to a management of change (MOC) process, changes to drilling procedures in the weeks and days before implementation are typically not subject to any such peer-review or MOC process. At Macondo, such decisions appear to have been made by the BP Macondo team in ad hoc fashion without any formal risk analysis or internal expert review.
• Maximizing Returns.BP decided to reduce the risk of lost returns in exchange for a less-than-optimal rate of cement flow. In the days leading up to the final cementing process, BP engineers focused heavily on the biggest challenge: the risk of fracturing the formation and losing returns. BP Wells Team Leader John Guide explained after the incident that losing returns “was the No. 1 risk.”The BP team’s concerns led them to place a number of significant constraints on Halliburton’s cementing design. The first compromise in BP’s plan was to limit the circulation of drilling mud through the wellbore before cementing. Optimally, mud in the wellbore would have been circulated “bottoms up”—meaning the rig crew would have pumped enough mud down the wellbore to bring mud originally at the bottom of the well all the way back up to the rig. There are at least two benefits to bottoms up circulation. Such extensive circulation cleans the wellbore and reduces the likelihood of channeling. And circulating bottoms up allows technicians on the rig to examine mud from the bottom of the well for hydrocarbon content before cementing. But the BP engineers feared that the longer the rig crew circulated mud through the casing before cementing, the greater the risk of another lost-returns event. Accordingly, BP circulated approximately 350 barrels of mud before cementing, rather than the 2,760 barrels needed to do a full bottoms up circulation.BP compromised again by deciding to pump cement down the well at the relatively low rate of 4 barrels or less per minute. Higher flow rates tend to increase the efficiency with which cement displaces mud from the annular space. But the increased pump pressure required to move the cement quickly would mean more pressure on the formation (ECD) and an increased risk of lost returns. BP decided to reduce the risk of lost returns in exchange for a less-than-optimal rate of cement flow.

  BP made a third compromise by limiting the volume of cement that Halliburton would pump down the well. Pumping more cement is a standard industry practice to insure against uncertain cementing conditions: more cement means less risk of contamination and less risk that the cement job will be compromised by slight errors in placement. But more cement at Macondo would mean a higher cement column in the annulus, which in turn would exert more pressure on the fragile formation below. Accordingly, BP determined that the annular cement column should extend only 500 feet above the uppermost hydrocarbon-bearing zone (and 800 feet above the main hydrocarbon zones), and that this would be sufficient to fulfill MMS regulations of “500 feet above the uppermost hydrocarbon-bearing zone.” However, it did not satisfy BP’s own internal guidelines, which specify that the top of the annular cement should be 1,000 feet above the uppermost hydrocarbon zone.

• Temporary Abandonment Procedures. Drilling the Macondo well had required a giant offshore rig of Deepwater Horizon’s capabilities. By contrast, BP, like most operators, would give the job of “completing” the well to a smaller (and less costly) rig, which would install hydrocarbon-collection and -production equipment. To make way for the new rig, the Deepwater Horizon would have to remove its riser and blowout preventer from the wellhead—and before it could do those things, the crew had to secure the well through a process called “temporary abandonment.”
• BP’s Macondo team had made numerous changes to the temporary abandonment procedures in the two weeks leading up to April 20. For example, in its April 12 drilling plan, BP had planned (1) to set the lockdown sleeve before setting the surface cement plug and (2) to set the surface cement plug in seawater only 6,000 feet below sea level (as opposed to 8,367 feet). The April 12 plan did not include a negative- pressure test. On April 14, Morel sent an e-mail entitled “Forward Ops” setting forth a different procedure, which included a negative-pressure test but would require setting the surface cement plug in mud before displacement of the riser with seawater. On April 16, BP sent an Application for Permit to Modify to MMS describing a temporary abandonment procedure that was different from the procedure in either the April 12 drilling plan, the April 14 e-mail, or the April 20 “Ops Note” (see below). There is no evidence that these changes went through any sort of formal risk assessment or management of change process.
• At 10:43 a.m. on April 20, a BP engineer e-mailed an “Ops Note” to the rest of the Macondo team listing the temporary abandonment procedures for the well. It was the first time the BP Well Site Leaders on the rig had seen the procedures they would use that day. BP first shared the procedures with the rig crew at the 11 a.m. pre-tour meeting that morning.

Safety Culture

• BP has proclaimed the importance of safety for its vast worldwide operations. “Our goal of ‘no accidents, no harm to people and no damage to the environment’ is fundamental to BP’s activities,” stated the company’s Sustainability Review 2009. Since 1999, injury rates and spills have reduced by approximately 75%.” Yet despite the improvement in injury and spill rates during that decade, BP has caused a number of disastrous or potentially disastrous workplace incidents that suggest its approach to managing safety has been on individual worker occupational safety but not on process safety. These incidents and subsequent analyses indicate that the company does not have consistent and reliable risk-management processes.
  • Between May 29 and June 10, 2000, BP’s Grangemouth Complex on Scotland’s Firth of Forth suffered three potentially life-threatening accidents: a power- distribution failure leading to the emergency shutdown of the oil refinery; the rupture of a main steam pipe; and a fire in the refinery’s fluidized catalytic cracker unit (which turns petroleum into gasoline).
  • In November 2003, a gas line ruptured on BP Forties Alpha platform in the North Sea, flooding the platform with methane. It was a windy day and there was no spark to ignite the gas.
  • On March 23, 2005, a blast at BP’s Texas City refinery—the third largest refinery in the United States—killed 15 people and injured more than 170.
  • In March 2006—one year after the Texas City refinery explosion and one year before the Chemical Safety Board report on it—BP had yet another significant industrial accident. Its network of pipelines in Prudhoe Bay, Alaska, leaked 212,252 gallons of oil into the delicate tundra environment—the worst spill ever recorded on the North Slope.22 The leak went undetected for as long as five days. Upon analysis, the pipes were found to have been poorly maintained and inspected.
  • A survey of the Transocean crew regarding “safety management and safety culture” on the Deepwater Horizon conducted just a few weeks before the accident hints at the organizational roots of the problem. The research, conducted at Transocean’s request, involved surveys and interviews with hundreds of employees onshore and on four rigs, including Deepwater Horizon, which was surveyed from March 12 to March 16. The reviewers found Deepwater Horizon “relatively strong in many of the core aspects of safety management.” But there were also weaknesses. Some 46 percent of crew members surveyed felt that some of the workforce feared reprisals for reporting unsafe situations, and 15 percent felt that there were not always enough people available to carry out work safely. Some Transocean crews complained that the safety manual was “unstructured,” “hard to navigate,” and “not written with the end user in mind”; and that there is “poor distinction between what is required and how this should be achieved.” According to the final survey report, Transocean’s crews “don’t always know what they don’t know. Front line crews are potentially working with a mindset that they believe they are fully aware of all the hazards when it’s highly likely that they are not.”

Communications

• All of the issues above — government oversight, disregard for data, testing, changes to processes and procedures, and safety culture — have communications implications.
• BP, Transocean, and Halliburton failed to communicate adequately. Information appears to have been excessively compartmentalized at Macondo as a result of poor communication. BP did not share important information with its contractors, or sometimes internally even with members of its own team. Contractors did not share important information with BP or each other. (See, for example, the subhead “Cement Testing” above.) As a result, individuals often found themselves making critical decisions without a full appreciation for the context in which they were being made (or even without recognition that the decisions were critical).
• Transocean failed to adequately communicate to its crew lessons learned from an eerily similar near-miss on one of its rigs in the North Sea four months prior to the Macondo blowout. On December 23, 2009, gas entered the riser on that rig while the crew was displacing a well with seawater during a completion operation. As at Macondo, the rig’s crew had already run a negative-pressure test on the lone physical barrier between the pay zone and the rig, and had declared the test a success. The tested barrier nevertheless failed during displacement, resulting in an influx of hydrocarbons. Mud spewed onto the rig floor—but fortunately the crew was able to shut in the well before a blowout occurred.