They Ran It 174 Times

I was home in Jacksonville from college for Christmas break when it happened. On December 19, 2007, a reactor on the north side of town let go with the force of about 1,400 pounds of TNT, and people felt it downtown, fifteen miles away.

Four men died. Two of them were chemical engineers. One of those two owned the company.

So when you reach for the usual explanation, that somebody cut a corner to save money, ask yourself who exactly you think was fooling whom.

T2 Laboratories was a small chemical manufacturer founded in 1996 by a chemical engineer and a chemist. It started as a solvent blending business and moved into making its own product: MCMT, an octane booster that goes into gasoline. Think of it as a fuel additive plant, the kind of operation that exists in industrial parks in every city in the country.

MCMT was made in batches, in one vessel, in three steps that ran back to back over roughly 48 hours. The step that matters is the first one, called metalation.

There's one piece of chemistry you need, and then I'm done with it. Metalation is exothermic, meaning the reaction makes heat all by itself. That's fine as long as you can pull the heat out faster than the reaction makes it. If you cannot, the mixture gets hotter, and heat makes the reaction run faster, and running faster creates more heat. Reach a certain point and you have a runaway reaction. On a graph the line goes vertical. Practically, it means a vessel full of liquid turns into a vessel full of gas in seconds, and then it's down to whether the vessel holds.

Two things stand between that and the people in the building. Cooling, to take the heat away. And a relief system, to let the pressure out if cooling fails.

T2 had both. Neither was any use against the reaction they actually had.

By December 2007 the plant was running three batches a week to keep up with demand. On December 19 they were making Batch 175.

At 1:23 p.m. the process operator had someone call the owners and ask them to come back to the site. The problem he reported was cooling.

There was nothing in the procedure to tell him what to do about it. The operating procedures in use that day carried no emergency instructions for a loss of cooling. An earlier version of those same procedures had carried them, and told the operator to open the water supply valve and the manual bypass valve all the way. The backup water sitting on site was not something he could get to quickly. That instruction had existed once. By Batch 175 it was not in the book any more, and the operator watching the temperature climb had nothing written to follow.

The owners returned. The owner who was a chemical engineer went into the control room to help. Then he walked out to the reactor, looked at it, told an outside operator there might be a fire and waved people away from the area. Then he went back into the control room.

At 1:33 p.m., ten minutes after the first phone call, the reactor burst.

The explosion killed the owner and the process operator in the control room, and two outside operators who were leaving the reactor area. Thirty-two people were injured. Twenty-seven of them were at work in the nine businesses around the plant, and one more was a truck driver making a delivery. Debris landed up to a mile away. Buildings within a quarter mile were damaged, the City of Jacksonville condemned four of the surrounding businesses' buildings, three of those companies relocated while they rebuilt, and the trucking company next door never reopened.

That trucking company had an office trailer 250 feet from the reactor, and the blast destroyed it. Nobody was inside. Christmas was six days out and the crew had been sent home early. The CSB says plainly that if those people had been at their desks they would likely have been seriously injured or killed. Four dead is not the ceiling of that afternoon. It is what was left after a piece of luck nobody had planned.

The blast was equivalent to 1,400 pounds of TNT.

The easy story here isn't operator error. It's something more comfortable, and I have heard educated people say it out loud about this incident: a little outfit was running a dangerous process on a shoestring budget , and small companies like that cut corners, and eventually it caught up with them.

Check the boxes. Small firm, no big-company safety department, irregular production paid for out of sales, batch size increased to make the economics work, three batches a week by the end. It reads like a story about money.

It falls apart on one fact. The people making those decisions were the ones standing next to the reactor. The owner who approved the process was a chemical engineer, and he died in the control room. The plant manager was a chemical engineer. By 2006 the company had hired two more chemical engineers to run the process on weekday shifts. This was not a plant where management was insulated from the hazard and the workers absorbed it. Everyone in the blast radius was, in the most literal sense, invested.

You can't tell a story about greed when the people who would have profited are the people who were killed. So the interesting question is the one the CSB actually asked. What did four chemical engineers not know?

The answer is uncomfortable, and the CSB stated it as the root cause in one sentence: T2 did not recognize the runaway reaction hazard associated with the MCMT it was producing.

Not "ignored." Did not recognize.

Now watch how many chances there were to recognize it.

The very first full-scale batch, on January 9, 2004, produced an exotherm nobody expected. T2 noted it, adjusted the recipe, added cooling to the first step, and started the next batch. Then Batch 5 ran away, an uncontrolled exothermic reaction in the same step that would kill everyone three years later. In Batch 10 the temperature climbed past expectations again, less violently, from the same exotherm. Three of the first ten batches produced exotherms T2 was not expecting. Yields over that stretch ran from nothing saleable to about 70 percent.

Here is the detail that tells you what kind of investigation was not happening. T2 changed the recipe on every one of those first ten batches. No two were run the same way. So when a batch misbehaved there was no way to learn anything from it, because every variable had moved at once.

In May 2004, after Batch 11, the owner who was a chemical engineer wrote to the investors declaring a successful startup and full-scale production.

Then, on July 28, 2005, on Batch 42, they increased the batch size by a third, and every batch from then until Batch 175 ran at the bigger size.

That is three unexpected exotherms, one of them a full runaway, all survived, all adjusted around, and then a scale-up of 33 percent with no analysis that would tell anyone how much heat the reaction could make or how quickly. A third more material is a third more energy. They ran another 133 batches at the new size, and the CSB found no record of any chemical or process analysis done to support the change.

That is the trap, and it has a name. Peter Susca puts it in one line that I think about constantly: a bad process, carrying significant risk, can consistently provide good outcomes. They ran it 174 times without killing anybody. That is not evidence that a process is safe. It is evidence that the conditions that kill you had not lined up yet.

The two contributing causes are what the good outcomes were hiding.

The cooling system had a single point of failure. There was no redundancy in the design, so one failure took cooling away entirely. The CSB listed six plausible ways for it to happen, from a valve sticking to mineral scale building up inside the jacket. Employees told investigators that T2 ran cooling components until they broke and did no preventive maintenance on them. On a reaction that generates its own heat, cooling is not a utility. It is the safeguard.

The relief system opened far too late. I had this backwards on my first read, and it's the correction I got most out of the report. The reactor's rupture disk was four inches wide, which was fine. It was set at 400 psig, and the owner got that number from the maximum hydrogen the process gave off on a normal day. He sized the emergency device for a good batch.

So the CSB took T2's actual recipe into a laboratory and ran it twelve times, across two kinds of test rig, and found a second reaction nobody at T2 knew existed. Above about 390°F, the sodium starts reacting with the diglymethe, the solvent it's dissolved in, not the material it's supposed to react with. That second reaction is far more violent. In a sealed cell it drove pressure up 32,000 psi and temperature up 2,340°F in a single minute, hard enough to burst the test cells. T2 ran the first step at about 350°F. Forty degrees stood between a normal batch and a reaction they never knew was there.

Once that second reaction starts, the CSB calculated, no relief device of any size would have saved the vessel. Not a bigger disk, not a better one. The only thing that would have worked was opening earlier, during the first reaction, and letting the contents boil and vent so the heat and the reactants left the vessel together. A rupture disk set at 75 psig instead of 400 would likely have done it. Same disk. Same four inches. A different release setpoint.

So the question this issue is named for has a specific answer, and the CSB put it in its findings: the owners were likely unaware of the second exothermic reaction sitting in their own recipe.

They had chances to meet it. The owner who was a chemist developed the process by running about 110 test batches in a one-liter glass reactor. He reported that he never saw extreme exothermic behavior, and that his test temperatures never went above 380°F. The reaction that destroyed the plant begins above 390°F. The laboratory program stopped ten degrees short of it.

Some of it was sitting in the literature the whole time. The CSB found a patent, one already on the list of patents T2 built its own process from, that flagged a sodium-diglyme reaction just above T2's normal operating temperature. And a year before Batch 5 ran away, the Center for Chemical Process Safety published Essential Practices for Managing Chemical Reactivity Hazards, free online, walking through exactly how to characterize a runaway and how to pick a relief set point that would survive one.

Nobody at T2 went looking for either one, because nothing in their training had told them this was a category of hazard that required its own analysis.

The investigative move here is one I had not seen used this way before, and it's worth stealing when your records run out.

They rebuilt the chemistry experimentally. The plant was gone, the people who ran it were dead, and the fire had burned the records of the earlier batches. So the CSB ran the recipe itself, measured how much heat came out and how fast, and used that to derive both the most likely failure scenario and the relief setting that would have stopped it. That converts an argument into a measurement. Before the testing you have a plausible story about a runaway. Afterwards you have 75 against 400, and that's not a matter of opinion.

It wasn't the first time the agency had been here. The CSB had already investigated four runaway reactions caused in part by a company failing to recognize the hazard: Morton International in 1998, Concept Sciences in 1999, MFG Chemical in 2004, and Synthron in 2006. Two of them read like rehearsals for this one. MFG scaled from a 30-gallon test reactor to a 4,000-gallon production reactor without appreciating how differently the bigger one sheds heat. Synthron took an order slightly larger than usual, scaled the recipe to match, and more than doubled the rate of energy released in the vessel. Those four incidents and T2 killed ten people between them and injured more than two hundred.

Then they did the thing that makes this issue matter. Having established that four chemical engineers did not recognize a runaway hazard, they asked why not, and they did not stop at T2's front gate. They followed it back to academia.

The CSB issued exactly three recommendations for this investigation. Not one of them went to T2, which no longer existed. Not one went to OSHA or the EPA. All three went to the American Institute of Chemical Engineers and to ABET, the body that accredits engineering degrees, and what they asked for was this: add reactive hazard awareness to the requirements for a bachelor's degree in chemical engineering.

That is a root cause analysis that ran all the way past the company, past the regulator, and into the curriculum.

Competence is not the same thing as coverage.

Every one of those engineers was qualified. They had accredited degrees. They could run a reaction, size a pump, read a P&ID. What they had never been taught was that a reaction's own heat output is a design input you have to go and measure, and that "we've made it 174 times" tells you nothing about it.

This wasn't one unlucky company's blind spot. In 2006 the Mary Kay O'Connor Process Safety Center surveyed 180 chemical engineering departments in the United States and found that 11 percent required any process safety education in the core degree. Another 13 percent offered it as an elective. Around three quarters of American chemical engineering programs required none of it at all.

And then the detail that stopped me cold. The owner who died in the control room sat on his own university's chemical engineering curriculum advisory board. He was one of the people the profession asks what its graduates ought to be taught.

I would like to say I'd have caught it. I'm not confident I would have. I have signed off on processes where I checked the equipment, the procedure, the training, the permits, and never once asked whether anyone had characterized what the chemistry could do on its worst day. That question wasn't in my toolkit either, and it wasn't absent because I was careless. It was absent because nobody put it there. I didn't put it there.

Which is why the ending of this article is different from every other teardown I have written.

All three recommendations are closed. Two of them, the ones asking AIChE and ABET to put reactive hazard awareness into baccalaureate chemical engineering requirements, are closed at "Exceeds Recommended Action," which is the highest classification the CSB has. In April 2012 the agency publicly applauded AIChE's response.

Two weeks ago you read about Bio-Lab Conyers, where the CSB reiterated a recommendation it first made to the EPA in 2002 asking for reactive hazards to be covered by federal rule, and that one is still open, twenty-four years on. Same agency. Same hazard class. One recommendation went to regulators and is still waiting. One went to educators and was over-delivered inside four years.

I don't think that's a coincidence, and I don't think the lesson is that regulators are useless. But look at what the regulatory route actually delivered here. OSHA inspected T2 the day after the explosion and issued its citations six months later, all of them serious, none of them willful, with proposed penalties totalling $17,550. T2 contested. The case settled. The final figure was $8,600, for four dead men and twenty-eight injured neighbours. That's the machinery the CSB could have aimed its recommendations at.

Instead it found the one lever in this system that nobody had a financial reason to fight, and pulled it. Whether that generalises is the argument I'd like to have with you.

Save this one, or screenshot the three checks below. They are a ten-minute conversation with whoever owns your reaction chemistry.

1. Pick your most exothermic process and ask who measured it, and where they stopped. Not who modelled it, not who has run it safely. Who put the actual reaction mass in a calorimeter and produced a number for heat of reaction and rate. Then ask what temperature the testing stopped at, because that's the part that killed T2. Their own chemist tested to 380°F. The reaction that leveled the plant starts above 390°F.

2. Check what pressure your relief device opens at, and what scenario chose that number. Not just whether it's big enough. T2's disk was set from the hydrogen given off by a normal batch, and the CSB found that at that setting no device of any size would have saved the vessel, while the same disk set lower probably would have. Ask what scenario the calculation assumed, and whether anyone has redone it since the last time the batch size, the recipe, or the rate changed.

3. Go find the near-miss you scaled up past. Somewhere in your plant's history is an excursion that got adjusted around and then followed by an increase in rate, size, or frequency. T2 had three in its first ten batches, and scaled up at Batch 42 anyway. This is the one I expect pushback on, because "we fixed it and moved on" is genuinely how good plants operate. The test I'd apply: was the fix based on understanding what happened, or on the fact that it stopped happening?

The four men who died in Jacksonville were not careless and they were not gambling. They were competent people operating confidently inside a gap in what their profession had taught them. On the 175th run, that gap killed them.

What the CSB decided to do about that was not to write another rule for another plant. It was to change what the next generation of chemical engineers is required to learn before anyone hands them a reactor.

Most of these teardowns end with a recommendation nobody acted on. This one ends with a degree requirement that changed. I wanted to write one of those.

What's the hazard category your training never covered, that you only learned about from an incident? I read every reply.

Before Human Error is a weekly incident teardown for industrial EHS leaders who know "the operator screwed up" is where a lazy investigation stops, not where a real one starts.

Built on the U.S. Chemical Safety Board's final report on T2 Laboratories, approved September 2009, on OSHA's own inspection record for the site, and on the current status of the CSB's three recommendati9ons, checked this week. Two or more independent sources on every factual claim.

Written in a personal capacity. The views are mine, not my employer's, and nothing here is written on their behalf.

Sources

All three figures are from the CSB's investigation report and are credited to the CSB.