Minus Two Minutes

Our best work files itself as nothing.

At 1:24 in the morning on 26 March 2024, the lights went out on a container ship called the Dali, and the Francis Scott Key Bridge had about five minutes left to stand.

The ship had left its berth at Seagirt a little before, bound for Sri Lanka, two harbour pilots aboard and the better part of five thousand containers stacked on deck. Then a complete blackout. Not a flicker. Nearly a hundred thousand tonnes of steel turned into dead weight in the channel, carried by its own momentum and the current and aimed, with the indifference of physics, at one of the piers holding the bridge up.

What happened in the time it had left is the only part of the night the spreadsheets cannot hold.

At around 1:27, the crew got a mayday out. A pilot came over the radio and asked, in the panicked voice of a man watching something he has no way to stop, for the bridge to be closed to traffic. The dispatcher who took it did not open a ticket or convene a working group. He told the officers to hold all traffic until somebody got the ship back under control. Maryland Transportation Authority cars rolled to both ends of the span and stopped the traffic where it sat. Somewhere on the recording a voice remembers the men working on the deck and asks whether anyone can reach the foreman and get them off in time.

There were eight of them, filling potholes for a contractor called Brawner Builders at one in the morning on a bridge the whole city drove over without thinking. The officers had time to stop the cars. They did not have time for the men. At 1:29 the bow met the pier and the central span came down in something close to thirty seconds, and it took the crew into the Patapsco with it. Six of them did not come back up.

Read the timeline again, because the number that matters is the gap. Roughly two minutes between the mayday and the collapse. Two minutes in which the people responsible for that bridge knew, with total clarity, that it was going to be struck, and acted on that knowledge before a single beam had moved. The response to the incident began before the incident happened. By the time the structure gave way, the most consequential decision of the night had already been made and carried out. The cars were stopped. The mayor would say afterwards that the mayday saved many lives, and he was right, and you can prove it by counting the empty lanes.

Now try to log it.

Open whatever your shop uses to record incidents and find me the field for that. There is a time you detected the problem, a time you resolved it, and a duration in between, and the whole apparatus assumes, without once saying so aloud, that detection comes after things start to break. Time to impact is meant to be a positive number. The bridge does not offer a positive number. The bridge offers minus two minutes, and most tooling, handed minus two minutes, will reject it, blink, and quietly record a zero.

That zero is the lie this entire piece is about. The two minutes that decided who lived are the one interval the instrument cannot see.

What the bridge gave us has a name. Negative time to impact: the response reaches the world before the damage does, and the blast radius comes out at zero because somebody got there first. It has a quieter sibling, negative time to detect, where you clock the trouble before it has even started to go wrong, reading the trajectory and calling it while every dial still says green. The fact that we need names for these at all should worry you.

Strip the jargon back and the ideas are almost insultingly simple. Time to detect is the gap between a thing starting to go wrong and somebody noticing. Time to impact is the gap before the trouble lands on someone who never asked for it. Both are assumed, always, to be positive numbers. The trouble comes first, you arrive second, and the whole grammar of the discipline takes it for granted that you are late. Turn either one negative and you have described the best work anyone in this field ever does. The pilot on the Dali was working in negative time. So is every team that ever caught a poisonous change and pulled it back before it reached more than a handful of users. They were responding to an incident that had not happened yet.

Here is the problem, and it is not a software problem, though software is where you will find the body. Every tool we have built to manage incidents encodes the same little machine, and the machine has three stops. Something breaks. You notice. You fix it. Detected, mitigated, resolved, with a clutch of timestamps hung off each stop so we can compute the famous numbers. We cannot even agree what the R in MTTR stands for, repair or recovery or resolution, and we will argue about it in good faith at a conference until the bar closes, but every faction agrees on the one thing that matters here. Whatever the R is, it comes after the break. The clock is bolted to the floor and it only counts up. There is no field for the response that arrived before the failure, because the schema was built by people who never imagined you might be early.

You can watch this happen in a room. Describe negative time to detect to a dozen competent engineers and half of them will stall, not because they are slow but because you have handed them an idea their model has no slot for. You can watch them try to file it under “detected” and find nothing in front of it. The blank look is not stupidity. It is data. A vocabulary that cannot hold an idea makes the idea hard to think, and a profession that cannot think an idea has no way to reward the people who keep doing it anyway.

So we have built an entire measurement discipline on the quiet assumption that we are always too late, and then we act surprised when being early shows up as nothing at all. The tool did not decide that detection comes after the break. People decided that. People who believed, somewhere underneath the process diagrams, that the correct moment to respond to a fire is the moment you can finally smell the smoke.

I know the type. I used to be one of them.

The world I am about to describe is mostly gone now, paved over by continuous delivery and pipelines that ship a hundred times a day, and good riddance to most of it. But there was a time, not so long ago, when change was a scheduled event. You did not ship on a Tuesday afternoon because you felt like it. You booked a window. A slot, approved weeks out by a committee that met on Thursdays, in which you were permitted to touch production while the rest of the company slept. Two in the morning on a Saturday, four hours, rollback plan attached. The window was sacred. The window was the whole liturgy.

And every so often a team would blow straight through it.

Not the good teams. The good teams were home in bed by three. The ones who overran were the ones who had planned the work on the back of a serviette, brought the wrong people, or brought no people, and discovered at the worst possible hour that step seven of eleven did not do what the runbook promised. By rule, any change that ran past its window stopped being a change and became an incident. That was the bright line, and I was the person standing on it with a clipboard.

Here is the part I am not proud of. These teams could see the wall coming. They were not always competent but they were not blind, and somewhere around hour two of a four-hour window a sensible engineer can do the arithmetic and work out that this is not going to land. So they would call. Sometimes hours before the window closed, they would try to pull an incident manager onto a bridge, to get help standing by for the moment it tipped over. And I would tell them no. Not yet. The window has not ended. This is your change and your mess, and I am not the cleanup squad for an afternoon of bad planning. Come back when it breaks.

I thought I was defending something. The integrity of the process. The principle that the incident channel was not a crutch for people who could not run a deployment. It felt like rigour. It felt like holding the line.

It was a boomerang, and I had thrown it myself. The second the clock struck the end of the window, the thing I had refused to look at became, by definition, an incident, and it landed on the side of my head with the full weight of however many hours I had spent insisting it was not my problem. Except now the team was exhausted, the change had been bleeding quietly into production for half the night, and the easy rollback we could have run at hour two was a tangled horror at hour five. I had not prevented anything. I had taken a recoverable situation, made everyone watch it deteriorate until a clock gave me permission, and then run the incident I could have got ahead of, slower and sicker and later than it ever needed to be.

That is the whole disease in one anecdote. Those teams planned like clowns, and they were still right about the one thing that mattered. They read the trajectory and called it before the impact. That is negative time to detect, delivered by the least disciplined people in the building, and I sent it back with a lecture about planning. I had been handed the exact response this entire essay is in praise of, and I refused it on principle, because the principle said you do not get to call it an incident until it has finished becoming one.

I was a clock bolted to the floor. I only counted up.

Here is the cruel arithmetic of doing this well. The better you are at it, the less it looks like you did anything at all. A fire put out before it spreads is indistinguishable, on the incident report, from a fire that was never going to spread. Stop the disaster early enough and you have not averted a catastrophe, you have merely had a quiet night, and quiet nights do not get budget, and they do not get thanks, and after enough of them somebody starts asking what they are paying you for. This is the prevention paradox, and it is the structural condition of the entire trade. The reward for being early is to be doubted.

The purest case the world has ever run was Y2K.

For most of the 1990s an enormous body of people went line by line through the planet’s ageing software, expanding two-digit years into four, because a great many systems built when memory was expensive had no way to tell 2000 apart from 1900. The bill ran to somewhere between three and six hundred billion dollars, depending on whose accounting you trust, with the better part of a decade of labour behind it. Then midnight came on the first of January, and the planes stayed in the sky, and the grids held, and the banks opened on Monday as if nothing had happened. Which, as far as anyone could see, it had not.

So the verdict came in fast and it came in cruel. Hoax. Hysteria. The greatest racket consultants ever ran. People who had not slept properly since 1997 read in the paper that they had spent six hundred billion dollars frightening themselves over a calendar. And the maddening part, the part that makes Y2K the perfect parable and not merely a sad one, is that the sceptics could not be cleanly proven wrong. A few countries spent almost nothing and rolled into the new century about as smoothly as the ones that had spent fortunes. There is no second Earth where nobody did the work, sitting in a lab so we can compare. A disaster that was prevented and a disaster that was never coming leave behind precisely the same evidence, which is to say none, and an absence will not testify on your behalf.

You do not need a calendar rolling over to see it. It happens in your pipeline every week. A change goes out to one percent of traffic, the error rate lifts its head, and a guardrail or a human paying attention rolls it back before it ever reaches the other ninety-nine. The blast radius is a rounding error. Nobody writes a postmortem for it, because there is no post and there was no mortem, and the engineer whose instinct caught it does not get an incident with their name on it the way they would have if they had let it burn and put it out heroically at three in the morning. We have built a discipline that pays out for the heroic recovery and stays silent on the quiet save, and then we wonder why people learn to wait for the fire.

If you want the shape of the thing in something heavier than software, put two volcanoes side by side.

In 1991 Mount Pinatubo in the Philippines woke up after five centuries. Volcanologists from the local institute and the United States Geological Survey watched it for weeks, read the seismographs and the gas, called the big eruption before it came, and moved more than sixty thousand people off the mountain. The eruption was one of the largest of the century. A few hundred died, most of them under roofs that gave way beneath wet ash. The forecast is reckoned to have saved somewhere between five and twenty thousand lives, and the whole monitoring effort cost under a million and a half dollars. Those saved thousands are not a figure you will find on any memorial, because saved lives do not gather in one place to be counted. They went home. The catastrophe is invisible precisely because it did not happen.

You want to know what it would have looked like if it had. Look six years earlier and a continent across.

In 1985 the Nevado del Ruiz volcano in Colombia gave every warning a mountain can give. Scientists had watched it for months. A hazard map went out in October marking the town of Armero, in the valley below, as sitting directly in the path of any mudflow the eruption would throw down. The map was correct in every particular. When the volcano erupted on the thirteenth of November, the authorities, weighing the cost of evacuating a profitable farming town against the embarrassment of a false alarm, decided to wait. A storm took out the communications that night. A priest reportedly told a frightened parishioner to enjoy the ash, it was a beautiful thing and she would never see its like again. The lahars reached Armero near midnight and buried it under five metres of mud moving at the speed of a car. The town held some twenty-nine thousand people. About twenty-three thousand of them died. The detection had worked perfectly. The response was the thing that was withheld.

That is what Pinatubo’s saved thousands would have looked like, laid out in a valley. Same class of event, same chain of cause, the detection achieved in both cases. The only variable that moved between a quiet evacuation and a buried town was whether anyone acted on the warning before the impact arrived.

And in case that reads as a fluke of two different mountains, the same mountain settled it. Four years after it buried Armero, Nevado del Ruiz stirred again, and this time the monitoring was watched and the evacuation was ordered and the valley was emptied. Nobody died. The hazard had not changed. The mountain was the mountain. The response was the variable, and when the response came before the impact, the death toll was a number the instruments record as nothing at all.

A volcano observatory is a serious place full of serious instruments, and you do not have one. What you have is a web form, and the web form does not merely fail to notice your best work. It deletes it on purpose, and it hands you the pen and makes you sign.

Here is how it goes. You have just done the good thing, the early thing. A change went sideways, somebody read the trajectory and called it, the rollback ran, and the failure was pulled back before a single customer felt a thing. The impact, in the only sense that matters, never happened. So you go to write it down, because writing it down is the job, and the tool asks you for the time the impact began and the time it ended, and you discover that the moment you were proudest of is a moment the form was built to reject.

Because the response landed before the impact, the end comes before the start. You enter what happened and the field turns red. End time cannot be before start time. The tool has rules, and the first rule, written by someone who never once imagined you might be early, is that nothing ends before it begins. So you sit there with the adrenaline still draining out of you, being told by a dialog box that the night did not happen the way you watched it happen. And you do the only thing the form will accept. You drag the start forward until it agrees with the end, the duration computes out to zero, and the form goes quiet and lets you save. You have been conscripted into falsifying your own timeline by a validation rule. The lie has two authors, and you are the one who pressed save.

Multiply that by a quarter. Every clean save the team made, every early call that worked, every fire smelled before it caught, each one filed as a zero, because zero is the only number the schema will hold. Then somebody senior opens the dashboard and finds a flat line along the bottom of the impact graph. Nothing to see. A quiet quarter, which from that altitude is never the good news it looks like. I have watched this land on people who deserved a great deal better: teams that pulled the cord while every dashboard still glowed green, that took a thing which would have cost the company a full day of broken payroll and turned it into a non-event by getting in front of it, and whose work arrived on the executive’s desk wearing the exact same face as no work at all. The context, the counterfactual, the cost they had quietly eaten so the business would not have to, was gone the instant the duration clamped to zero. The machine did it on everyone’s behalf and called it data hygiene.

Hand that machine a negative number, the one figure that says what actually happened, and it gives you back a zero and files it under routine. That is not a measurement failing to capture something. That is a measurement manufacturing the opposite of the truth, signing your name to it, and carrying it upstairs to the people who decide what you are worth.

You will want, by now, to fix the form.

It is the natural instinct of anyone who has read this far and works for a living. Find the validation rule. Let the field accept a negative number. Add a checkbox for responded before impact and a column for the counterfactual, ship it next sprint, and the problem is solved. And you should do it. It will take an afternoon, and it is worth the afternoon, and I am not going to stand here and tell you better tooling is a waste of time after five movements spent cursing a dialog box.

But the field was never the disease. The field is a symptom. It is where the disease shows on the skin, and the disease is the assumption: that you are, by your nature, late. That response is a thing which happens to you after the break, in the wreckage, by torchlight. The entire discipline is built on it. The metrics assume it, the tools enforce it, the war stories celebrate it, and somewhere a long way down, the people doing the work come to believe it about themselves. We have organised a whole profession around the conviction that its practitioners arrive second.

The negative number is heresy because it says otherwise. It says the response can come first. It says the best people in this trade are not the ones who run fastest toward a fire already burning, but the ones who can read a building and call it before there is any smoke to smell, and that those two things are not the same skill and never were. The recovery at three in the morning is cleanup. Skilled, necessary, the kind that saves the quarter and earns the bonus, but cleanup, and cleanup means there was already something to clean. The response that lands before the impact is the only version of the work that leaves nothing behind it, because it got there before the wreckage could be made. That is the summit. That is the whole of it.

So here is the verdict, and it is not a complaint about a form. Any instrument that reads zero when the work is at its best is not measuring the work badly. It is measuring the wrong thing. It is pointed at failure, and at the cleaning up of failure, and it is calling that reliability, and reliability is not the speed of your recovery. Reliability is the disaster that never arrived: the lanes that stayed empty, the town that went home, the quarter with nothing on the graph. We have built our rulers to measure the wreckage, named the absence of wreckage a quiet quarter, and gone looking for someone to blame for the quiet.

Which brings it back to the water.

The Dali was bearing down at eight knots in the dark and there was nothing anyone could do about the bridge. The bridge was already lost the moment the lights went out. What was not yet lost was everyone who would have been on it two minutes later, the ordinary traffic of a city that does not stop at one in the morning, and they are alive because a dispatcher with no time and no script said hold the cars, and the cars held. Six men still died, the ones already out on the deck, the ones the warning reached too late because they were standing on the impact before anyone knew it was coming. The pre-impact response is not a miracle. It does not always win, and it did not fully win that night.

But the lanes were empty. That is the evidence. That is what the most important work of that entire night looks like in the record it leaves behind: a stretch of empty road and a clock that ran backwards. Minus two minutes, the truest number anyone produced on the Patapsco that night, and the one number no system we have built will agree to write down. Learn to write it down. Until then we will keep measuring our people by the wreckage they leave, and paying our best ones in zeros, and wondering, from the calm of an empty graph, what exactly it is they do.