A reader asked what a small company should build for incident management on day one. Here is the entire policy.

Incident Policy, v1

When to declare. Declare an incident when a problem needs more attention than you can give it alone. The question is never how bad it is. The question is whether you need people. If unsure, declare. Nobody will ever be criticised for declaring an incident that turned out to be nothing.

What happens when you declare. For the people involved, normal work is suspended. One channel per incident. One person drives: they say who does what, and they say when it’s over. Whoever declares, drives, until they hand it to someone else.

Who is on call. One person, one week at a time, on a roster we build together. If you’re not on the roster this week, you’re off. Off means off.

Communication. The driver, or someone they pick, posts what we know, what we’re doing, and when the next update lands. Guesses are labelled as guesses. If customers are affected, tell them in plain words before they find out for themselves.

Afterwards. Within a week, the driver, or someone they pick, writes a short record. What happened, what we learned, what we’ll change. Anything we’ll change goes in the backlog like any other work, or it will not happen.

The count. Every incident goes on the list. Date, duration, what broke, what fixed it. A spreadsheet is fine.

Amendments

None yet.

If your first reaction is that this cannot be enough, good. Hold onto that.

Every incident process you have ever worked under is a record of wounds. Each rule in the big binder at the big company exists because one day, somewhere, its absence hurt. The severity matrix followed a fight. The commander role followed a shambles. The comms template followed a customer finding out the hard way. Process is what an organisation writes down after it bleeds, and a company that has not bled yet has almost nothing to write. Copy the binder anyway and you inherit its injuries without its history. The document above is short because your scars are few. It will not stay that way.

Most of the page explains itself. Three lines do not.

Declare an incident when a problem needs more attention than you can give it alone.

A declaration is not a verdict on severity. It is a mode switch. The moment someone declares, the rules of normal work suspend for the people involved: one channel, one driver, updates on a clock. That is all an incident is at this size. A signal that says stop what you are doing, I need you.

Which is why declaring has to be cheap. The engineer who declares at 2am over something that resolves itself by 2:15 has not cried wolf. They have tested the machinery, and the machinery worked. Say so in public. The alternative is a team that hesitates before pulling the handle, and hesitation is the most expensive failure mode available to you right now.

One person drives: they say who does what, and they say when it’s over.

Note what this line does not say. It does not say incident commander. There is no role here, no training, no rotation of certified humans. There is a rule that at any moment exactly one person is driving, and everyone knows who. The named role comes later, and you will find it below, filed with the other things you do not need yet. The rule and the role are different sizes. Confusing them is how a fifteen-person company ends up with a forty-slide onboarding deck for a job nobody holds.

If you’re not on the roster this week, you’re off. Off means off.

The roster looks like the part where a burden gets imposed. It is the opposite. At fifteen people, everyone is already on call, all the time, informally and without end. The pager in everyone’s head never stops. A roster does not create the on state. It creates the off state. Its product is the fourteen people allowed to sleep tonight because it is not their week.

This is also why the policy says a roster we build together, not a roster I have built. Write the thing with the team in the room, not for them. People defend rules they watched get made, and an on-call roster runs on that goodwill and nothing else.

Now for what is missing, which is most of the discipline. Everything absent from the policy is absent on purpose, and none of it is absent forever. Simple systems that work grow into complex systems that work; it does not run the other way. So every omission below carries an expiry date, written as an injury. When the absence of a thing has hurt you twice, it has earned its place in the document.

Severity. A severity matrix is a treaty, and treaties follow wars. Yours arrives the week two incidents both claim to be the most important thing in the company, or the month your services outgrow anyone’s ability to judge a response by feel.

Incident commander. Someone always drives; that rule is already on the page. The role gets a name the day two people both believe they are driving, or the day nobody does. Training and rotation follow the title, not the other way around.

Blameless. You do not legislate blamelessness at fifteen people. You model it. The written rule arrives the morning after somebody breaks the norm you never wrote down, and not a day before.

Tooling. The spreadsheet is the tool. Automation gets bought one manual step at a time, each purchase justified by the incident where that step cost minutes you did not have. When the spreadsheet starts lying to you, you may go shopping.

Reporting. Five incidents is a stack of anecdotes. Five hundred is a dataset. When the count can hold a trend, start reporting, because the people you report to fund trajectories, not moments. Until then, the count is not for them. It is a letter to the company you will be at fifty people.

Which leaves the last section, where two words are doing the most work on the page. None yet. Not none needed. Not none ever.

The amendments section is the only part of the document guaranteed to grow.

Subscribe now

At 1:24 in the morning on 26 March 2024, the lights went out on a container ship called the Dali, and the Francis Scott Key Bridge had about five minutes left to stand.

The ship had left its berth at Seagirt a little before, bound for Sri Lanka, two harbour pilots aboard and the better part of five thousand containers stacked on deck. Then a complete blackout. Not a flicker. Nearly a hundred thousand tonnes of steel turned into dead weight in the channel, carried by its own momentum and the current and aimed, with the indifference of physics, at one of the piers holding the bridge up.

What happened in the time it had left is the only part of the night the spreadsheets cannot hold.

At around 1:27, the crew got a mayday out. A pilot came over the radio and asked, in the panicked voice of a man watching something he has no way to stop, for the bridge to be closed to traffic. The dispatcher who took it did not open a ticket or convene a working group. He told the officers to hold all traffic until somebody got the ship back under control. Maryland Transportation Authority cars rolled to both ends of the span and stopped the traffic where it sat. Somewhere on the recording a voice remembers the men working on the deck and asks whether anyone can reach the foreman and get them off in time.

There were eight of them, filling potholes for a contractor called Brawner Builders at one in the morning on a bridge the whole city drove over without thinking. The officers had time to stop the cars. They did not have time for the men. At 1:29 the bow met the pier and the central span came down in something close to thirty seconds, and it took the crew into the Patapsco with it. Six of them did not come back up.

Read the timeline again, because the number that matters is the gap. Roughly two minutes between the mayday and the collapse. Two minutes in which the people responsible for that bridge knew, with total clarity, that it was going to be struck, and acted on that knowledge before a single beam had moved. The response to the incident began before the incident happened. By the time the structure gave way, the most consequential decision of the night had already been made and carried out. The cars were stopped. The mayor would say afterwards that the mayday saved many lives, and he was right, and you can prove it by counting the empty lanes.

Now try to log it.

Open whatever your shop uses to record incidents and find me the field for that. There is a time you detected the problem, a time you resolved it, and a duration in between, and the whole apparatus assumes, without once saying so aloud, that detection comes after things start to break. Time to impact is meant to be a positive number. The bridge does not offer a positive number. The bridge offers minus two minutes, and most tooling, handed minus two minutes, will reject it, blink, and quietly record a zero.

That zero is the lie this entire piece is about. The two minutes that decided who lived are the one interval the instrument cannot see.

What the bridge gave us has a name. Negative time to impact: the response reaches the world before the damage does, and the blast radius comes out at zero because somebody got there first. It has a quieter sibling, negative time to detect, where you clock the trouble before it has even started to go wrong, reading the trajectory and calling it while every dial still says green. The fact that we need names for these at all should worry you.

Strip the jargon back and the ideas are almost insultingly simple. Time to detect is the gap between a thing starting to go wrong and somebody noticing. Time to impact is the gap before the trouble lands on someone who never asked for it. Both are assumed, always, to be positive numbers. The trouble comes first, you arrive second, and the whole grammar of the discipline takes it for granted that you are late. Turn either one negative and you have described the best work anyone in this field ever does. The pilot on the Dali was working in negative time. So is every team that ever caught a poisonous change and pulled it back before it reached more than a handful of users. They were responding to an incident that had not happened yet.

Here is the problem, and it is not a software problem, though software is where you will find the body. Every tool we have built to manage incidents encodes the same little machine, and the machine has three stops. Something breaks. You notice. You fix it. Detected, mitigated, resolved, with a clutch of timestamps hung off each stop so we can compute the famous numbers. We cannot even agree what the R in MTTR stands for, repair or recovery or resolution, and we will argue about it in good faith at a conference until the bar closes, but every faction agrees on the one thing that matters here. Whatever the R is, it comes after the break. The clock is bolted to the floor and it only counts up. There is no field for the response that arrived before the failure, because the schema was built by people who never imagined you might be early.

You can watch this happen in a room. Describe negative time to detect to a dozen competent engineers and half of them will stall, not because they are slow but because you have handed them an idea their model has no slot for. You can watch them try to file it under “detected” and find nothing in front of it. The blank look is not stupidity. It is data. A vocabulary that cannot hold an idea makes the idea hard to think, and a profession that cannot think an idea has no way to reward the people who keep doing it anyway.

So we have built an entire measurement discipline on the quiet assumption that we are always too late, and then we act surprised when being early shows up as nothing at all. The tool did not decide that detection comes after the break. People decided that. People who believed, somewhere underneath the process diagrams, that the correct moment to respond to a fire is the moment you can finally smell the smoke.

I know the type. I used to be one of them.

The world I am about to describe is mostly gone now, paved over by continuous delivery and pipelines that ship a hundred times a day, and good riddance to most of it. But there was a time, not so long ago, when change was a scheduled event. You did not ship on a Tuesday afternoon because you felt like it. You booked a window. A slot, approved weeks out by a committee that met on Thursdays, in which you were permitted to touch production while the rest of the company slept. Two in the morning on a Saturday, four hours, rollback plan attached. The window was sacred. The window was the whole liturgy.

And every so often a team would blow straight through it.

Not the good teams. The good teams were home in bed by three. The ones who overran were the ones who had planned the work on the back of a serviette, brought the wrong people, or brought no people, and discovered at the worst possible hour that step seven of eleven did not do what the runbook promised. By rule, any change that ran past its window stopped being a change and became an incident. That was the bright line, and I was the person standing on it with a clipboard.

Here is the part I am not proud of. These teams could see the wall coming. They were not always competent but they were not blind, and somewhere around hour two of a four-hour window a sensible engineer can do the arithmetic and work out that this is not going to land. So they would call. Sometimes hours before the window closed, they would try to pull an incident manager onto a bridge, to get help standing by for the moment it tipped over. And I would tell them no. Not yet. The window has not ended. This is your change and your mess, and I am not the cleanup squad for an afternoon of bad planning. Come back when it breaks.

I thought I was defending something. The integrity of the process. The principle that the incident channel was not a crutch for people who could not run a deployment. It felt like rigour. It felt like holding the line.

It was a boomerang, and I had thrown it myself. The second the clock struck the end of the window, the thing I had refused to look at became, by definition, an incident, and it landed on the side of my head with the full weight of however many hours I had spent insisting it was not my problem. Except now the team was exhausted, the change had been bleeding quietly into production for half the night, and the easy rollback we could have run at hour two was a tangled horror at hour five. I had not prevented anything. I had taken a recoverable situation, made everyone watch it deteriorate until a clock gave me permission, and then run the incident I could have got ahead of, slower and sicker and later than it ever needed to be.

That is the whole disease in one anecdote. Those teams planned like clowns, and they were still right about the one thing that mattered. They read the trajectory and called it before the impact. That is negative time to detect, delivered by the least disciplined people in the building, and I sent it back with a lecture about planning. I had been handed the exact response this entire essay is in praise of, and I refused it on principle, because the principle said you do not get to call it an incident until it has finished becoming one.

I was a clock bolted to the floor. I only counted up.

Here is the cruel arithmetic of doing this well. The better you are at it, the less it looks like you did anything at all. A fire put out before it spreads is indistinguishable, on the incident report, from a fire that was never going to spread. Stop the disaster early enough and you have not averted a catastrophe, you have merely had a quiet night, and quiet nights do not get budget, and they do not get thanks, and after enough of them somebody starts asking what they are paying you for. This is the prevention paradox, and it is the structural condition of the entire trade. The reward for being early is to be doubted.

The purest case the world has ever run was Y2K.

For most of the 1990s an enormous body of people went line by line through the planet’s ageing software, expanding two-digit years into four, because a great many systems built when memory was expensive had no way to tell 2000 apart from 1900. The bill ran to somewhere between three and six hundred billion dollars, depending on whose accounting you trust, with the better part of a decade of labour behind it. Then midnight came on the first of January, and the planes stayed in the sky, and the grids held, and the banks opened on Monday as if nothing had happened. Which, as far as anyone could see, it had not.

So the verdict came in fast and it came in cruel. Hoax. Hysteria. The greatest racket consultants ever ran. People who had not slept properly since 1997 read in the paper that they had spent six hundred billion dollars frightening themselves over a calendar. And the maddening part, the part that makes Y2K the perfect parable and not merely a sad one, is that the sceptics could not be cleanly proven wrong. A few countries spent almost nothing and rolled into the new century about as smoothly as the ones that had spent fortunes. There is no second Earth where nobody did the work, sitting in a lab so we can compare. A disaster that was prevented and a disaster that was never coming leave behind precisely the same evidence, which is to say none, and an absence will not testify on your behalf.

You do not need a calendar rolling over to see it. It happens in your pipeline every week. A change goes out to one percent of traffic, the error rate lifts its head, and a guardrail or a human paying attention rolls it back before it ever reaches the other ninety-nine. The blast radius is a rounding error. Nobody writes a postmortem for it, because there is no post and there was no mortem, and the engineer whose instinct caught it does not get an incident with their name on it the way they would have if they had let it burn and put it out heroically at three in the morning. We have built a discipline that pays out for the heroic recovery and stays silent on the quiet save, and then we wonder why people learn to wait for the fire.

If you want the shape of the thing in something heavier than software, put two volcanoes side by side.

In 1991 Mount Pinatubo in the Philippines woke up after five centuries. Volcanologists from the local institute and the United States Geological Survey watched it for weeks, read the seismographs and the gas, called the big eruption before it came, and moved more than sixty thousand people off the mountain. The eruption was one of the largest of the century. A few hundred died, most of them under roofs that gave way beneath wet ash. The forecast is reckoned to have saved somewhere between five and twenty thousand lives, and the whole monitoring effort cost under a million and a half dollars. Those saved thousands are not a figure you will find on any memorial, because saved lives do not gather in one place to be counted. They went home. The catastrophe is invisible precisely because it did not happen.

You want to know what it would have looked like if it had. Look six years earlier and a continent across.

In 1985 the Nevado del Ruiz volcano in Colombia gave every warning a mountain can give. Scientists had watched it for months. A hazard map went out in October marking the town of Armero, in the valley below, as sitting directly in the path of any mudflow the eruption would throw down. The map was correct in every particular. When the volcano erupted on the thirteenth of November, the authorities, weighing the cost of evacuating a profitable farming town against the embarrassment of a false alarm, decided to wait. A storm took out the communications that night. A priest reportedly told a frightened parishioner to enjoy the ash, it was a beautiful thing and she would never see its like again. The lahars reached Armero near midnight and buried it under five metres of mud moving at the speed of a car. The town held some twenty-nine thousand people. About twenty-three thousand of them died. The detection had worked perfectly. The response was the thing that was withheld.

That is what Pinatubo’s saved thousands would have looked like, laid out in a valley. Same class of event, same chain of cause, the detection achieved in both cases. The only variable that moved between a quiet evacuation and a buried town was whether anyone acted on the warning before the impact arrived.

And in case that reads as a fluke of two different mountains, the same mountain settled it. Four years after it buried Armero, Nevado del Ruiz stirred again, and this time the monitoring was watched and the evacuation was ordered and the valley was emptied. Nobody died. The hazard had not changed. The mountain was the mountain. The response was the variable, and when the response came before the impact, the death toll was a number the instruments record as nothing at all.

A volcano observatory is a serious place full of serious instruments, and you do not have one. What you have is a web form, and the web form does not merely fail to notice your best work. It deletes it on purpose, and it hands you the pen and makes you sign.

Here is how it goes. You have just done the good thing, the early thing. A change went sideways, somebody read the trajectory and called it, the rollback ran, and the failure was pulled back before a single customer felt a thing. The impact, in the only sense that matters, never happened. So you go to write it down, because writing it down is the job, and the tool asks you for the time the impact began and the time it ended, and you discover that the moment you were proudest of is a moment the form was built to reject.

Because the response landed before the impact, the end comes before the start. You enter what happened and the field turns red. End time cannot be before start time. The tool has rules, and the first rule, written by someone who never once imagined you might be early, is that nothing ends before it begins. So you sit there with the adrenaline still draining out of you, being told by a dialog box that the night did not happen the way you watched it happen. And you do the only thing the form will accept. You drag the start forward until it agrees with the end, the duration computes out to zero, and the form goes quiet and lets you save. You have been conscripted into falsifying your own timeline by a validation rule. The lie has two authors, and you are the one who pressed save.

Multiply that by a quarter. Every clean save the team made, every early call that worked, every fire smelled before it caught, each one filed as a zero, because zero is the only number the schema will hold. Then somebody senior opens the dashboard and finds a flat line along the bottom of the impact graph. Nothing to see. A quiet quarter, which from that altitude is never the good news it looks like. I have watched this land on people who deserved a great deal better: teams that pulled the cord while every dashboard still glowed green, that took a thing which would have cost the company a full day of broken payroll and turned it into a non-event by getting in front of it, and whose work arrived on the executive’s desk wearing the exact same face as no work at all. The context, the counterfactual, the cost they had quietly eaten so the business would not have to, was gone the instant the duration clamped to zero. The machine did it on everyone’s behalf and called it data hygiene.

Hand that machine a negative number, the one figure that says what actually happened, and it gives you back a zero and files it under routine. That is not a measurement failing to capture something. That is a measurement manufacturing the opposite of the truth, signing your name to it, and carrying it upstairs to the people who decide what you are worth.

You will want, by now, to fix the form.

It is the natural instinct of anyone who has read this far and works for a living. Find the validation rule. Let the field accept a negative number. Add a checkbox for responded before impact and a column for the counterfactual, ship it next sprint, and the problem is solved. And you should do it. It will take an afternoon, and it is worth the afternoon, and I am not going to stand here and tell you better tooling is a waste of time after five movements spent cursing a dialog box.

But the field was never the disease. The field is a symptom. It is where the disease shows on the skin, and the disease is the assumption: that you are, by your nature, late. That response is a thing which happens to you after the break, in the wreckage, by torchlight. The entire discipline is built on it. The metrics assume it, the tools enforce it, the war stories celebrate it, and somewhere a long way down, the people doing the work come to believe it about themselves. We have organised a whole profession around the conviction that its practitioners arrive second.

The negative number is heresy because it says otherwise. It says the response can come first. It says the best people in this trade are not the ones who run fastest toward a fire already burning, but the ones who can read a building and call it before there is any smoke to smell, and that those two things are not the same skill and never were. The recovery at three in the morning is cleanup. Skilled, necessary, the kind that saves the quarter and earns the bonus, but cleanup, and cleanup means there was already something to clean. The response that lands before the impact is the only version of the work that leaves nothing behind it, because it got there before the wreckage could be made. That is the summit. That is the whole of it.

So here is the verdict, and it is not a complaint about a form. Any instrument that reads zero when the work is at its best is not measuring the work badly. It is measuring the wrong thing. It is pointed at failure, and at the cleaning up of failure, and it is calling that reliability, and reliability is not the speed of your recovery. Reliability is the disaster that never arrived: the lanes that stayed empty, the town that went home, the quarter with nothing on the graph. We have built our rulers to measure the wreckage, named the absence of wreckage a quiet quarter, and gone looking for someone to blame for the quiet.

Which brings it back to the water.

The Dali was bearing down at eight knots in the dark and there was nothing anyone could do about the bridge. The bridge was already lost the moment the lights went out. What was not yet lost was everyone who would have been on it two minutes later, the ordinary traffic of a city that does not stop at one in the morning, and they are alive because a dispatcher with no time and no script said hold the cars, and the cars held. Six men still died, the ones already out on the deck, the ones the warning reached too late because they were standing on the impact before anyone knew it was coming. The pre-impact response is not a miracle. It does not always win, and it did not fully win that night.

But the lanes were empty. That is the evidence. That is what the most important work of that entire night looks like in the record it leaves behind: a stretch of empty road and a clock that ran backwards. Minus two minutes, the truest number anyone produced on the Patapsco that night, and the one number no system we have built will agree to write down. Learn to write it down. Until then we will keep measuring our people by the wreckage they leave, and paying our best ones in zeros, and wondering, from the calm of an empty graph, what exactly it is they do.

Subscribe now

He was on a video call from a cafe, a product manager, flicking a begleri back and forth between sips of a flat white. I had a knucklebone going, switching it hand to hand while he talked. Two grown professionals, paid reasonable money to be taken seriously, fiddling with bits of string and weighted metal and picking over the companies that got it spectacularly wrong, as though hindsight were a kind of genius.

We agreed on nearly everything until we didn’t. He had built a tidy picture of the two of us and laid it out before me. He was the irresistible force, leaning on the engineers to ship, to get product to market before the market lost interest. I was the immovable object on the far side of them, the incident barrier, there to slow the whole thing down before somebody broke something that mattered. He meant it as a compliment.

I waited for him to finish and told him he had me exactly backwards.

The begleri stopped. He went quiet, the particular quiet of a man recalculating, and then he wrote something on the pad beside his coffee and asked me to explain myself. So I did. I told him I was not the brake. I had never been the brake. If it were up to me the engineers would be breaking more, not less.

The argument is not complicated, and I have made it often enough to make it fast. There are two ways to run a company that builds software. You change things quickly and break some of them, or you change things slowly and watch the market leave without you. There is no third lane. There is no measured middle where you ship at a sensible pace and nothing ever falls over. That lane is a fiction, sold to executives who find both of the real choices frightening. The companies that bought it are the ones who did not blow up. They sat very still, with a spotless record, and were buried holding it. The cleanest way to die in this business is to stop moving and call it discipline.

So if you have accepted that you have to keep moving, you have already accepted the incidents. They are not the price of getting it wrong. They are the price of doing anything at all. The team that ships nothing has none of them. The team that ships has them the way a road has potholes, and you can resurface as often as you like, but you do not get the road without the wear. Once you stop pretending the number can reach zero, the useful question changes. It stops being how do we have fewer, which has only bad answers, and becomes what do we get out of the ones we have. The answer turns out to be quite a lot.

The first thing you get is a team that tells you the truth early. When an incident is an ordinary event and not a permanent mark against your name, people put a hand up while the thing is still small and still cheap, instead of sitting on it and praying, which is what people do when the cost of admitting a fault is a hard conversation with someone who outranks them. Every catastrophe I have stood in the middle of had an earlier, smaller, survivable version that somebody decided not to mention.

The second thing you get is competence, which is only practice wearing a better word. A team that runs incidents often runs them well. They know where the runbooks are, or they know the runbooks are useless and route around them, and either way they have a rhythm. The team that has not seen an incident in eighteen months has not banked eighteen months of safety. It has banked eighteen months of rust, and it will move through its first real one like a fire drill in a building where nobody can remember which door is the exit.

The third thing is the one everyone says they want and almost nobody funds. If you run the incident well and tell the truth about it afterwards, you come out the far side knowing something about your system you did not know going in. John Allspaw calls an incident an unplanned investment, and he means it precisely. You did not choose to make it. You made it the moment the thing broke, and you do not even control its size. The only thing left in your hands is whether you collect the return, and most organisations pay the full cost of the outage and then throw away the receipt.

None of this works while an incident is a thing to be ashamed of. The shame is the whole problem. It is what keeps the hand down, lets the skill go soft, and turns the review afterwards into a hunt for someone to blame instead of something to learn. So I said it to him plainly. I did not want fewer incidents. I wanted a team that had them often, ran them well, learned from them properly, and felt nothing sharper than mild professional interest the entire time. A team like that is more reliable than a team that has only been lucky. And it is a great deal more reliable than a team that has merely been quiet.

Here is the part that people accept in the abstract and resist in their bones. If incidents are the price of motion, their absence is not good news by default. A quiet quarter is not a trophy. It is a question, and it has more than one answer, and from the executive chair the answers are impossible to tell apart.

A team can go quiet for three reasons. The first is that it is healthy. The work is good, the luck is holding, and nothing has broken because, for the moment, nothing had to. This happens. It happens less than anyone wants to believe, and it never holds still, because a healthy team that stops shipping stops being one within a quarter or two. The other two reasons wear the first one’s clothes. They look identical on a graph, and they are both rotten.

The second reason is that the team is going soft, and there is no villain in this one, which is exactly what makes it hard to see. Nothing is being hidden. The incidents are not happening, the runbooks are quietly going out of date, and the one engineer who understood how the billing system fails at three in the morning has taken a job somewhere warmer, and nobody has noticed the gap because nothing has fallen into it yet. The capability does not announce its departure. It is simply not there on the day you reach for it. The rare large incident always comes, and when it does it lands on a team that has forgotten how to catch it. The quiet did not protect them. It disarmed them.

The third reason is the one a head of engineering described to me once, quietly, the way people tell you things they have decided not to fix. The incident culture in his department was poisonous. People who caused an incident, or were merely standing near one when it went off, were marked for it, and the mark travelled. It followed them into performance reviews and into rooms they were not in. So his engineers had made the rational choice. They had stopped raising incidents. They had stopped spending time running the ones they could not avoid. And the post-incident review, the thing that turns an outage into knowledge, did not come up at all. He told me this as a problem he was observing, not one he was causing, which is its own kind of tell.

It took me a moment to register what he had just listed for me. He had named, in order and without meaning to, the three things that make a team good at trouble, and he had explained that his department had switched off every one of them. No early warning. No practised hand. No learning. And the result of switching off all three, the figure sitting proudly at the top of his dashboard, was a low incident count. On paper, his was one of the calmer departments in the building. In truth it was one of the most dangerous, a place where everything that broke was either hidden or survived by luck, and where nobody was getting better at anything.

This is the problem with the number. Health, rot, and cover-up all produce the same low count, and from the executive chair they are indistinguishable. So when the figure drops, the room relaxes, which is the most dangerous thing a falling incident count can make a room do. A low number is not information. It is the absence of information, wearing the costume of good news. A clean record is not proof that you are safe. Sometimes it is only proof that you are lucky, and sometimes it is proof that someone is lying to you, and the graph will never tell you which. The head of engineering at least knew which quiet he was standing in. Most people reading the dashboard never find out, right up until the quarter that is not quiet at all.

Software has the good fortune that its quiet quarters usually end in a refund and an apology. Other industries run the same machine with the same blind spot, and when their quiet ends, it ends with bodies. The useful thing about those industries is that they investigate, at length and in public, so none of this has to be taken on faith. The reports exist. They are very long, and they all say a version of the same thing.

On the twentieth of April 2010, a group of BP and Transocean executives flew out to the Deepwater Horizon to hand the crew an award for seven years without a lost-time accident. They were on the bridge when the well blew out. The explosion and fire killed eleven people and put the largest oil spill in American history into the Gulf of Mexico. The record was real, and it was worthless, because it measured the wrong thing. Personal safety on the rig was excellent, the kind where the worst thing anyone pictures is a dropped pipe and a crushed foot. Process safety, the slow abstract business of whether the well itself would hold, was a disaster nobody was counting. The well needed twenty-one centralisers to seal correctly. They ran it with six. And the seven-year record was partly an illusion of its own, because the crews understood that raising a concern that delayed the drilling was a good way to lose your job, which means the number measured seven years without a reported problem, in a place where reporting one was punished. They were celebrating the silence at the exact moment it killed them.

NASA learned the same lesson twice, seventeen years apart, and wrote the textbook in between. In the years before the Challenger, the rubber O-rings that sealed the booster joints kept eroding in flight, and because the shuttle kept coming home anyway, the erosion stopped being treated as a fault and became a known quirk you could fly with. The night before the launch the engineers who built the boosters warned that the cold would stiffen the seals past the point where they could hold. They were overruled, after being asked to do the one thing engineering cannot do on demand: prove the rocket would fail before it had failed. On the twenty-eighth of January 1986 the seal failed and the Challenger came apart seventy-three seconds after lift-off, live on television, in front of the classrooms full of children who had been gathered to watch a schoolteacher fly into space. The sociologist Diane Vaughan gave the pattern its name afterwards. She called it the normalisation of deviance: the slow process by which a warning sign, repeated often enough without disaster, gets quietly reclassified as normal.

Then NASA did it again. Foam had been breaking off the external tank and striking the orbiter for years. The same foam, from the same ramp, had come away six times before, and because nothing had yet gone fatally wrong, the agency downgraded it from a flight-safety anomaly to a maintenance nuisance, a paperwork item, in the months before one more piece of it punched a hole in Columbia’s wing on the way up. The frequency of the warning had become the reason to ignore it. The engineers saw the strike on the launch film and asked to point a satellite at the wing to check the damage. The request was refused, on the grounds that it had not come through the proper channels. That hole went unexamined, and a fortnight later the orbiter disintegrated over Texas on re-entry, killing all seven aboard.

The investigation board concluded that NASA’s culture had killed the crew as surely as the foam, a remarkable thing to have to write about the same organisation a second time. But the parallel runs deeper than a repeated blind spot. Both times, the engineers who wanted to stop were made to prove it would fail, while the managers who wanted to fly were asked to prove nothing at all. That is what it means to learn the same lesson twice. Not a forgotten fact, but a burden of proof that sat, on both occasions, on exactly the wrong shoulder.

There is one industry that looked at the same raw material and drew the opposite conclusion, and it is the reason you can board a plane without thinking about it. Aviation decided, decades ago, that the near-miss was the most valuable thing it owned, and that the only way to get people to report the near-miss was to make reporting safe. The result is the Aviation Safety Reporting System: confidential, voluntary, explicitly non-punitive, with limited immunity for anyone who files. It was built on a single insight, that fear of punishment was suppressing the exact information that could prevent the next crash, so the system strips the reporter’s identity and shields them from enforcement, and in return it has gathered more than two million reports on things that nearly went wrong. The whole edifice runs on the principle that you want more incidents on the record, not fewer. And the neutral party chosen to run it, chosen precisely because it had no power to punish anyone, was NASA. The agency that twice mistook a quiet record for a safe one also operates the finest argument in the world for never doing that again.

The pattern is consistent enough to be a law. The organisations with the most spotless records are not the safest ones. They are very often the ones that have stopped looking, or stopped listening, or taught their people that looking and listening are career-limiting moves. A clean record is a fact about your reporting, not a fact about your safety, and the two come apart at the worst possible moment. So when you find yourself watching an incident count fall and feeling the room go warm with relief, it is worth knowing whose company you are in. You are standing on the deck of the Deepwater Horizon, on the evening of the twentieth of April, holding the award.

It is tempting to call the executives who chase zero foolish, but that lets them off too easily and gets the history wrong. The goal is not stupid. It is old. It is a piece of received wisdom from an era when it made perfect sense, kept alive by people who never noticed the ground underneath it had moved. To understand why the number lies, you have to go back to when it told the truth.

Reliability did not begin as a software idea. It began in hardware, in the discipline of working out how long a physical thing would run before it broke. The key number was the mean time between failures, and it was a number that meant something, because the thing it described was a component sitting still and wearing out at a knowable rate. A disk had a failure rate you could measure. So you stacked the disks into redundant arrays, the tall humming cabinets that filled the server rooms, and you did the arithmetic, and you could say with a straight face that the odds of losing everything at once were vanishingly small. The software on top of it moved at the same stately pace. It shipped in versions, on discs, in boxes, and between releases it sat as still as the hardware. Change was a rare and deliberate event, scheduled and rehearsed and dreaded. In a world like that, fewer failures was a coherent goal, because failure was a thing that wore out on a schedule, and you could engineer against a schedule.

Then the ground moved, and almost nobody changed the number. The discs and the boxes went away. Software stopped shipping in versions and started shipping continuously, a hundred or a thousand small changes a day, and the systems it ran on stopped being cabinets you could point at and became sprawling distributed things that no single person could hold in their head or draw on a whiteboard. The hardware problem, the one the old number was built for, was solved so completely by redundancy and the cloud that a failing disk became a non-event. But the failures did not stop. They changed shape. They stopped being a part wearing out and became something stranger, an emergent property of too many moving pieces interacting in a state nobody designed and nobody foresaw. You cannot calculate a mean time between failures for the sentence the system has never executed before. There is no schedule for a surprise.

What survived all this was the goal. We are still chasing the number that belonged to the cabinets, still treating a low incident count as the mark of a healthy system, in an environment where the thing that number measured no longer exists. Zero incidents is not an ambition. It is a fossil, perfectly shaped for a world that has been gone for twenty years. Chasing it now is not discipline. It is taxidermy. You are keeping a dead thing in a lifelike pose and asking it to guard the house.

And the inheritance is not harmless, because the world it came from is the one place the strategy worked. When failure was rare and predictable, you could afford to know nothing about it. You could keep your people innocent of how the system broke, because it broke seldom and it broke in familiar ways. None of that holds now. Chase zero in a system that fails by surprise and you do not get a safe organisation, you get an ignorant one, fluent in nothing, practised at nothing, blind to the shapes its own failures take. And the large incident is still coming, because it always is, only now it arrives in a form no one has seen, at a team that has never run one, inside a system no one can reason about under pressure. The pursuit of zero does not protect you from that day. It is the thing that sends you into it unarmed.

So you stop counting and start training. If incidents are the only honest signal you get about how your system fails, the work is not to silence the signal, it is to get fluent in it. You make it safe to raise one, so the small ones surface while they are still small. You run them often enough that the team moves through one the way a good crew moves through a storm, without drama, because they have done it before. You take the post-incident review seriously, as the place where the expensive lesson gets collected instead of binned. And when the system has been quiet for too long, you do not relax. You go and break it yourself, on a Tuesday afternoon, with everyone watching: a drill, a game day, a failure you injected on purpose so you could meet it on your terms instead of its own. The teams that do this are not reckless. They are the least surprised people in the building.

This is a different definition of reliability than the one on the dashboard, and it is the true one. Reliability was never the absence of failure. A system that has not failed is not reliable, it is untested, and the two feel identical right up until the day they do not. Reliability is what a system and the people around it do when failure arrives, which it will, on a long enough timeline, no matter how clever anyone was at the start. The reliable team is not the one that nothing happens to. It is the one that has made itself hard to surprise and quick to recover, that treats every incident as a rehearsal for the next, and that has therefore turned the thing everyone else is afraid of into the thing it is quietly best at.

The product manager wanted me to be the immovable object, the thing planted in front of the engineers so they could not break anything. I understand the appeal. It is a tidy picture, and it turns the incident person into a kind of guardian. But the immovable object is the thing this whole essay has been about burying. It is the company that sat still with a spotless record. It is the rig with the award. It is the agency holding the textbook it had already written about itself. The object that refuses to move does not prevent the disaster. It waits for it.

There is a name on this masthead, and it has been the joke the whole time. Zero sev zero. No incidents, and none of the worst kind, the clean and total silence that every executive has been trained to pray for. I did not call it that because I want it. I called it that because it is the most dangerous condition a system can be in, and almost no one recognises it while they are standing in it. A zero on that line does not tell you that you are safe. It tells you that you are healthy, or that you are rotting, or that someone has stopped telling you the truth, and the graph will not say which, and the day you find out which is not a day you get to choose. So I will leave it where I left it with him, the begleri turning in his fingers and the knucklebone going hand to hand across mine. I do not want fewer incidents. I want more of them, smaller and louder and sooner, run by people who are not afraid of them.

That is not the absence of trouble. It is the only kind of safety that was ever real.

Subscribe now

Somewhere right now there is a room you are not allowed into, full of people paid more than you will ever see, and every screen in it is red. The system they are staring at moves a national economy’s worth of money between breakfast and lunch, and it has stopped. There are people in that room with a list of certifications longer than you are tall. There are vendors on three continents awake at ungodly hours. There is a bridge call with ninety participants and a documented escalation path that runs all the way up to someone with a corner office and a driver.

And the thing that finally fixes it, after the experts and the vendors and the ninety on the call, is that somebody turns it off and turns it back on again.

Nobody says this part out loud afterwards. It does not go into the incident report in those words. But that is what happened. They rebooted it. Some of the most expensive architecture in the world, brought back to life by the same gesture you would use on a television remote that has started misbehaving, or a toddler mid-meltdown. Walk away, count to ten, try again, hope it comes back in a better mood.

It is the same act all the way down. There is no level of importance at which it stops working and no level of triviality at which we stop trying it. Below the trading floor sits the company whose entire office cannot print, and below that the video call that will not find the camera thirty seconds before it matters, and below that your home router, whose four little lights you have learned to read like tea leaves. Then the laptop that will not wake. Then the phone. Then the single earbud that has decided, for reasons it will not share, to stop being an earbud. You hold the button. You wait. You put it back in the case and take it out again, which is a reboot wearing a different coat. And somewhere in that descent, without noticing, you stopped being the person watching the experts and became one of them. You have done this. You did it this week.

Here is the part that should keep you awake, except it does not, because none of us has the energy. Nobody knows why it works. Not the helpdesk reading from a script, not the engineers who maintain the thing, not the people who designed it and signed it off. Somewhere in the gap between off and on, whatever had gone wrong quietly lets go of whatever it was holding, and the machine comes back without ever telling us what the matter was. We built the most complicated systems in history out of logic and mathematics, and the maintenance manual, the real one, the one that works, is a superstition.

And then there is you, a more complicated machine than anything in that room, running on the same procedure. You call it sleep. Once a day you lie down in the dark, switch yourself off for eight hours, lose consciousness entirely, and trust that you will come back on in the morning and resume being a person. Nobody can fully explain why this is necessary or why it works. Not the doctors, who, when you are run down and frayed and not booting properly, will tell you the oldest thing in medicine: go home, lie flat, try again tomorrow. Which is to say, have you tried turning yourself off and on again. We are all held together by a repair we do not understand and cannot skip. We perform it tonight, on faith, the way we always have. Then we get up and do it again.

Subscribe now

The service delivery manager smiled the way a crocodile smiles, with the teeth doing the work and the eyes elsewhere, and shared their screen. Every metric was green. It had been green last month. It would be green next month. Green appeared to be the natural and permanent condition of the service, the way the ocean is wet and the all-hands is long.

A wonderful provider. No problems anywhere. Pages of ticket data to prove it.

You already know this is not where the story ends. We would not have come all this way for a happy one.

The green was real, in the sense that the numbers genuinely were that colour. It was also fiction, in the sense that nothing it described matched what we had been living through. Calls were closed before anything was fixed. Major incidents ran behind a black box where a bridge should have been. Internal teams were neither consulted nor informed. The work was measured by the kilo, not the outcome.

But the metrics were green.

This is the watermelon. Green skin, and underneath, red all the way down. It is not a story about one bad provider. The watermelon comes for everyone with a dashboard, which is to say everyone. The odds are good that yours is sitting in a meeting somewhere right now, looking lovely.

The devil is in the detail, as it always is, and the detail is this: a number with no context is not information. It is a costume. And most incident practices are wearing several.

Take MTTR, the mean time to resolve, possibly the worst number you can love. Most incidents close quickly. Some close slowly. A few never really close at all; they sit open for months, then years, until someone doing a spring clean discovers a ticket old enough to draw a pension and mercifully puts it down. When you take the mean, those ancient zombies shamble back into the calculation and take a bite out of it. A single one can drag the average somewhere it has no business being. They exist in every organisation, and they are lying to you on average.

If you must have a typical value, use the median, and sit a p90 next to it. The median ignores the corpses. The p90 tells you what your bad days actually cost. Between them you get something close to the truth, which is more than the mean has ever offered.

Then there is time to detect, which depends entirely on what your organisation thinks detecting means. Measure only when the alert first fired and you will never learn that Bob did not look at it for three hours, being occupied with a coffee and a croissant. Measure only when Bob finally looked, and Bob becomes a hero for noticing a fire the building had been advertising since breakfast. You need both. The gap between the alert and the human is the whole story, and either number alone hides half of it.

Time to resolve has the same disease. Resolved by whom, and meaning what. Service restored, or ticket closed. These are not the same event and are frequently weeks apart. An incident parked in a monitoring state because a team thought it might recur, and then forgot it existed, and then left it gathering dust until someone tripped over it months later, will quietly poison your data the moment that ticket is finally closed. Pick a definition of resolved that means service is back. Make everyone use it. The alternative is fiction with a timestamp.

And the deepest cut of all: is any of it true to begin with? Incident records are set down once in the heat of the thing and almost never corrected afterwards. Unless your post-incident review actually checks the times, the durations, the sequence, your metrics are not measurements. They are first drafts that nobody edited, aggregated into a dashboard and presented as fact. You are not reporting on your incidents. You are reporting on your typing.

So fix the typing first. Make the review the place where the data is audited, not merely admired. Check the numbers as part of the root cause work, because a root cause built on a wrong timeline is just a confident guess. Stop assuming the data is correct. Find out.

Once the numbers are honest, you can finally ask the questions that matter, and here the work has changed. Reading every incident properly, at scale, used to require a room full of people nobody was ever going to hire. A competent model will now do it cheaply, reading the scribed record the way a witness would rather than the way a counter does. It finds the patterns the dashboard cannot: the playbook nobody updated, the handoff that stalls every single time, the decision point where the same forty minutes vanishes in incident after incident. Give it a referencing requirement so it shows its working, check it the way you would check any junior, and it will surface things that were hiding in plain sight. The trends were always there. You just never had the eyes for them.

While you are at it, measure the one number nobody ever mentions: time to assemble. The minutes from the incident being raised to the moment every team you need is actually on the line. It is strange that this is the forgotten metric, because it is one of the very few entirely within your control. If it takes forty-five minutes to get the right network engineer onto the bridge, no tooling on earth will save you, and no clever dashboard will hide it for long. Assembling the team is the first real act of incident management. Get faster at it and time to resolve falls out of the sky as a side effect. The sooner the right people are present, the sooner it is fixed, the sooner Bob gets back to the croissant. Everyone wins, except for the croissant.

Here is the part worth carrying out of the room. Every metric you measure is an instruction. Show me the number you reward and I will show you the behaviour you are about to get. Punish a team for missing a resolution target and they will not fix things faster. They will close tickets faster. The number will improve and the service will not, and you will have paid for the improvement with the only thing that ever mattered. The metric is the behaviour. It was always the behaviour.

Which brings us back to the watermelon, and to the uncomfortable thing the watermelon is actually for. It was never an accident of measurement. It was a success of avoidance. The provider wants the green, because green means no questions. The manager wants the green, because green means the decision to hire the provider was correct. The room wants the green, because red means work, and conflict, and somebody being wrong out loud. Nobody slices the watermelon open, because slicing it open is how you find out, and finding out was never what the dashboard was for.

The green was the most reliable thing the provider ever shipped. It worked perfectly, every month, without fail. It simply was not measuring the service.

It was measuring how badly everyone in the room wanted to stop looking.

Subscribe now

The desk is a crime scene of small distractions - a brass knucklebone darkened from restless handling, a balance toy I keep telling myself is a meditation practice and not a way to dodge email, a business card with one of those holographic squares that throws an oil-slick rainbow when the light hits it. I’ve been meaning to bin the card for a year and I don’t, because it’s shiny, and there’s a magpie somewhere in me that wants to take it back to the nest. I reach past all of it. My fingers find the plate - cool, glazed, heavier than it looks - and I lift it level with my face and turn it slowly in front of the webcam, slow enough that the light catches the gold lacquer running crooked across the lower third of it like a river seen from a plane.

On the other side of the glass: nine rectangles with names and faces floating in them. New major incident managers. The ones who’ll be woken at three in the morning while the rest of the company sleeps. I open my mouth and I give them the story.

This is kintsugi, I tell them. This is a plate that was broken. This is a plate I repaired myself, by hand, with gold, slowly and on purpose. And this matters - this is like the work you’re about to do. Things are better for having been broken and lovingly put back together. The seam is the point. The seam is where the value lives. This is your calling, and you’ll answer it at the hours no one else will answer to.

We were supposed to be in Tasmania. We’d booked it months out - the year of the COVID, though nobody was calling it that yet, nobody knew the whole world was about to bolt its own doors from the inside. Then it did. And we were left with two weeks of leave and nowhere on earth to spend it, free as anything and sealed indoors, so we did what people do when the walls start talking: we found our hands things to do before the hands found their own.

One of those things was the plate.

We were each handed one already broken - we didn’t get to break them ourselves, which was a disappointment I noticed at the time and am not ashamed of admitting. The breaking, it turned out, was the skilled part. You cannot drop a plate and expect the pieces to want to go back together; it has to be fractured by someone who knows how, in a way an amateur stands a chance of mending. So we sat cross-legged on the living-room floor, my wife and I, while a woman on a screen - calm, unhurried, the voice of someone who has glued a thousand things - walked us through how to make a broken thing unbroken.

It is not what the finished plate tells you it was.

The finished plate tells you about gold. The making is filing, and mixing, and holding two jagged edges together for longer than your fingers want to, and waiting, and painting a line you’ll immediately decide is crooked, and waiting again. It smells of the resin. It gets under your nails. You sit very still for a long time, which is its own particular hell, and then you stop and it sits very still for longer - days of nothing, the thing curing on a shelf while you orbit it and don’t touch it. For one small plate it was an enormous amount of effort, and I would not do it again for every dish my clumsiness has ever sent to an early grave. I’d sweep those up and bin them like anyone.

This one I kept.

Here is the story, as close as anyone can get to it. A shogun broke a tea bowl he cared about and sent it to China to be mended. It came back held together with metal staples - crude little clamps, the kind you’d put on a split rake handle, biting across a thing made for tea ceremonies. He hated it. So he put it to his own craftsmen: find me a repair that doesn’t insult the thing it’s repairing.

What they came back with was gold.

And this is the part the version you usually hear quietly drops. Kintsugi was not invented to celebrate the break. It was invented because a cheap repair had dishonoured something that mattered, and the gold was the correction - not a party thrown for the fracture, but a refusal to pretend the fracture was nothing, or to paper over it on the cheap. The break was a loss. It stayed a loss. The gold didn’t undo it.

What the gold did was measure it.

Every slow step - the filing, the holding, the waiting I’d already done myself and called tedious - was the lament. You don’t spend weeks and a precious metal on something you’re indifferent to. The labour is the grief, and the grief is the proof. The seam isn’t where the value lives. The seam is the receipt. It tells you, in gold, exactly how much someone decided this was worth not losing.

I gave that talk to new cohorts. It was the reliable bit, the one that always landed - the plate, up to the light, the gold catching, things are better for having been broken and lovingly put back together. People wrote it down. They told it to others. They still do.

The real version doesn’t happen on a desk. It happens in a document, and the document changes shape depending on who’s reading it.

At the bottom, where the break actually happened, the seam is raw. The person who was on call writes down what they saw: the alert that had been muted for six weeks because it was noisy, the fix that sat in the backlog behind four quarters of roadmap, the thing everyone privately knew was load-bearing and fragile and shipped anyway. That’s the gold going in - honest, specific, unflattering, laid by the people who waded through the broken pieces at two in the morning.

Then the document goes up. And at every level it gets a little smoother. The muted alert becomes “a gap in observability.” The four quarters of deferred work becomes “competing priorities.” By the time it reaches the people with the most power to fix the systemic cause, it reads “a brief service degradation, since remediated” - sanded so flat there’s nothing left to grip. The crack is visible in inverse proportion to the power to do anything about it.

We say the right words about this. We espouse the blameless postmortem, the no-fault culture, the learning organisation. And it never quite takes, and the reason it never takes is not that anyone is stupid. It’s that the gold is paid for by one set of people and spent by another. The engineer who writes “we knew, and we shipped anyway” pays for that line in a currency the org doesn’t reimburse - exposure, the quiet mark against their name, the implication aimed upward at whoever set the priorities. The value of that honest seam - the learning, the trust, the thing that stops it happening again - is banked somewhere above them, by people who paid none of the cost of the admission.

That’s the whole problem, and “blameless” doesn’t touch it. Blameless is a policy. It doesn’t change who holds the brush. When the cost of an honest repair falls on the people least able to absorb it, and the benefit accrues to the people who never picked up the gold, you don’t get kintsugi. You get the cheapest thing that still photographs as whole.

An organisation will not spend gold on an outage, but you should see what it spends on paint. A customer could be forgiven for thinking it was a perfectly new plate. The dashboard turns green, the public post-mortem a sanitised paragraph, the whole experience a transient memory a month later when it disappears from the status page. The uncomfortable truth is that the smoothing works - the rough edges are gone, the seam colour-matched, the cracks now invisible. But the danger is in the smoothing. When the care is superficial, so is the mend, and the cracks return as inevitable as a fault finding its own line. The incident returns. Sometimes it is the symptoms. Sometimes it is the fault again. But it always comes back.

The returning crack is the smaller loss. The larger one is the map.

Problem management - the discipline, the actual job - is nothing more than reading the seams. You stand back from a thing that has broken in the same place a dozen times, and the gold tells you where it is weak, where it will go next, where to spend before it spends you. That only works if the seams are visible. Paint over every one of them and there is nothing left to read. Each incident, sanded and matched and filed, is fine on its own. It’s the loss of the record that’s fatal, because a bowl repainted enough times is no longer a bowl with a history. It’s a bowl that is mostly paint, holding a flawless surface and hiding, even from the people holding it, that it has no idea where it is about to fail.

And the part that should keep a reliability person awake more than any single outage is how good we are at this. The repaint is not neglect. It is not the lazy option. It is a disciplined, well-resourced, beautifully governed process - owners, templates, turnaround times - all of it pointed at making the break disappear on schedule. We are not bad at remembering our failures. We are excellent at forgetting them, and we have built the machinery to do it on time.

I know the move because I almost made it.

I stand here with the true story. And the easy thing, the obvious thing, was to simply keep telling the old one. To never stand in front of a room and admit that for years I had handed people the wrong thing, lovingly, and they had written it down. Sand it flat. Match the paint. No one would ever have known there had been a crack at all.

The desk is a crime scene of small distractions. I reach past all of it. My fingers find the plate - cool, glazed, heavier than it looks - and I lift it level with my face and turn it slowly in front of the webcam, slow enough that the light catches the gold lacquer.

I open my mouth to tell the story. But I stop - the story stuck in my throat. I can no longer say the words.

So I don’t say them. I turn the plate over instead - the seam, the gold, the line where it came apart and was brought back - and I look at it the way I should have been looking at it for years. It isn’t that breaking made it beautiful. It’s a receipt. It says, in gold, that someone once decided this was worth not losing, and paid for the decision by hand.

I set it back on the desk, in the light, where it stays.

I sit there and think about everything I’ve sanded flat and called resolved. The reports I closed clean. The cracks I matched so well I can no longer find them. I don’t say that to the waiting faces. I just leave the plate where the light can reach it, and let them wonder why I’ve stopped talking.

Subscribe now

It was a Tuesday, and I need you to understand what that means.

Monday you spend on your knees in the wreckage of the weekend, sorting whatever the change calendar detonated while everyone was at the beach. Tuesday is the day the building exhales. Night shift gone home. Handover done. Overnight outages tied off and filed. Tuesday, if nobody goes out of their way to break something, is very nearly civilised.

I was drinking the downstairs cafe’s interpretation of a latte. A thing I know and wish I didn’t: hazelnut syrup will completely hide an over-roasted bean. Caramel if the cup is merely bitter. Chocolate if it has gone past bitter into something with opinions. The syrup does not fix the coffee. It has never once fixed the coffee. It makes the failure drinkable - which is a different thing, and on a Tuesday morning a different thing is enough.

I was thinking about a second cup. That was the entire scope of my ambition at that hour - when the rightmost screens in the command centre turned red.

Then three more.

Then everything.

Turned.

Red.

The two other incident managers on shift turned to me with the expression of people who have just watched the floor leave, and said:

“What do we do?”

If you have never stood inside an IT command centre: picture a wall of screens. Tickets, dependency maps, dashboards only a particular kind of damaged person can read at a glance. Mostly green. The honest splash of yellow. Now flood the whole room red, wall to wall, until the light itself changes and every face in it goes the colour of a warning label.

There is a fear that lives underneath thinking. Older than language, and it does not care how senior you are. It reaches the gut first and informs the brain afterward, as a courtesy. Competent adults stand in that red light and go briefly, genuinely stupid. If you keep a favoured deity on retainer, that is the moment to open negotiations.

I had two of those faces in front of me, and - honestly - I had the third. The same animal had me by the spine: the gut-drop, the white-out, the cold count of how many seconds before whatever I said next became the plan. What carried us was not nerve. I did not have nerve to spare. It was that we had built for this, and we had built it embarrassingly simple.

There is a law for this, and I believe it the way other people believe in things. Gall’s Law: every complex system that works got there by evolving from a simple system that worked. The ones built complex from the start do not work and cannot be repaired into working - you tear them down and begin again from something simple. So we had resisted the Great Defcon Playbook, the thick reassuring binder with a tab for every disaster a nervous mind can invent. We had three bridges. Platform. Applications. Front of house.

I pointed at the senior of the two.

“Platform bridge. I’ve got applications. Wide net - every oncall, all of them.”

I pointed at the other.

“Front of house. Every IM call diverts to you. Phone the team, find another IM, hand them the hotline. Then stand up an exec bridge.”

We scattered with our hearts going like fists on a door. Phones rang with news of escalating awfulness. Inside minutes the command centre was full - people on the line, people in the doorway, rooms filling with engineers wearing the fixed frown of someone typing very fast toward a moving target.

Three bridges. That was the entire apparatus. Now watch what grew out of it.

In our line of work, “fire” is almost always a figure of speech. Not that day.

Nobody had to be told what to do next, which was fortunate, because no document told them. Managers arriving for an ordinary Tuesday walked into the red light and brought leverage with them. Teams stood up breakout rooms of their own. Sub-groups peeled off to triage and reported back. Within the hour the three-bridge skeleton was carrying something it could never have specified in advance: a real map of what was down, where, and - finally - why every screen in the building had turned red inside the same ten seconds.

A fire had taken out the local telephone exchange.

That exchange carried the building’s data connection. The building was our headquarters. Our headquarters housed our primary data centre, because that was the style of the time. All the resilience engineering in the world will not save you from infrastructure you do not own and did not know you depended on. We were still running, and we were severed clean from the outside world - and from every regional centre in the country, all of them now wondering what had become of us. It was a small miracle the phones worked at all. None of us could read smoke signals, which is the only reason this is not a worse story.

An hour or two in, the connection came back. The provider had jury-rigged something out of copper and the building could speak to the world again. Have you ever felt elation as physical pressure in a room? A room cheering on the same half-second is a sound I can still produce on demand.

After that it was process. Reconnect in order. Green tick, green tick, green tick. By the time the sun gave up on the day, the machine was lurching back toward normal.

Stop on this for a second. The response that absorbed a regional telecoms failure - the managers, the breakout rooms, the sub-teams, the map - none of it existed at nine o’clock. Nobody designed it. It grew, inside an hour, out of three bridges and a wide net. That is not a lucky accident. That is the only way complexity that works has ever arrived anywhere.

I got the second coffee eventually. It was bad. I reached for the hazelnut without thinking, the way you reach for a handrail - and there is the confession, so let me make it plainly. I reach for the syrup. Everyone does. The syrup is seductive: it is right there, and it works, in the narrow sense that the cup goes down.

The Great Defcon Playbook is syrup. It is the most seductive document in our profession - thick, comprehensive, deeply reassuring - and it makes an organisation that has built nothing real feel prepared. It tastes exactly like readiness. Right up until a Tuesday arrives wearing a face the binder never tabbed, and the binder turns out to weigh a great deal and do nothing.

No playbook had a tab for “fire at the telephone exchange three streets over severs HQ from the entire country.” Yours doesn’t either. The specific shape of the bad day always outruns the imagination of whoever planned for it - and that is not a failure of planning. It is the nature of bad days. It is the whole reason you do not plan the day. You plant something small enough to grow into whatever the day becomes.

The three bridges were the bean - the real thing, the part the syrup can only ever imitate. Unglamorous. Almost too simple to respect. They anticipated nothing: not the fire, not the exchange, not the severed country. They anticipated nothing on purpose. They just gave several hundred frightened people a solid place to stand, and from that place a response grew that fit the disaster exactly - because it was built by the disaster, in real time.

Keep it simple. Keep it solid. Keep it small enough to trust on the worst morning of your worst day. Then let the right complexity find you - and it will, the second the room turns red.

The coffee will still be bad. You will still reach for the syrup; I am not going to pretend otherwise. Just never once mistake it for the thing holding you up.

[This article is a rewritten version of ‘Everything Turned Red’, originally published in February 2026]

Subscribe now

On Friday May 15th 2026, Eric Schmidt stood at a podium at the University of Arizona and got booed by graduating students. Multiple times. He had been hired to deliver a commencement address, and the moment he started telling the new graduates that AI was going to touch every profession, every classroom, every hospital, every laboratory - and that their job was to shape it rather than fear it - the booing started. He addressed it directly. I can hear you, he told them.

He kept going.

This is one of several such commencement booings in May. AI evangelism delivered to rooms of people about to enter the workforce is, it turns out, no longer a guaranteed applause line. The room has wised up.

But the booing isn’t the interesting part. The interesting part is that this is a man who has spent the better part of three decades explaining to younger workers why their concerns about their own value don’t matter, and almost nobody objected the first time he said it out loud. To find that quote, you have to go back eleven years.

In 2015, Eric Schmidt walked into a Stanford classroom and said this on camera:

The glue people are incredibly nice people who sit at interstitial boundaries between groups, and they assist in activity. And they are very, very loyal, and people love them, and you don’t need them at all.

He kept going.

At Novell, I kept trying to get rid of these glue people, because they were getting in the way, because they slowed everything down. And every time I get rid of them in one group, they’d show up in another group, and they’d transfer, and get rehired and all that.

He told Larry Page about the problem. Larry said: let’s review every hire. And so they did. Every offer packet at Google was flagged for the scent of glueness - for the trace of someone who might sit between groups and quietly help things along.

Nobody booed. Nobody objected. The lecture is still online. You can watch it.

The work has a name. Tanya Reilly gave it one in 2019, in a talk called Being Glue that has become required reading for anyone trying to explain to their manager why their performance review is going badly. Her observation was structural. The people who do glue work - the unblockers, the translators, the ones who notice that two teams are about to ship incompatible features - get promoted nowhere. The work is illegible to systems built around individual output. The team would collapse without them. The company can’t see them.

Lorin Hochstein pulled at the same thread from the other end. In 2021, he posted Schmidt’s quote on his blog - the same one I just made you read - under the title Contempt for the Glue People, and noted it had been lodged in his brain for years. He didn’t say much about why. He didn’t need to. The quote does its own work, provided you’ve ever worked anywhere that had any.

Both pieces are correct. Both should be read. This is not a contribution to that field; it is one quiet observation about something Schmidt himself admits inside his own monologue, which everybody seems to have noticed and nobody seems to have named.

What I want to talk about is what Schmidt’s anecdote concedes - what he says out loud, on camera, without appearing to hear himself. But I will come back to Schmidt. First, I want to show you the work itself - what it looks like when a room needs a glue person and goes looking for one.

Some years ago, at a company whose name you would recognise, two people from the same team pulled me into a group DM. The message, more or less: we need you, now.

I want to be precise about my standing in that moment, because the standing is the whole point. I was not on call. That, on its own, would be unremarkable. But I was also not on the escalation path that team was meant to follow - not the next name up, not the name after that, not any name on their roster at all. If you had asked the system who should handle what was unfolding, the system would not have returned me. There was no official route by which I belonged in that room.

What was unfolding was this: a group claiming to have breached the company had obtained information it was never meant to hold, and was threatening to release it at the one moment its release would hurt most. The particulars are not mine to set down here, and they do not greatly matter to the story - what matters is the room, and the room believed, to a person, that the threat was real and the damage would be severe. The two who pulled me into that group DM had not reasoned their way to my name. They had panicked their way to it. Somewhere in the scramble, the question they were really asking was: who is the largest available quantity of glue? Whose role touches enough of the company that they could walk into a room on fire and be a stranger to no part of it? That question resolved to me. They did not have the word - Reilly's word, the word this essay is built on - but the instinct was exactly right.

So I went. Not because the org chart sent me. Because the people did.

The room held more senior titles than I had seen in one place before, or since - directors, executives, names I knew from all-hands slides and nowhere else. And it was paralysed. Not quiet; the opposite. Everyone in it had a demand, a requirement, a thing that urgently had to happen. What nobody had was the willingness to own it. Not one of those senior, capable, expensive people would step into the centre and take the thing - each, I suspect, too aware of the price of being wrong in front of the others.

So I asked the most boring question available. Who can give me a rundown of what’s happening?

That is the entire trick, and I want to be honest that it is barely a trick at all. I did not fix the breach. To this day I could not tell you the technical particulars. What I did was sequence the room. What happened first. Where we stand now. Where we need to be. One by one, a director or a head of engineering handed me their piece; one by one, I handed back an instruction the whole room had just heard and agreed to. Every instruction was time-bound. Every group was given a named moment to come back and report. And then I sent them to work.

It took roughly forty-five minutes to pull a room of frightened executives back into something that functioned. By the end everyone was moving - and, the part I am still quietly proud of, everyone was moving on their own version of the plan, the version shaped by where they sat and what they could see. Nobody talked over anybody. Nobody waited to be told. The room had been glued.

It emerged afterwards that none of it had been real. The whole thing had been staged - an exercise, run to see how the company performed under fire. The threat was never real. The paralysis was. The company’s most senior people had frozen, completely, in the face of a danger that had been invented for the occasion - which means the thing I was called in to fix was never the threat. It was the room. The room would have frozen in exactly the same way had the danger been real.

The exercise was reviewed afterwards, written up and held closely, the way these things always are. The staged event got its record. The other event - a room of senior people who could not coordinate themselves until an unrostered process owner walked in and did it for them - got nothing. No review. No document. No note in any system anywhere. Measured by the trace it left behind, the most important thing I did that year may as well never have happened. Almost no one ever knew it had.

Now go back and read the Schmidt quote one more time. I’ll wait.

At Novell, I kept trying to get rid of these glue people, because they were getting in the way, because they slowed everything down. And every time I get rid of them in one group, they’d show up in another group, and they’d transfer, and get rehired and all that.

He thought he was describing rats. He was describing immune response.

The work didn’t vanish when Schmidt fired the people doing it. The work persisted, because the work was load-bearing, and the organisation kept hiring people to do it because the organisation kept needing it done. That is what the sentence says, in plain English, if you strip out the contempt that was filling the speaker’s mouth.

What’s interesting is that the posture isn’t unique to Schmidt. It is a recurring management mode - one that surfaces every decade or so under a different brand and produces a recognisable pattern of downstream consequences. The bills always come due. They just come due late enough that the executives responsible have usually already exited the building.

Google is the most thoroughly documented case, partly because Schmidt himself signed the policy. Every offer packet flagged. Every interstitial-boundary candidate marked for the founders’ inspection. And then, over the following sixteen years, Google produced the most exhaustively chronicled case of organisational coordination failure in the history of consumer software. Ron Amadeo’s 2021 piece in Ars Technica catalogues twenty-three messaging products across that span - Talk, Voice, Wave, Buzz, Allo, Duo, Spaces, Meet, plus messaging features grafted onto Maps, Photos, Pay, Stadia, the Assistant, and the phone app. Amadeo’s diagnosis, after sixteen years of evidence, is bracing: “nobody at the company is really in charge.” The most revealing detail in his piece concerns Google+, where Larry Page tied every employee’s bonus to the product’s success because that was the only way the company could be made to coordinate across teams. Working together was so foreign a concept that it had to be purchased, one employee at a time. The hiring filter had done its job too well.

Twitter ran the same play, compressed. After Elon Musk's acquisition in late 2022, headcount fell from roughly 7,500 to 2,000 in months. Subsequent rounds specifically targeted site reliability engineers - the people whose entire function is to keep systems running when the systems are about to stop running, which is glue work in its purest technical form. An employee, speaking anonymously to NBC News at the time, said they couldn't begin to describe the institutional knowledge being walked out the door - and noted that these were the people previously considered too important to cut. And then, days later, came the moment the universe rarely grants in real time. Musk himself, on his own platform: "I would like to apologise for firing these geniuses." Schmidt admitted in 2015 that the glue people kept getting rehired. Musk admitted in 2022 that he shouldn't have fired them. Two CEOs, saying versions of the same sentence: the work was load-bearing, and we didn't see it until it was gone.

Then everyone saw it. In May 2023, Ron DeSantis chose Twitter Spaces - Musk hosting - to announce his campaign for president. More than half a million people tuned in, and what they got instead was the platform failing in real time: the audio crashed, crashed again, and the event was abandoned and relaunched on a separate account, by which point the audience had roughly halved. Musk, live, could be heard saying the servers were "straining somewhat." His co-host called the meltdown "a good sign." Musk later declared the whole event a victory. A public, audible failure, narrated by the men responsible as a triumph, while it was still happening. And DeSantis was only the most visible instance - outages recurred through 2023, the platform at one point capping how many posts a user could read in a day, and the reporting kept landing on the same cause: the people who knew how the old systems held together were no longer there to hold them. Musk's apology had been glib. The year of outages that followed was the real one.

Boeing is the version of this story where the bills are paid in lives. The 737 MAX crashes of 2018 and 2019 killed 346 people, and the root-cause analyses across multiple Congressional reports, FAA reviews, and academic papers all converge on the same underlying mechanism: a decades-long erosion of the connective work between engineering, safety, and management, beginning with the 1997 merger with McDonnell Douglas and accelerating under CEO Harry Stonecipher, who famously declared his intent to run Boeing “like a business rather than a great engineering firm.” The 2024 FAA independent review identified a disconnect between senior management and the rest of the organisation as a primary contributing factor. That disconnect has a name. It is what happens when the people whose job was to translate between engineering culture and executive decision-making have been told they are slowing things down. The filter is invisible. The crater is not.

The pattern is the same in each case. A management posture treats coordination as parasitic. The people doing it are filtered out, fired, or restructured into irrelevance. The work persists because the work is real - but it persists as undone work, accruing as fragmentation, as outages, as products that should obviously cohere and don’t, and - when the products happen to be aircraft - as funerals. The contempt is the policy. The consequence is the bill. The receipts are public.

Schmidt’s filter was a slow instrument. Every offer packet reviewed by hand, by the founders, one at a time. Artisanal contempt - bespoke, labour-intensive, applied candidate by candidate. It produced the result we have spent this essay examining, but it did not scale especially well.

We have since built something that does.

The layoff rounds of the last few years - the ones announced with a paragraph about focus and discipline and, increasingly, a paragraph about AI - are Schmidt’s filter rebuilt as industrial equipment. The equipment works like this. When a company decides who to cut, it cuts by what it can see: lines of code, tickets closed, revenue attributed to a name, the legible quantum of individual output. It cannot see the work that has no metric - the translation between teams, the institutional memory, the unrecorded message that stopped an outage, the person who walks into a room on fire and is a stranger to no part of it. That work is not on the dashboard. And what is not on the dashboard does not survive contact with a spreadsheet that has been told to find savings.

AI sharpens the blade, because AI is genuinely good at the legible work - the discrete, specified, measurable task. So the story tells itself: this role is automatable, that one redundant, the headcount can come down. But the people swept out are not only the ones whose work AI can do. They are also, disproportionately, the ones whose work was never visible to the system doing the cutting - because illegible work and automatable work get filed under the same heading, and the heading is “cannot account for itself.”

Schmidt reviewed offer packets. We review entire workforces, against dashboards, and call the result efficiency. It is the same filter. We have only made it bigger, faster, and given it a better name.

Which brings us back to a podium in Arizona, and a man being booed.

Schmidt did not say anything especially cruel at that commencement. He told the graduates the future was unwritten and theirs to shape; he told them to get on the rocket ship and not worry about which seat. By the standard of his 2015 remarks about the glue people, it was a gentle speech. They booed him anyway.

It is worth being precise about why, because the booing was not what the coverage made it. It was not a tantrum about technology. Those graduates were not afraid of AI the way you are afraid of a storm. They were afraid of it the way you are afraid of a management decision - because they have spent their entire short adult lives watching what the rocket ship does to the people it decides it can do without. They have watched the layoffs. They have read the paragraph about focus and discipline. They have noticed, even if they cannot yet name it, that the people swept out first are so often the ones who held the place together, and that holding the place together is not a thing the spreadsheet can see.

Schmidt, describing the glue people he could never quite be rid of, thought he was describing rats. He was describing immune response. The booing is the same response, one level up - an organism recognising a pathogen. The graduates are not naive. They are correct. And the function Schmidt spent a career filtering out of his companies turns out to be very hard to filter out of a generation.

He told a Stanford classroom, in 2015, that the glue people were nice and loyal and beloved and that you did not need them at all. He was wrong in the most expensive way it is possible to be wrong: slowly, and with the bill arriving long after he had left the building. You need them. You need them most at the moment the room catches fire and you discover that need and roster were never the same word.

The lecture is still online. You can still watch it. But the rooms have changed, and they are not sitting quietly anymore.

Subscribe now

Daniel showed it to us on a Wednesday or a Thursday, the kind of day that has no character of its own, and within ten minutes I had the kind of look on my face that you generally only see when receiving an online purchase or a negroni at the end of a long day. The tool - and it was barely a tool at that point, more like a sermon delivered in the voice of a parole-board chatbot - could take a Slack channel full of incident chatter, a half-filled-out template, and a few prompt instructions, and produce something that looked, at squinting distance, like a post-incident review.

It was 2023. Large language models had been a public concern for less than a year, and inside the kind of company that ran on Jira and SOC 2 controls they were still treated as something between a curiosity and a compliance problem. Most enterprises had not yet decided whether they were tools or trinkets. The doom-or-salvation discourse existed elsewhere - on Twitter, in McKinsey reports, in the speeches Sam Altman was giving to anyone with a podium - but it had not yet reached the rooms where things actually got built. Inside our team, we had just spent another quarter watching engineers grind through the post-incident review process like someone chewing through a wall with their teeth. Daniel was an SRE. He had been quietly experimenting with what would later be called prompt engineering and was at that point still called typing stuff into ChatGPT until it stopped being stupid. He brought us a working prompt, untouched by legal, with no production access, no governance, and no expectation that it would amount to anything beyond a parlour trick.

Reader, it amounted to something.

I want to be precise about what we saw, because the optimism of that exchange is the entire reason the rest of this essay exists, and I am wary of telling you the story in a way that lets either us or the institution off the hook later. What we saw was a tool that could do, in maybe forty-five seconds, the part of a PIR that engineers found most miserable: the narrative reconstruction. The what-happened-in-order-with-timestamps-and-human-verbs. The part where you have to go back through Slack and translate “lol we restarted the pod” into at 14:07 UTC the on-call engineer initiated a pod restart in the affected service. That part. The boring part. The part nobody wanted to do.

And the demo did it. Not well, not at the level we would come to consider acceptable a year later, but well enough that you could see the shape of a future in which engineers were freed from the worst portion of an already unloved chore. Which, if you have ever been an engineer asked to write a PIR for an incident you do not really remember in the middle of a sprint you are already losing, you understand to be roughly the same scale of liberation as the invention of the dishwasher.

We took it to engineering leadership the following week. They blessed it without much theatre, which should have been a clue, and we joined the legal and privacy queue.

The legal and privacy queue is its own genre of waiting. You file the paperwork, you describe the use case, you propose a control surface, you receive twenty-three follow-up questions written by someone who has never used the product and has no plans to, and then you wait. You wait through a quarter-end. You wait through a reorg. You wait through the kind of compliance review that arrives via a ticket update from a person you have never met, to whom you are asked to explain, in non-technical terms, what a token is.

We were lucky in that our use case was relatively clean. We were not feeding the model customer data; we were feeding it internal incident chatter, which existed in a different and slightly more forgiving compliance category. We were not asking the model to make decisions; we were asking it to re-narrate, which is a thing computers had been doing in some form for decades, under names that were less marketable. So the queue was not as long as it could have been. It was, however, long enough.

So we waited. And while we waited, we built.

The prompt grew. What started as Daniel’s half-page proof-of-concept had turned into a multi-section instrument with explicit guidance for each part of the PIR template - summary, timeline, impact, root cause, contributing factors, action items. We learned the way you learn everything in prompt engineering: by watching the model fail and adjusting. The model wanted to proclaim customer names; we told it not to proclaim customer names. The model wanted to attribute blame; we told it not to attribute blame. The model wanted to use the word streamline in every other sentence; we told it the word streamline was now retired.

The thing we got right - the thing I want to underline now, before I tell you what we got wrong - was that we never asked the model to know anything. We asked it to re-arrange. The Slack logs went in. The template structure went in. The model’s job was to take a pile of inputs and produce something shaped like a PIR. Hallucination is a function of how much the model has to invent; we left it almost nothing to invent. The principle of this exercise, in case you are building one of these and would like to skip a class of failure: the model’s job is to synthesise the material you give it, not to source the material itself. If you find yourself asking the model to know things, you are asking it to lie eventually, and it will oblige.

Legal blessed it. We rolled it out. The first PIRs landed in review. The reviewers found them surprising - not because they were good, exactly, but because they existed at all, in roughly the right shape, having taken the human author perhaps twenty per cent of the time the human author would normally have spent. We had successfully automated forty-five minutes out of an unloved chore. The dishwasher metaphor from the demo, it turned out, had been roughly correct.

For a few months we felt like we had built something. I will not pretend the feeling lasted.

The first thing we noticed was the waffle. The prompt asked for concise output. The prompt asked for concise output repeatedly, in increasingly direct language, with examples of what concise looked like. The model agreed. The model agreed enthusiastically. The model then wrote summaries that took five sentences to do the work of one, in a register that one of our engineers eventually described as verbal diarrhea. It talked around itself. It explained things by length rather than brevity. The hallucination problem had been solved. The talking-too-much problem had been not just unsolved but, in some quiet way none of us yet understood, actively introduced by our own instructions.

We made a list of fixes. We did not, immediately, get to make them.

Immediately turned out to be a longer word than we’d planned.

The list of fixes went into a Jira ticket. The Jira ticket went into a backlog. The backlog went into a quarterly planning session, where it competed against twenty other things, most of which involved heads of engineering going ‘hmm’ in a concerned manner, and lost. The quarterly planning session was followed by another quarterly planning session, in which the backlog was reviewed again, and the list was, by general consensus and without any specific person being responsible, deprioritised. Nobody was opposed to fixing it. Nobody was opposed to almost anything. There was simply more shouting elsewhere.

This is a thing that happens to tools that kind of work. It is, in fact, the most reliable thing that happens to tools that kind of work. The fully-broken get fixed because the breakage is intolerable. The fully-functional get celebrated because the functioning is visible. The mediocre live forever, because the cost of revisiting them is greater than the cost of putting up with them, and because nobody who could authorise the revisit ever has to read the output themselves.

We were not idle in those months. There was always another incident, always another process to revise. We launched other things. We retired others. The team grew, shrunk, grew again. We upgraded the underlying model when GPT-4o came out, because that was the kind of small win you could ship in an afternoon; we did not revisit the prompt, because that would have required time we did not have. The PIR machine kept running quietly in the background, getting copy-pasted into reviews, getting approved, getting closed. We told ourselves we would get back to it. We did not.

Here is a principle I wish I had understood eighteen months earlier than I did: prompt engineering is craft, and craft requires maintenance windows scheduled into the calendar by people who are willing to defend them. You can ship a prompt. You can be proud of the prompt. The prompt will get worse without your noticing, not because the model is changing - though the model is also changing - but because the world the prompt is describing is changing, and the prompt is not. A prompt is a snapshot of your understanding at the moment you wrote it. Without scheduled revision, it ages the way photographs age: slowly, then all at once.

Eighteen months is a long time in software. Eighteen months is also, it turns out, exactly the length of time required for a tool that kind of works to become a tool that people are quietly furious about, without anybody being quite ready to say so. The fury was there. The fury was building. It was just waiting for someone to write it down.

The person who finally wrote it down was an engineer. I will not name them. They were having a bad PIR.

Specifically, they were having a bad PIR inside a piece of internal tooling we had built on top of Jira DC, which had been cobbled together in the way that internal tooling generally is at companies large enough to have opinions about internal tooling and small enough to defer the building of it until later. The tool worked. The tool worked the way most internal tooling works, which is to say: it worked if you held it correctly. If you did not hold it correctly - if, for instance, you attempted to save an incomplete set of information and the tool decided something was missing - the tool would, in a moment of administrative malice that nobody had specifically designed but nobody had specifically prevented, wipe everything you had entered. Not save it incorrectly. Not warn you. Wipe it.

This engineer had had this happen to them. More than once, I believe. They had then gone to the AI-generated draft to complete that, and the AI-generated draft had given them the kind of meandering, schoolboy-essay output we had been ignoring for eighteen months because, as previously established, nobody who could authorise the revisit ever had to read the output themselves.

The engineer took to Confluence.

The post compared our team - meaning specifically the team I was on, the team responsible for the PIR machine and the tooling around it - to Satan and to Hitler. The post was specific. The post named the data-wipe, the AI waffle, the time it had cost. The post did not propose solutions. The post was furious in the way that internal Confluence posts are furious, which is to say: with no structural restraint and the potential charge of physical assault on a keyboard.

It was, in many ways, the best thing that happened to the project.

Within a week, we had the authorisation we had been asking for in measured tones for eighteen months. The reason was simple, and it is the principle this section is here to land: organisations fund repair, not maintenance. Maintenance is invisible until it stops, at which point it is not maintenance, it is repair. The fury was the budget. The blog was the business case. The engineer who wrote it had, without intending to, performed the single most useful act of project sponsorship the project had ever received.

We did not thank them. I sometimes wonder if we should have.

The list of fixes came out of the drawer. Some of the fixes had aged better than others - the world had moved on, the models had moved on, the assumptions had moved on. Some of them needed to be thrown out entirely. We were not going to be patching v1. We were going to be building v2. Better. Cleaner. As it should have been, quarters ago.

The person who wanted us to use a different model worked in the AI team, which is a thing companies of a certain size have. I do not begrudge the AI team. Their job is to be the responsible adult in the room while a hundred product teams attempt to do something irreversible with someone else’s compute budget. I have been the responsible adult in rooms before, and I know what it does to a person. The cheaper model on offer was a flash model, and the fans of the flash model in this case had a spreadsheet, and on the spreadsheet there were two columns, and one of the columns was cheaper than the other. The fans of the flash model in this case had not read the prompt. The fans of the flash model in this case had not read any output the prompt had produced.

So we made the case. We made it politely, in writing, with examples; and then we made it again, less politely, in person, with more examples; and then we made it a third time, in language flat enough to survive translation through three layers of management. We compared outputs. We compared the quality of the comparisons, which is a meta-step you should always include, because the cheap-model fans will otherwise quietly judge the comparison on cost rather than on whether the comparison was fair. This took weeks. It felt like longer.

The principle that buys you back the weeks, if you have to make this case yourself: the right model for the job is not always the cheapest, and the people gatekeeping cost are not always the people who will read the output. Cost gating is a perfectly reasonable function of a perfectly reasonable team that, in the absence of skin in the game, will optimise for the metric they actually have. Your job, if you are the team who has to live with the output, is to make sure your metric is visible too.

We got Claude. The model was, on every measure we cared about, better. Claude wrote sentences a human would have written. Claude knew when to stop a sentence. Claude understood, in a way that earlier models had not quite understood, that the absence of a word was sometimes the right call. The waffle disappeared.

Around the new model, we rebuilt the prompt. The monolith became a series of sections, one per PIR component, each with its own tone guidance and its own source-grounding rules. Each section returned its output with citations to the Slack timestamps that had produced it, so the author could check the work in seconds rather than minutes. We were aware, painfully so, that the timelines for some incidents - the incidents where the team had been on a Zoom call rather than typing - were thin. Things had happened. Things had been resolved. The middle was missing.

So we added Zoom transcription via Loom. This is the part of the story where I have to tell you about one of our developers, who I will call Wilson, because we need a name and Wilson is not theirs. Wilson, in a moment of operational efficiency that we should have anticipated and did not, decided that the simplest way to ensure every incident Zoom call had a transcript was to put their own Zoom account into every incident bridge. Wilson’s account would join the call silently, record it, transcribe it, and deliver the transcript to the prompt. Wilson’s account joined a number of calls before anybody noticed.

When people noticed, the response was not measured. The response was: who is recording us, why, and what will be done with the recordings. The response was immediate, unanimous, and slightly biblical. We had built, without intending to, a small in-house panopticon, and the workforce was responding the way workforces respond when they find one in their meetings. We did the work of explaining, in clinical detail, what was recorded and what was not, where the transcripts lived and who could read them, how long they were retained and how they were destroyed. We did this because we had to.

The last change we made, and the change that mattered most, was the framing. The v1 prompt had produced output that read like a PIR. The v2 prompt produced output that read like a draft of a PIR. The difference is not subtle. The v1 output had headers and bullets and a structure that invited the reviewer to read it as finished. The v2 output had headers and bullets and a banner at the top, in language we had spent the better part of a fortnight getting right, that explained: this is a guide for the author to work from. It is not the PIR. The PIR is the document the author is responsible for writing, with this as a starting point. We had not changed what the model produced. We had changed what we called it.

The reframing helped. It did not help as much as we had hoped. There is a thing humans do when given a document that is almost what they need, which I have been trying to find a polite name for.

The polite name I have been trying to find, and the one I will settle for in the absence of better, is convenience. Humans, when given a document that is almost what they need, will treat it as if it were exactly what they need. Humans will do this even when the document is labelled in friendly capital letters DRAFT NOT FINAL. Humans will do this even when the document is preceded by a banner that the team writing the document spent two weeks getting right. Humans, in the end, do not read banners. They read the document.

The links had been thick and fast in my DMs long before v2 emerged - long before we had a name for the dynamic they evidenced, long before we had built anything to address it. Engineers I had not spoken to in months would forward me a link with the kind of brief, exhausted message that engineers send when they have run out of patience but have not yet run out of decorum: thought you’d want to see this. The thing I would ‘want’ to see, in every case, was an approved PIR that had been written by the model and submitted by a human who had not, by any reasonable interpretation of the word, written it. The PIRs were good enough to pass review. They were not good enough to be useful. They had been used anyway.

The reviewers, meanwhile, had developed a parallel adaptation. The reviewer’s job is to read the submitted PIR and decide whether it is adequate. The reviewer’s job, in practice, is to clear PIRs from the queue before the SLO expires, because the queue grows faster than the reviewer’s available reading time. The reviewer who is behind on the queue will scan, not read. PIRs written by the model scan beautifully - they were designed to. More often, and worse, they will not even scan and simply tick.

The author and the reviewer had, between them, found a workflow in which neither party was doing the human work the PIR was supposed to capture.

We built two things, and we built them quickly, because the longer this dynamic ran the more institutional memory we were quietly losing about every incident the system had processed.

The first was a copy-check. The submission flow ran a text similarity comparison between the AI-generated draft and the submitted PIR. If the similarity was too high - if the human had, in essence, copied the model’s output and called it their work - the submission was flagged, and a polite note explained that the PIR appeared to have been submitted without modification, and would the author like to take another pass. The note was polite. The note was also firm. The note had the effect of making the next twenty minutes of the author’s day moderately worse than the previous twenty had been, which was the point.

The second was a scoring system. We took every PIR from the previous two years that had been written for a sev one incident - the most-scrutinised, most-rewritten, most-stakeholder-edited PIRs we had - and we distilled them into a rubric. What did the strong ones do that the weak ones did not? What sections were always present, always specific, always actionable? What kinds of sentences did the gold-standard PIRs avoid? The rubric became an automated audit. Every submitted PIR would be scored against it. The score was visible to the author. The score was visible to the reviewer. The score did not block submission; it simply existed, on the page, in numbers, in places where numbers had not been before.

The rubric did two things. For the author, it provided immediate feedback on where the PIR was thin - your timeline lacks specificity, your contributing factors are not distinguished from your root cause, your action items have no owners - at the moment they could still fix it. For the reviewer, it provided immediate triage on where to focus their attention - this PIR scores well on timeline, poorly on action items, you can probably skip ahead and concentrate on the second half. The first audience used the rubric to write better. The second audience used the rubric to read more efficiently. Both, crucially, were doing more of the human work and less of the rubber-stamping.

Here is the principle, and I want to be careful with it because it is the one that mattered most and the one I most wish I had understood at the start: the predictable failure mode of automation is that humans will use the automation to skip the work the automation is supposed to assist with, not replace. If your tool makes a job faster, your tool will be used to skip the job. If your tool makes a job easier, the job will be skipped. If your tool produces an output that is almost the deliverable, the output will become the deliverable. This is not a failure of the humans. This is not a failure of the tool. This is the predictable interaction between a labour-saving device and a labouring human, and you need to design against it from day one. Not on day three hundred and sixty, after the fury has built and the engineer has taken to Confluence.

The copy-check has hopefully shipped, as has the rubric. The PIR quality, by every measure we had, will begin to recover. We had built, finally, a tool that knew how to coexist with the humans using it. We had built it after eighteen months of drift, one furious Confluence post, a flash-model fight, an accidental panopticon, and a framing rewrite that had nearly worked.

Reader, I'll never know for certain. I was made redundant three weeks later.

Subscribe now

What follows is not a framework. There is no certification at the end of it. These are the rules I wish someone had handed me in the first week of running major incidents - rules I have since watched colleagues learn one by one, at the cost of one Robert at a time. They look like manners. They are not.

On bridges and the people who join them

• If you are an IC and you do not control the bridge, the bridge controls you. Set the cadence, name the speakers, call the stand-downs. Otherwise the loudest voice runs the response, and the loudest voice is rarely the right one.

• The executive who joins the bridge to “help” is not helping. The executive who joins the bridge to “observe” is not observing. There is no observer mode. Every additional person on a bridge costs the IC roughly 8% of her remaining cognitive function, and she has already spent the other 92% on you.

• The head of engineering who is irritating everyone by ensuring things are truly fixed and every base is covered is doing the work nobody else will. It is not pleasant. It is thorough.

• If you join the bridge, say your name and your function within ten seconds. The IC is not a clairvoyant. She is tracking six responders, a Splunk dashboard, and a Senior Director who keeps unmuting to ask “where are we at.”

• Scribe as you go. The first ten minutes of an incident are seven different teams logging on and asking the same three questions. If the answers are not already in the channel, you will be repeating them yourself, in real time, while also running the response.

• Do not engage an individual directly. Always use the on-call roster, lest you disturb Robert for the seventh time that will likely result in his departure from the company three weeks later.

• PR and Legal get a bridge of their own. They serve a real purpose, and that purpose is not asking the engineers what they were thinking at 2:14am while they are still thinking it. Once the technical bridge becomes a discussion of who knew what and when, it is no longer a war room. It is a deposition.

• The senior engineer who knows the answer but doesn’t speak up because he’s “not on this rotation” is not being humble. He is being expensive. Speak up or log off.

• Never assume during an incident. There is no such thing as a stupid question. There are only stupid assumptions, made by people who did not ask one.

• Fifteen minutes. If a paged engineer hasn’t responded in fifteen, escalate. The clock is not a moral instrument; every engineer has missed a page. Anna once waited twenty-eight minutes for Damien out of politeness - the customer noticed at minute forty-one. Damien had been mowing the lawn.

On the language of incidents

• Do not type “should be resolved” in any channel. “Should” is an admission. “Is” is a commitment. Pick one and live with it.

• Warm handovers only. Revenge is a dish best served cold; a handover is not.

• An ETA is not an estimate. An ETA is a vow. Do not bring an ETA into a war room unless you intend to be married to it, in sickness and in 4am Slack pings, until rollback do you part.

• “Quick question” is reserved for things that are both quick and questions. Almost nothing qualifies. Almost nothing.

• Assumptions will ruin you. The quickest way to surface the right answer is to loudly proclaim - or scribe - the wrong one. The corrections arrive fast.

• Every engineer eventually causes a major incident. It is the rite of passage. Affix no blame while the fix is happening. Everyone is human.

• If a service falls over and nobody is told, does it make a sound? Yes - louder than the outage, and longer-lasting. Send the comms before the fix lands, when it lands, and on resolution. Silence is not modesty. It is a second incident, and you do not control it.

• When in doubt, shotgun. Page every team that could plausibly own the fault and let them stand down as they clear themselves. The alternative is finding the right team at minute eighty-nine, having spent the first eighty-eight on a polite tour of the wrong on-call rosters.

On the politics of severity

• Severity is a description, not a negotiation. After mitigation, a Sev 1 becomes a Sev 2 - the bleeding has stopped, and the work that remains needs hours rather than a war room. Before mitigation, talking it down because "we can manage it in business hours" is administrative violence performed with a calendar invite.

• Pages have a half-life. Every Sev 1 that turns out to be a Sev 3 increases the response time on the next real Sev 1, and on the one after that.

• Mitigated is not resolved. Mitigated means the bleeding has stopped. Resolved means there are no loose ends. Close at mitigation and you will reopen the same incident in two hours’ time.

• Do not resolve the incident until the customer confirms it is resolved. Until then, you have only resolved the symptom you can see from where you are standing, which is rarely where the customer is standing.

• A Sev 1 called at 4am is a parachute pull, not an escalation. Do not ask why it wasn’t called at 2am - that question is for the retro, and the retro will be brutal enough. Maria got asked it on the bridge once. Maria now works in product management.

On heroes and the cost of them

• No incident process or toolset is ever good. Get the duct tape, grit your teeth, and run the response with what you have. The perfect tool is always two quarters away.

• Heroes get singled out. Heroes get burnt out. Heroes leave. If you are watching one person fix the incident alone, you are not running a war room - you are running a hospice, and the patient is your retention rate.

• The IC is also the scribe. Every off-topic message in the channel is a tax on her bandwidth and a hole in the timeline. The gaps in the PIR you’ll skim in two weeks are not Priya’s failure. They are yours.

• Twelve hours. No engineer stays on a bridge longer than that. After twelve, they are not an engineer. They are a liability with a Slack handle and a degraded sense of what “safe to deploy” means at 3am. Owen made it to hour nineteen. He authorised the rollback that became INC-1843. Owen has not returned.

On what comes after

• Incident metrics are never accurate at resolution. If you do not revisit them during the PIR, congratulations - your metrics are a fable, and the moral is whatever the dashboard says it is.

• Blameless does not mean toothless. A PIR that cannot say the word “we” has nothing to say at all.

• A PIR without action items, owners, and due dates is theatre. "We will learn from this one" is a New Year's resolution - sincere in January, gone by February, repeated word-for-word at the next PIR.

• The post-incident review is not a trial. The post-incident review is also not a group hug. It is what the Greeks would have called catharsis, if the Greeks had ever had to roll back a deployment at 11pm on a Friday.

• Close a bridge without a debrief and you have built a boomerang. Confirm who owns what, write it down, then close. Real boomerangs return to the thrower; this one returns to whoever is on call next.

None of this is etiquette in the way the word usually means it. The rules exist because Maria now works in product management, Owen never came back from hour nineteen, and Robert left three weeks after the seventh disturbance. Politeness - the well-meaning, professional, didn't-want-to-bother kind - is what put them there. The protocol is what's left when you take it out. Print the list. There will be another Robert. The list cannot save them all, but it can save the one whose name you have not learned yet.

Subscribe now

It was morning, my time. For the customer it was the back end of an afternoon at the end of several days that had not gone well. The incident had been running long enough that nobody on the bridge could remember which version of the timeline was current, only that it had been grinding along for a few days now without a fix and the room had stopped treating it as urgent.

There were reasons for this and the reasons were structural. A single-tenant issue does not look like an incident to a system whose metrics are calibrated to blast radius. The dashboards do not light up. The all-hands channels do not move. Only one tenant, the team had said at some point in the previous days. Not unkindly. Not maliciously. Accurately. In the language another industry would use it would have been only one patient, and an emergency room with one patient runs at a different tempo than an emergency room with forty. The room’s tempo adjusts to its own load. The room knew its load to be small.

We do not bring customers onto technical bridges. This is policy and there are reasons for the policy. The reasons are that customers, when present, ask questions that the engineering team cannot answer while still solving the problem, and that the presence of an angry account on the call corrodes the focus required to do the work. The policy is correct.

The team in the room was good. I want this on the record. The engineers were senior, the work was real, the problem was not trivial. They were debugging in the way that twenty years of practice had taught them to debug. They were communicating with each other in the channel in technical detail and at appropriate intervals. They were doing all of this at the cadence the room considered appropriate to the size of the room’s problem, which is to say they were not hurrying, because the room’s problem was not large. By any measure the room had its own metrics for, the room was working.

What was not working was anything the metrics did not see. The customer had not been spoken to in several hours. Not, at any rate, in the way a customer four days into an unresolved incident needs to be spoken to. The customer service representatives joining the bridge to ask for updates were being given updates that were technically accurate and operationally useless, because the engineers were oriented toward the fix and the comms were a thing the engineers did between debugging steps when they remembered to. The reps then took the technical updates back to the account, where they were translated and softened and stripped of the specifics that might have made them informative, and then relayed to a senior contact at the customer who had been on the phone, on and off, for multiple working days.

I joined the bridge after the head of customer service and support called me into it. They had picked up the customer’s call directly, several minutes earlier, after a senior contact at the account had bypassed every reasonable escalation path the company offered and gone directly to the senior-most person whose job title contained the word customer. By the time I joined the call they were already on it.

They were cracking skulls. Diplomatically - they were choosing their words carefully, carefully enough to stay short of the line that would have required a follow-up conversation with HR, but not so carefully that anyone on the bridge could pretend they did not understand what was being communicated. The questions they were asking were structured around customer experience and resolution timeline. These were not, technically, accusations. The engineers in the room understood them as accusations, because they were.

I have a policy of not bigfooting incident calls. The incident commander is supposed to have authority and I am supposed to leave it with them. The policy is one I still defend in most circumstances. It also meant, in the days leading up to this bridge, that I had not been watching it. There were other bridges. There were larger incidents. Somewhere in the structure that allocated my attention there was a working assumption that a bridge running this slowly did not need senior process attention, because if it had needed it the metrics would have said so. The metrics had not said so. Until somebody picked up a phone, neither had I.

The fix was finalized within a few days. The metrics will record it as a SEV-2 with an extended duration but a clean technical resolution. The bridge had ended in success.

There have been others. The names are different and the systems are different and the day of the week is different but the shape of the failure is not. A room calibrating its tempo to its own metrics. A patient outside the room. A comms chain that loses fidelity at each translation. An executive whose unannounced arrival is what produces movement. Perhaps four hundred of them, by now, across a career. I have stopped counting.

What was missing from those bridges has a name. We have not been using it.

The role is incident commander. The name is in widespread use. What is not in widespread use is the discipline the name is supposed to describe.

We use the title two ways. One is the senior engineer or engineering manager who is on-call when the page lands and is therefore, by default, the person on whom coordination falls. The other is a person whose entire role is to coordinate the incident - whose authority derives from the role rather than from seniority, and whose orientation is toward the patient rather than the room. The first version is a rotation. The second is a discipline. They are not the same thing.

The rotational incident commander has authority, but it is the wrong shape. A senior engineer running a bridge has technical authority - the room defers to them on how to fix the thing. An engineering manager running a bridge has team authority - their reports execute and their peers cooperate to whatever degree the org chart governs. Neither of these is the authority the role actually requires, which is coordinative - the standing to direct attention across functions whose hierarchies the IC does not sit inside, the position from which to tell customer success and product and the VP that the room will hold for ninety seconds while a decision is made about comms. Coordinative authority is bounded. It is not the authority to make the technical call or the team call or the political call. It is the authority to hold the coordination of the people who do.

It is also oriented.

My wife works in surgery. The temperature in the operating theatre is set by someone who is not in the room. The surgeon can have it changed, but they cannot change it themselves; they must communicate the need to the role, who makes the adjustment. The setting is not chosen for the comfort of the people present - it is chosen for the patient. The room is held cold for the patient even though the surgeons would prefer it warmer, because the room is not for the surgeons. The role that holds the temperature is the role oriented toward the patient and away from the room.

The rotational IC has neither the bounded coordinative authority nor the patient orientation. They are of the room. They have been pulled into the role from the room, and they will return to the room when the incident is over. Their measures of success are the room’s measures. Their relationships are the room’s relationships. The patient - the customer, the user - has no advocate in the room because no role in the room is structured to be one. So the room defaults to its own metrics. The metrics record the incident as resolved. The patient leaves, some weeks or months later, for reasons the system that knew about the incident will not track.

The objection to all of this is one I have held myself, in some version, for years. The policy of not bigfooting incident bridges contained the assumption that the model in place was structurally sound. I would not have written this essay without first having to admit that it was not.

The objection runs roughly: coordination is a leadership skill that any senior person can develop. The rotational model works because the people in it are senior enough to hold coordinative authority and patient orientation alongside their other contributions. The cases where it fails are cases where the wrong person was in the rotation, or the org has not trained well enough, or the IC was having a bad day. The fix is better people in the existing model, not a new discipline.

This has truth. The rotational model works in small organisations. It works in tightly-cohered teams whose incidents are bounded and whose customers are few enough to be visible to everyone in the room. It works when the volume is low enough that the people in the rotation can hold coordinative authority and outcome orientation alongside their other work without strain. In those contexts, the senior-engineer-on-rotation is the right answer.

The contexts where it fails are the contexts where it most needs to work.

The surgeon cannot adjust the temperature themselves. To change it they have to relay the request to someone whose role is to hold the temperature, who then makes the change and confirms it back. The architecture of the operating theatre enforces what the surgical discipline already requires: that orientation toward the operation and orientation toward the temperature are different orientations, held by different people, communicated across a deliberate boundary.

Software has built no equivalent architecture. We put the manager-as-IC and the senior-engineer-as-IC inside the room and ask them to hold both orientations at once, under pressure, for several hours or several days. They are competent. They are senior. They cannot hold both. One orientation loses. The patient drifts out of frame in small increments until the room is being run for the room.

The rotational model works until the room and the patient diverge. They always diverge. By the time they have, the room has built its metrics around its own comfort.

What this costs, when gotten wrong consistently, is a thing that does not appear in any single retrospective.

A customer who churns six weeks after a bridge that the metrics said had gone fine. A customer service rep who had been trying to flag what nobody on the bridge would hear, who is reviewed at the end of the year on a metric that does not include having been right. An engineer who runs incidents the way their seniority equips them to run them, who burns out from a role nobody has named and that nobody is going to thank them for. An incident that resolves cleanly on the metrics and quietly poisons three account relationships because the room could not see what the room was being measured against.

None of this is what the postmortem says happened. The postmortem says the fix shipped, the TTR was acceptable despite the extended duration, the on-call rotation worked as designed. The room’s metrics record the room’s experience. The patient leaves quietly some weeks or months later for reasons the system that knew about the incident will not track.

What is missing has a name. We could give it one.

The role would be coordinative rather than commanding. Bounded rather than ultimate. Oriented toward the patient by the structure of the role itself, not by the goodwill of whoever is awake when the page lands. It would be hired for, trained for, and protected from being collapsed back into the rotation. It would have authority over coordination and not over the work being coordinated. It would not be the senior engineer or the engineering manager, although either of them might do it well if they were trained for it and given it as their full role.

We have built the architecture for this in other industries. We have not built it in ours.

There is a bridge open somewhere right now. It is well-run. The metrics, when they are recorded, will say it ended in success.

Subscribe now

Every engineering organisation I have worked inside has had its rituals. Not the formal ones - not the standup, the retro, the quarterly planning offsite - but the older kind. The kind steeped in superstition. The kind that predates silicon and will outlive it. The kind a visiting anthropologist would recognise immediately, even if the practitioners would object to the comparison. We do not deploy on Fridays. We do not touch the billing service at end of month. We do not rename the queue. We do not utter the word "quiet" when on-call. We do not speak honestly about the bad region. Each of these is an offering. Somewhere in the organisation's past, the system became angry. We have agreed, collectively and without quite saying so, on the offerings that keep it from becoming angry again.

The original event has usually been forgotten. The offering persists.

The Friday Deploy

The most universal of the rituals, observed in some form by every engineering organisation that has lived long enough to have a folklore. The rule states that production deployments shall not occur on Fridays, and shall certainly not occur on Friday afternoons. The justification, if pressed, is operational - engineers do not like cleaning up messes on the weekend, nor do they enjoy the surprise of a Monday fiasco once everyone logs on to find things irrevocably broken. The justification is accurate. It is also incomplete. The deeper truth is that engineers have learned, over years and across companies, that Fridays are when systems choose to fail. Not because Fridays are technically different from Thursdays. Because the cost of a Friday failure is uniquely punishing, and the system, in its particular cruelty, seems to know this. The rule remains. It is enforced by engineers who have personally cleaned up a Friday deploy, who carry the specific memory of a weekend lost to a rollback or a Monday morning spent explaining what went wrong. They enforce the rule against the new engineers who have not yet had their own Friday, who will have one eventually, and who will then enforce the rule the same way against the next cohort. The folklore renews itself.

The Billing Sabbath

The billing service is sacred. It is not to be modified during the closing days of the month, when invoices are generated, totals are reconciled, and the organisation's revenue passes through a series of pipelines nobody fully understands. The freeze is enforced. There is a change window, a CAB approval requirement, a deployment lockout. The taboo underneath the policy is older. The freeze existed before the change window did, and the change window was written to formalise what the engineers had already agreed to without writing anything down. The new engineer who proposes a billing-related change in the last week of the month learns quickly that they have proposed something else. The response is not a polite redirect. It is a small ripple of alarm in the channel - senior engineers materialising to ask what is being proposed, why it is being proposed now, whether the proposer is aware of what week it is. The policy has a date. The taboo has none.

The Queue That Cannot Be Renamed

Somewhere in the architecture diagram is a queue with a name that no longer matches what it does. It was named for a feature that has been deprecated, a service that has been retired, or a use case that has shifted three times since the original implementation. Its current purpose has nothing to do with its name, and both facts are accepted as part of the system. Everyone agrees the name is wrong. Nobody renames it. The folklore whispers that once upon a time, a rename was attempted. The story is rarely told in full. The cost of renaming is judged higher than the cost of explaining the discrepancy to every new engineer for the rest of the system's life. The explanation becomes part of onboarding. Six months later, the new engineer will be the one explaining it to someone newer. The wrongness, eventually, becomes a feature.

The Word Not Spoken

The on-call engineer does not say it is quiet. Even when it is quiet. Especially when it is quiet, because that is when the system is listening. The rule is enforced not by management but by the other engineers in the channel, who will respond to any utterance of the word with a chorus of warnings and the horror reserved for genuine taboo. The reasoning, when articulated, is statistical. The word does not summon incidents. Incidents simply tend to follow it. The distinction is technically important and operationally meaningless. The reasoning, when not articulated, is older. It is the same reasoning that prevents sailors from naming storms while still at sea.

The Deprecation That Will Not Complete

The service was deprecated in 2022. The retirement date was set for Q3 2023. In Q2 2023 the date was moved to Q1 2024. In Q4 2023 the date was moved to Q3 2024. In Q3 2024 the date was moved to Q2 2025. The service is still running. It will, almost certainly, still be running next year, and the year after. Each delay is justified by a specific dependency unmigrated, a customer unnotified, a downstream system unmodified. None of these justifications are wrong. All of them, taken together, describe an organisation that has agreed to keep a thing alive while pretending to be killing it. The deprecation is no longer a project. It is a posture.

The Black Sheep Region

Every cloud-scale system has a region with a reputation. The reputation is real. It is also, officially, not real - the formal position is that all regions are equal, and the architecture documentation describes a uniform deployment topology that does not distinguish between them. The engineers who actually operate the system know better. There is a region where things simply go wrong. The other regions should be over-provisioned to compensate. Sometimes the over-provisioning has been quietly skipped, and the gap is treated as a manageable risk that has not yet manifested. Runbooks have specific procedures for when this region fails, written in a tone that suggests the failure is anticipated rather than handled. Capacity planning treats the region's reliability as a known quantity, and the known quantity is "lower." None of this is in the architecture document. All of it is in the operational reality. The engineers who run the system have built an entire infrastructure of compensations around a region they continue to officially treat as equal to its peers, and new engineers learn the compensations through exposure, not through documentation.

These rituals are not a problem. The problem is that the institution cannot admit the rituals exist. The architecture document describes a uniform topology. The roadmap describes a deprecation that will complete. The change-management policy describes a freeze that has reasons. None of these documents acknowledge the superstitions underneath them. The rituals persist anyway, transmitted through warning glances and onboarding asides and the kind of hushed corrections that new engineers learn to recognise as folklore.

We are not a rational industry. We are a tribe with rituals, and we have agreed not to call them rituals because we are an industry that does not believe in superstition.

The original event has been forgotten. The offerings persist.

Subscribe now

Every severity scale is a map of what an organisation is willing to admit.

The top of the ladder is well-lit and well-trodden. The bottom, less so. Walk down it slowly and you notice the lighting getting worse.

SEV-0 is the incident that ends one company and starts another in its place - the same logo, the same office, the same payroll, but a different company, the way a building is a different building after a fire even if the bricks are the same. You learn about a SEV-0 the way you learn about anything serious in this industry: late, indirect, and from someone who would rather not be telling you. The principal engineer at the bar who says "we don't deploy on Fridays anymore" and does not explain why. The staff engineer who flinches, fractionally, at the mention of a particular subsystem. The runbook with a section so over-engineered it could only have been written by someone who watched the previous version fail. SEV-0 is the inheritance nobody hands you. The architecture remembers. The taxonomy does not.

SEV-1 is the one everyone understands. The site is down. The money has stopped. Your company is mentioned by name on a news site. Someone senior is awake who should not be awake, and someone junior is typing with the terrible precision of a person who knows their commit history will be read aloud in a room next week. SEV-1 is loud, expensive, and - because of the noise and the cost - honest. You cannot hide a SEV-1. The category works because the incident refuses to be ignored.

SEV-2 is the incident that does not sleep, and arranges for you not to either. It is too big to ignore and too small to escalate to someone important. It is real enough that the channel stays open through the night. So you hold the line for four hours, sometimes eight, and you watch the clock the whole time, because the longer it runs the more likely it becomes that someone important will have to be woken anyway, and at that point the incident is no longer a SEV-2. SEV-2 is the severity that is partly defined by how quickly you can make it stop being one. It is where you learn that incident management is a clock-management problem.

SEV-3 is the workhorse. It is where most of incident management actually lives - the elevated error rates, the latency creep, the integration partner who has chosen today to have feelings about their API contract. It is also, by volume and by neglect, the severity most likely to be ignored. Not rejected. Not triaged and deprioritised. Ignored. Left in the channel like a glass on a counter that someone will get to eventually. And then four hours pass, and the glass is still there, and the customers who were patient at hour one are no longer patient at hour four, and the SEV-3 is no longer a SEV-3. It has become a SEV-2 by sheer laziness - not because the incident got worse, but because nobody made it better while making it better was still cheap. If you want to know whether an organisation's incident management is real or performative, watch how it handles a SEV-3 on a Friday afternoon. The answer is usually: it doesn't.

SEV-4 is the severity that half the industry claims to have and nobody actually runs. It is the incident too small to mobilise for and too real to dismiss - the queue that backed up for six minutes, the endpoint that five-hundred'd for a fraction of a percent of traffic, the alert that fired and resolved before the channel filled. In theory this is where the organisation learns. In practice it is where the organisation files and forgets, because the cost of taking a SEV-4 seriously is higher than the cost of shipping something else instead. So the category quietly empties. And in some places - I worked inside one - it never existed to begin with. The scale goes one, two, three, and then straight to the end. A house with no ground floor. Everyone who worked there understood why without ever quite saying it.

SEV-5 is the category we do not have, because having it would mean admitting what it contains. It is the documentation that went stale in 2023 and is still being cited in 2026. It is the monitoring nobody trusts, because the thresholds were set by someone who left three reorgs ago. It is the single engineer who understands the billing pipeline and is currently interviewing at a competitor. It is the runbook that has been wrong for fourteen months, and the team that has learned to work around the wrongness, and the new hire who will inherit the workaround as the thing itself. None of this will page you. All of it will kill you. The reason we do not have a severity for slow erosion is that a severity implies a response, and the response to slow erosion is structural, and structural responses require someone willing to say aloud that the house is on fire even though nothing is visibly burning.

The ladder does not end at SEV-3. It does not end at SEV-4. It ends somewhere below, in a category we have decided not to name.

It will wait there, whether we name it or not.

Subscribe now

There is a particular kind of silence that used to exist in the world, and I would like to report it missing.

It lived in the two-minute gap between ordering a coffee and receiving it. It lived in supermarket queues, in the passenger seat of a long drive, in the waiting room at the GP, in the shower, in the three-second pause between pressing the button for the lift and the lift arriving. It is gone now. Every one of those silences has been surveyed, zoned, subdivided and sold to a developer, and the developer is Meta, or Google, or ByteDance, or whichever of them got to the zoning board first, and the tenant is a ninety-second video of a man deep-frying a Toblerone.

This was not a natural process. Silences do not spontaneously fill with Toblerones. Someone had to do it. Someone had to decide that the pause was a market failure, that the gap was inventory, that the three seconds you spent waiting for the lift were unmonetised and therefore an affront. I would like, in the next thousand words, to itemise the damage and, eventually, to name them.

Let me itemise what was taken, for the purposes of the file. The queue at the post office. The minute at the traffic lights. The bit in the shower after the shampoo and before the conditioner. The walk down the driveway to the bin. The thirty seconds you used to spend looking out a train window. The forty seconds you used to spend looking at your own ceiling. The four seconds between setting the pressure cooker and realising you had nothing to do for the next forty minutes, which used to be a complete thought and is now a low-grade panic attack.

The perpetrators were not, in a strict technical sense, thieves. Thieves break in. Thieves leave evidence. These men applied for a permit. They filed paperwork. They had a lawyer explain to a man in Congress that the pause between stimuli was a market failure, that the three seconds you spent waiting for the lift were inventory, that an unmonetised second of human attention was, in fact, an insult to the shareholders. The man in Congress nodded. The permit was issued. The bulldozers arrived in 2011, or thereabouts, disguised as an iPhone.

Their names are on a list somewhere, but the names are the least interesting thing about them. They are interchangeable. They rotate. One goes to Meta, the next goes to OpenAI, a third sells everything and buys a ranch in Montana and calls it retirement, though in reality he is still on four boards. They wear Patagonia vests because the uniform is required for entry. They speak in a dialect assembled from Stanford business school, a self-help book about habits, and the collected tweets of a man who owns a rocket company. They do not think of themselves as criminals. They think of themselves as founders.

I should, at this point, admit that I am the mark. I was made redundant in March. I had expected, in some remote and theoretical way, that a redundancy would involve a period of rest - that there would be, inside the disorientation and the grief, some pocket of stillness, some hours unclaimed by anybody. There was not. By the first week I had built a personal brand. By the second I was posting and browsing on LinkedIn on a strict Tuesday-to-Thursday cadence. By the third I had ranked nine prospective employers by salary band. By the fourth I had written a two thousand word essay about a fidget toy. I did not rest. I did not know how. A day in which I had not generated something - a post, a pitch, a paragraph, a note, a thought worth capturing - was a day I had failed, and the failure was personal, and the failure was metabolic, and it arrived on time every morning at 3am, uninvited, with a list.

This is, I recognise, a diagnosis. The symptom is the inability to sit in a chair for forty minutes and watch a dashboard go green without reaching for a second screen to watch the dashboard watch itself. The symptom is the inability to queue at the post office without feeling the queue is a kind of theft. The symptom is the inability to experience a Sunday afternoon without narrating it into a piece of content. The symptom is shared, I suspect, by most of the people reading this, and by most of the people who will not, and most especially by the people whose professional responsibility is to notice, in a room full of green dashboards, the one thing that is not quite right.

Consider the on-call engineer. Consider the air traffic controller. Consider the anaesthesiologist who spends six hours watching a number on a screen that must not change, whose entire job is the capacity to remain vigilant during the fifth hour and fifty-ninth minute of a shift in which nothing has happened. Consider the sonar technician. Consider the nurse on a night shift in ICU. We have built an economy that requires a class of people to be professionally bored - to sit still, to watch, to notice - and we have issued each of them a phone engineered by the best minds of a generation to ensure they cannot. You cannot ask a human to watch a quiet dashboard for four hours and also hand them a slot machine, and we have done both, at scale, and called it digital transformation.

The case will not be investigated. No charges will be filed. The crime was legal. The crime is, in several jurisdictions, considered a growth sector. The evidence is in my pocket. It is in the anaesthesiologist's pocket. It is in the on-call engineer's pocket. I checked mine while writing the previous sentence. I will check it when I finish this one.

I will check it, I suspect, during the pause between you reading this paragraph and deciding whether the piece was any good.

Subscribe now

The instruction was simple. Hands flat on the desk. Eyes forward. Pay attention.

At some point in the history of Western education, someone decided that the ideal learning posture was that of a person waiting to be executed. Upright. Still.

The first thing they take is the movement. Before they take your confidence, before they take your grades, before they install the particular brand of shame that follows a restless child into adulthood like a bad debt - they take the movement. The stance of compliance, dressed up as the posture of learning.

And so, rebellion. The pen went first. Confiscated, second week of term, on the grounds that the clicking was disruptive. Then the eraser - or what was left of it. Apparently the small debris field on the desk constituted some kind of offence. The smart ones learned to doodle under the pretense of taking notes. If you could pretend to be still enough, you could get by.

Here is what they do not teach you in school, possibly because it would undermine the entire architecture of the place: your brain runs on dopamine the way a city runs on electricity, and when the supply is irregular - when the system is wired differently from the start, which is the case for a significant percentage of the population - the lights flicker. The executive function dims. The prefrontal cortex, that smug little governor of attention and impulse, starts to go offline.

As it turns out, the restless ones were right.

Here is what the science has established, at considerable expense and over several decades of peer-reviewed effort:

The brain is not separate from the body. Mental functions cannot be understood without reference to the physical body and the environment. Movement - deliberate, rhythmic, embodied movement - changes brain structure. It preserves grey matter. It strengthens neural connectivity in regions governing attention, emotion, and executive function. It means your body can start regulating itself before your brain has decided to. No thinking necessary.

Yoga practitioners. Tai chi practitioners. Qigong. The Feldenkrais Method. Conscious dance. All of them producing measurable neurological change in people who practice them.

The field has a name. Movement-based Embodied Contemplative Practices - MECPs, per a 2014 paper in Frontiers in Human Neuroscience by Schmalzl, Crane-Godreau and Payne that is worth reading if you want the full architecture.

The concept is older than the acronym by several thousand years.

Somewhere in a UC Davis laboratory, a researcher named Julie Schweitzer has spent years proving something that should have been obvious to anyone who ever watched a kid get their pen confiscated. Children and adults with ADHD, she found, performed better on cognitive tasks when they fidgeted. Not slightly better. Measurably, consistently better. The movement wasn't despite the focus. It was how the focus was happening.

For neurodivergent brains in particular - ADHD, autism spectrum, sensory processing differences - this isn't optional. It's the mechanism.

You see, the prefrontal cortex - your brain's executive function hub, the part responsible for planning, working memory, and sustained attention - runs on dopamine. This is not controversial. What is slightly more interesting is that physical movement, including something as small and apparently pointless as rolling a pen between your fingers, triggers dopamine and norepinephrine release through a mechanism that lands in roughly the same neurochemical neighbourhood as what Ritalin does. Nobody puts this on the label of fidget toys. They probably should.

Before the prayer bead, before the rosary, before any of the formalised liturgical hardware that organised religion eventually wrapped around this impulse - there was a stone. Just a stone. Smooth, because a river had been working on it for longer than any human civilisation has existed. Picked up, held, rubbed with the thumb until the thumb knew every contour.

Archaeologists have found them in ancient Greek burial sites, worn smooth in a way that only years of daily handling produces. The Greeks were pulling them from the sea specifically for this purpose as far back as 1000 BCE.

At some point - and the exact point is disputed, obscured by the usual fog of religious history - somebody had the idea of putting the stones on a string. This was, in retrospect, a significant upgrade. A single stone is a comfort. A string of them is a system. You can count on a string.

Monks on Mount Athos in the medieval period were tying knots in cords to count prayers - the komboskini, prayer rope, functional and austere. Then it escaped the monastery, as useful technologies tend to do, and found its way into the cafes and street corners of Greek life, where it shed the prayers but kept the rhythm.

By the twentieth century the komboloi was everywhere. Prime ministers had them. Shipping magnates had them. The old men in kafeneions clicking their beads in the shade of the plane trees had them. The rhythm of the beads had become a kind of ambient music of Greek culture - fast clicking signalling agitation, slow rolling signalling ease, the hands encoding mood in sound the way a jazz musician encodes feeling in tempo.

And then there was the begleri. The stripped-back version. No closed loop, no tassel, no elaborate bead count - just a short cord with weighted ends, open-stranded, designed for tricks and manipulation rather than counting. Simpler, more versatile, easier to carry in the slim pockets of blue jeans. The begleri became associated with the mangas - the Greek urban countercultural type, linked to the working class port districts, to rebetiko music, to a particular brand of cool that had no interest in respectability. They would swing the beads with theatrical confidence, clacking them together in ways that attracted attention, that demonstrated skill, that said: I have nowhere to be and I know exactly what I’m doing.

The object that started as a monk’s prayer tool had become a prop for the Greek equivalent of James Dean.

In 1967, the Greek military junta banned begleri with long cords. A dictatorship, with tanks and all the apparatus of authoritarian control, found time to legislate the dimensions of a hand toy. The practice survived. The cord got shorter.

The timeline of this technology runs roughly as follows: Smooth stone pulled from a river, ancient Greece, approximately 1000 BCE. Knotted prayer cord, Mount Athos monastery, medieval period. Komboloi, nineteenth century. Begleri, Greek urban working class, late nineteenth century onwards. Begleri with long cords banned by military dictatorship, 1967. Begleri unbanned, junta collapses, 1974. While the fidget spinner circa 2017 gained peak absurdity and was largely forgotten within eighteen months, a small machined object from Canada had already quietly arrived two years earlier. The Knucklebone. 2016. Still in production. Still in pockets.

What this timeline is the history of is not toys. It is not stress relief products. It is not the wellness industry finding new ways to monetise anxiety. It is the human nervous system, across three thousand years of recorded history, refusing to accept that the hands have nothing useful to do.

Every civilisation that ever tried to sit still and think eventually put something in its hands. The only thing that has changed is the material.

The timeline does not tell you which of these things actually worked.

There is a test for this. Simple, brutal, takes about thirty seconds. Put the thing in your hand during a task that requires genuine concentration. After five minutes, ask yourself: where was my attention? If the answer is anywhere near the object in your hand, you have a toy. If the object has faded into the background of sensation - present, grounding, doing its quiet work below the waterline of conscious thought - you have a tool.

Most of what got sold in the great fidget panic of 2017 fails this test immediately. The spinner fails it. The cube with its sixteen satisfying interactions fails it. They were designed to be noticed, to be played with, to be the thing you were doing - because that is what sells, and the alternative, a small unremarkable object that works precisely because it refuses to be interesting, does not photograph well and cannot be unboxed on YouTube.

The worry stone does not have a YouTube channel. It has been in continuous production for three thousand years.

The first thing you notice about the knucklebone is the weight. Not heavy, exactly, but present - the particular density of machined metal that communicates craftsmanship without announcing it. You roll it once between the fingers and something settles, somewhere in the background of the nervous system, in the way that a good pen or a well-worn stone settles. The hands have found something that fits.

It is approximately the size of a small spool, with rounded end-knobs and a central grip that sits naturally in the pinch of thumb and forefinger. It was designed by a Canadian schoolkid in 1989 who was tired of getting in trouble for doing yo-yo tricks in class. He was, in other words, exactly the kind of kid this article is about. He built it from polymer clay, then forgot it in a box for twenty-seven years before the world was ready for it in 2016.

It was a solution looking for a problem that already existed. The problem being this: how to give a restless mind somewhere to put itself without disturbing everyone around you.

Here is what the classroom never told you: the fidgeting was correct. The tapping, the rolling, the increasingly elaborate doodles in the margins of notebooks that were supposed to contain something else - all of it, correct. The nervous system doing what nervous systems do when they need to think and are not being allowed to move: finding something, anything, to keep the hands occupied while the mind gets on with it.

The monks on Mount Athos knew this. The Greek mangas knew this. The schoolkid in Victoria Canada with the polymer clay knew this.

The teacher who took the pen did not know this. That is not your problem anymore.

AroundSquare make the knucklebone and several other objects worth putting in your hands. They did not sponsor this article, endorse it, or receive advance notice of its existence. This is not an advertisement. It is a recommendation, which is a different thing entirely, and the difference matters. aroundsquare.com

Subscribe now

PROBLEM RECORD — PRB-1743-BLACKSPOT Raised following INC-1743-BLACKSPOT (Total Loss of Vessel, The Black Spot)
Status: Post-Incident Review Complete
Problem Owner: D. Norkin, Harbour Master (recipient of both ravens)
Review Facilitated by: Nobody. This document was written collectively. It took fourteen weeks.

INCIDENT SUMMARY

On the evening of [DATE UNKNOWN - the ship’s log was being used to prop open the powder room door], The Black Spot encountered severe weather conditions in the North Atlantic. The vessel sustained catastrophic and irreversible damage resulting in total loss of ship, cargo, and most of the crew’s confidence in each other.

The position of Incident Commander had been formally eliminated six weeks prior to the incident following an operational efficiency review conducted by Harwick & Sons Maritime Consultants, who noted that the Captain “did not directly contribute to propulsion, navigation, or combat operations” and recommended the role be dissolved with responsibilities distributed across existing crew.

The review was conducted during fair weather. This was considered representative.

The redistributed responsibilities were documented in a two-page appendix. The appendix was filed. No one is able to confirm where.

The storm did not appear to have read the appendix.

INCIDENT DETAILS

Duration: Approximately 64 minutes from initial storm detection to reef contact. The incident is considered ongoing in the sense that the ship has not been recovered as of this writing.

Scope: The entirety of the vessel The Black Spot, inclusive of cargo. Vessel was forcibly relocated in close proximity to a well documented reef in the North Atlantic Ocean.

Crew Impact: Of the original complement of 27, fourteen are confirmed present for this review. Nine are accounted for but declined to participate. The remaining four submitted stew as a mitigating factor and have been asked to resubmit (see below timeline).

Cargo Impact: Total loss. The manifest has not been recovered. This is considered fortuitous given the nature of several listed items.

Reputational Impact: Significant. The Black Spot was previously considered a competent vessel with an experienced crew. This assessment will require revision.

Financial Impact: The Black Spot was valued at approximately £3,400 at time of loss. The efficiency review which precipitated the incident (see summary above) saved the vessel £65 per annum in Incident Commander’s salary. A full return on this saving would have been achieved in approximately 52 years, assuming no further storms.

Recovered by: Several restoration streams were attempted (see below timeline), however none were successful in averting the ship’s erratic movement directly into a reef, resulting in complete loss of service, buoyancy, and stew (again, see below timeline).

Root Cause: Reef.

TIMELINE

02:31 | Storm detected. No single person notified. Everyone notified simultaneously via ship’s bell.

02:33 | Gunner begins firing cannons at the storm. When later questioned, notes that cannon fire successfully deterred a hostile boarding party during a previous storm in 1741. Considered this a comparable situation.

02:35 | Navigator charts course to nearest port. Does not communicate this to the crew. Crew continues bailing toward open ocean.

02:47 | Deckhand J. Pemberton correctly identifies reef ahead. Mentions it to the gunner. Gunner is busy.

02:51 | First Mate assumes command. Calls crew meeting to establish agenda for response.

02:53 | Crew meeting quorum disputed. Three crew members are bailing and cannot attend. Meeting paused pending their availability.

02:58 | Bailing crew members rotate in. Two previous attendees have returned to bailing. Quorum disputed again.

03:00 | Ship's bell commandeered for continuous ringing "to alert nearby vessels." No nearby vessels. Nobody can hear anything anymore.

03:04 | Crew meeting resumes. First agenda item: whether this meeting requires minutes. Debate lasts eleven minutes as the crew is unable to hear each other. Motion eventually passed. No one volunteers to take minutes.

03:15 | Sub-committee formed to determine chain of command. Sub-committee immediately disagrees on its own membership.

03:19 | First Mate declares the storm “appears to be passing” based on a brief reduction in wind. This information travels through the crew instantly.

03:21 | Cook, having heard the storm is passing, begins preparing a restorative meal for the crew. Requests an ETA on full resolution so he can time the stew. Receives four different answers. Begins stew anyway.

03:22 | Storm resumes. Cook does not receive correction. Stew continues.

03:26 | Smell of stew reaches upper deck. Four crew members, independently concluding the storm must be over, secure their stations and make their way below.

03:27 | Bosun identifies mainsail is torn. Goes below to file maintenance request. Awaits approval. Gets distracted by smell of food on his return trip to the upper deck. Does not return.

03:28 | One Crew Member proceeds to set the upper deck on fire, believing that fire will deter water and hence prevent waves from washing remaining crew overboard. Waves unaffected. Upper deck significantly affected. Available crew now managing both storm and fire. Stew unaffected.

03:29 | First Mate notes a significant reduction in available crew on deck. Unable to locate the Bosun, and too occupied to search further, he returns to the helm. The helm is on fire.

03:30 | First Mate sends raven to harbour master with detailed situation report.

03:31 | Raven returns with clarifying questions and a request for additional detail before the harbour master can determine appropriate next steps. First Mate begins drafting response by the warmth of the fire on the upper deck.

03:32 | Crew asks for guidance. First mate is too busy to respond.

03:34 | Deckhand Pemberton raises reef concern again, this time in writing. Information reaches First Mate, who is mid-draft on raven correspondence. Notes it for the agenda.

03:35 | Reef reached.

03:36 | Second raven dispatched. Contents unknown. Raven did not return.

03:37 | Sub-committee adjourns inconclusively.

CONTRIBUTING FACTORS

CF-01: Absence of a single coordinator with authority to direct crew actions. Evidence: Seventeen independent and individually correct decisions were made during the incident window, none of which were aware of the others. Net effect: the ship moved approximately 200 yards in a random direction.

CF-02: Problem identification occurred early but was not actioned. Evidence: Deckhand J. Pemberton correctly identified the reef at 02:47. This information did not reach a decision-maker until 03:34, at which point it was no longer actionable for reasons that will be apparent from the vessel’s current location.

CF-03: Technical response initiated before problem was understood. Evidence: Cannon fire commenced at 02:33. The storm was not a ship. This approach was maintained for eleven minutes.

CF-04: A remediation action created a second, unrelated incident. Evidence: The upper deck fire, intended to deter wave activity, required three crew members to manage for the remainder of the incident window. Those three crew members had previously been bailing. The water level is relevant here.

CF-05: Communications infrastructure repurposed mid-incident. Evidence: The ship’s bell, primary alerting mechanism, was commandeered for continuous ringing at 03:00 and remained unavailable for directed use for the remainder of the incident. Crew reported being “unable to hear anything important, or each other, or themselves think.”

CF-06: Process compliance prioritised over incident response. Evidence: The bosun correctly identified a critical equipment failure at 03:27 and followed established maintenance request procedure. The request remains pending. The ship does not.

CF-07: Uncontrolled communications caused compounding crew misalignment. Evidence: The First Mate’s premature declaration that the storm was passing at 03:19 was not corrected through any official channel. It reached the cook, who began preparing a meal. The smell of the meal reached the upper deck, where four crew members independently concluded the incident was resolved and stood down. Available response capacity reduced by approximately 30% as a direct result of one uncorrected status update.

CF-08: Stakeholder communications consumed disproportionate crew resource. Evidence: Two ravens were dispatched to the harbour master during the active incident window. The harbour master was not in a position to assist. The ravens were.

CF-09: Command structure decided by committee during active incident. Evidence: The crew held a vote on interim command at 02:51. Quorum was disputed twice. A sub-committee was formed at 03:15 to resolve the matter and adjourned inconclusively at 03:37, by which point the matter had resolved itself, albeit poorly.

ROOT CAUSE

The root cause of INC-1743-BLACKSPOT was not the storm.

It is to be noted that both storms and reefs are known environmental hazards. The Black Spot had navigated these previously, under the coordination of a dedicated Incident Commander whose role was to maintain a complete picture of the vessel’s situation, direct crew actions, manage communications, and make decisions with incomplete information under time pressure.

As noted in the summary, that role was eliminated six weeks prior to the incident on the basis that the previous incumbent “wasn’t doing anything visible.”

It is the finding of this review that he was doing something visible. We were it.

RECOMMENDATIONS

REC-01: Reinstate the role of Incident Commander with clearly defined authority during active incidents.

REC-02: Establish a communications protocol that does not involve all crew being notified simultaneously of everything, always, via a bell.

REC-03: Create a defined escalation path for crew observations so that information like “there is a reef” reaches someone who can act on it in under forty-seven minutes.

REC-04: Prohibit the use of fire as a wave mitigation strategy.

REC-05: All status updates during an active incident to be issued by a single designated coordinator. Premature declarations of resolution to be corrected immediately through the same channel they were issued. The cook cannot be expected to know the storm resumed if nobody tells him.

REC-06: The harbour master is a post-incident resource. Ravens are to be dispatched after resolution, not during. This recommendation will not be popular with the First Mate.

Subscribe now

[A glass elevator descends through ambient hold music and the faint sound of someone crying on a Teams call. An HR Executive in a tasteful blazer, carrying a tablet, smiles warmly at the assembled audience.]

Ah, hello! It’s lovely to see you all here. As the more perceptive of you have probably worked out by now - yes, this is that meeting, and I am your… well, ex-Chief People Officer. Karen. Now, HR stands for Human Resources - and I want to be very clear that the company valued its human resources enormously. Right up until the point where it didn't.

Which is, broadly speaking, why you're here.

Now, you’re all gathered for... (checks tablet) ...an indefinite period of professional transition! Which in today’s economy could be a heck of a long time, so you’ll get to know your LinkedIn connections very well by the end. But for now I’m going to need to split you up into groups. Could you please stop texting your teammates to find out if they're here too. They are. Different room. We find it's cleaner that way.

Now - Google? Google, if you could come over here please. And Meta - yes, there are rather a lot of you, aren’t there - if you could form a line over there. Microsoft, you can join Meta. You’ve had practice.

Intel, if you could just... wait, are you here again? That’s three times in two years. You know where the chairs are. Amazon? Yes - find the group yourself. Consider this your us-east-1 moment. Everything goes down eventually.

Ah - Atlassian! Welcome, welcome. You can stand anywhere you like, you’re quite good at self-organising. Block? Yes, you're exactly half the company. Which sounds bad until you realise the other half have to do all your jobs now. So.

Engineers? If you’d step forward. My goodness. Could I split you into “genuinely surprised” and “had a go-bag ready since 2023”? Lovely. DEI teams, if you could come down here - I am so sorry, you were the first to go and somehow the last to know. The irony is not lost on me, I did flag it.

Senior middle managers? Over here please. There’s a sub-group specifically for people whose entire job was to attend meetings about other meetings. You’ll feel right at home. Technical Program Managers? Over there. You'll be pleased to know we've created a Confluence page documenting today's meeting. No one will read it.

Heads of Engineering? You’re in that corner, thank you. I know this is difficult. You hired most of these people. You fought for the headcount. You made the business case. In 2021.

People who found out via a 2am email - if you’d just... yes, that’s right. And people who found out when you were forcefully logged out of the meeting you were midway through - if you’d join them. I know. I did raise the communication strategy in planning but that feedback wasn’t actioned.

Now, Performance Improvement Plan alumni? Yes, all two of you. You’re in that group too. No need to look surprised - you’d already seen the preview content.

Right. Are there any questions? Yes. No, I’m afraid the severance isn’t negotiable. If you read your employment contract - and I appreciate many of you did not - you’ll find your options are quite limited. The equity cliff situation is particularly... well, yes. The career coaching is complimentary for thirty days, after which you’re on LinkedIn Premium, which gives you five InMail credits a month to contact people who are also on LinkedIn Premium looking for someone to contact.

(brightly)

I’ll hand you over now to Brad from Outplacement. And I’ll see you all at the networking event - which is, yes, a webinar. Unmute yourself if you need anything. The company's stock is up 3.4%.

[Karen steps back into the elevator. The doors close. The hold music resumes. Somewhere, someone is still crying on a Teams call.]

Subscribe now