Customer portal: L2 support runbook
Last reviewed: 14 March 2017
Check the portal dashboard for error rate and login latency (new link TBC).
Restart the application service on the active node.
Do not restart both nodes at once. The standby takes over the active node’s sessions; if you restart both then every logged-in customer is dropped.
Clear the session cache.
Do not fail over to the DR site.
Check the certificate expiry on the load balancer.
NOTE (2018): do not run the overnight batch before 6am.
If login latency stays above two seconds, disable the retry queue.
That’s the end of it. There’s no last line, only the latest one.
Step three has its reason. Steps five and seven don’t. The page was last reviewed in 2017 and has grown since, from the bottom, one step at a time, each added after something broke.
No runbook ends with “if the above does not work”, because what comes next isn’t a step. The reader gets there anyway, at 3am, one line past the end.
I’ve read a lot of these, most of them written for support teams. I’ve written my own, and cut them down on purpose.
Written well, a “do not” carries its reason, the way step three does. As a runbook grows, they stand between a tired reader and the next step, at an hour when that reader has no time for them. So they come out, and for the reader at 3am that’s the right call. Hollnagel calls this the efficiency-thoroughness trade-off, and argues it’s normal: work usually gives up thoroughness to gain efficiency.
The next reader inherits the step without the reason. Steps five and seven get followed to the letter by people who couldn’t tell you what they prevent. We fill the boxes, and we don’t know why.
At a new company, part of the job is learning which steps to skip and which links go nowhere. The runbook can’t tell you. Someone who’s been there longer can, and it usually starts the same way: “Well, but…”. Every runbook has a spoken edition. That’s where the judgement went.
You can hear a runbook end. At 1:50am the engineer reads out step eight, waits, and says that’s everything in the doc. Logins are still failing, about one in five. Someone suggests running step two again. I ask what changed yesterday. Nobody on the bridge knows. An engineer from the platform team opens the deploy log. A load balancer config change, 4:40 the previous afternoon, from a team that isn’t on the call. It’s rolled back at 2:22, and logins recover by 2:30.
The runbook gets a step nine the following week.
Past the newest entry, the work changes. The questions are where it’s failing, what the failures share, what the symptoms are, and what changed. Usually it’s a recent change or a deploy. The people who know their tools go looking and find it. The people who were handed the senior’s guide stop where the guide stops.
They were given the document. Nobody gave them the rest.
A handover note is the same document, written live. On a long-running incident, someone about to go off shift writes it at 2am, for someone taking over a thing that’s still moving. It carries the status, the actions in flight, the owners and the time of the next update. It’s written in a hurry, by someone who’s been awake too long.
Done that way, it carries the facts and nothing else, and the next shift spends its whole shift doing what the last one did, without knowing it. The same problems get solved again. Nothing moves. In incident management it goes this way more often than not.
I handed over at 6:00am. The note was good: impact, the three workstreams, who owned each, next update at 7:00. At 11:40 I rejoined the bridge and heard the day shift paging the storage vendor. We’d paged them at midnight and ruled storage out by one. The note said “storage: no fault found”. It didn’t say how sure we were, or who had proved it. They spent the morning proving it again.
The facts made it across. The rest didn’t:
Which streams are weak, and why.
Who’s tired.
Which people to protect, because they’re carrying the investigation.
Whose analysis is helping, and whose is getting in the way.
A newer commander takes over the facts and works from them, because the facts are what a handover looks like. The nuance that moves things forward doesn’t make the trip. A veteran re-learns it on shift, if they’re veteran enough.
There’s a case for the loss. The incoming commander also arrives without the outgoing one’s fixation. Patterson, who studied handovers in shuttle mission control and dispatch rooms, notes that bringing someone new in is one way to break a fixation. A note with only the facts loses both. It can’t tell the judgement worth keeping from the fixation worth losing.
Some of it can’t be written. Polanyi’s starting point, in 1966, was that we can know more than we can tell. Klein’s fire commander is the case that gets retold. His crew’s water did nothing to what looked like a kitchen fire, he had a bad feeling he couldn’t account for, and he pulled them out. The floor gave way into a basement fire, in a house he hadn’t known had a basement. He put it down to ESP. There is no step for that.
Some of it shouldn’t be written. Who’s tired is true tonight and unfair in a record that keeps. So are the frank, human things: what the commander needs to watch for, and who needs a closer eye. None of it would survive a HR meeting, and it shouldn’t have to. Judgement about people goes off in a day. A document keeps.
And some of it wouldn’t be read. I always know more than I write down. If I wrote it all, it would be a five-hundred-page tome, the first five pages read a hundred times and the rest read once. It’s the trade that took the reasons out of steps five and seven.
Each of these is a good reason, and the people writing make the call on purpose. The gaps in the document are decisions, and most of them are right. If a runbook had a last line, it would be a name: whoever knows what the steps don’t.
So the rest goes out loud. The handovers that work have three parts: a note in the channel, a talk-through, and the incoming commander talking it back. The note carries the facts. The talk-through carries what the note can’t: the responders who need oversight, the wrong stream people keep drifting back to, the executive nudging the incident the wrong way. The talk-back is how you know it landed.
Then everyone on the bridge hears who has command.
At 6:40am, fourteen hours in, the night commander handed over. Her note was a page: impact, the three streams, who owned each, next update at 7:00. Then she talked me through it. Storage was closed: the vendor’s engineer had proved it at one, and reopening it would waste the morning. Network was the weak stream, on its third theory. Leon had been up since the previous afternoon and was the only one getting anywhere on the database, so keep him off the exec call. The ops director would push to fail over at eight. Don’t. I read it all back to her, and she stayed on the bridge until 7:15. Nobody paged the storage vendor. The database fix went in at 9:20.
The industries that can’t afford a bad handover settled this a long time ago. In Patterson’s study, the outgoing person kept responsibility until the handover was complete and stayed for anything that broke during it. At the ambulance desk, you hadn’t taken over until your headphones were plugged into the console.
The runbook’s “Well, but…” is the same thing, slower: judgement handed over by voice, one new starter at a time.
My name is at the bottom of several runbooks, at companies I’ve left. That’s a runbook doing its job: the process outlives its author.
What I knew and didn’t write down left when I did.