Offerings
Field notes from the server room
Every engineering organisation I have worked inside has had its rituals. Not the formal ones - not the standup, the retro, the quarterly planning offsite - but the older kind. The kind steeped in superstition. The kind that predates silicon and will outlive it. The kind a visiting anthropologist would recognise immediately, even if the practitioners would object to the comparison. We do not deploy on Fridays. We do not touch the billing service at end of month. We do not rename the queue. We do not utter the word "quiet" when on-call. We do not speak honestly about the bad region. Each of these is an offering. Somewhere in the organisation's past, the system became angry. We have agreed, collectively and without quite saying so, on the offerings that keep it from becoming angry again.
The original event has usually been forgotten. The offering persists.
The Friday Deploy
The most universal of the rituals, observed in some form by every engineering organisation that has lived long enough to have a folklore. The rule states that production deployments shall not occur on Fridays, and shall certainly not occur on Friday afternoons. The justification, if pressed, is operational - engineers do not like cleaning up messes on the weekend, nor do they enjoy the surprise of a Monday fiasco once everyone logs on to find things irrevocably broken. The justification is accurate. It is also incomplete. The deeper truth is that engineers have learned, over years and across companies, that Fridays are when systems choose to fail. Not because Fridays are technically different from Thursdays. Because the cost of a Friday failure is uniquely punishing, and the system, in its particular cruelty, seems to know this. The rule remains. It is enforced by engineers who have personally cleaned up a Friday deploy, who carry the specific memory of a weekend lost to a rollback or a Monday morning spent explaining what went wrong. They enforce the rule against the new engineers who have not yet had their own Friday, who will have one eventually, and who will then enforce the rule the same way against the next cohort. The folklore renews itself.
The Billing Sabbath
The billing service is sacred. It is not to be modified during the closing days of the month, when invoices are generated, totals are reconciled, and the organisation's revenue passes through a series of pipelines nobody fully understands. The freeze is enforced. There is a change window, a CAB approval requirement, a deployment lockout. The taboo underneath the policy is older. The freeze existed before the change window did, and the change window was written to formalise what the engineers had already agreed to without writing anything down. The new engineer who proposes a billing-related change in the last week of the month learns quickly that they have proposed something else. The response is not a polite redirect. It is a small ripple of alarm in the channel - senior engineers materialising to ask what is being proposed, why it is being proposed now, whether the proposer is aware of what week it is. The policy has a date. The taboo has none.
The Queue That Cannot Be Renamed
Somewhere in the architecture diagram is a queue with a name that no longer matches what it does. It was named for a feature that has been deprecated, a service that has been retired, or a use case that has shifted three times since the original implementation. Its current purpose has nothing to do with its name, and both facts are accepted as part of the system. Everyone agrees the name is wrong. Nobody renames it. The folklore whispers that once upon a time, a rename was attempted. The story is rarely told in full. The cost of renaming is judged higher than the cost of explaining the discrepancy to every new engineer for the rest of the system's life. The explanation becomes part of onboarding. Six months later, the new engineer will be the one explaining it to someone newer. The wrongness, eventually, becomes a feature.
The Word Not Spoken
The on-call engineer does not say it is quiet. Even when it is quiet. Especially when it is quiet, because that is when the system is listening. The rule is enforced not by management but by the other engineers in the channel, who will respond to any utterance of the word with a chorus of warnings and the horror reserved for genuine taboo. The reasoning, when articulated, is statistical. The word does not summon incidents. Incidents simply tend to follow it. The distinction is technically important and operationally meaningless. The reasoning, when not articulated, is older. It is the same reasoning that prevents sailors from naming storms while still at sea.
The Deprecation That Will Not Complete
The service was deprecated in 2022. The retirement date was set for Q3 2023. In Q2 2023 the date was moved to Q1 2024. In Q4 2023 the date was moved to Q3 2024. In Q3 2024 the date was moved to Q2 2025. The service is still running. It will, almost certainly, still be running next year, and the year after. Each delay is justified by a specific dependency unmigrated, a customer unnotified, a downstream system unmodified. None of these justifications are wrong. All of them, taken together, describe an organisation that has agreed to keep a thing alive while pretending to be killing it. The deprecation is no longer a project. It is a posture.
The Black Sheep Region
Every cloud-scale system has a region with a reputation. The reputation is real. It is also, officially, not real - the formal position is that all regions are equal, and the architecture documentation describes a uniform deployment topology that does not distinguish between them. The engineers who actually operate the system know better. There is a region where things simply go wrong. The other regions should be over-provisioned to compensate. Sometimes the over-provisioning has been quietly skipped, and the gap is treated as a manageable risk that has not yet manifested. Runbooks have specific procedures for when this region fails, written in a tone that suggests the failure is anticipated rather than handled. Capacity planning treats the region's reliability as a known quantity, and the known quantity is "lower." None of this is in the architecture document. All of it is in the operational reality. The engineers who run the system have built an entire infrastructure of compensations around a region they continue to officially treat as equal to its peers, and new engineers learn the compensations through exposure, not through documentation.
These rituals are not a problem. The problem is that the institution cannot admit the rituals exist. The architecture document describes a uniform topology. The roadmap describes a deprecation that will complete. The change-management policy describes a freeze that has reasons. None of these documents acknowledge the superstitions underneath them. The rituals persist anyway, transmitted through warning glances and onboarding asides and the kind of hushed corrections that new engineers learn to recognise as folklore.
We are not a rational industry. We are a tribe with rituals, and we have agreed not to call them rituals because we are an industry that does not believe in superstition.
The original event has been forgotten. The offerings persist.