Jamie Turner

2 months ago

What to Do When You Mess Up Prod (Incident Response Guide)

If you've got a product people care about, you're going to have incidents. No amount of testing or carefulness stops something from eventually breaking in production. We've both spent a lot of our careers on the other side of that moment: on call, staring at a dashboard, trying to figure out why something we shipped is now on fire. This is episode three of the second season of Databased. The theme is how to be on call, how to build an incident response process, and how to become the kind of operationally mature company that handles it well. It's not documented well anywhere. Most of what's written is either too abstract to use at 4 a.m. or written for a company ten times our size.

The boring stuff first

We'll start with the least important part of incident response, which also happens to be the part everyone thinks about first: SEV levels, on-call roles, status pages, postmortems. It's worth leveling-set on the vocabulary, but this is the most straightforward part of the whole topic, so we'll move quickly.

At larger companies, an incident gets a SEV level, short for severity, and the level is what tells the rest of the company how much to care without anyone having to have that conversation live. Here's the rough breakdown:

A SEV isn't there to make anyone feel better or worse about what happened. It's there to trigger a response that matches how bad things actually are, which is why a severity level that doesn't trigger any real action might as well not exist.

One rule of thumb that scales down to smaller teams: SEV 2 means someone should look into this, SEV 1 means the team should stop and look into this, and SEV 0 means the company needs to look into this. If you're four people just starting a company and something breaks, don't waste time agreeing on a SEV level. Just fix it. There's no point performing ceremony you don't need yet. The value shows up later, once you're big enough that you need a signal to tell everyone how much to care.

Out of the SEV process come two roles that matter more than the level itself. There's the tech lead on call, which we'll call the T-lock, and the incident manager on call, the IMO for most of this conversation (sometimes shorthanded IMOC, worth flagging since both terms point at the same role). At a larger organization these are two different people, and for good reason. When something's down, two things have to happen at once. Someone has to fix it, and someone has to manage communication. Try to do both with one person and it breaks down. Talking to the CEO or to support pulls the fixer off the technical problem, and miscommunication follows. The T-lock owns making sure the problem gets fixed. The IMO owns messaging to the rest of the org and to customers, and protects the T-lock's time and attention. The higher the stakes, the more stakeholders start asking for updates and piling on stress. At Convex, we're often on the annoying end of that dynamic. We're both engineers and like to think we're reasonable, but there's a point in a real incident where someone has to tell us to go away, and we have to listen. The IMO has to be technical enough to know when and how to interrupt the T-lock without being an annoying manager about it, while fielding everything coming from outside the room.

The last piece of the mechanics is the status page, which you keep updated, and knowing who's on call in the first place, since it's surprisingly common to lose a few minutes of an incident just figuring that out. That's the process. What we want to spend the rest of this on is the why. How to think about the art of communication and leadership during an incident. That's the part that determines whether people trust you afterward.

The five stages of incident response

Once something breaks, there's a rough order of operations: detection, triage, stopping the bleeding, debugging, and remediation. Incidents are stressful because you're doing the rest of your job when something interrupts you at the worst possible time. The temptation is to jump straight into reading source code and fixing the underlying problem. That's usually the wrong move. Think of it as a journey of discovery. Stop the impact first, take away the urgency, then fix what's broken. Skipping steps to save time is the classic way an incident ends up taking way longer than it needed to, because you double back once you realize you fixed the wrong thing.

The five stages of incident response: detection, triage, stopping the bleeding, debugging, and remediation, in sequence

1. Detection

Detection is where an incident starts. Metrics and logs do different jobs here, and it's worth keeping them separate in your head. Metrics are time series. They're good for triage and detection, and they're the thing you put alerts on. Logs (stack traces, exceptions) are for debugging. They're high-volume, often delayed by a minute or more, and hard to alert on cleanly. That's part of why streaming logs to a dedicated destination matters once you outgrow the built-in dashboard logs view. The signals for "something is wrong" should be small in number and close to the user experience, not a zoomed-in system metric. A service can look perfectly healthy on an internal metric while being completely unreachable. We've seen systems that never paged because their error graph read zero, purely because requests weren't arriving at all. Monitoring availability from the client side, not just the server side, catches that blind spot.

Across a lot of systems and a lot of years, the pattern holds. A company ends up leaning on maybe three core alerts that really matter, even with far more metrics instrumented than that. Everything else is useful for triage and debugging, not for waking someone up. If your alert channel is firing constantly, the alerts become background noise and get ignored exactly when you need them.

This is the idea we think of as the power of zero. Complex systems tend to run with some low background level of errors, say fourteen a second, that everyone gets used to and stops really looking at. Then a jump to seventeen sparks a debate about whether it means anything. The fix is to design your topline metric so that when nothing is wrong, the number is genuinely zero. Any movement off zero is real signal. If there's a known, hard-to-fix source of noise, like encoding errors caused by client behavior you don't control, categorize it out so the main metric can sit at zero. We've both led expensive, multi-quarter efforts to drive a category of error to zero: filesystem errors in one case, desktop client errors in another. The payoff wasn't just fewer errors, it was speed. Once you know Tuesday was clean, a new error on Wednesday tells you exactly when something happened. Without that baseline, every fluctuation costs you time proving whether it's real.

The same idea extends to data integrity. If you're running a database or storage system, a meaningful fraction of your workload should be background validation jobs scanning for corruption. The power of zero is worthless if you only discover the corruption months after it happened. Good systems engineering here means having enough confidence that yesterday was fine and today isn't, so a new problem announces itself instead of hiding in a slowly rotting number. If there are a few known-bad, ancient items you're never going to clean up, quarantine them explicitly and reset the metric around them. That way new noise doesn't hide behind old noise you've already accepted.

2. Triage

Triage means figuring out what broke and where it's coming from. It's a genuinely hard skill, because it tempts engineers toward heroics. Clicking seven dashboards deep and hunting through logs at 4 a.m. doesn't work well while stressed and half-awake. That's why it's worth investing ahead of time in a small number of dashboards that point straight at the likely source of a problem: which service is throwing errors, has load increased, is one customer doing something unusual, did the network drop, is the database slow. Somewhere around six graphs is enough to spot the issue quickly. More than that and you're back to hunting.

Two things actively hurt triage. First, someone loudly proposing a fix ("we could build a new system") before anyone knows what's wrong. Engineers naturally jump to solutions, and that jump derails the process of figuring out what happened. Second, blame. A zero-blame culture is operationally necessary here, not a nicety. If there's any fear of being blamed or disciplined for causing an incident, people stop calling out what they're seeing, and anything that discourages someone from raising a hand makes the whole team worse at responding. Between the two of us, across a lot of incidents, we can't think of a single time someone was disciplined for causing one. If your SEV designation is doing its job, the incident room stays a zone of safety to find the issue. Any "why did this happen" conversation gets written down for later, not raised in the moment.

We're cautious about leaning on AI heavily during triage. Some tools genuinely help. At the end of the day, though, you have to find and fix the real problem yourself, and you can't blame a model for getting it wrong. AI also tends to be unusually confident in its assertions during triage, which is the opposite of what the process needs. Human triage often works because someone notices a metric that's a little high, which nudges someone else to go look at something adjacent, and those small observations compound. AI wants to resolve the ambiguity and declare an answer. Accept that too quickly and you've effectively skipped triage, moving on before you know what's wrong.

Simplicity matters here too, more than it gets credit for. Complex systems are hard to triage almost by definition, and it's possible to build something so complicated that no one can reason about it during an incident. A useful design question, well before anything breaks: can someone else understand this system at 4 a.m., freshly woken up, well enough to fix it?

3. Stop the bleeding

Stopping the bleeding is distinct from debugging, and it's worth treating as its own step. You can often make a problem's impact go away quickly without understanding the root cause yet. If a system is corrupting data, turn it off. If a database is overloaded, shut down background load or pause a migration. If a service is optional, have it return a safe fallback. This can happen in parallel with debugging. Once the impact stops, everyone can breathe, and the team can work the real problem without the pressure of live customer impact.

This is where preparation pays off directly. In any large system there's background load (test traffic, benchmarking jobs, remediation scripts) that's a good candidate to shut off first. The value is in having a script or a button ready ahead of time, rather than trying to figure out how to stop a table compaction job under duress. At Convex, we have tools and playbooks that let the team run a single command to pause all background work and take load off the system, buying time and clarity to find the real problem. Segmenting live-site-critical services from things you can safely pause is a design decision worth making before you're in an incident, not during one.

4. Debugging

Once you've triaged and stopped the bleeding, debugging is regular engineering again. If triage narrowed the scope, debugging is a familiar kind of problem to work. It only gets brutal when you try to solve it without knowing what "it" actually is. Simple systems are easier to debug, and the engineers who deeply understand the system's structure are the ones who move fastest here. Part of the T-lock's job is making sure someone actually fixes it, or handing off explicitly to someone who will. Ambiguity about who's in charge means everyone acts in parallel and nobody's coordinated.

After triage, if the cause isn't environmental, AWS, DNS, or a new exploit, look hard at what changed recently. In our experience, something that changed in the last week or two is the cause roughly 80% of the time.

5. Remediation

Remediation is the "how do we prevent this from happening again" step, and it's where you have to be careful not to make things worse. Both debugging and remediation are technical work with no real shortcuts. A lot of what makes them tractable happens beforehand: tested backups, the ability to pause non-critical load, disaster recovery drills that actually exercise restoring from backup. Decide ahead of time, as a team, how you'll handle the awkward cases, so you're not arguing about it live during an outage. The same goes for compliance. Build in an exemption path for incident circumstances, with proper documentation, instead of arguing with your CISO while the company is down. Plan what happens when the VPN doesn't work, and know who has root on your infrastructure. The answer to all of this is practice. Companies that go through incidents get better at responding to them, so it's worth deliberately building that muscle, including through simulated incidents, without over-relying on the same one or two people who happen to be great at it.

Live communication is the multiplier

During an incident, this isn't the moment for email, and it might not even be the moment for Slack. Get on a call or into a room. There's real value in live communication for the response team specifically, not the whole company, just the people who are relevant. Anyone who isn't needed in the room should leave. The person who cracks an incident is often a junior engineer with less context, who hasn't made the same assumptions everyone else has and notices a spike in a log nobody else is watching. That kind of observation surfaces on a live call in a way it doesn't in a Slack thread. If someone says something like "huh, that's weird," the T-lock's job is to immediately ask what they saw, because that offhand noise is often exactly where the answer is hiding.

The culture in the room matters as much as the process. Incidents can go two ways. They can become sources of guilt, shame, and fear, none of which help anyone resolve anything faster. Or they can become gallows humor, joking, and the camaraderie of solving a hard problem together under pressure. Some of our best career memories come from exactly these rooms. During a massive outage at Dropbox, one engineer joked that it was a good thing their resume wasn't stored in Dropbox, and it landed because that same engineer was heads-down grinding on the fix. As long as people are working hard, a little levity is fine, even useful. Save the judgment for the postmortem, not mid-incident.

None of this means unlimited endurance is the goal. We've both been in incidents that ran multiple days, including stretches of 36-plus hours with no real break. It can happen, but it shouldn't happen often, and sustaining the team matters. If your best debugger has been at it for twelve hours, tell them to sleep, even if they argue. You want gas left in the tank for the next one.

Talking to customers without lying to them

Once you have a product people rely on, you have to tell them when something's wrong, and there's real nuance in how. The difference between a customer thinking "they've got this" and a customer thinking "I'm moving off this platform" often comes down entirely to communication, not to how bad the incident actually was.

The instinct to declare victory early is strong, a lot like the pressure in sales, because reassuring customers feels good in the moment. Resist it. The first message should just be "we're having an incident," sent as early as possible, without speculating about the cause before you know it. Overstate your confidence and you'll have to walk it back later, which costs you more trust than the delay would have. Customers need to know that you know something's wrong, that you're working on it, and that you have some sense of how long it might take, without overpromising on that last part.

The technical team and the communications side should run on different personas during the incident. The technical side stays focused on solving the problem, not on managing feelings. The person handling communication has to be the empathetic one, thinking about what the customer, or the customer's customer, actually needs to hear. That's rarely technical detail. Telling someone "the service hit a file descriptor limit" doesn't help them. Telling them whether they need to take action right now, and when to expect resolution if not, does. Run about ten to twenty minutes behind the war room in both specificity and confidence. Early conclusions in the room often change, and you don't want to have told customers something you then have to retract.

One of us got burned by this directly. During a big outage at Dropbox, someone DM'd asking if things were fixed, and the answer was "my filesystem is running." True personally, but not true company-wide, and it read as an official all-clear it was never meant to be. Speaking for yourself during an incident can easily be heard as speaking for the company. The better move is hedged language that's still useful: "we believe the issue is resolved, and we're monitoring for regressions," rather than declaring it closed before you're sure. Incidents are one of the clearest moments a customer gets to judge whether they trust you, and overpromising during one is a fast way to lose a customer who'd otherwise have stuck around.

Postmortems, and where this leaves off

The last phase is the postmortem, and it's easy for this to become a checkbox exercise. The real point of a postmortem is reflection that prevents the same thing from happening again, not a punitive record. Making a team that's already struggling spend half its time writing up a detailed timeline instead of fixing things is its own kind of waste. A good postmortem answers three things: what happened, why it happened, and what changes as a result. If there's genuinely nothing meaningful to say, it doesn't need to be long.

Postmortems are the close of the loop, not a formality bolted onto the end of it. The bigger throughline across all five stages is the same one that opened the episode. The goal was never to prevent every incident, since that's not achievable. It's to get good enough at responding to one that customers come away trusting you more, not less.