It's 8 PM on a Friday night, and I've just been paged for an incident affecting our members. Ten engineers join the call, reporting elevated latency, increased errors, and failures across multiple services. Nothing has recently changed, there's no obvious cause, and everyone has a different theory about what's happening.
As the Incident Commander (IC), it's my job to make sense of these competing perspectives, coordinate the response, and set a direction. How do I make a good decision when the information is incomplete, the pressure is mounting, and the right path isn't obvious?
Over the years, I've found myself coming back to six questions and techniques when making these kinds of decisions during incidents:
- How much do we really know?
- What would change our minds?
- Can we undo this decision if we need to?
- What happens if we wait?
- Who makes the decision?
- When do we change course?
How much do we really know?
Certainty is an illusion. Yet when faced with a difficult decision, we often look for it and talk like we have it. Ask someone "Are you sure?" and you'll probably get a yes or no. Ask "How sure are you?" and you'll get a much more useful answer.
Picture yourself as the Incident Commander and a responder tells you:
I think it's the database. We should restart it, and that won't introduce any issues.
Now imagine they say:
- I'm 90% confident the database is the source of the errors.
- I'm 70% confident restarting it will restore service.
- I'm 40% confident restarting it won't introduce additional problems.
That changes the conversation. We're fairly confident about the diagnosis, but much less confident about the proposed fix and its consequences. I can dig a little deeper: "You're 70% confident restarting it will work. What would make you 80% confident?"
That question can uncover what's missing from our understanding and whether there's something worth investigating before we act.
But sometimes the information we need simply isn't available. In How to Make a Big Decision, Adam Savage talks about the relationship between information and decision-making. When we don't have enough information to make the decision in front of us, sometimes we need to reconsider the decision itself.
During an incident, that might mean asking a smaller question. We might not know enough to fix the underlying problem, but do we know enough to reduce its impact?
What would change our minds?
During an incident, decisions aren't isolated events. Our systems are constantly changing, often because of the actions we're taking. What we knew five minutes ago might already be wrong. We need to keep updating our understanding as the incident unfolds.
This approach is captured in the OODA loop; Observe, Orient, Decide, Act, developed by US Air Force Colonel John Boyd. Drawing on his experience as a fighter pilot and military strategist, Boyd described decision-making as a continuous, interconnected process rather than a linear sequence. Importantly, Boyd's OODA loop describes decisions as hypotheses and actions as tests. Speed matters, but so does our ability to learn from each action, challenge our assumptions, and adjust as circumstances change.
In 1989, United Airlines Flight 232 lost all its hydraulic controls after a catastrophic engine failure. The pilots could no longer steer the aircraft normally, and there was no procedure for what to do next. They began experimenting with the remaining engine throttles, observing how the aircraft responded and adjusting their approach. They discovered they could influence the aircraft's direction by varying engine power. After roughly 40 minutes of improvisation, they reached an airport.
The NTSB investigation documents how the crew improvised their way through a situation nobody had prepared them for. What interests me is how they kept learning. Each adjustment gave them information that influenced what they tried next.
During an incident, we should be clear about what we expect to happen when we take action. If we restart a service, what should improve? How quickly? And what would convince us that our original diagnosis was wrong?
AI is making this more interesting. Responders can now spin up multiple agents to investigate different theories, producing more findings, hypotheses, and recommendations than ever before. But ten AI investigations don't necessarily produce a better decision than one. They might contradict each other, repeat the same assumptions, or generate findings that have little bearing on what we should actually do.
As an IC, I don't need every possible explanation. I need to know which findings might change our decision, which assumptions are worth challenging, and what evidence we're still missing. The quality of a decision depends on what we do with the information, not how much of it we collect.
Can we undo this decision if we need to?
Two decisions with similar chances of success can carry very different risks depending on how easily we can undo them. When considering a course of action, I find it useful to ask not just "What could go wrong?" but "How difficult would it be to reverse this decision if we're wrong?"
In Amazon's 2015 shareholder letter, Jeff Bezos described decisions as one-way and two-way doors. Two-way doors are relatively easy to reverse, while one-way doors are difficult, costly, or impossible to undo. Treating every decision like a one-way door can slow us down unnecessarily.
In 1948, the Soviet Union blockaded West Berlin, cutting off road and rail supplies to more than two million people. American leaders considered sending armed convoys through the blockade, but military commanders feared a confrontation with Soviet forces could start another war. Instead, the Western Allies supplied the city by air.
The Berlin Airlift kept the city supplied until the blockade was lifted in May 1949. The operation could be expanded, scaled back, or suspended. An armed confrontation, on the other hand, could have escalated into something impossible to undo.
As an IC, understanding reversibility helps me decide how much scrutiny an action deserves. If we can easily undo something, I'm generally more comfortable moving quickly. If we can't, I want to spend more time understanding the risks.
Of course, I've also made decisions that I thought were reversible, only to discover otherwise. Oops. Restarting a service or reverting a configuration might be easy, but that doesn't mean we can undo the customer impact or other consequences.
While we're considering the risks of acting, there's another risk we often overlook: doing nothing.
What happens if we wait?
When making decisions with incomplete information, it's tempting to keep investigating until we're confident enough to act. The moment when you have that confidence might never come, and meanwhile the situation could be getting worse.
During the Apollo 13 emergency in 1970, NASA shut down the command module to conserve its remaining electrical power. The astronauts needed that power to get home, but restarting the module too early could have left them without enough for reentry. Engineers spent three days developing a new startup procedure, something that would ordinarily have taken three months and the crew returned safely.
I like this example because there must have been enormous pressure to get everything working again. But restarting the command module immediately would have been a terrible idea.
During incidents, I find it useful to set checkpoints for important decisions. Give responders some time to investigate, then ask what we've learned and whether another ten minutes is likely to help. What's happening to our customers while we wait? Are we running out of options?
Sometimes we have good reasons to wait. I just want to make sure we've actually discussed them, rather than finding ourselves another half hour into an incident without having made a decision.
And even when we know it's time to act, there's still the question of who makes the call.
Who makes the decision?
During an incident, the IC is rarely the person with the deepest technical understanding of the systems involved. Nor do they need to be. My job is to ask the right questions, understand the risks, and make sure a decision gets made.
In 2010, 33 miners became trapped underground in Chile. Rescue leader André Sougarret was coordinating engineers, geologists, drilling specialists, and other experts who had never worked together. When geologist Felipe Matthews arrived with new technology for measuring drilling trajectories, his measurements contradicted those of other teams. Rather than dismiss his findings, the team tested the competing methods, discovered Matthews' equipment was more accurate, and put him in charge of measurements for the rescue. All 33 miners were eventually brought to safety.
In a 2010 study published in Science, Anita Woolley and colleagues found that group performance wasn't strongly correlated with the intelligence of its smartest member. Instead, factors like social sensitivity and more equal participation mattered.
You can have a Slack channel full of brilliant engineers and still make terrible decisions if nobody challenges assumptions or draws out different opinions. As an IC, I need to make sure the people with relevant expertise are heard, even when they aren't the most senior or outspoken person in the room. Sometimes I'll make the call, sometimes it belongs to a technical expert. Either way, I want to make sure we're listening to the right people.
The McDonald's method
Sometimes the problem isn't getting people to listen to different opinions. It's getting anyone to offer one in the first place.
Imagine you're with a group trying to decide where to go for lunch. Nobody has any suggestions, so you propose McDonald's. Suddenly everyone has an opinion. Someone wants sushi, someone else suggests Italian, and another person knows a great place around the corner.
Jon Bell described this in 2013 as McDonald's Theory. It's similar to the provocative techniques described in Edward de Bono's work on lateral thinking, where an unexpected suggestion can break people out of their usual patterns of thinking.
I've used a similar approach during incidents when the team is stuck. I'll throw out an option with something like, "Hey, I know this isn't a great idea, but what if we tried...?" Often, that's enough to get people talking. Someone explains why it won't work, someone else suggests an alternative, and suddenly we have a few options worth discussing.
And who knows? Maybe it really is 10-for-1 McFlurry Day, and McDonald's was the best idea all along.
When do we change course?
Making a decision doesn't mean sticking with it forever. But changing course can be difficult, especially when we've already invested time and effort. This is the sunk-cost fallacy, explored by Arkes and Blumer in their 1985 research.
During the 1996 Mount Everest disaster, climbers continued toward the summit long after their planned turnaround time. After months of preparation, considerable expense, and hours of climbing, giving up was a difficult prospect. Several climbers pushed on, and eight people died in the disaster. Researchers have examined how sunk costs, group dynamics, and poor decision-making contributed to what happened.
This is why I like agreeing on abandonment criteria before taking action. During an incident, what do we expect to happen? How long will we give a mitigation? What evidence would convince us to stop?
It's much easier to answer those questions before we've spent an hour convincing ourselves we're on the right path.
Good decisions don't guarantee good outcomes
In Thinking in Bets, Annie Duke discusses the fiending of the 2015 Super Bowl. Seahawks coach Pete Carroll called for a pass instead of a run. The pass was intercepted, and Seattle lost. Despite the outcome, there were reasonable strategic arguments for the decision.
Duke calls judging decisions by their outcomes resulting.
In 1983, Soviet officer Stanislav Petrov was on duty when an early-warning system reported American nuclear missiles heading toward the Soviet Union. Rather than treating the alert as confirmation of an attack, he judged it to be a malfunction. The satellites had mistaken sunlight reflecting off clouds for missile launches.
Petrov couldn't know for certain that the warning was false. But an attack involving just a handful of missiles didn't make sense to him, and he had doubts about the reliability of the warning system. He reported it as a false alarm. He turned out to be right, but he didn't have that certainty when he made the call.
The important question isn't whether Petrov was right. It's whether his judgment was reasonable given what he knew at the time. The evidence he considered matters more than the fact that the alarm turned out to be false.
There's an organizational responsibility here, too. We can't expect responders to make difficult decisions under pressure and then punish them whenever things go wrong. We should hold people accountable for how they make decisions, not simply how those decisions turn out. If someone understood the risks, considered the alternatives, and made a reasonable call with the information available, the organization needs to stand behind them, even when the outcome is bad.
Six questions, not six answers
These six questions won't guarantee the right decision. I've made calls that prolonged incidents, and I'm sure I'll make more that don't turn out as expected. But they've helped me challenge assumptions, understand the risks, draw on the expertise around me, and recognize when it's time to change course.
What I like about these questions is that they don't require me to have all the answers. They help me work out what we know, what we need to learn, and what we can reasonably do next. Sometimes that means acting quickly. Other times it means slowing down, asking another question, or reconsidering a decision we've already made.
Making decisions with incomplete information is unavoidable, especially during incidents. We'll never have perfect information, and even well-considered decisions will sometimes go wrong. The best we can do is make decisions we can explain, remain willing to change our minds, and learn from the results.