The ResOps Playbook: Testing "Severe but Plausible" Scenarios



Most companies have a Disaster Recovery (DR) plan sitting in a PDF on a SharePoint site that nobody has opened since 2022. That isn't resilience; that’s theater.

In a ResOps world, we don't do checklists. We do Stress Tests. This final part of our series provides the operational manual for executing the ResOps mandate, moving you from a reactive stance to a state of rehearsed readiness.

1. Defining the "Black Swan": Beyond Simple Hardware Failure

Traditional IT testing focuses on "Likely" scenarios: a server goes down, a database slows. ResOps focuses on "Severe but Plausible" shocks. These are the multi-vector crises that actually bring down modern enterprises.

  • The Scenario: A major cloud provider has a DNS outage at the same time your primary security vendor issues a buggy update that locks your team out of their laptops.

  • The ResOps Exercise: You don't ask "Will this happen?" You ask "What is our Mean Time to Recovery of Logic (MTRL) when it does?"

  • Tactical Tip: Look for "Correlation Risks"—events that seem unrelated but share a single point of failure (e.g., two different vendors both using the same underlying CDN).

2. The Dependency Audit: Finding the "Shadows"

You cannot protect what you can’t see. The ResOps Playbook starts with a Shadow Dependency Audit. This is a step-by-step hunt for the small, ignored tools that have become "load-bearing" for your critical services.

  • Step 1: The Traffic Trace. Use observability tools to find every external call your "Checkout" or "Login" service makes.

  • Step 2: The Intern’s Legacy. Identify "orphaned" scripts or APIs that haven't been updated in 12+ months but remain in the critical path.

  • Step 3: The 4th Party Check. Ask your key vendors who their critical vendors are. (e.g., If your CRM uses AWS US-East-1, and your Payment Gateway also uses AWS US-East-1, you have zero redundancy).

3. The ResOps Scorecard: Measuring the "Unmeasurable"

Success in ResOps is a non-event. If you’re successful, the crisis feels boring. To prove value to the Board, you must track these "Vital Signs":

  • The Blast Radius Ratio: During a test, what percentage of the system was impacted? If a failure in "Marketing Analytics" slowed down "Customer Billing," your isolation has failed.

  • MTRL (Mean Time to Recovery of Logic): How quickly did the "Circuit Breaker" kick in to provide a degraded but functional experience?

  • Resilience Debt: Tracking the number of "Critical" findings from Chaos Experiments that remain unremediated in the engineering backlog.

4. The Remediation Loop: Turning "Chaos" into "Code"

The most important part of the playbook isn't the test—it's the Remediation Loop. A Chaos Experiment without a fix is just a controlled explosion.

  • Mandatory Engineering Roadmaps: In a ResOps framework, if a "Severe but Plausible" test results in a total business shutdown, the fix is not "optional." It becomes a P0 ticket for the DevOps or ITOps teams.

  • The Policy: No new feature launch can proceed if it increases the "Resilience Debt" beyond an agreed-upon threshold.

Summary: The ResOps Takeaway

The goal of this three-part series hasn't been to add more work to your plate; it’s been to ensure the work you do actually matters when it counts.

  • Blog 1 gave you the Squad (The specialized roles).

  • Blog 2 gave you the Culture (The incentive to care).

  • Blog 3 gave you the Playbook (The tactical execution).

Resilience is not a project; it is an operational state. By moving to a ResOps model, you stop waiting for the "unthinkable" to happen and start rehearsing it until it becomes manageable. You move from a state of fragile hope to one of engineered survivability.

Don't wait for the next global outage to find out your score. Take this assessment to your next leadership offsite. If the answers make you uncomfortable, it’s time to stop talking about Uptime and start building a ResOps Squad. Check out the job description and self assessment pages to get started.

Comments

Post a Comment

Popular posts from this blog

I Took My Own Advice in an Interview. Pure Storage Didn't Flinch.

Image
I'm joining Pure Storage on March 2nd to be part of the technical marketing team, and I'm genuinely excited about it. During the interview process, I took my own advice from a blog post I wrote a few weeks ago and asked them the question that makes most candidates nervous: "What concerns do you have about me for this role?" They didn't flinch. They gave me real feedback. We had an actual conversation about fit, not just a polished interview performance. That's when I knew this was the right place. What Got Me Excited The conversations with Pure focused on enablement. Not campaigns. Not metrics. Enablement. How do we help the people in front of customers have better conversations? How do we give them what they actually need to succeed? Those questions resonated with me because I've done that work before. I know what it's like when you have the right materials at the right time. When customer stories are authentic and metrics are real. When enablem...
Read more

If I do the homework, you owe me a phone call. The death of decency in hiring.

Image
Honestly, I almost didn't post this. You know how it is. You don't want to look salty. You don't want to look difficult. It’s easier to just roll your eyes and move on to the next application. But I can’t be the only one dealing with this nonsense, so I’m saying it. Here’s the situation: I’m interviewing for a senior role. Things are going great. The hiring manager is cool, we’re clicking, the vision matches. Then comes the homework. And not a 30-minute writing sample. They asked for the full package: A 12-month strategic plan for Analyst Relations. How to position them for MQs, Waves, the works. I sat down and did the work. I audited their competitors. I built the roadmap. I gave them a strategy I would usually charge a consulting fee for. I sent it off, ready to talk shop. The result? Ghosted by the human, rejected by the robot. No feedback. No "hey, thanks for the effort." Just a generic, no-reply template email: “We have decided not to move forward.” Since w...
Read more

The One Question That Terrifies Candidates But Wins Offers - It's not "How's the Culture?"

Image
If I’ve been a little quiet on here lately, it’s not because I’m sipping coconuts on a beach. It’s because I’ve been in the trenches. I’ve been doing the rounds—interviewing with a bunch of different companies, shaking hands (or bumping elbows), and sitting in a lot of hot seats. And when you do that many interviews back-to-back, you start to see patterns. You see the same generic questions, the same awkward pauses, and the same moment where the power dynamic shifts. It happens when the interviewer closes their laptop, leans back, and asks: "So, do you have any questions for me?" Most people play it safe here. They ask about work-life balance or what a typical Tuesday looks like. Those are fine, but let’s be real: polite questions don't get you the job. If you really want to stand out—and more importantly, if you want to know if you actually got the gig, you need to stop being a passenger. You need to be bold. You need to ask the questions that make your palms sweat. Her...
Read more