Missing Metrics in Crisis Exercises, Measuring Team Decision Quality

Emergency preparedness has come a long way, thanks in part to structured frameworks like FEMA’s Homeland Security Exercise and Evaluation Program (HSEEP). This framework offers a consistent methodology for planning, conducting, and evaluating exercises. It also standardizes reporting through After Action Reports (AARs) and Improvement Plans (IPs), helping organizations close capability gaps and improve performance over time.

But while HSEEP is excellent at answering the question, "Did we follow the plan?", it falls short in answering a far more critical one: "Did we make the right decisions as a team—when it mattered most?"

HSEEP’s Strengths and Its Blind Spots

The HSEEP doctrine focuses heavily on process:

• Establishing objectives

• Designing realistic scenarios

• Conducting and controlling exercises

• Evaluating outcomes

• Documenting lessons learned

This structured approach has elevated the professionalism and effectiveness of emergency management exercises across public and private sectors. But despite its depth, HSEEP doesn’t define standardized metrics for evaluating decision-making quality, especially at the team level, where the most impactful choices are made in real time.

Why Decision Quality Matters

In real-world emergencies—whether a cyberattack, natural disaster, or critical infrastructure failure, decision-making under pressure is the single most important determinant of outcomes.

A team may follow procedures perfectly but still fail to achieve operational objectives if their decisions are:

• Too slow

• Based on incomplete information

• Disconnected from risk realities

• Poorly communicated

• Not adapted to dynamic conditions

Exercises that don’t measure how decisions were made, only what actions were taken, miss a vital part of performance.

What Would Decision-Making Metrics Look Like?

To improve the quality of team-based crisis exercises, we need core metrics for evaluating team decision-making. These metrics could include:

These indicators would allow evaluators to move beyond binary pass/fail criteria and toward a nuanced understanding of performance under uncertainty.

Why HSEEP Should Evolve Now More Than Ever

We’re at a tipping point. Organizations are starting to adopt AI-driven simulations, virtual tabletop exercises, and digital twin environments for training. These platforms offer unprecedented opportunities for:

• Capturing detailed behavioral data

• Analyzing decision paths

• Replaying scenarios for coaching and feedback

But without a shared language and structure for measuring decision quality, we risk leaving these powerful tools underutilized—or worse, using them to reinforce flawed assumptions.

The shift to virtual and AI-supported preparedness should be matched by a shift in evaluation standards. It's time to integrate decision-making metrics into the exercise lifecycle.

Where Do We Go from Here?

If we want exercises to build true resilience, we need to evaluate not just what teams did, but how they thought.

This means expanding current frameworks like HSEEP to include:

• Decision-quality KPIs

• Behavioral analytics

• Real-time feedback loops

• Post-exercise decision audits

Whether you're designing internal crisis drills, regional preparedness exercises, or cross-sector simulations, start asking:

• How can we define and measure quality decisions?
• How can we coach teams not just to act—but to think better in real time?

It’s a shift in mindset—and one that will separate the truly prepared from the merely practiced.

Want to integrate decision-making metrics into your next crisis exercise?
Learn how our AI-driven tabletop tools work