Your ATT&CK Coverage Map Is Lying to You

Someone senior asked for a number, so you built the Navigator layer. Green where you have a rule, yellow where you sort of do, red where you do not. Seventy-eight percent green. It went in the board pack.

It is not true. Not because anyone lied, but because the thing you counted is not the thing you claimed to measure.

Four ways the map deceives

One rule does not equal one technique

T1059.001 - PowerShell - is a single cell in the matrix. Behind that cell:

• Encoded commands from an Office parent
• Download cradles over HTTP
• In-memory assembly loads with no command line at all
• PowerShell via WMI, so powershell.exe never appears as a child process
• Constrained-language-mode bypasses
• Custom runspace hosts that are not powershell.exe

You have a rule for the first one. The cell is green. You are covered against roughly fifteen percent of what that cell actually represents, and the map has no way to say so.

Meanwhile T1490 - Inhibit System Recovery - is a genuinely small technique. Four or five command-line variants, all high-signal. One good rule gets you most of the way. Same green cell, radically different truth.

The matrix has no depth axis. Every cell is one unit wide regardless of whether it contains a doorway or a continent.

Coverage is asserted, not demonstrated

Walk up to any green cell and ask: when did we last prove this fires?

The honest answer in most estates is “when it was written.” Since then:

• The Sysmon config was updated and dropped Event ID 10
• The log source moved to a new index and the rule still points at the old one
• An EDR upgrade renamed process.parent.name to process.parent.executable
• Someone added a tuning exclusion so broad it excludes the technique

None of these turn the cell red. The rule still exists. It has not fired in eight months and nobody noticed, because “has a rule” and “would detect an attack” are different claims and the map only checks the first.

Detection and alerting get conflated

There are at least five distinct states behind one green cell:

State	What it means	What it is worth
Logged	The telemetry exists somewhere	Forensics only, after the fact
Queryable	A hunter could find it	Depends entirely on someone hunting
Detected	A rule matches	Nothing, if it goes to a dead queue
Alerted	A human sees it	Something, if they know what to do
Responded	A playbook exists and has been rehearsed	The actual goal

Most coverage maps mark green at “Detected.” The distance between Detected and Responded is where breaches live. I have seen a ransomware deployment where the shadow-copy-deletion rule fired correctly, forty minutes before encryption, into a queue with a nine-hour backlog.

It counts what is easy

Process-creation techniques are easy to write rules for, so the Execution and Persistence columns look great. Collection, Exfiltration, and Command & Control are hard - they need network telemetry, decryption, or behavioural baselining - so they look sparse.

The map then drives the roadmap: “we are weak in Exfiltration, let us write more exfil rules.” But the reason those cells are red is usually that you lack the telemetry, not the rules. Writing ten rules against logs you do not collect turns cells green and changes nothing.

What to measure instead

Four metrics. None of them fit in a heat map, which is precisely why they are useful.

1. Validated coverage, with a decay clock

A technique counts as covered only if a detonation has proved it fires - and the proof expires.

technique_status(T) =
    VALIDATED    if last_successful_detonation(T) < 90 days ago
    STALE        if last_successful_detonation(T) < 365 days ago
    UNVALIDATED  if a rule exists but has never been detonated
    GAP          if no rule exists

Run this as a scheduled job against your Atomic Red Team or Caldera results. The first time you run it, expect the green to collapse by half. That collapse is the most valuable thing the metric will ever produce - it is the difference between what you believed and what is true.

Report it as four numbers, never as a percentage:

31 validated · 44 stale · 22 unvalidated · 61 gap

A single percentage invites averaging away the distinction. Four numbers force the conversation about which bucket to move.

2. Telemetry coverage, tracked separately

Before asking “do we have a rule,” ask “do we have the data.” Maintain a log-source inventory mapped to ATT&CK data sources:

Data source	Coverage	Retention	Gap
Process creation (Sysmon 1)	94% of endpoints	90d hot	6% - legacy OT segment
Process access (Sysmon 10)	94%	30d	Short retention blocks hunting
DNS query (Sysmon 22)	41%	30d	Not deployed to servers
Network connection	12%	7d	Sampled NetFlow only
Cloud audit (CloudTrail)	100% orgs	400d	-

That DNS row explains your red C2 column better than any rule-writing sprint will. Fixing telemetry is unglamorous and it is usually the highest-leverage work available.

3. Detection quality per rule

Coverage counts rules. Quality tells you whether they should exist.

For each rule, weekly:

• Precision - true positives ÷ total fires
• Volume - fires per week
• Time to triage - median minutes from fire to analyst disposition
• Last true positive - when this rule last caught something real

A rule at 2% precision firing 300 times a week is worse than no rule. It consumes the analyst attention that the good rules need, and it trains the team to close alerts reflexively. Cutting it raises your real coverage while lowering your map score.

Set a floor and enforce it: any rule below 5% precision for four consecutive weeks goes back to the engineer or gets disabled. No exceptions for rules someone is fond of.

4. Time to detect, measured by detonation

The number that actually answers “are we getting better.”

Run an unannounced detonation monthly. Measure:

• Time from execution to first telemetry
• Time from telemetry to alert fire
• Time from alert fire to analyst acknowledgement

Three numbers, three different owners: platform, detection engineering, SOC operations. When the total gets worse, you know which team’s number moved. That is the entire value - a single MTTD figure tells you something degraded but not where.

What the board pack should say

Not this:

78% ATT&CK coverage, up from 71% last quarter.

This:

We have validated detection for 31 of the 158 techniques we consider relevant to our threat model, with proof from detonations in the last 90 days. A further 44 have rules that were validated over a year ago and need re-testing. Our biggest constraint is not rules - it is DNS and network telemetry, which covers 41% and 12% of the estate. Closing the DNS gap would make 14 currently-unwritable detections possible. Median time from detonation to analyst acknowledgement is 34 minutes, down from 51.

It is longer. It is less comfortable. It tells the reader what to fund.

Keep the map

None of this means throw away Navigator. A heat map is a good communication device and a genuinely useful way to spot structural blind spots - a whole column with nothing in it is worth seeing at a glance.

It is a bad measurement device. The failure is treating the picture as the metric.

Colour the map by validation age rather than rule existence and it starts telling the truth. Cells fade as their proof expires. Watching a board go pale over a quarter is a far more honest artefact than watching it go green.