The Detection-as-Code Pipeline That Actually Ships

Putting your detection rules in Git is the easy part. It is also where most teams stop, and six months later they have a repository that nobody trusts, a SIEM full of rules that do not match the repository, and a quarterly ritual of reconciling the two by hand.

The repository is not the point. The pipeline is the point. Here is one that survives contact with a real SOC.

What the pipeline has to do

Four things, in order:

1. Reject rules that cannot parse. Non-negotiable, instant, free.
2. Reject rules that do not detect what they claim to detect. This is where the work is.
3. Deploy the survivors to the right place, automatically. No console clicking.
4. Tell you when a deployed rule stops working. The part everyone skips.

Stage 4 is what separates a pipeline from a deployment script. A rule that fired 40 times a week and now fires zero is either a fixed environment or a broken detection, and you need to know which within days, not at the next audit.

Repository layout

Flat directories beat clever ones. Detection engineers navigate by technique, not by product.

detections/
  credential-access/
    T1003.001-lsass-handle-access.yml
    T1003.006-dcsync-replication.yml
    T1558.003-kerberoasting-rc4.yml
  execution/
    T1059.001-encoded-powershell-office-parent.yml
  impact/
    T1490-shadow-copy-deletion.yml
tests/
  T1003.001-lsass-handle-access/
    true-positive.evtx.json
    false-positive-defender.evtx.json
  ...
pipelines/
  lint.yml
  test.yml
  deploy.yml
docs/
  ADR-0004-why-sigma-as-source-of-truth.md

Two decisions worth making early, and writing down:

Sigma as the source of truth, backend queries as build artifacts. You write the rule once, sigma convert emits SPL, KQL, and EQL. The alternative - maintaining native queries per platform - is fine until your second SIEM, at which point it is not fine at all. The cost is real: Sigma’s abstraction cannot express everything, and you will keep an escape-hatch/ directory of native rules. Ten percent native is healthy. Sixty percent means Sigma is not buying you anything.

One rule per file, filename carries the technique. Makes git log readable, makes coverage scriptable, makes review diffs small.

Stage 1: lint

Runs in under ten seconds on every push. Anything slower gets bypassed.

name: lint
on: [push, pull_request]

jobs:
  sigma-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - run: pip install sigma-cli pysigma-backend-splunk pysigma-backend-microsoft365defender

      - name: Schema and syntax
        run: sigma check detections/

      - name: House rules
        run: python pipelines/house_rules.py detections/

      - name: Backends can actually compile this
        run: |
          sigma convert -t splunk   -p splunk_windows   detections/ > /dev/null
          sigma convert -t kusto    -p microsoft_xdr    detections/ > /dev/null

sigma check catches malformed YAML and schema violations. house_rules.py catches the things that are legal Sigma but bad detections:

• No rule without a falsepositives block. If you have not thought about false positives, you have not finished the rule.
• No rule without an attack.t* tag. Untagged rules are invisible to coverage reporting.
• No level: critical without a linked response playbook in the description.
• No bare CommandLine|contains on a single common string. contains: 'powershell' is not a detection, it is a log search.
• UUID must be unique across the repository, and must not change once merged. Changing a rule ID orphans every historical alert.

That last one has bitten every team I have worked with. Enforce it in CI:

# pipelines/house_rules.py (excerpt)
def check_stable_ids(repo_root: Path, base_ref: str = "origin/main") -> list[str]:
    """A rule's UUID is its identity. Renaming a file is fine; changing the id is not."""
    errors = []
    for path in (repo_root / "detections").rglob("*.yml"):
        current = yaml.safe_load(path.read_text())
        previous = git_show(base_ref, path)          # None if newly added
        if previous and previous.get("id") != current.get("id"):
            errors.append(
                f"{path}: rule id changed "
                f"{previous.get('id')} -> {current.get('id')}. "
                f"Historical alerts reference the old id."
            )
    return errors

Stage 2: test

This is the stage that makes the whole thing credible, and the stage that almost nobody builds.

A detection unit test is three files: a rule, an event that must match, and an event that must not. Run the rule against both, assert the outcome. No SIEM required - the matching happens locally against the Sigma AST.

# tests/test_detections.py
import pytest
from sigma.collection import SigmaCollection
from pipelines.matcher import matches          # thin wrapper over pySigma evaluation

CASES = discover_cases("tests/")               # (rule_path, event, should_match)

@pytest.mark.parametrize("rule_path,event,should_match", CASES, ids=case_id)
def test_rule_behaviour(rule_path, event, should_match):
    rule = SigmaCollection.load_ruleset([rule_path]).rules[0]
    assert matches(rule, event) is should_match, (
        f"{rule.title}: expected match={should_match} for {event.get('_case_name')}"
    )

Where do the test events come from? Three sources, in descending order of value:

1. Real telemetry from an Atomic Red Team detonation. Run the atomic in a lab, export the resulting events, strip hostnames and usernames, commit as the true positive. This is the gold standard because it is a recording of the actual technique, not your mental model of it.
2. Real telemetry from the false positive that woke someone at 3am. Every tuning exclusion you add should arrive with a test proving it excludes the thing it was meant to exclude - and nothing else.
3. Hand-written events. Fine for edge cases, weak as primary evidence. You will write the event that matches your rule rather than the event the technique produces.

The discipline that matters: every tuning change ships with a test. Someone adds filter_backup_agent to the shadow-copy rule, they also add tests/T1490-shadow-copy-deletion/fp-veeam-prune.json. Six months later when a different engineer widens that filter, the test tells them what they just broke.

Stage 3: deploy

Merge to main converts and pushes. Nothing exotic:

deploy:
  needs: [lint, test]
  if: github.ref == 'refs/heads/main'
  runs-on: ubuntu-latest
  environment: production          # requires approval for the first 90 days
  steps:
    - uses: actions/checkout@v4

    - name: Build backend queries
      run: |
        sigma convert -t splunk -p splunk_windows \
          --output build/splunk/ detections/
        sigma convert -t kusto  -p microsoft_xdr \
          --output build/sentinel/ detections/

    - name: Push to Splunk
      run: python pipelines/push_splunk.py build/splunk/
      env:
        SPLUNK_TOKEN: ${{ secrets.SPLUNK_TOKEN }}

    - name: Push to Sentinel
      run: python pipelines/push_sentinel.py build/sentinel/
      env:
        AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}

Two things worth building into the push scripts:

Deploy new rules in shadow mode first. Enabled, logging matches, generating no tickets. Let it run for a week, look at the volume, then promote. A rule that fires 4,000 times in its first week should never have reached the queue, and you will not know that from a unit test.

Make deletion explicit. A rule removed from the repository should be disabled in the SIEM, not deleted, and the script should refuse to disable more than three rules in one run without a --force flag. Someone will eventually merge a bad rebase that drops half the directory.

Stage 4: rule health

The stage that turns a deployment script into an operational system.

For every deployed rule, track weekly: fire count, true-positive count, median time-to-close, and whether the underlying log source is still arriving. Then alert on the second derivative - not on the numbers, on the changes in the numbers.

-- Rules that have gone quiet: fired reliably for 8 weeks, then stopped.
WITH weekly AS (
  SELECT rule_id,
         date_trunc('week', fired_at) AS wk,
         count(*)                     AS fires
  FROM alerts
  WHERE fired_at > now() - interval '10 weeks'
  GROUP BY 1, 2
),
baseline AS (
  SELECT rule_id,
         avg(fires)  AS avg_fires,
         count(*)    AS active_weeks
  FROM weekly
  WHERE wk < date_trunc('week', now() - interval '1 week')
  GROUP BY 1
)
SELECT b.rule_id, b.avg_fires, coalesce(c.fires, 0) AS last_week
FROM baseline b
LEFT JOIN weekly c
       ON c.rule_id = b.rule_id
      AND c.wk = date_trunc('week', now() - interval '1 week')
WHERE b.active_weeks >= 8
  AND b.avg_fires >= 3
  AND coalesce(c.fires, 0) = 0;

A rule going silent has three causes, and you have to check them in this order:

1. The log source stopped arriving. Sysmon config pushed without the ProcessAccess block, a forwarder died, an agent upgrade changed the channel name. This is the most common cause and the most dangerous, because it silently removes every detection built on that source, not just the one you noticed.
2. The environment genuinely changed. You fixed the thing. Legitimately fewer fires. Close the loop and write it down.
3. The rule broke. A schema change, a field rename, a backend upgrade that altered operator semantics.

Only the second is good news, and it is the least likely.

What this costs

Honest numbers from building this three times:

• Two weeks to get stages 1 and 3 working for a single backend.
• Six to ten weeks to build stage 2 into something with real coverage, mostly spent capturing clean test telemetry.
• Ongoing: roughly a day a week of maintenance for a library of 200–400 rules, most of it triaging stage 4 output.

That last figure is the one to put in front of whoever approves headcount. Detection engineering is not a project that finishes. A rule library is a garden, and stage 4 is the part that tells you which beds have gone to weed.

Start here

If you have nothing today, do not build all four stages. Do this:

1. Move your ten highest-value rules into a repository. Just ten.
2. Add sigma check in CI. One afternoon.
3. Write true-positive tests for those ten, from Atomic Red Team detonations.
4. Automate the deploy for those ten only.
5. Add rule-health tracking before you add rule eleven.

The failure mode is migrating 400 rules into Git with no tests and calling it detection-as-code. That is version-controlled technical debt. Ten rules that are linted, tested, deployed automatically and monitored for decay beats four hundred that are merely stored.