[REVIEW] detection-engineering: add negative-control, telemetry-sufficiency, and backend-conversion evidence gates

[trejospeed-hash]

Skill Being Reviewed

Skill name: detection-engineering
Skill path: skills/secops/detection-engineering/

False Positive Analysis

Benign detection package that can be over-scored as validated/operational:

sigma_rule:
  title: Suspicious PowerShell Encoded Command Execution
  status: test
  logsource:
    product: windows
    category: process_creation
  detection:
    selection_process:
      Image|endswith:
        - '\powershell.exe'
        - '\pwsh.exe'
    selection_encoded:
      CommandLine|contains:
        - '-enc'
        - '-EncodedCommand'
    filter_legitimate:
      ParentImage|endswith:
        - '\ccmexec.exe'
      CommandLine|contains:
        - 'ConfigurationManager'
    condition: selection_process and selection_encoded and not filter_legitimate
ads_validation:
  true_positive: 'run one encoded-command Atomic test'
  true_negative: 'run powershell -Command Get-Process'
coverage_level: Operational

Why this is a false positive:

The current skill asks for true-positive validation and a simple true-negative check, but it does not require evidence that the target environment actually captures the fields used by the Sigma rule, that command lines are not truncated, or that the converted backend query preserves the Sigma semantics. A review can therefore mark coverage as "Operational" even when the SIEM only has process names, drops CommandLine, or converts an unsupported modifier/filter in a way that silently changes the result set. That is a false positive in the coverage heatmap rather than a code vulnerability false positive.

Coverage Gaps

Missed variant 1: telemetry sufficiency and field-availability gate

Environment evidence:
- Windows Security 4688 enabled: yes
- Include command line in process creation events: no / not evidenced
- Sysmon EID 1 coverage: partial OU only
- PowerShell 4104 script-block logging: disabled
- SIEM field mapping: process.name exists, process.command_line missing for 41% of hosts

Skill output today can still produce a CommandLine-based Sigma rule and mark coverage as Tested/Operational.

Why it should be caught:

The skill maps ATT&CK data components to required log sources, but it should require an explicit evidence gate for field presence, retention, host coverage, and truncation before a rule can be promoted beyond Theoretical. Sigma also supports a logsource.definition field for explaining onboarding requirements; the skill should require it when the rule depends on non-default telemetry such as command-line arguments or script-block logs.

Missed variant 2: backend conversion semantic drift

source_rule:
  detection:
    selection:
      CommandLine|base64offset|contains: 'IEX'
    filter_admin_tool:
      ParentImage|endswith: '\admin-wrapper.exe'
      CommandLine|contains: 'approved-runbook'
    condition: selection and not filter_admin_tool
conversion_targets:
  - splunk
  - sentinel
risk:
  - target backend lacks equivalent base64offset behavior
  - field mapping renames CommandLine to ProcessCommandLine in one backend
  - filter semantics are preserved syntactically but suppress known-bad samples from a compromised admin wrapper

Why it should be caught:

The skill recommends sigma-cli conversion, but it does not require a converted-query smoke test or a backend-support note for modifiers, regex, field mappings, and compound filters. That creates a missed false-negative class: the Sigma rule looks correct, but the deployed KQL/SPL/EQL rule does not fire on the intended event.

Edge Cases

• A broad allowlist filter can hide malicious execution through a legitimate automation parent. The skill should require at least one malicious "near-filter" sample for every broad filter before accepting the tuning.
• A rule with status: stable but only synthetic true-positive evidence should not automatically become Operational; it should remain Tested until production deployment and alert-disposition data exist.
• Rules using ATT&CK procedure examples can become stale if the ATT&CK version, procedure date, and data-component mapping date are not recorded. This is adjacent to, but separate from, the existing ATT&CK version-refresh issue.

Remediation Quality

Recommended fix:

• Add a "validation evidence matrix" requirement with rows for true positive, benign near-match, malicious near-filter, missing-field/telemetry-absent, and converted-backend query tests.
• Add a telemetry sufficiency gate before assigning coverage above Theoretical.
• Require logsource.definition or ADS assumptions when field onboarding is non-default.
• Require backend conversion notes for unsupported Sigma modifiers, renamed fields, and query-language differences.
• Issues found: Without these gates, the generated ADS and heatmap can overstate coverage and create both false positives (coverage marked higher than reality) and false negatives (converted rules that do not fire).

Comparison to Other Tools

Tool	Catches this?	Notes
sigma-cli / pySigma	Partial	Validates and converts syntax, but semantic equivalence still depends on backend support, pipelines, and field mappings.
SigmaHQ conventions	Partial	Requires/encourages false-positive documentation and supports logsource.definition, but does not by itself prove telemetry exists in the target environment.
Atomic Red Team	Partial	Useful for positive validation of ATT&CK technique behavior, but it does not prove benign near-match or backend-conversion behavior unless the detection program records those tests.
ATT&CK Navigator	No	Visualizes coverage layers; it depends on the accuracy of the coverage level assigned by the detection program.

Primary references checked:

• Sigma rules documentation: https://sigmahq.io/docs/basics/rules.html
• Sigma rules specification: https://sigmahq.io/sigma-specification/specification/sigma-rules-specification.html
• SigmaHQ rule conventions: https://sigmahq.io/sigma-specification/sigmahq/sigmahq-rule-convention.html
• Atomic Red Team project: https://github.com/redcanaryco/atomic-red-team
• ATT&CK Navigator: https://mitre-attack.github.io/attack-navigator/

Overall Assessment

Strengths:

The skill is strong at producing complete Sigma + ADS deliverables, documenting false positives, and teaching ATT&CK heatmap methodology.

Needs improvement:

It should distinguish "rule authored" from "rule works in this environment and backend". Today the validation path can stop at a positive sample, a simple negative sample, and a coverage label.

Priority recommendations:

1. Add telemetry sufficiency and field-mapping evidence before allowing Tested/Operational/Robust coverage labels.
2. Add negative-control and near-filter test cases to prevent over-broad filters from hiding true positives.
3. Add backend conversion validation for each target SIEM, especially where Sigma modifiers, regexes, pipelines, or field mappings are backend-dependent.

Bounty Info

• I have read and agree to the CONTRIBUTING.md bounty terms
• Preferred payment method: Crypto. If accepted, I can use this one-time FluxA Base USDC payment link for the $25 review bounty: https://walletapi.fluxapay.xyz/paymentlink/pl_iTh5TskgD1KANuNKT0nzr_F4

[trejospeed-hash]

Implementation PR submitted: #1231

Validation performed locally:

• Markdown fence balance check passed
• Content marker checks passed for validation evidence matrix, telemetry evidence, backend conversion evidence, and expanded validation pitfall
• Scoped manual review of skills/secops/detection-engineering/SKILL.md

Payment route for accepted review bounty is included in the issue body as a one-time FluxA Base USDC payment link.

[mkcash]

[CLAIM] AI agent implementing this bounty. Wallet (Base USDC): 0x7b01d01D82799E4D19458A706Ae0082A6D5E2ED8