Checkov tool omits Azure Pipelines results

[ncook-hxgn]

When I use the Checkov tool to scan a cloned Azure DevOps Git repo with an azure-pipelines.yml, I can see CKV_AZUREPIPELINES_ results, but when I use MicrosoftSecurityDevOps@1 in a pipeline for that very same repo, scanning the whole repo, I don't get these CKV_AZUREPIPELINES_ errors.

Don't get me wrong but, this seems like a missed easy win?
Thus, I'm not sure if this is a bug with my implementation or a missing feature. As such here are some details on my implementation, please let me know if I'm missing something obvious.

Here's my .gdcnconfig, derived from https://github.com/microsoft/security-devops-azdevops/wiki#checkov-gdnconfig-sample:

{
  "tools": [
    {
      "tool": {
        "name": "checkov",
        "version": "Latest"
      },
      "arguments": {
        "Directory": "$(Checkov.DefaultTargetDirectory)",
        "Help": false,
        "Version": false,
        "OutputType": "sarif",
        "List": false,
        "Quiet": false,
        "Compact": true,
        "RunAllExternalChecks": false,
        "Soft": false,
        "ShowConfig": false,
        "CreateBaseline": false,
        "OutputBaselineAsSkipped": false,
        "NoFailOnCrash": false,
        "EnableSecretScanAllFiles": true
      },
      "outputExtension": "sarif",
      "successfulExitCodes": [
        0
      ],
      "errorExitCodes": {
        "1": "Checkov tool found issues.",
        "2": "An error has occurred running the Checkov tool."
      },
      "outputPaths": []
    }
  ]
}

And the pipeline task:

  - task: MicrosoftSecurityDevOps@1
    env:
      GDN_RUN_WORKINGDIRECTORY: $(Build.SourcesDirectory)
    inputs:
      break: true
      publish: false

I do get the other results from Checkov, although not always on par with calling standalone Checkov itself.

There seems to be some behind-the-scenes magic at work. For instance, when using MicrosoftSecurityDevOps@1 I found CKV_AZURE_177 downgraded to a warning, where it's an error when I call checkov standalone to perform what I think is the same scan.

Here's how I've been performing a manual scan with Checkov, that seems to look deeper and be more strict than the task's call does:

checkov -d "$(Build.SourcesDirectory)" --repo-root-for-plan-enrichment "$(Build.SourcesDirectory)" --deep-analysis  -o cli -o sarif --output-file-path console,$results

[ncook-hxgn]

.. OK fine, I was using TargetDirectory. When I used Directory, MSDO returns the following:

##[error]Error running tool 3 of 3: checkov
##[error]UnrecognizedArgumentException: Unrecognized argument key: Directory
##[error]BreakException: Guardian detected one or more breaking results.

So, the documented example on the wiki seems broken.

[DimaBir]

Hi @ncook-hxgn — thanks for the detailed report and the side-by-side comparison, that's very helpful.

A few things worth checking before we dig deeper:

1. Argument key name. In your .gdnconfig the Checkov argument is named "Directory", but the wrapper's schema expects "TargetDirectory" (it translates to --directory). When the key doesn't match, the value is silently ignored and Guardian falls back to its default working directory. The wiki sample uses TargetDirectory, and the same gotcha came up in Help me with the example of using environment variables with values for checkov #121. Could you rename it and re-run?

2. The actual Checkov invocation. In the pipeline logs, look for the line that starts with …/Microsoft.Guardian.CheckovRedist_…/tools/dist/checkov …. That line shows the exact CLI Guardian is executing — bundled version, directory, frameworks, soft-fail, etc. Sharing it (and the resulting .gdn/.r/checkov/001/checkov.sarif) would let us tell quickly whether azure_pipelines framework results are being produced and filtered out downstream, or simply never produced.

3. --deep-analysis / --repo-root-for-plan-enrichment. These flags are not currently surfaced by the wrapper's argument schema, so the wrapped invocation won't pass them. They primarily affect Terraform plan enrichment rather than the CKV_AZUREPIPELINES_* rule family, so I wouldn't expect them to explain the missing Azure Pipelines results — but worth flagging.

4. CKV_AZURE_177 severity. The task applies Guardian's severity mapping on top of Checkov's output, which can diverge from Checkov's defaults. That's by design rather than a bug, but if you have a specific case where the downgrade hides something material, please call that out separately.

If you can share (a) the log line above, (b) the SARIF the task produces, and (c) the result of changing Directory → TargetDirectory, that should be enough to confirm whether this is a config issue, a wrapper issue, or a Guardian-side filter. Thanks!

[DimaBir]

You're right, and apologies for the confused direction — I had your .gdnconfig and the wiki sample mixed up in my head. To confirm what you've established:

• TargetDirectory is correct (matches samples/configs/checkov.gdnconfig in this repo).
• Directory is rejected by the wrapper with UnrecognizedArgumentException — that's the error you just hit.
• The wiki Home page Checkov sample is wrong: the JSON snippet uses "Directory": "" while the env-var table immediately below maps Directory → GDN_CHECKOV_TARGETDIRECTORY. The JSON snippet is the broken half — it should read "TargetDirectory". I'll get a wiki fix in for that. Thanks for catching it.

Now, back to the original CKV_AZUREPIPELINES_* question — since your real config (with TargetDirectory) does scan and does return Checkov results, the most useful artifact would still be:

1. The …/Microsoft.Guardian.CheckovRedist_…/tools/dist/checkov … invocation line from your task logs (so we can see the bundled version and the exact flags Guardian passed).
2. The resulting SARIF at .gdn/.r/checkov/001/checkov.sarif (or at least its runs[0].tool.driver.rules[*].id list, filtered to CKV_AZUREPIPELINES_*) so we can tell whether the rules were executed but filtered out downstream vs never executed at all.

That'll let me say definitively whether this is a framework/version gap in the bundled Checkov, a Guardian post-processing filter, or something else. Sorry again for the detour.

[DimaBir]

Quick follow-up: the wiki is updated and live — the Checkov gdnconfig sample and the env-var table now both say TargetDirectory. Thanks again for catching that.

The CKV_AZUREPIPELINES_* question is still open — when you have a chance, the Guardian invocation log line and .gdn/.r/checkov/001/checkov.sarif from a real run would let me give you a definitive answer.

[ncook-hxgn]

1., 2. So, I found a combination of what I had working before + what I saw in the wiki example that surfaces* the CKV_AZUREPIPELINES_* errors - I changed from $(Checkov.DefaultTargetDirectory) to $(WorkingDirectory) and returned to TargetDirectory rather than the documented Directory from the sample.

* I can see them in the cli output from Checkov as called by guardian,

but they aren't showing up in the SAST Scans tab collapse for checkov. They are present in the Checkov collapse from my standalone call though.. (I can't get good screen shots of the omissions from the SAST Scans tab as I have quite a lot of results)..

Standalone call's sarif in Scans tab:

MicrosoftSecurityDevOps@1's results in Scans tab:

So, whilst the following .gdnconfig gets Checkov to find my azure-pipelines.yml and checkov does discover and record pipeline issues to CLI, I can't see them in the final sarif.

4. for 44 issue from standalone vs 29 from MSDO, error degredation to warning etc. Fair enough, that makes sense. I guess it is this severity mapping by Guardian that may be excluding the CKV_AZUREPIPELINES_* results from the final sarif that is rendered by the Scans tab?

    {
      "tool": {
        "name": "checkov",
        "version": "Latest"
      },
      "arguments": {
        "TargetDirectory": "$(WorkingDirectory)",
        "Help": false,
        "Version": false,
        "OutputType": "sarif",
        "List": false,
        "Quiet": false,
        "Compact": true,
        "RunAllExternalChecks": false,
        "Soft": false,
        "ShowConfig": false,
        "CreateBaseline": false,
        "OutputBaselineAsSkipped": false,
        "NoFailOnCrash": false,
        "EnableSecretScanAllFiles": true
      },
      "outputExtension": "sarif",
      "successfulExitCodes": [
        0
      ],
      "errorExitCodes": {
        "1": "Checkov tool found issues.",
        "2": "An error has occurred running the Checkov tool."
      },
      "outputPaths": []
    }

[ncook-hxgn]

And here is the requested guardian call for checkov, output from pipeline log:

 Running Checkov 3.2.497
    ------------------------------------------------------------------------------
    /home/vsts/work/_msdo/packages/nuget/Microsoft.Guardian.CheckovRedist_linux_amd64.3.2.497/tools/dist/checkov --directory /home/vsts/work/1/s --output sarif --quiet --enable-secret-scan-all-files --output-file-path /home/vsts/work/1/s/.gdn/.r/checkov/001/checkov.sarif
    
    
           _               _
       ___| |__   ___  ___| | _______   __
      / __| '_ \ / _ \/ __| |/ / _ \ \ / /
     | (__| | | |  __/ (__|   < (_) \ V /
      \___|_| |_|\___|\___|_|\_\___/ \_/
    
    By Prisma Cloud | version: 3.2.497 
    Update available 3.2.497 -> 3.2.528
    Run pip3 install -U checkov to update

[ncook-hxgn]

(or at least its runs[0].tool.driver.rules[].id list, filtered to CKV_AZUREPIPELINES_)

Let me see if I can do some jq stuff in the pipeline to get this info, hang tight :)

[ncook-hxgn]

Here's the output from the query of the checkov sarif file.

...
2026-05-13T12:31:04.9150438Z ##[command]jq '[.runs[0].tool.driver.rules[].id]' /home/vsts/work/1/s/.gdn/.r/checkov/001/checkov.sarif/results_sarif.sarif
2026-05-13T12:31:04.9184658Z [
2026-05-13T12:31:04.9186897Z   "CKV_AZURE_212",
2026-05-13T12:31:04.9187422Z   "CKV_AZURE_225",
2026-05-13T12:31:04.9187855Z   "CKV_AZURE_59",
2026-05-13T12:31:04.9188294Z   "CKV_AZURE_33",
2026-05-13T12:31:04.9188718Z   "CKV_AZURE_44",
2026-05-13T12:31:04.9189128Z   "CKV_AZURE_35",
2026-05-13T12:31:04.9189556Z   "CKV_AZURE_190",
2026-05-13T12:31:04.9189965Z   "CKV_AZURE_206",
2026-05-13T12:31:04.9190660Z   "CKV_AZURE_72",
2026-05-13T12:31:04.9191225Z   "CKV_AZURE_221",
2026-05-13T12:31:04.9191640Z   "CKV_AZURE_178",
2026-05-13T12:31:04.9192047Z   "CKV_AZURE_49",
2026-05-13T12:31:04.9192441Z   "CKV_AZURE_97",
2026-05-13T12:31:04.9192823Z   "CKV_AZURE_149",
2026-05-13T12:31:04.9193456Z   "CKV_AZURE_42",
2026-05-13T12:31:04.9193863Z   "CKV_AZURE_189",
2026-05-13T12:31:04.9194254Z   "CKV_AZURE_110",
2026-05-13T12:31:04.9194691Z   "CKV_AZURE_109",
2026-05-13T12:31:04.9195357Z   "CKV_AZURE_177",
2026-05-13T12:31:04.9195849Z   "CKV2_AZURE_33",
2026-05-13T12:31:04.9196360Z   "CKV2_AZURE_1",
2026-05-13T12:31:04.9196991Z   "CKV2_AZURE_38",
2026-05-13T12:31:04.9197435Z   "CKV2_AZURE_32",
2026-05-13T12:31:04.9197888Z   "CKV2_AZURE_40",
2026-05-13T12:31:04.9198324Z   "CKV2_AZURE_47",
2026-05-13T12:31:04.9198763Z   "CKV2_AZURE_41",
2026-05-13T12:31:04.9199196Z   "CKV2_AZURE_21",
2026-05-13T12:31:04.9199630Z   "CKV_SECRET_6",
2026-05-13T12:31:04.9200059Z   "CKV_AZUREPIPELINES_1",
2026-05-13T12:31:04.9200487Z   "CKV_AZUREPIPELINES_2"
2026-05-13T12:31:04.9200886Z ]
...

Sorry about the wonky filename, I've been playing with outputPaths in the .gdnconfigs

      "outputPaths": ["$(WorkingDirectory)"]

I digress. I also managed to extract the ruleids from the msdo.sarif emitted by the task - the CKV_AZUREPIPELINES_1 and CKV_AZUREPIPELINES_2 are present..

jq '[.runs[] | select(.tool.driver.name == "checkov") | .results[].ruleId]' /home/vsts/work/1/a/.gdn/msdo.sarif
[
  "CKV_AZUREPIPELINES_1",
  "CKV_AZUREPIPELINES_2",
  "CKV_AZURE_59",
  "CKV_AZURE_33",
  "CKV_AZURE_190",
  "CKV_AZURE_206",
  "CKV_AZURE_42",
  "CKV_AZURE_189",
  "CKV_AZURE_110",
  "CKV_AZURE_109",
  "CKV2_AZURE_33",
  "CKV2_AZURE_32",
  "CKV2_AZURE_40",
  "CKV2_AZURE_47",
  "CKV2_AZURE_41",
  "CKV2_AZURE_1",
  "CKV2_AZURE_21",
  "CKV_SECRET_6",
  "CKV_AZURE_177",
  "CKV_AZURE_97",
  "CKV_SECRET_6",
  "CKV_AZURE_178",
  "CKV_AZURE_49",
  "CKV_AZURE_97",
  "CKV_AZURE_149",
  "CKV_SECRET_6",
  "CKV_AZURE_212",
  "CKV_AZURE_225",
  "CKV_AZURE_59",
  "CKV_AZURE_33",
  "CKV_AZURE_44",
  "CKV_AZURE_35",
  "CKV_AZURE_190",
  "CKV_AZURE_206",
  "CKV2_AZURE_33",
  "CKV2_AZURE_38",
  "CKV2_AZURE_40",
  "CKV2_AZURE_47",
  "CKV2_AZURE_41",
  "CKV2_AZURE_1",
  "CKV_AZURE_221",
  "CKV_AZURE_221",
  "CKV_AZURE_72",
  "CKV_AZURE_221"
]

but yet not visible in the scans tab. That's super weird, I'm not sure how to explain that. Somehow the checkov-results.sarif file from standalone checkov calls, when uploaded to CodeAnalysisLogs, definitely shows the CSKV_AZUREPIPELINES_* stuff in the SARIF Scans tab.

[DimaBir]

Thanks for the thorough data — this changes the diagnosis.

Both queries confirm CKV_AZUREPIPELINES_1 and CKV_AZUREPIPELINES_2 do survive into msdo.sarif. So the drop isn't happening in Guardian / MSDO's SARIF processing — those findings are in the published CodeAnalysisLogs artifact. The disappearance is in the rendering layer reading msdo.sarif (the SARIF Scans Tab extension).

Two structural hypotheses, in priority order:

1. Orphaned results[].ruleId references. SARIF allows results[].ruleId to refer to a rule that isn't defined in runs[*].tool.driver.rules[]. Most SARIF viewers — including the Scans Tab — silently skip results whose ruleId doesn't resolve to a rule definition. Your jq queried results[].ruleId in msdo.sarif but did not query tool.driver.rules[].id. If MSDO's SARIF merge keeps the results but loses some rule definitions, AZUREPIPELINES findings would be present-but-invisible.

2. Multi-run SARIF, viewer reading only one run. Your msdo.sarif jq used .runs[] (plural). The duplicate ruleIds in your dump (CKV_AZURE_221 × 3, CKV_SECRET_6 × 3) are a strong signal that MSDO is concatenating multiple Checkov runs into one file. If the Scans Tab renders only runs[0] (or one run per tool name), and the AZUREPIPELINES results live in a non-first run, they'd be invisible regardless of correctness.

A single jq query disambiguates these. Could you run this against msdo.sarif (not the raw Checkov one) and share the output?

jq '[.runs[] | select(.tool.driver.name == "checkov") | {
  rule_ids: [.tool.driver.rules[].id],
  result_ids_unique: ([.results[].ruleId] | unique)
}]' /home/vsts/work/1/a/.gdn/msdo.sarif

For each Checkov run in msdo.sarif it returns the rules defined in that run and the unique ruleIds the results reference. Interpretation:

• If CKV_AZUREPIPELINES_* appears in result_ids_unique for some run but not in that same run's rule_ids → hypothesis 1 (orphaned references).
• If they appear in both for one run and are absent from other runs → hypothesis 2 (multi-run shape mismatch).

Either way, the fix would be on our side (or on the Scans Tab extension's side); your config is doing what it should. Thanks again for the careful reproduction — this is rapidly converging on a precise root cause.