Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 149 additions & 0 deletions .github/workflows/action-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
name: Action Test

# Validates the composite GitHub Action (install → scan → exit-code/outputs) on
# real runners, lints all workflow files, and exercises the new `github` output
# format with a binary built from this branch. The run-action job installs a
# released binary via the action, pointing release-repo at the current repo so it
# works both pre- and post-org-transfer.
on:
push:
branches: [main]
paths:
- 'action.yml'
- '.github/workflows/action-test.yml'
- 'internal/output/github/**'
pull_request:
paths:
- 'action.yml'
- '.github/workflows/action-test.yml'
- 'internal/output/github/**'
workflow_dispatch:

permissions:
contents: read

jobs:
actionlint:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
with:
persist-credentials: false
- uses: actions/setup-go@44694675825211faa026b3c33043df3e48a5fa00 # v6.0.0
Comment thread
coderabbitai[bot] marked this conversation as resolved.
with:
go-version: '1.25.10'
- name: Install actionlint
run: go install github.com/rhysd/actionlint/cmd/actionlint@v1.7.12
# shellcheck is preinstalled on ubuntu runners, so actionlint also lints
# the run: scripts inside the workflows.
- name: Run actionlint
run: |
GOBIN="$(go env GOPATH)/bin"
"$GOBIN/actionlint" -color

run-action:
name: run-action (${{ matrix.os }})
needs: actionlint
runs-on: ${{ matrix.os }}
permissions:
contents: read
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
steps:
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
with:
persist-credentials: false

# Build the fixture at runtime so no secret-shaped literal is ever committed
# (which would trip push protection / secret scanning). Reassembled at
# runtime, the canonical AWS docs example key triggers the
# aws-access-key-id detector.
- name: Create fixtures
shell: bash
run: |
set -euo pipefail
mkdir -p _lwtest _lwclean
printf 'AWS_ACCESS_KEY_ID=%s%s\n' 'AKIA' 'IOSFODNN7EXAMPLE' > _lwtest/leak.env
echo 'just some harmless text, no secrets here' > _lwclean/ok.txt

- name: Scan dirty fixture (expect a finding)
id: detect
uses: ./
with:
scan-type: fs
path: _lwtest
format: sarif
no-verify: 'true'
fail-on-findings: 'false'
# Download from the repo running this workflow (works pre- and
# post-org-transfer); consumers use the default HodeTech/Leakwatch.
release-repo: ${{ github.repository }}

- name: Assert a finding was reported
shell: bash
env:
COUNT: ${{ steps.detect.outputs.findings-count }}
SARIF: ${{ steps.detect.outputs.sarif-file }}
run: |
set -euo pipefail
echo "findings-count=$COUNT sarif-file=$SARIF"
if [ "$COUNT" != "1" ]; then
echo "::error::expected findings-count=1, got '$COUNT'"; exit 1
fi
if [ -z "$SARIF" ] || [ ! -f "$SARIF" ]; then
echo "::error::expected SARIF file at '$SARIF'"; exit 1
fi
echo "OK: finding detected and SARIF written"

- name: Scan clean fixture (expect no findings)
id: clean
uses: ./
with:
scan-type: fs
path: _lwclean
format: table
no-verify: 'true'
fail-on-findings: 'true'
release-repo: ${{ github.repository }}

- name: Assert no findings reported
shell: bash
env:
COUNT: ${{ steps.clean.outputs.findings-count }}
run: |
set -euo pipefail
[ "$COUNT" = "0" ] || { echo "::error::expected findings-count=0, got '$COUNT'"; exit 1; }
echo "OK: clean directory reported no findings"

# The released binary the action installs predates the `github` output format,
# so exercise that format with a binary built from this branch.
cli-github-format:
name: cli-github-format
needs: actionlint
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
with:
persist-credentials: false
- uses: actions/setup-go@44694675825211faa026b3c33043df3e48a5fa00 # v6.0.0
with:
go-version: '1.25.10'
- name: Build leakwatch from this branch
run: go build -o "${RUNNER_TEMP}/leakwatch" .
- name: github format emits inline annotations
shell: bash
run: |
set -uo pipefail
mkdir -p _fx
printf 'AWS_ACCESS_KEY_ID=%s%s\n' 'AKIA' 'IOSFODNN7EXAMPLE' > _fx/leak.env
out="$("${RUNNER_TEMP}/leakwatch" scan fs _fx --format github --no-verify 2>/dev/null)"
echo "$out"
echo "$out" | grep -q '^::error .*aws-access-key-id' \
|| { echo "::error::expected an ::error annotation for aws-access-key-id"; exit 1; }
echo "OK: --format github emitted an inline annotation"
23 changes: 16 additions & 7 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,20 @@ on:
pull_request:
branches: [main]

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
go-version: ['1.25.10']
steps:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
with:
persist-credentials: false
- uses: actions/setup-go@44694675825211faa026b3c33043df3e48a5fa00 # v6.0.0
Comment thread
coderabbitai[bot] marked this conversation as resolved.
with:
go-version: ${{ matrix.go-version }}
cache: true
Expand All @@ -23,7 +28,7 @@ jobs:
run: |
COVERAGE=$(go tool cover -func=coverage.out | grep total | awk '{print $3}' | tr -d '%')
echo "Total coverage: ${COVERAGE}%"
if [ $(echo "$COVERAGE < 70" | bc -l) -eq 1 ]; then
if [ "$(echo "$COVERAGE < 70" | bc -l)" -eq 1 ]; then
echo "::error::Coverage ${COVERAGE}% is below 70% threshold"
exit 1
fi
Expand All @@ -32,8 +37,10 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
with:
persist-credentials: false
- uses: actions/setup-go@44694675825211faa026b3c33043df3e48a5fa00 # v6.0.0
with:
go-version: '1.25.10'
- run: go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.11.4
Expand All @@ -42,8 +49,10 @@ jobs:
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
- uses: actions/checkout@1af3b93b6815bc44a9784bd300feb67ff0d1eeb3 # v6.0.0
with:
persist-credentials: false
- uses: actions/setup-go@44694675825211faa026b3c33043df3e48a5fa00 # v6.0.0
with:
go-version: '1.25.10'
- run: go install golang.org/x/vuln/cmd/govulncheck@latest
Expand Down
28 changes: 28 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,31 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
HOMEBREW_TAP_TOKEN: ${{ secrets.HOMEBREW_TAP_TOKEN }}

# Move the floating major tag (e.g. v1) to this release so consumers can
# pin `uses: HodeTech/Leakwatch@v1` and always get the latest v1.x. Skipped
# for pre-releases (tags containing a hyphen, e.g. v1.5.0-rc.1). Uses the
# REST API via gh so it works with the persist-credentials: false checkout.
- name: Update major version tag
if: ${{ !contains(github.ref_name, '-') }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TAG: ${{ github.ref_name }}
REPO: ${{ github.repository }}
SHA: ${{ github.sha }}
run: |
set -euo pipefail
# Only move the major tag for a proper vMAJOR.MINOR.PATCH release tag, so
# a malformed tag (e.g. vnext.1) can never force-move an unrelated ref.
if ! printf '%s' "$TAG" | grep -Eq '^v[0-9]+\.[0-9]+\.[0-9]+$'; then
echo "Tag '$TAG' is not a vX.Y.Z release tag; skipping major-tag move."
exit 0
fi
MAJOR="${TAG%%.*}" # v1.5.0 -> v1
echo "Pointing ${MAJOR} at ${SHA} (${TAG})"
if gh api "repos/${REPO}/git/refs/tags/${MAJOR}" >/dev/null 2>&1; then
gh api -X PATCH "repos/${REPO}/git/refs/tags/${MAJOR}" -f sha="${SHA}" -F force=true >/dev/null
else
gh api -X POST "repos/${REPO}/git/refs" -f ref="refs/tags/${MAJOR}" -f sha="${SHA}" >/dev/null
fi
echo "Major tag ${MAJOR} now points at ${TAG}."
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
## [Unreleased]

### Added
- **GitHub Action is now Marketplace-ready and installs a prebuilt binary** — the action metadata moved from `action/action.yml` to the repository root `action.yml` so it can be published to the GitHub Marketplace and consumed as `uses: HodeTech/Leakwatch@v1`. Instead of compiling from source with `go install` on every run, the action now downloads the platform's prebuilt release archive and verifies its SHA-256 checksum before running (Linux and macOS runners). New inputs: `output`, `remediation`, `config`, `scan-diff`, `extra-args`, `working-directory`, `release-repo`. Composite `outputs` now declare `value:` mappings, so `findings-count` and `sarif-file` are actually exposed to downstream steps (previously always empty). The download is checksum-verified and retried; `extra-args` rejects (by prefix, so combined shorthand like `-fcsv` is caught too) flags the action manages (`--format`/`--output`/`--config`/`--show-raw`); the assembled command is not echoed (path/extra-args may carry credentials); `scan-diff` is validated and `auto` degrades to a full scan with a warning when the base commit is absent (e.g. a shallow checkout) instead of hard-failing; and the `github` format always writes annotations to stdout even if an output file is configured. The nested `upload-sarif` and all CI workflow actions are SHA-pinned.
- **Pull-request diff scanning in the Action** — for `git` scans, `scan-diff: auto` (default) limits the scan to commits introduced by the event (`pull_request` base..HEAD or `push` before..HEAD) via `--since-commit`, so CI surfaces only newly added secrets. Requires `actions/checkout` with `fetch-depth: 0`.
- **GitHub Actions job summary** — the action writes a findings summary (counts and a per-finding table parsed from SARIF) to `$GITHUB_STEP_SUMMARY`.
- **`github` output format** — `--format github` emits GitHub Actions workflow commands (`::error`/`::warning`/`::notice`) so findings appear as inline annotations on pull requests. The raw secret is never emitted (redacted only), and command data/properties are percent-escaped. New `internal/output/github` formatter with full unit-test coverage.
- **Floating major version tag** — releases now move the `vN` tag (e.g. `v1`) to the latest `vN.x.y` so consumers can pin `uses: HodeTech/Leakwatch@v1`. Pre-releases (tags containing `-`) are skipped.
- **Action self-test workflow** — `.github/workflows/action-test.yml` runs the composite action against fixtures on Linux and macOS and lints all workflows with `actionlint` (which also shellchecks the `run:` scripts).
- **Custom rules are now loaded from `.leakwatch.yaml`** — the documented `custom-rules:` block is finally wired into the scan. Previously `custom.RegisterCustomRules` existed and was tested but never called, so user-defined detectors were silently ignored. Registration is duplicate-safe: a rule whose ID collides with a built-in detector (or another custom rule) is skipped with a warning instead of panicking. (Resolves ROADMAP "Known Gaps" P0 #1.)
- **Inline ignore (`# leakwatch:ignore` / `# leakwatch:ignore:<detector-id>`) is now honored** — the marker is checked on each finding's source line during scanning and ignored findings are dropped before verification, so they never trigger a network call. Repeated occurrences of the same secret are resolved to their own lines, so an ignore on one copy never suppresses a genuine leak elsewhere in the file. The library helpers existed but were never invoked by the engine. (Resolves ROADMAP "Known Gaps" P0 #2.)
- **Line numbers are now reported for findings** — the engine computes the 1-based line of each match per occurrence (from its byte offset within the chunk). Previously every finding reported `line: 0` in JSON/SARIF/CSV/table output, and repeated matches of the same bytes would all have collapsed onto the first occurrence's line. This is also the prerequisite that makes inline ignore correct.
Expand All @@ -17,6 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
- **SARIF results carry location-independent `partialFingerprints`** — GitHub Code Scanning tracks an alert across line moves instead of closing and reopening it. (Important now that findings report real line numbers instead of `line: 0`.)

### Changed
- **CI coverage-gate script quoting** — quoted the `bc` command substitution in the coverage check (`ci.yml`) so the new `actionlint`/shellcheck job passes (SC2046).
- **Config validation hardening** — `output.severity-threshold` is validated against the known severity set (a typo no longer silently falls back to "low"); a unit-less `verification.timeout` (e.g. `30`, which YAML decodes as 30 nanoseconds) is rejected with a hint to use a unit; a disabled `verification:` block no longer fails validation on leftover non-positive values; nested config keys are now overridable via environment variables (e.g. `LEAKWATCH_OUTPUT_SEVERITY_THRESHOLD`).
- **`detector.RegisterIfAbsent`** — new atomic check-and-insert used by custom-rule registration to avoid a check-then-register race and the panic on duplicate IDs.
- **Finding IDs include the line number** — disambiguates two findings that share the same redacted value in the same file (e.g. two private keys with identical redaction on different lines).
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ leakwatch/
│ ├── config/ # Viper-based configuration
│ └── filter/ # .leakwatchignore, inline ignore
├── pkg/ # Public packages (finding model)
├── action/ # GitHub Action definition
├── action.yml # GitHub Action definition (Marketplace, repo root)
├── Formula/ # Homebrew formula
├── Dockerfile # Multi-stage Docker build
├── docs/ # Documentation
Expand All @@ -71,6 +71,7 @@ Architecture decisions are documented in ADR format under `docs/decisions/`. The
| [ADR-0006](docs/decisions/ADR-0006-container-library.md) | go-containerregistry | Daemonless, layer-based analysis |
| [ADR-0007](docs/decisions/ADR-0007-license.md) | MIT | Enterprise adoption, open-core compatibility |
| [ADR-0008](docs/decisions/ADR-0008-concurrency-model.md) | Worker Pool | Fixed worker count, channel-based |
| [ADR-0009](docs/decisions/ADR-0009-github-marketplace-action.md) | Marketplace Action | Root `action.yml`, prebuilt-binary composite, `@v1` |

## Coding Standards

Expand Down
27 changes: 16 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Go Report Card](https://goreportcard.com/badge/github.com/HodeTech/leakwatch)](https://goreportcard.com/report/github.com/HodeTech/leakwatch)
[![Go Reference](https://pkg.go.dev/badge/github.com/HodeTech/leakwatch.svg)](https://pkg.go.dev/github.com/HodeTech/leakwatch)
[![GitHub Marketplace](https://img.shields.io/badge/Marketplace-Leakwatch%20Secret%20Scanner-2ea44f?logo=github)](https://github.com/marketplace/actions/leakwatch-secret-scanner)

> Next-generation secret scanning platform — fast, accurate, open source.

Expand All @@ -15,18 +16,21 @@

| Feature | Leakwatch | TruffleHog | Gitleaks |
|---------|-----------|------------|----------|
| **License** | MIT | AGPL-3.0 | MIT* |
| **License** | MIT | AGPL-3.0 | MIT [^gl-action] |
| **Secret Verification** | Yes (54 verifiers, 51 packages) | Yes | No |
| **Container Scanning** | Yes | Yes | No |
| **Aho-Corasick** | Yes | Partial | No |
| **Entropy Analysis** | Hybrid | Yes | Filter |
| **YAML Custom Rules** | Yes | No (Go) | TOML |
| **SARIF Output** | Yes | Yes | Yes |
| **SARIF Output** | Yes | No [^th-sarif] | Yes |
| **Aho-Corasick Prefilter** | Yes | Yes | Yes |
| **Entropy Analysis** | Yes | Yes | Yes |
| **Custom Rules** | YAML | YAML (config) | TOML |

[^gl-action]: The Gitleaks CLI is MIT-licensed. The official `gitleaks-action` GitHub Action, however, runs under a commercial EULA and requires a (free) license key for **organization** accounts (personal accounts are exempt).
[^th-sarif]: TruffleHog emits JSON / plain / GitHub-Actions output; it has no native SARIF formatter (SARIF requires an external converter). All three tools use Aho-Corasick keyword pre-filtering and Shannon-entropy filtering, and all three support custom rules (Leakwatch: YAML, TruffleHog: `config.yaml` `detectors:` block, Gitleaks: TOML).

**What makes Leakwatch different:**
- **Verification + MIT license** — A unique combination in the open source world
- **MIT license _with_ verification** — Among these tools, Leakwatch is the only one that is both permissively licensed (MIT, unlike TruffleHog's AGPL-3.0) and performs live secret verification (unlike Gitleaks, which is detection-only)
- **85.7% verification coverage** — 54 of 63 detectors have live API or format validation verification
- **Hybrid detection engine** — Low false positives with Aho-Corasick + Regex + Entropy
- **Verification + container + SARIF in one MIT binary** — TruffleHog lacks SARIF; Gitleaks lacks verification and container scanning
- **Easy extensibility** — YAML for simple rules, Go plugin for advanced ones
- **Single binary, zero dependencies** — Runs on every platform
- **Scan summary** — Every scan prints a summary to stderr (date, source, target, files scanned, duration, findings)
Expand All @@ -48,8 +52,8 @@ go install github.com/HodeTech/leakwatch@latest
# Docker
docker run --rm -v $(pwd):/scan ghcr.io/hodetech/leakwatch:latest scan fs /scan

# Binary download
curl -sSfL https://github.com/HodeTech/Leakwatch/releases/latest/download/leakwatch_$(uname -s)_$(uname -m).tar.gz | tar xz
# Binary download — pick the archive for your OS/arch from the releases page:
# https://github.com/HodeTech/Leakwatch/releases (e.g. leakwatch_1.5.0_linux_amd64.tar.gz)
```

### Quick Setup
Expand Down Expand Up @@ -190,10 +194,11 @@ leakwatch scan fs . --remediation
### GitHub Actions

```yaml
- uses: HodeTech/leakwatch-action@v1
- uses: HodeTech/Leakwatch@v1
with:
scan-type: git
only-verified: true # only report verified live secrets (action default: false)
no-verify: false # turn verification ON (required for only-verified)
only-verified: true # report only secrets confirmed live
sarif-upload: true
```

Expand Down
Loading
Loading