Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions .dumplingconf.example
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,17 @@ salt = "${DUMPLING_GLOBAL_SALT}"
#
# Each column maps to an anonymizer spec: { strategy = "…", <options> }
# ---------------------------------------------------------------------------
# Faker strategy: `faker = "module::Type"` matches the Rust `fake` crate layout.
# Crate docs: https://docs.rs/fake/latest/fake/
# Faker modules: https://docs.rs/fake/latest/fake/faker/index.html
# Upstream repo: https://github.com/cksac/fake-rs
[rules."public.users"]
# email — random-looking email at example.com; force quoted string output
email = { strategy = "email", domain = "customer_identity", unique_within_domain = true }
# name — random placeholder full name
full_name = { strategy = "name" }
first_name = { strategy = "first_name" }
last_name = { strategy = "last_name" }
# email — fake email via Rust `fake` crate; force quoted string output
email = { strategy = "faker", faker = "internet::SafeEmail", domain = "customer_identity", unique_within_domain = true }
# name — locale-aware full name (see `locale`); other generators use `faker = "module::Type"`
full_name = { strategy = "faker", faker = "name::Name" }
first_name = { strategy = "faker", faker = "name::FirstName" }
last_name = { strategy = "faker", faker = "name::LastName" }
# phone — US-style (xxx) xxx-xxxx
phone = { strategy = "phone" }
# ssn — SHA-256 hex of original; use per-column salt for extra protection
Expand All @@ -58,7 +62,7 @@ wake_time = { strategy = "time_fuzz", min_seconds = -3600, max_seconds = 360
# credit card — redact entirely; force as quoted string
credit_card = { strategy = "redact", as_string = true }
# keep the same anonymized email as users table via shared domain
customer_email = { strategy = "email", domain = "customer_identity" }
customer_email = { strategy = "faker", faker = "internet::SafeEmail", domain = "customer_identity" }

[rules."public.audit_log"]
# unqualified table name also works (matches any schema)
Expand Down
35 changes: 35 additions & 0 deletions .github/workflows/docs-pr.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# mdBook verification on pull requests only (no GitHub Pages upload or deploy).
# Pages build + deploy live in docs.yml and run on pushes to main.
name: Docs (PR)

on:
pull_request:
paths:
- "README.md"
- "book.toml"
- "docs/**"
- ".github/workflows/docs.yml"
- ".github/workflows/docs-pr.yml"

permissions:
contents: read

concurrency:
group: docs-pr-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
verify:
name: Build mdBook (verify)
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Install mdBook
uses: peaceiris/actions-mdbook@v2
with:
mdbook-version: "0.4.52"

- name: Build documentation site
run: mdbook build
20 changes: 9 additions & 11 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
# Build and deploy the mdBook site to GitHub Pages (main branch only).
# Pull-request verification runs in docs-pr.yml — this workflow does not run on PRs.
name: Docs

on:
pull_request:
paths:
- "README.md"
- "book.toml"
- "docs/**"
- ".github/workflows/docs.yml"
push:
branches:
- main
Expand All @@ -18,16 +14,16 @@ on:

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: docs-${{ github.ref }}
group: docs-pages-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Checkout
uses: actions/checkout@v4
Expand All @@ -40,15 +36,17 @@ jobs:
- name: Build documentation site
run: mdbook build

- name: Upload docs artifact
- name: Upload Pages deployment artifact
uses: actions/upload-pages-artifact@v3
with:
path: docs/book

deploy:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
/target/
/docs/book/
/.tools/
18 changes: 17 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,8 @@ Follow these steps in order. Do not skip any step.

8. **`README.md`**: Add a row to the "Anonymization strategies" table.

**`faker` strategy:** Config only carries string identifiers; Dumpling never evaluates user Rust from config. To ship a new generator, add dispatch in `src/faker_dispatch.rs` and validation in `validate_anonymizer_spec` for the `faker` branch. Upstream reference: [`fake` on docs.rs](https://docs.rs/fake/latest/fake/), [`fake::faker` module index](https://docs.rs/fake/latest/fake/faker/index.html), [source on GitHub](https://github.com/cksac/fake-rs).

---

## How to Add a New Row Filter Predicate Operator
Expand Down Expand Up @@ -274,15 +276,29 @@ Follow these steps in order. Do not skip any step.

This is a pure Rust CLI project with **no external services** (no database, Docker, or network dependencies). The Rust stable toolchain (rustc + cargo) is the only prerequisite.

### One-shot environment (agents and humans)

From the repository root:

```bash
./scripts/setup-dev.sh
```

This installs the **stable** toolchain with **rustfmt** and **clippy** (via `rustup` when available), runs **`cargo fetch`**, and installs a pinned **mdBook** binary under `.tools/` (same version as the Docs CI workflow) so you can run `mdbook build` without a global install. Add `.tools` to `PATH` for convenience, or invoke `.tools/mdbook build` directly.

The repo root **`rust-toolchain.toml`** pins **stable** and the **components** CI uses, so `cargo` automatically selects the right toolchain in fresh checkouts.

### Quick reference

| Task | Command |
|------|---------|
| Setup (toolchain + fetch + mdbook) | `./scripts/setup-dev.sh` |
| Build | `cargo build` |
| Test | `cargo test --all-targets --all-features` |
| Lint | `cargo clippy --all-targets --all-features` |
| Format check | `cargo fmt --all -- --check` |
| Auto-format | `cargo fmt` |
| Docs site (mdBook) | `mdbook build` or `.tools/mdbook build` after setup |
| Run CLI | `./target/debug/dumpling --help` |

### Running the CLI
Expand All @@ -295,6 +311,6 @@ Dumpling is fail-closed by default — it exits non-zero without a config file.

### Notes

- All 94 tests are inline `#[cfg(test)]` modules; there are no separate test files or fixtures to manage.
- All tests are inline `#[cfg(test)]` modules; there are no separate test files or fixtures to manage.
- The update script uses `cargo fetch` to pre-download crate dependencies. A full `cargo build` or `cargo test` will then compile from the local cache without network access.
- No environment variables or secrets are required for building, testing, or running the CLI locally.
8 changes: 8 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,14 @@ For AI coding agents: also read `AGENTS.md`, which contains more detailed techni
- **Rust stable toolchain** — install via [rustup.rs](https://rustup.rs/).
- No database, Docker, or external services are required. Dumpling is a pure CLI tool.

### One-shot setup (recommended)

```bash
./scripts/setup-dev.sh
```

Installs stable + `rustfmt` + `clippy`, prefetches crates, and downloads a pinned **mdBook** under `.tools/` (for `mdbook build`, same version as CI). Optional: `export PATH="$PWD/.tools:$PATH"`.

### Build and run

```bash
Expand Down
21 changes: 14 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,8 @@ salt = "${DUMPLING_GLOBAL_SALT}"

# Rules are keyed by either "table" or "schema.table"
[rules."public.users"]
email = { strategy = "email", domain = "customer_identity", unique_within_domain = true }
name = { strategy = "name", locale = "de_de" } # German-locale name
email = { strategy = "faker", faker = "internet::SafeEmail", domain = "customer_identity", unique_within_domain = true }
name = { strategy = "faker", faker = "name::Name", locale = "de_de" } # German-locale name
ssn = { strategy = "hash", salt = "${env:DUMPLING_USERS_SSN_SALT}", as_string = true } # SHA-256 of original (salted)
age = { strategy = "int_range", min = 18, max = 90 }

Expand Down Expand Up @@ -132,15 +132,20 @@ token = "high"
| `redact` | Replace with `REDACTED` (string) |
| `uuid` | Random UUIDv4-like string |
| `hash` | SHA-256 hex of original value; supports per-column `salt` and global `salt` |
| `email` | Random-looking email at `example.com` |
| `name` / `first_name` / `last_name` | Locale-aware fake name (configurable via `locale`); defaults to English |
| `faker` | Values from the Rust [`fake`](https://crates.io/crates/fake) crate ([docs.rs](https://docs.rs/fake/latest/fake/), [`faker` modules](https://docs.rs/fake/latest/fake/faker/index.html)), chosen by a **string identifier** only (`faker = "module::Type"`, e.g. `internet::SafeEmail`). Config is **data only**: nothing from TOML is compiled or executed as Rust at runtime. Use `locale` for locale-aware generators; optional `min`/`max`, `length`, `format` as documented. Unsupported targets fail at config load. New generators require a **new Dumpling release** (or your own fork), not config-side code. |
| `phone` | Locale-aware fake phone number (configurable via `locale`); defaults to English format |
| `int_range` | Random integer in `[min, max]` |
| `string` | Random alphanumeric string (`length = 12` by default) |
| `date_fuzz` | Shifts a date by a random number of days in `[min_days, max_days]` (defaults: `-30..30`) |
| `time_fuzz` | Shifts a time-of-day by a random number of seconds in `[min_seconds, max_seconds]` with 24h wraparound (defaults: `-300..300`) |
| `datetime_fuzz` | Shifts a timestamp/timestamptz by a random number of seconds in `[min_seconds, max_seconds]` (defaults: `-86400..86400`) |

**`faker` reference (upstream `fake` crate):** Dumpling’s `faker = "module::Type"` strings mirror the Rust [`fake`](https://crates.io/crates/fake) crate’s [`faker`](https://docs.rs/fake/latest/fake/faker/index.html) module layout. Use these when picking or extending generators:

- [docs.rs — `fake` crate root](https://docs.rs/fake/latest/fake/) (overview, `Fake` / `Dummy` traits, locales)
- [docs.rs — `fake::faker` module index](https://docs.rs/fake/latest/fake/faker/index.html) (per-domain submodules: `address`, `internet`, `name`, …)
- [GitHub — `cksac/fake-rs`](https://github.com/cksac/fake-rs) (source, README with the CLI’s generator name list)

### Secret references

Dumpling resolves secret references in string config fields so plaintext salts/keys
Expand Down Expand Up @@ -188,7 +193,9 @@ dumpling --security-profile hardened --input dump.sql --check
- `unique_within_domain`: when true, different source values are assigned unique pseudonyms within the configured `domain`. NULL values are unaffected and always remain NULL.
- `min_days` / `max_days`: used by `date_fuzz`.
- `min_seconds` / `max_seconds`: used by `time_fuzz` and `datetime_fuzz`.
- `locale`: selects the language/regional format for the `name`, `first_name`, `last_name`, and `phone` strategies. Supported values: `en`, `fr_fr`, `de_de`, `it_it`, `pt_br`, `pt_pt`, `ar_sa`, `zh_cn`, `zh_tw`, `ja_jp`, `cy_gb`. Defaults to `en` when not specified.
- `locale`: selects the language/regional format for the `faker` and `phone` strategies. Supported values: `en`, `fr_fr`, `de_de`, `it_it`, `pt_br`, `pt_pt`, `ar_sa`, `zh_cn`, `zh_tw`, `ja_jp`, `cy_gb`. Defaults to `en` when not specified.
- `faker`: required when `strategy = "faker"`. A plain string `"module::Type"` (case-insensitive) that maps to a **built-in** generator compiled into Dumpling—not arbitrary Rust or expressions. Names follow [`fake::faker`](https://docs.rs/fake/latest/fake/faker/index.html) (e.g. `internet::SafeEmail` → `faker::internet::SafeEmail` in the crate).
- `format`: used with `faker = "number::NumberWithFormat"`; pattern uses `#` (0–9) and `^` (1–9) per the [`fake` crate docs](https://docs.rs/fake/latest/fake/).

> **Note:** `table_options` are no longer supported; use explicit `rules` and optional `column_cases`.

Expand Down Expand Up @@ -332,7 +339,7 @@ Define default strategies in `rules."<table>"` and add ordered per-column cases
```toml
[rules."public.users"]
email = { strategy = "hash", as_string = true } # default
name = { strategy = "name" }
name = { strategy = "faker", faker = "name::Name" }

[[column_cases."public.users".email]]
when.any = [{ column = "is_admin", op = "eq", value = "true" }]
Expand Down Expand Up @@ -383,7 +390,7 @@ salt = "${DUMPLING_HMAC_KEY}"

[rules."public.users"]
ssn = { strategy = "hash", as_string = true }
email = { strategy = "email", domain = "users" }
email = { strategy = "faker", faker = "internet::SafeEmail", domain = "users" }
```

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/src/ci-guardrails.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ violations to stderr, and exits:
| `empty-rules-table` | warning | A `[rules]` entry has no column rules. Likely a stale or incomplete config section. |
| `empty-column-cases-table` | warning | A `[column_cases]` entry has no column cases. |
| `unsalted-hash` | warning | A `hash` strategy is used with no salt (neither per-column `salt` nor global `salt`). Unsalted hashes are reversible via precomputed lookup tables for low-entropy inputs (names, emails, common IDs). |
| `inconsistent-domain-strategy` | error | The same domain name is used with two or more different strategies. This breaks referential integrity: a domain shared between `email` and `name` would try to maintain a bidirectional map between incompatible pseudonym types. |
| `inconsistent-domain-strategy` | error | The same domain name is used with two or more different strategies. This breaks referential integrity: a domain shared between incompatible generators (for example `faker` with different `faker` targets, or `faker` vs `hash`) cannot maintain a single stable mapping. |
| `uncovered-sensitive-column` | error | A column listed in `[sensitive_columns]` has no matching anonymization rule or case. The column will pass through unmodified, making the sensitive declaration misleading. |

---
Expand Down
12 changes: 11 additions & 1 deletion docs/src/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,24 @@ If no configuration is found, Dumpling fails closed by default and exits non-zer
Error output includes every checked location. If you intentionally want a no-op
run, pass `--allow-noop`.

## Faker strategy and the `fake` crate

When you use `strategy = "faker"` with `faker = "module::Type"`, those names align with the Rust [**`fake`**](https://crates.io/crates/fake) crate’s [`faker`](https://docs.rs/fake/latest/fake/faker/index.html) modules (for example `name::FirstName` ↔ `fake::faker::name::raw::FirstName`). Use the upstream docs to discover available generators and options:

- [docs.rs — `fake` (crate overview)](https://docs.rs/fake/latest/fake/)
- [docs.rs — `fake::faker` (all faker submodules)](https://docs.rs/fake/latest/fake/faker/index.html)
- [GitHub — `cksac/fake-rs` (source + README)](https://github.com/cksac/fake-rs)

Dumpling only exposes a **subset** wired in `src/faker_dispatch.rs`; unsupported `module::Type` pairs fail at config load.

## Baseline config template

```toml
salt = "${DUMPLING_GLOBAL_SALT}"

[rules."public.users"]
email = { strategy = "hash", salt = "${env:DUMPLING_USERS_EMAIL_SALT}", as_string = true }
name = { strategy = "name" }
full_name = { strategy = "faker", faker = "name::Name" }

[sensitive_columns]
"public.users" = ["employee_number", "tax_id"]
Expand Down
4 changes: 3 additions & 1 deletion docs/src/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@

## Prerequisites

- Rust stable toolchain (edition 2021 compatible)
- Rust **stable** toolchain (`rustup` recommended). The repo includes `rust-toolchain.toml` (stable + `rustfmt` + `clippy`) so CI and local `cargo` stay aligned.
- `cargo` on your `PATH`

Optional: run **`./scripts/setup-dev.sh`** once from the repo root — it installs toolchain components, **`cargo fetch`**, and a pinned **mdBook** under `.tools/` for the same docs build CI uses.

## Build

```bash
Expand Down
6 changes: 3 additions & 3 deletions docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ This documentation covers the operating model for day-to-day use:

## Documentation quality gate

All documentation is built with `mdBook` in CI:
The mdBook site is built in CI as follows:

- pull requests must pass the docs build job,
- pushes to `main` automatically publish docs to GitHub Pages.
- **Pull requests:** the **Docs (PR)** workflow runs `mdbook build` when docs-related paths change (no deploy).
- **`main`:** the **Docs** workflow builds and deploys to GitHub Pages when docs-related paths change.

This keeps the docs in a continuously deployable state instead of drifting from the codebase.
3 changes: 3 additions & 0 deletions rust-toolchain.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[toolchain]
channel = "stable"
components = ["rustfmt", "clippy"]
Loading
Loading