Generate a complete, analytics-ready retail dataset inspired by the ContosoRetailDW schema - with configurable dimensions, realistic sales behavior, budget forecasts, inventory snapshots, wishlists, and complaints. Designed for BI, analytics, data engineering, and data modeling scenarios.
Every run is deterministic, schema-stable, and idempotent, making the generator ideal for repeatable demos, training environments, and benchmarking.
The generator produces a full dimensional model across dimension and fact tables (lightly snowflaked, with multiple fact tables, like ContosoRetailDW).
Dimension tables:
| Group | Tables |
|---|---|
| Customers & accounts | Customers, CustomerProfile, OrganizationProfile, Plans, CustomerSubscriptions, LoyaltyTiers, CustomerAcquisitionChannels |
| Products & catalog | Products, ProductProfile, ProductCategory, ProductSubcategory, Suppliers, Promotions |
| Locations & org | Stores, Employees, EmployeeStoreAssignments, Geography |
| Time | Dates (calendar + fiscal + weekly fiscal), Time |
| Currency | Currency, ExchangeRates |
| Other lookups | Channels, ReturnReason |
Fact tables: Sales (flat or split into OrderHeader + OrderDetail), Returns, BudgetYearly, BudgetMonthly, InventorySnapshot, CustomerWishlists, Complaints
| Format | Description |
|---|---|
csv |
CSV files + auto-generated SQL Server and PostgreSQL bootstrap scripts (CREATE TABLE, load, constraints, views, indexes) |
parquet |
Merged Apache Parquet with configurable compression, row groups, and dictionary encoding |
deltaparquet |
Delta Lake tables partitioned by Year/Month |
Each run produces a self-contained output folder under generated_datasets/ with all tables, SQL scripts (for CSV), and a Power BI Project template ready to open.
- Python 3.13 (pinned via
.python-versionfor reproducible builds; uv will auto-install it if missing) - Git
Optional:
- uv - recommended for fast, locked dependency installs (
pip install uv) - Power BI Desktop (to explore the included
.pbipproject template)
Verify Python:
python --versiongit clone https://github.com/SharmaAntriksh/ContosoForge.git
cd ContosoForgeThis project uses uv for locked, reproducible installs. Install it once if you don't have it:
- With Python:
pip install uv - Without Python (Windows):
irm https://astral.sh/uv/install.ps1 | iex - Without Python (macOS / Linux):
curl -LsSf https://astral.sh/uv/install.sh | sh
uv then provisions the correct Python itself - no separate Python install needed.
Just installed uv and still see "uv is required but was not found on PATH"? The installer adds uv to your PATH, but already-running shells don't pick that up. Opening a new tab isn't enough; Windows Terminal / VS Code give new tabs the old environment. Fully quit and reopen the terminal app, or just set it in the current session and re-run:
$env:Path = "$env:USERPROFILE\.local\bin;$env:Path" .\scripts\create_venv.ps1
Windows (PowerShell), first-time setup:
Step 1: allow local scripts (one time only). Windows blocks unsigned .ps1
files by default, so the setup scripts won't run until you permit them for your
user account. This needs no admin rights and you only ever do it once:
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned(If you'd rather not change the policy, skip this step and prefix each script
with a one-off bypass, e.g. powershell -ExecutionPolicy Bypass -File .\scripts\create_venv.ps1.)
Step 2: create the environment and activate it:
.\scripts\create_venv.ps1 # creates .venv + installs locked deps via uv
. .\scripts\activate_venv.ps1 # activate itTo update dependencies later: .\scripts\sync_venv.ps1
macOS / Linux:
# Locked, reproducible - includes SQL Server + Postgres import drivers
uv sync --extra sql --extra postgres
# Core only (no database import drivers)
uv syncImporting CSV output into SQL Server / Postgres? The
pyodbcandpsycopgdrivers are optional extras. Thecreate_venv.ps1/sync_venv.ps1scripts install them by default; with bareuv syncadd--extra sql --extra postgres. A plainuv sync(no extras) will remove them from an existing venv.
Activate the venv first (your prompt should show
(.venv)), otherwisepython main.py ...runs against your system Python and fails withModuleNotFoundError: No module named 'pyarrow'. The dependencies live in the venv, not globally. Use. .\scripts\activate_venv.ps1, or skip activation entirely withuv run python main.py ...(uv picks the right interpreter). Note: if VS Code's terminal was already open when you created the venv, reopen the window so its Python extension auto-activates.venv.
First run feels slow (~10-20s to start)? That's a one-time cold start, not a hang: Windows Defender scans the freshly installed package DLLs, Python compiles bytecode (
.pyc) for the large libraries (pyarrow, pandas, numpy, deltalake), and the OS warms its file cache. Subsequent runs start in ~1-2s.
The fastest way to get started - run with default settings from config.yaml:
.\scripts\run_generator.ps1Or run directly via CLI with overrides:
python main.py \
--format parquet \
--sales-rows 100000 \
--customers 5000 \
--stores 50 \
--products 500 \
--start-date 2022-01-01 \
--end-date 2025-12-31 \
--workers 8 \
--cleanThe 5 most-used flags are --format, --sales-rows, --customers, --workers, and --clean. For the full CLI surface (every flag, common patterns, override precedence), see the CLI reference.
Generated datasets land in generated_datasets/ with a timestamped folder name like:
generated_datasets/
└── 2026-03-07 02_30_45 PM Customers 5K Sales 100K CSV/
├── Sales PBIP/
│ └── Sales.pbip ← open directly in Power BI
├── config/
│ ├── config.yaml
│ └── models.yaml
├── dimensions/
│ ├── customers.csv
│ ├── products.csv
│ ├── stores.csv
│ ├── dates.csv
│ └── ... (22 dimension tables total)
├── facts/
│ ├── sales/
│ ├── returns/ ← if returns enabled
│ ├── budget/ ← if budget enabled
│ ├── inventory/ ← if inventory enabled
│ ├── customer_wishlists/ ← if wishlists enabled
│ └── complaints/ ← if complaints enabled
└── sql/ ← CSV mode only - one folder per dialect
├── sqlserver/ ← SQL Server bootstrap
│ ├── schema/ ← CREATE TABLE, constraints, views, verify
│ ├── load/ ← BULK INSERT scripts
│ └── indexes/ ← columnstore (CCI) helper
└── postgres/ ← PostgreSQL bootstrap
├── schema/ ← CREATE TABLE, constraints, views (DDL)
├── load/ ← COPY scripts
├── admin/ ← manage_primary_keys procedure
└── indexes/ ← btree + BRIN indexes
The generated SQL is always PascalCase. Identifier casing is chosen at import:
the SQL Server / PostgreSQL import scripts take -IdentifierCase pascal|snake and
re-render the SQL in that case (snake gives quote-free PostgreSQL querying). Only
the emitted scripts change, never the CSV/parquet data.
The generator is driven by two YAML files at the project root.
-
config.yaml- controls the shape and scale of the dataset: row counts, entity counts, date ranges, output format, parallelism, and feature toggles. Full reference: CONFIG_GUIDE. -
models.yaml- controls how sales behave: demand curves, pricing dynamics, basket sizes, brand popularity, return patterns, and the overall business shape via trend presets. Not overridable via CLI - edit directly or via the web UI. Full reference: MODELS_GUIDE.
CLI flags override config.yaml values for the current run only - they are not persisted.
For tuning the customer/row/date balance to get visually interesting charts, see Scaling tips.
Post-generation utilities for tuning, repartitioning, and importing generated datasets. Each script has its own reference doc with full flag coverage, recipes, and troubleshooting.
| Task | Script | Docs |
|---|---|---|
| Re-compress / re-row-group Parquet output | scripts/optimize_parquet.py |
parquet-optimization |
| Compact small Delta Lake files | scripts/optimize_delta.py |
delta-optimization |
| Change Delta Lake partition layout | scripts/repartition_delta.py |
delta-repartitioning |
| Import CSV output to SQL Server | scripts/run_sql_server_import.ps1 |
sql-server-import |
| Import CSV output to PostgreSQL | scripts/run_postgres_import.ps1 |
postgres-import |
| Provision a SQL login for SSAS / Power BI | (same import script) | tabular-user |
| Post-import admin & verify procedures | (generated SQL) | post-import-procedures |
A web UI (FastAPI + React) is also available for interactive generation:
.\scripts\run_web.ps1
Each output includes a Power BI Project (.pbip) template with pre-configured folder paths. Open the .pbip file directly in Power BI Desktop - no manual path setup required.
The project includes an extensive test suite covering config validation, pricing pipeline, quantity model, geography, trend presets, version store, state management, determinism guarantees, edge-case guards, web API, packaging, sales logic, schema validation, product dimensions, sales writer, SQL tools, and date dimension edge cases.
# Run all tests
pytest
# Run a specific file or class
pytest tests/test_geography.py
pytest tests/test_config_loader.py::TestParseDate
# Stop on first failure
pytest -x
# Rerun only previously failed tests
pytest --lf| Topic | Doc |
|---|---|
| Full CLI flag reference | cli-reference |
config.yaml reference |
CONFIG_GUIDE |
models.yaml reference + trend presets |
MODELS_GUIDE |
| Pipeline architecture | PIPELINE_FLOWCHART |
| Operations (parquet, delta, SQL Server / PostgreSQL import) | operations/ |
This project is licensed under the PolyForm Noncommercial License 1.0.0.
You are free to use, modify, and share it for any noncommercial purpose. Commercial use, including selling it, is not permitted. You must keep the attribution notice (Copyright (c) 2026 Antriksh Sharma). See LICENSE.md for the full terms.