API Data Warehouse

A beginner-friendly Data Engineering project that demonstrates an ETL pipeline using Python and SQLite.

Architecture

API ↓ extract.py ↓ raw/products.json ↓ transform.py ↓ processed/products_clean.csv ↓ load.py ↓ SQLite Warehouse ↓ Analytics Queries

Tech Stack

• Python
• SQLite
• SQL
• Pandas
• Requests

Project Structure

src/
data/
database/
sql/

Setup

pip install -r requirements.txt

Run pipeline:

python src/run_pipeline.py

Features

• API ingestion
• Data validation
• Data cleaning
• SQLite loading
• SQL analytics