Web Attack Prediction

Detects and classifies HTTP web attacks (SQLi, XSS, LFI, CMDi) from request/response pairs using rule-based heuristics and supervised ML models. Built as a bachelor thesis comparing classical heuristic WAF logic against trained classifiers across 12 intentionally vulnerable web applications.

Tech stack: Python · scikit-learn · Streamlit · pandas · requests · PyYAML

---

Pipeline

Vulnerable app          HTTP Recorder          Feature Extractor       Classifier
(DVWA, bWAPP,    →    src/capture/record.py  →  src/features/       →  Heuristic / ML
 Mutillidae, …)        configs/*.yaml            extract.py             src/training/
        ↓
  data/raw/*.jsonl  →  data/features/features.csv  →  data/results/

1. Capture — sends labelled probes (benign + attack payloads) to vulnerable apps and records full request/response pairs as JSONL
2. Extract — derives 28 binary/numeric features per record (payload signals, param-name signals, response signals)
3. Train — runs StratifiedGroupKFold CV (grouped by endpoint to prevent data leakage) and saves the best model
4. Demo — interactive Streamlit app for single-request inference and CSV batch prediction

---

Results

Macro F1 across all attack types (5-fold cross-validation, grouped by endpoint):

Method	CMDi	LFI	SQLi	XSS	Macro F1	Accuracy
Heuristic A (payload only)	0.41	0.56	0.62	0.47	0.514	0.499
Heuristic B (pay + response)	0.42	0.60	0.65	0.45	0.527	0.517
ML — structural features	0.46	0.60	0.71	0.64	0.599	0.615
ML — signal features	0.72	0.78	0.81	0.64	0.738	0.740
ML — all features	0.78	0.76	0.83	0.68	0.762	0.763

Best models: Extra Trees and Random Forest (tuned via cross-validated grid search over max_depth).

Key finding: ML with signal features (+44 pp macro F1 vs. Heuristic A) outperforms classical WAF-style rule matching, while remaining interpretable through feature importances.

---

Data

HTTP traffic was captured against 12 intentionally vulnerable web applications:

App	Attack types covered
DVWA	SQLi, XSS, LFI, CMDi
bWAPP	SQLi, XSS, LFI, CMDi
Mutillidae	SQLi, XSS, LFI, CMDi
Hackazon	SQLi, XSS, LFI, CMDi
DVNA	SQLi, CMDi
XVWA	SQLi, LFI, CMDi
Juice Shop	SQLi
Bricks	SQLi
SQLi-Lab	SQLi
DSVPWA / DSVW	LFI, CMDi
Vulhub	LFI
WackoPicko	LFI

Each record contains the full HTTP request (method, URL, params, headers) and response (status, body snippet, timing), with attack-type labels applied at capture time.

The extracted feature dataset (data/features/features.csv) is also published on Kaggle: Web Attack Prediction - HTTP Features.

Dataset size: 947 labelled records · 72 endpoints · 4 attack classes (SQLi 314, XSS 247, CMDi 196, LFI 190)

Each endpoint receives multiple probe types: benign baselines, targeted attack payloads, and cross-type probes (e.g. an XSS payload sent to a SQLi endpoint) to test the classifier's robustness against confounding signals.

---

Project Structure

.
├── configs/            # YAML endpoint + payload configs for each app
├── data/
│   ├── features/       # features.csv (extracted from raw JSONL)
│   └── results/        # per-model metrics, confusion matrices, importances
├── notebooks/
│   ├── eda.ipynb       # dataset exploration and feature distributions
│   └── comparison.ipynb # heuristic vs. ML comparison plots
├── src/
│   ├── capture/
│   │   ├── record.py   # CLI: probe runner, JSONL writer
│   │   ├── recorder.py # Recorder class with session/auth management
│   │   └── auth.py     # pluggable auth handlers (form login, cookie, etc.)
│   ├── features/
│   │   └── extract.py  # feature engineering from raw JSONL
│   ├── training/
│   │   ├── train_all.py # trains all 6 models × 4 feature sets, saves best_signal.joblib
│   │   ├── train_ml.py # ML pipeline: 6 models, hyperparam tuning, CV, export
│   │   └── heuristic.py # rule-based baseline (Heuristic A + B)
│   └── app/
│       └── app.py      # Streamlit demo (manual input + CSV batch)
├── models/             # saved .joblib models (gitignored, reproducible)
└── requirements.txt

---

Setup

pip install -r requirements.txt

Tested with: Python 3.11.5 · scikit-learn 1.7.2 · pandas 2.3.3 · numpy 2.2.6

The vulnerable applications need to be running locally (e.g. via Docker) for data capture. Pre-extracted features (data/features/features.csv) and result files are included, so steps 1–2 can be skipped.

---

Usage

1. Capture HTTP data

python src/capture/record.py --config configs/dvwa_sqli.yaml

Outputs a timestamped JSONL file to data/raw/<app_name>/.

2. Extract features

python src/features/extract.py
# or with explicit paths:
python src/features/extract.py --raw-dir data/raw --out data/features/features.csv

3. Run heuristic baseline

python src/training/heuristic.py

4. Train ML models

Train all 6 models across all 4 feature sets (24 combinations). The best model on the signal feature set is saved as models/best_signal.joblib for the web app:

python src/training/train_all.py

Or train a single model manually:

# default: Extra Trees with signal features + hyperparameter tuning
python src/training/train_ml.py

# choose model and feature set:
python src/training/train_ml.py --model rf --feature-set signal
python src/training/train_ml.py --model svm --feature-set all --no-tune

Available models: et, rf, gb, lr, svm, knn
Feature sets: signal (default), all, structural, signal_noparam
Models are saved to models/{algorithm}/{algorithm}_{feature_set}.joblib.

5. Start the demo app

streamlit run src/app/app.py

Opens a browser UI for manually entering HTTP request features or uploading a CSV for batch classification.

Run tests

pytest

38 unit tests covering feature extraction (src/features/extract.py) and the heuristic classifier (src/training/heuristic.py).

---

License

• Code: MIT License — free to use, modify and distribute with attribution.
• Dataset: the feature table (data/features/features.csv), also published as the Kaggle dataset: CC-BY-4.0.

This project is intended for educational and defensive security research. The included attack payloads target intentionally vulnerable applications in a local lab environment — only use them against systems you own or are explicitly authorized to test.