Real-Time Network Intrusion Detection System (LSTM)

A real-time Network Intrusion Detection System (NIDS) using dual LSTM neural networks with attention mechanisms and concept drift detection. Classifies network flows as benign or attack-type in real time via an HTTP API, with automatic fallback to a drift monitor when the primary classifier's confidence drops.

How It Works

flowchart LR
    A["Live Traffic<br/>(pcap / port mirror)"] --> B["Feature Extraction<br/>CICFlowMeter"]
    B --> C["Temporal LSTM<br/>27 time-based features"]
    B --> D["Context LSTM<br/>54 connection features"]
    C --> E["Fusion + Attention<br/>Layer"]
    D --> E
    E --> F{"Confidence ≥ 0.75<br/>& Stability OK?"}
    F -- Yes --> G["Classifier<br/>Prediction"]
    F -- No --> H["Drift Monitor<br/>Prediction"]
    G --> I{"Anomaly > 0.6<br/>or Drift?"}
    H --> I
    I -- No --> J["✅ Benign / Known Attack"]
    I -- Yes --> K["🚨 Alert (SSE stream)"]

Loading

Pipeline: Raw traffic → 80+ CICFlowMeter features → dual LSTM streams (temporal + context) → attention-weighted fusion → classifier with drift-aware fallback → alert generation.

Performance

Both models evaluated on a held-out test set (20% of CICDDoS2019, random stratified split). Metrics reflect classification on the same 4 attack types seen during training.

CICDDoS2019 is a benchmark where 95–99%+ accuracy is commonly reported. Real-world deployment requires evaluation on live traffic with concept drift, imbalanced classes, and zero-day attacks.

Classifier (DualInputLSTM)

Metric	Value
Accuracy	99.01%
Mean Confidence	98.43%

Class	Precision	Recall	F1-Score	Support
DrDoS_NTP	99.998%	99.998%	99.998%	63,674
DrDoS_UDP	99.981%	98.039%	99.000%	164,123
Syn	99.990%	99.971%	99.981%	82,807
UDPLag	85.642%	99.799%	92.180%	19,389

Confusion Matrix (Classifier)
              Predicted
              NTP     UDP     Syn     UDPLag
Actual NTP   63673       0       0        1
Actual UDP       0  160904       0     3219
Actual Syn       0       0   82783       24
Actual UDPLag    1      30       8    19350

Drift Monitor

Metric	Value
Accuracy	99.89%
Mean Confidence	99.91%

Class	Precision	Recall	F1-Score	Support
DrDoS_NTP	100.000%	99.992%	99.996%	63,674
DrDoS_UDP	99.841%	99.959%	99.900%	164,123
Syn	99.990%	99.964%	99.977%	82,807
UDPLag	99.475%	98.613%	99.042%	19,389

Confusion Matrix (Drift Monitor)
              Predicted
              NTP     UDP     Syn     UDPLag
Actual NTP   63669       1       0        4
Actual UDP       0  164056       0       67
Actual Syn       0       0   82777       30
Actual UDPLag    0     261       8    19120

Notes

• UDPLag precision gap: Classifier achieves 85.64% vs drift monitor's 99.47%. The classifier's 2-layer bidirectional LSTM misclassifies 3,219 DrDoS_UDP samples as UDPLag (1.96% of 164K UDP samples). The drift monitor's simpler 1-layer architecture and dual training objective (classification + anomaly score regression targeting 1.0 − classifier_confidence) reduce this to 67 — a 48× improvement through implicit regularization.

Dataset

Trained on CICDDoS2019. Random 80/20 stratified train/test split. All metrics above are on the held-out test set.

Training classes (seen during training):

Class	Type
Syn	SYN flood
UDPLag	UDP flood with lag
DrDoS_UDP	Distributed reflection DoS (UDP)
DrDoS_NTP	Distributed reflection DoS (NTP)

Drift detection (unseen during training, handled at inference): DrDoS_DNS, DrDoS_SNMP, DrDoS_MSSQL, DrDoS_NetBIOS, DrDoS_SSDP, DrDoS_LDAP, TFTP

Model Architecture

Component	Classifier	Drift Monitor
Temporal hidden	128	64
Context hidden	64	32
Fusion hidden	96	48
LSTM layers	2	1
Bidirectional	Yes	No
Dropout	0.3	0.2
Attention	Feature + Temporal	Feature + Temporal
Params (est.)	~450K	~80K

Quick Start

docker compose -f docker/docker-compose-lstm.yml up

Simulate an attack

docker exec -it lstm-attacker-shell hping3 -S --flood -p 80 lstm-target-server

Query the API

# Health check
curl http://localhost:8090/health

# Classify a flow (POST)
curl -X POST http://localhost:8090/predict \
  -H "Content-Type: application/json" \
  -d '{
    "Flow Duration": 1000000, "Protocol": 6,
    "Source Port": 443, "Destination Port": 80,
    "Total Fwd Packets": 10, "Total Backward Packets": 8,
    "Flow IAT Mean": 500.0, "Flow IAT Std": 100.0,
    "Flow Bytes/s": 2000.0, "Flow Packets/s": 18.0
  }'

# Stream live alerts
curl -N http://localhost:8090/alerts/stream

# Statistics
curl http://localhost:8090/stats

API Endpoints

Endpoint	Method	Description
/health	GET	Health check
/predict	GET/POST	Classify a network flow
/stats	GET	Model usage statistics
/alerts	GET	Recent alerts (last 50)
/alerts/stream	GET	Live SSE alert feed
/drift	GET	Drift detection history

Project Structure

src/
├── lstm_config.py                      # Hyperparameters & paths
├── lstm_run_pipeline.py                # Training orchestrator
├── lstm_training/
│   ├── lstm_preprocess.py             # CICDDoS2019 preprocessing
│   ├── lstm_combine_and_clean.py       # Data merging & cleaning
│   ├── lstm_sequence_builder.py        # Sequence generation
│   └── lstm_train_model.py            # Model definitions & training
├── lstm_detection/
│   ├── lstm_realtime_detector.py       # HTTP API server
│   └── lstm_traffic_reporter.py        # /proc/net monitoring
└── lstm_evaluation/
    ├── lstm_drift_evaluation.py        # Drift detection eval
    └── lstm_evaluate_unknown.py        # Unknown attack eval

License

MIT License — see LICENSE.