SIEM Pilot

A modern Security Information and Event Management (SIEM) system with AI-powered threat detection and natural language query capabilities.

Overview

This platform provides security teams with intelligent log analysis, real-time threat detection, and conversational AI assistance for investigating security incidents. Built for both learning environments (Demo Mode) and production deployments (Real Mode).

Key Features

AI Assistant

• Natural language queries powered by Groq (Llama 3.3 70B)
• Automatic MITRE ATT&CK technique mapping
• Intelligent threat correlation and analysis

Threat Detection

• Real-time Windows Event Log monitoring via Winlogbeat
• Custom detection rules with MITRE ATT&CK alignment
• Automated alert generation with severity scoring
• Behavioral anomaly detection

Investigation Workflows

• Structured incident tracking
• Timeline reconstruction
• Evidence collection and documentation
• AI-assisted root cause analysis

Dual Mode Operation

• Demo Mode: Pre-populated data for testing and learning (works immediately)
• Real Mode: Production monitoring with live log ingestion

Quick Start

Prerequisites

• Python 3.12+
• Docker & Docker Compose
• Node.js 18+
• Groq API key (free at console.groq.com)

5-Minute Setup

# Clone and navigate
git clone https://github.com/tejcodes-rex/siem-pilot.git
cd siem-pilot

# Start infrastructure
docker-compose up -d

# Configure environment
cp .env.example .env
# Edit .env and add your GROQ_API_KEY

# Setup backend
cd backend
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
python init_db.py

# Start backend
uvicorn app.main:app --reload

Access the API at http://localhost:8000/docs

Default credentials: admin / admin123

Architecture

Frontend (React)
       ↓
Backend API (FastAPI)
       ↓
   ┌───┴────┬──────────┬─────────┐
   ↓        ↓          ↓         ↓
PostgreSQL  Elasticsearch  Groq   Winlogbeat
(Alerts)    (Logs)         (AI)   (Events)

Screenshots

Dashboard

Alert Management

AI Chat Interface

Investigation Timeline

Installation

Full Setup

1. Infrastructure Services

docker-compose up -d

This starts:

• PostgreSQL (port 5432) - Alert and investigation storage
• Elasticsearch (port 9200) - Log storage and search
• Kibana (port 5601) - Optional log visualization

2. Backend

cd backend
python -m venv venv
venv\Scripts\activate  # Windows
# source venv/bin/activate  # Linux/Mac

pip install -r requirements.txt
python init_db.py
python create_admin.py

uvicorn app.main:app --reload

3. Frontend (optional)

cd frontend
npm install
npm start

Winlogbeat Setup (Real Mode)

For production monitoring with actual Windows Event Logs:

# Download Winlogbeat 9.2.0
# Configure winlogbeat.yml to point to your Elasticsearch
# Install and start the service

See winlogbeat.yml for full configuration.

Configuration

Environment Variables

Copy .env.example to .env and configure:

# AI Configuration
GROQ_API_KEY=your-api-key-here
LLM_MODEL=llama-3.3-70b-versatile

# Elasticsearch
ELASTICSEARCH_URL=http://localhost:9200

# Database
DATABASE_URL=postgresql://siem_user:siem_password@localhost:5432/siem_db

# Security
SECRET_KEY=your-secret-key-here
JWT_SECRET_KEY=your-jwt-secret-here

Detection Rules

Custom detection rules are defined in backend/detection_rules/ using YAML format:

name: Suspicious PowerShell Execution
severity: high
mitre_techniques:
  - T1059.001
conditions:
  - field: event_id
    operator: equals
    value: 4104
  - field: script_block
    operator: contains
    value: ["Invoke-WebRequest", "DownloadString"]

Usage

Natural Language Queries

Navigate to http://localhost:8000/docs and try the /api/v1/query/ask endpoint:

"Show me failed login attempts in the last 24 hours"
"What PowerShell scripts executed today?"
"Find suspicious process creation events"
"Analyze authentication failures from IP 192.168.1.100"

API Examples

# Login
curl -X POST "http://localhost:8000/api/v1/auth/login" \
  -d "username=admin&password=admin123"

# Get alerts
curl "http://localhost:8000/api/v1/alerts" \
  -H "Authorization: Bearer <token>"

# Search logs
curl -X POST "http://localhost:8000/api/v1/query/search" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"query": "show me process creation events"}'

Switching Modes

Demo Mode (default) - Uses simulated security data Real Mode - Processes actual logs from Elasticsearch

Toggle via API:

curl -X POST "http://localhost:8000/api/v1/auth/toggle-mode" \
  -H "Authorization: Bearer <token>" \
  -d '{"is_demo_mode": false}'

Project Structure

siem-pilot/
├── backend/
│   ├── app/
│   │   ├── api/           # API endpoints
│   │   ├── core/          # Configuration
│   │   ├── models/        # Database models
│   │   ├── services/      # Business logic
│   │   └── utils/         # Utilities
│   ├── detection_rules/   # YAML detection rules
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   └── components/    # React components
│   └── package.json
├── docker-compose.yml
├── .env.example
└── README.md

Data Sources

Supported Log Types

1. Windows Events (via Winlogbeat)

   • Sysmon (Event ID 1, 3, 7, 8)
   • PowerShell Script Block Logging (4104)
   • Security logs (4624, 4625, 4688)

2. Network Logs

   • Firewall events
   • DNS queries

3. Authentication Logs

   • SSH attempts
   • RDP sessions
   • Web authentication

Adding Custom Sources

Configure your log shipper (Filebeat, Logstash) to send data to Elasticsearch with appropriate index patterns. Update backend/app/services/elasticsearch_service.py to include your indices.

Security

• JWT-based authentication
• bcrypt password hashing
• SQL injection protection via SQLAlchemy ORM
• CORS configuration
• Environment-based secrets management

Important: Always change default credentials and secret keys in production!

Testing

# Backend tests
cd backend
pytest tests/

# API health check
curl http://localhost:8000/health

Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/improvement)
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

Roadmap

• Machine learning-based anomaly detection
• Automated response playbooks
• Threat intelligence feed integration
• Email/Slack notifications
• Custom dashboard builder
• Multi-tenant support
• LDAP/SSO authentication

Known Issues

• First AI query may take 3-5 seconds while model initializes
• Large log volumes (100K+ events) can slow query response times
• Windows console encoding may affect log output formatting

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Built with:

• Groq - High-performance LLM inference
• Elastic - Elasticsearch and Winlogbeat
• MITRE ATT&CK - Threat intelligence framework
• FastAPI - Modern Python web framework
• React - Frontend library

Support

• Open an issue for bugs or feature requests
• Check the API docs at /docs endpoint
• Review configuration examples in .env.example

---

Disclaimer: This tool is for authorized security monitoring only. Always ensure proper authorization before monitoring systems and networks.