ipset-blacklist Python Implementation

A fast, enhanced Python implementation inspired by trick77/ipset-blacklist.

This is a complete rewrite in Python with no code from the original shell script. It maintains compatibility with the same configuration file format while providing significant enhancements.

Features

Core Functionality

• Fetches IP blocklists from URLs or local files
• Normalizes to CIDRs (/32, /128 for hosts)
• Advanced deduplication: Removes exact duplicates AND covered subnets (O(N·P) algorithm)
• Dual backend: Native nftables support with auto-detection, plus legacy ipset+iptables
• Generates ipset restore or nft batch files
• Atomic apply with --apply flag (ipset swap or nft batch)
• Import/export between ipset and nft formats

Major Enhancements Over Original

• Smart subnet optimization: Removes IPs/subnets covered by broader ranges (original only removes exact duplicates)
• Full IPv6 support with dual-stack handling (original is IPv4-only)
• Analysis mode: Audit existing ipset files for duplicates and covered subnets
• Dry-run mode: Test changes without modifying the system
• Private IP filtering: Configurable filtering of RFC1918, loopback, multicast ranges
• Retry logic: Automatic retry with exponential backoff for network failures
• Proper logging: Configurable verbosity levels instead of just echo
• Progress indicators: Won't flood SSH connections
• Configuration validation: Warns about potential issues
• Security: No shell injection vulnerabilities, fail-closed fetch, input validation
• nftables migration: Three-phase migrate/rollback/finalize with dual-write coexistence

Drop-in Replacement

The Python script is designed as a drop-in replacement for the original shell script:

# Original cron job:
/usr/local/sbin/update-blacklist.sh /etc/ipset-blacklist/ipset-blacklist.conf

# New cron job (just change the script name):
/usr/local/sbin/update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf

# Or use the drop-in wrapper (same name as original):
/usr/local/sbin/update-blacklist.sh /etc/ipset-blacklist/ipset-blacklist.conf

• Reads the same configuration file format
• Produces compatible ipset restore files (or nft batch files with --backend nft)
• Supports all original config variables (BLACKLISTS, MAXELEM, HASHSIZE, etc.)
• Adds --force flag equivalent to FORCE=yes in original
• Includes update-blacklist.sh wrapper for cron jobs that call the original script name

Requirements

• Python 3.7+
• nftables (recommended) or ipset (v6+ recommended) + iptables
• No Python packages required - uses only standard library

Development

# Run tests
python3 -m unittest test_update_blacklist -v

# Activate the pre-commit hook (run once after cloning)
git config core.hooksPath .githooks

Installation

# Automated install (stamps version, installs man page, runs preflight checks)
sudo ./deploy.sh

# Or install manually
sudo install -m 0755 update_blacklist.py /usr/local/sbin/update_blacklist.py
sudo install -d /usr/local/share/man/man8
sudo install -m 0644 update_blacklist.8 /usr/local/share/man/man8/
sudo mandb

# Use existing config from original ipset-blacklist
# Config at: /etc/ipset-blacklist/ipset-blacklist.conf

Basic Usage

Generate optimized restore file (no changes to firewall)

sudo update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf

Apply to firewall (atomic swap)

sudo update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf --apply

Test mode (dry run)

sudo update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf --dry-run --verbose

IPv4-only mode (like original)

sudo update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf --apply --ipv4-only

New Features

Analysis Mode

Analyze existing ipset for duplicates and covered subnets:

# Save current set
sudo ipset save blacklist > blacklist.dump

# Analyze
sudo update_blacklist.py --analyze blacklist.dump --set blacklist --show-removed

# Output:
# Total adds: 77283
# Unique adds: 64715
# Exact duplicates removed: 0
# Covered subnets removed: 12568

Extract clean CIDR list:

sudo update_blacklist.py --analyze blacklist.dump --set blacklist --format cidr > clean.txt

Private IP Filtering

By default, filters RFC1918 and reserved ranges. Disable with:

sudo update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf --no-filter-private

Enhanced Options

• --version - Show version number and exit
• --dry-run - Simulate without making changes
• --force - Create ipsets/rules if missing (like FORCE=yes)
• --verbose / --quiet - Control output verbosity
• --progress - Show progress bars
• --collapse - Additional CIDR aggregation
• --show-removed - Report what was deduplicated
• --allow-partial - Continue even if >50% of sources fail to fetch
• --backend {ipset,nft,auto} - Force a specific firewall backend
• --nft-table / --nft-set-v4 / --nft-set-v6 - Override nft table/set names
• --analyze-format {ipset,nft,auto} - Force parser for --analyze input
• --import-ipset FILE - Convert ipset dump to nft batch format
• --export-ipset FILE - Convert nft JSON dump to ipset restore format

Cron Setup

Update your cron to use the Python script:

# /etc/cron.d/update-blacklist
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

33 23 * * * root /usr/local/sbin/update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf --apply --ipv4-only >/dev/null

Memory Usage

• 100k IPs: ~35 MB
• 400k IPs: ~140 MB
• 800k IPs: ~280 MB
• 1.6M IPs: ~560 MB

Suitable for any system with 1GB+ RAM.

Performance

• O(N·P) optimization algorithm (P ≤ 32 for IPv4, ≤ 128 for IPv6)
• Processes 800k entries in seconds
• 3x retry with exponential backoff for reliability
• Progress updates at 5% intervals (SSH-friendly)

Credits

Inspiration: The original trick77/ipset-blacklist shell script provided the concept and configuration format.

Python implementation by:

• Kenneth Shane Hartman (kshartman @ GitHub, shane@ai.mit.edu)
• ChatGPT (OpenAI) - Optimization algorithms and initial implementation
• Claude (Anthropic) - Code improvements and documentation

Documentation

• man update_blacklist — full man page with all options, config keys, and examples (installed by deploy.sh)

License

MIT License - See LICENSE file for details.

This is a complete reimplementation in Python with no code from the original shell script.

nftables Migration

Migrate from ipset+iptables to nftables with zero downtime. Cron jobs keep running unchanged throughout.

Step 1: Migrate (both backends coexist)

sudo ./migrate-to-nftables.sh --conf /etc/ipset-blacklist/ipset-blacklist.conf

This creates an nft table alongside the existing ipset sets. update_blacklist.py auto-detects nft and dual-writes both backends on every cron run, keeping ipset fresh for rollback.

Step 2: Monitor

# Verify nft is being used
sudo nft list table inet blacklist

# Check cron logs
grep update_blacklist /var/log/syslog

Step 3: Finalize (or rollback)

# When confident:
sudo ./migrate-to-nftables.sh --finalize

# Or to revert:
sudo ./migrate-to-nftables.sh --rollback

Manual backend override

# Force nft in write-only mode
sudo update_blacklist.py --conf /etc/ipset-blacklist/ipset-blacklist.conf --backend nft

# Convert existing ipset dump to nft
sudo update_blacklist.py --import-ipset blacklist.dump --out blacklist.nft

# Convert nft JSON to ipset restore
sudo update_blacklist.py --export-ipset nft-dump.json --out blacklist.restore

Migration from Shell Script

1. Test first: Run with --dry-run to verify
2. Compare output: Check restore file matches expected format
3. Update cron: Change update-blacklist.sh to update_blacklist.py
4. Keep config: Same config file works unchanged
5. Monitor initially: Check logs after first few runs

The Python version will produce slightly different (better) results due to covered subnet removal, but the format and structure remain fully compatible.