A modular monitoring and alerting system for Storj storage node operators. Get email notifications when your nodes have issues, with automatic self-repair capabilities.
┌─────────────────────────────────────────────────────────┐
│ CRON SCHEDULER │
│ (runs every X min) │
└───────────────────────────┬─────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ run_checks.sh │
│ (Main Entry Point) │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ check_logs │ │ check_nodes │ │ check_disks │ │ check_noip │ Loaded from │
│ │ .sh │ │ .sh │ │ .sh │ │ .sh │ ◄── config.env │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ + nodes.json │
│ │ │ │ │ │
└──────────┼──────────────────┼──────────────────┼──────────────────┼─────────────────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌────────────────┐ ┌──────────────────┐ ┌────────────────┐ ┌────────────────┐
│ Docker Logs │ │ HTTP Request │ │ Disk Mount │ │ Public IP │
│ Analysis │ │ to :14002 etc │ │ Check │ │ vs NOIP DNS │
│ │ │ │ │ │ │ │
│ • Error count │ │ ┌──────────────┐ │ │ • /mnt/storj* │ │ • ipecho.net │
│ • FATAL/WARN │ │ │ Node 1 │ │ │ • mountpoint │ │ • noip2 -S │
│ • % threshold │ │ │ Node 2 │ │ │ • lsblk │ │ │
│ │ │ │ Node N... │ │ │ │ │ │
└───────┬────────┘ └ └──────────────┘ ┘ └───────┬────────┘ └───────┬────────┘
│ │ │ │
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ ISSUE DETECTED? │
│ │
│ IS_NODE_OPERATOR=true IS_NODE_OPERATOR=false │
│ ┌─────────────────────────────┐ ┌─────────────────────────────┐ │
│ │ SELF-REPAIR │ │ ALERT ONLY │ │
│ │ • docker restart container │ │ • No local access │ │
│ │ • mount -a (remount disks) │ │ • HTTP checks only │ │
│ │ • noip2 restart │ │ • Remote monitoring │ │
│ └─────────────────────────────┘ └─────────────────────────────┘ │
└──────────────────────────────────┬──────────────────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ AGGREGATE ALERTS │
│ (collect all issues) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ SEND EMAIL (ssmtp) │
│ │
│ To: ALERT_EMAIL │
│ Subject: X issues detected │
│ Body: Details + logs │
└──────────────────────────────┘
SCENARIO A: Node Operator Mode SCENARIO B: Central Monitor
(Install on each Storj node) (Install on separate VPS)
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ Raspberry Pi / Server │ │ Monitoring VPS │
│ ┌───────────────────────┐ │ │ │
│ │ StorjMailAlert │ │ │ ┌───────────────────────┐ │
│ │ IS_NODE_OPERATOR=true│ │ │ │ StorjMailAlert │ │
│ └───────────┬───────────┘ │ │ │ IS_NODE_OPERATOR=false│ │
│ │ │ │ └───────────┬───────────┘ │
│ ▼ │ │ │ │
│ ┌───────────────────────┐ │ │ │ HTTP only │
│ │ Local Checks: │ │ │ ▼ │
│ │ • Docker logs │ │ └──────────────┼──────────────┘
│ │ • Disk mounts │ │ │
│ │ • Self-repair │ │ ┌──────────────┴──────────────┐
│ └───────────────────────┘ │ │ │
│ + │ ▼ ▼ ▼
│ ┌───────────────────────┐ │ ┌────────┐ ┌────────┐ ┌────────┐
│ │ Remote Checks: │ │ │ Node 1 │ │ Node 2 │ │ Node 3 │
│ │ • HTTP status (as VPS)| │ │ :14002 │ │ :14002 │ │ :14002 │
│ └───────────────────────┘ │ └────────┘ └────────┘ └────────┘
└─────────────────────────────┘
- Docker Log Analysis - Monitor container logs for errors with configurable thresholds
- Node HTTP Status - Check node dashboard accessibility across multiple hosts
- Disk Mount Monitoring - Verify storage disks are properly mounted
- NOIP Dynamic DNS - Check and auto-repair dynamic DNS configuration
- Self-Repair - Automatic container restart and mount recovery
- Multi-Node Support - Monitor multiple nodes across different machines
- Flexible Deployment - Run as a node operator or as a remote monitor
- Ansible Deployment - Automated multi-node deployment with a single command
Install on a separate machine (VPS, home server) to monitor nodes remotely. Enables:
- HTTP status checking for all configured nodes
- Email alerts when nodes are unreachable
Install on each machine running Storj nodes. Enables all of the above plus:
- Local docker log checking
- Local disk mount monitoring
- Self-repair (auto-restart containers, remount disks, fix NOIP)
Preferences can be set per check to enable/disable features as needed.
ssmtp- For sending email alerts (configured with your SMTP server)jq- For parsing JSON configurationbc- For calculationscurl- For HTTP checks and IP detectioncrontab- For scheduled execution
nmap- For SSH port checkingnoip2- For dynamic DNS management with NOIP
Install dependencies:
sudo apt install ssmtp jq bc curl nmapgit clone https://github.com/tachyontec/storjMailAlert
cd storjMailAlertchmod +x .sh
./setup.shThe setup wizard will guide you through:
- Email configuration
- Check intervals and thresholds
- Node operator mode selection
- Feature toggles
- Cron job installation
Alternative: Manually copy and edit the configuration files:
cp config/config.env.example config.env
cp config/nodes.json.example nodes.json
# Edit config.env and nodes.json with your settingsEdit nodes.json with your node details:
{
"servers": [
{
"name": "MyServerNode",
"host": "myservernode.ddns.net",
"sshPort": 22,
"nodes": [
{ "port": 14002 }
]
},
{
"name": "MyMultiNode",
"host": "mymultinode.ddns.net",
"sshPort": 22,
"nodes": [
{ "port": 14002 },
{ "port": 14003, "containerName": "my-custom-node", "mountPoint": "/mnt/my-data" }
]
}
]
}Server fields:
name- Hostname identifier (should match/etc/hostnamefor self-repair)host- Domain or IP addresssshPort- SSH port for host-level checks
Node fields (per-node within a server):
port- (required) Node dashboard port to checkcontainerName- (optional) Docker container name. Falls back to naming convention if omitted.mountPoint- (optional) Disk mount path. Falls back to naming convention if omitted.
./run_checks.sh
# Or wait for the cron job to run automaticallystorjMailAlert/
├── ansible/
│ ├── generated/ # Generated configs (git-ignored)
│ ├── roles/
│ │ └── storjMailAlert/
│ │ ├── defaults/main.yml # Role default variables
│ │ └── tasks/main.yml # Deployment tasks
│ ├── inventory.ini.example # Inventory template
│ └── playbook.yml # Ansible playbook
├── config/
│ ├── config.env.example # Configuration template
│ └── nodes.json.example # Nodes list template
├── lib/
│ ├── common.sh # Shared functions
│ └── self_repair.sh # Self-repair functions
├── checks/
│ ├── check_logs.sh # Docker log analysis
│ ├── check_nodes.sh # Node HTTP status
│ ├── check_disks.sh # Disk mount checker
│ └── check_noip.sh # Dynamic DNS checker
├── logs/ # Log output directory
├── tmp/ # Temporary files
├── config.env # Your configuration (git-ignored)
├── nodes.json # Your nodes list (git-ignored)
├── run_checks.sh # Main entry point
├── setup.sh # Single-node setup wizard
├── setup-multi.sh # Multi-node Ansible setup wizard
└── README.md
| Variable | Description | Default |
|---|---|---|
ALERT_EMAIL |
Email to receive alerts | (required) |
CHECK_INTERVAL |
Minutes between checks | 15 |
ERROR_PERCENT_THRESHOLD |
Error % to trigger alert | 5 |
IS_NODE_OPERATOR |
Enable local checks/repair | true |
ENABLE_LOG_CHECK |
Check docker logs | true |
ENABLE_NODE_CHECK |
Check node HTTP status | true |
ENABLE_DISK_CHECK |
Check disk mounts | true |
ENABLE_NOIP_CHECK |
Check NOIP DNS | false |
CONTAINER_PREFIX |
Docker container prefix | storagenode |
DEBUG |
Enable debug logging | false |
By default, StorjMailAlert expects the following naming pattern for Docker containers and disk mount points:
| Node # | Container Name | Mount Point |
|---|---|---|
| 1st | storagenode |
/mnt/storj |
| 2nd | storagenode2 |
/mnt/storj2 |
| 3rd | storagenode3 |
/mnt/storj3 |
| Nth | storagenodeN |
/mnt/storjN |
- The first node has no numeric suffix (e.g.
storagenode,/mnt/storj) - All subsequent nodes append their index (e.g.
storagenode2,/mnt/storj2) - The container prefix is configurable via
CONTAINER_PREFIX(default:storagenode)
It is highly recommended to follow this convention for simplicity. However, custom names can be specified as needed.
Default example:
"nodes": [
{ "port": 14002 }, // uses storagenode, /mnt/storj
{ "port": 14003 } // uses storagenode2, /mnt/storj2
]
}
Custom names: If the default convention doesn't match your setup, you can override the container name and mount point per node in nodes.json using the optional containerName and mountPoint fields:
{
"nodes": [
{ "port": 14002 }, // uses storagenode, /mnt/storj
{ "port": 14003, "containerName": "my-node-2", "mountPoint": "/mnt/my-data" }
]
}When these fields are omitted, the default convention is used. When provided, they take priority. Self-repair, log checks, and disk checks all respect custom names.
| Feature | IS_NODE_OPERATOR=true | IS_NODE_OPERATOR=false |
|---|---|---|
| Log Check | Runs | Skipped |
| Disk Check | Runs | Skipped |
| Node Check | Runs + self-repair | Runs (HTTP only) |
| NOIP Check | Runs + auto-restart | Check only |
./run_checks.sh./run_checks.sh --check logs
./run_checks.sh --check nodes
./run_checks.sh --check disks
./run_checks.sh --check noiptail -f logs/storjMailAlert.log
tail -f logs/cron.log./setup.shOn each Pi (node operator mode):
# config.env
ALERT_EMAIL="you@example.com"
IS_NODE_OPERATOR=true
ENABLE_LOG_CHECK=true
ENABLE_NODE_CHECK=true
ENABLE_DISK_CHECK=trueEach Pi monitors itself with full self-repair capabilities.
On a separate VPS (monitor-only mode):
# config.env
ALERT_EMAIL="you@example.com"
IS_NODE_OPERATOR=false
ENABLE_LOG_CHECK=false
ENABLE_NODE_CHECK=true
ENABLE_DISK_CHECK=false// nodes.json - monitors all your nodes remotely
{
"servers": [
{"name": "storjNode1", "host": "storjNode1.ddns.net", "sshPort": 22, "nodes": [{"port": 14002}]},
{"name": "storjNode2", "host": "storjNode2.ddns.net", "sshPort": 22, "nodes": [{"port": 14002}]},
{"name": "storjNode3", "host": "storjNode3.ddns.net", "sshPort": 22, "nodes": [{"port": 14002}, {"port": 14003}]}
]
}For operators running Storj nodes across multiple machines, StorjMailAlert includes an Ansible-based deployment system. A single setup wizard collects all node information, generates configuration files, and deploys to every machine automatically.
All nodes receive the same nodes.json, so every node monitors every other node via HTTP checks while also performing local checks (docker logs, disk mounts, self-repair) on itself.
- Ansible installed on the control machine (the machine you run setup from)
# Install via pip (recommended) pip install ansible # Or via apt sudo apt install ansible
- SSH public key authentication to all target machines (recommended)
Set up passwordless SSH access to all your nodes:
# Generate a key pair (if you don't have one)
ssh-keygen -t ed25519
# Copy your public key to each node
ssh-copy-id user@node1.ddns.net
ssh-copy-id user@node2.ddns.net
ssh-copy-id user@node3.ddns.net
# Verify you can connect without a password
ssh user@node1.ddns.net hostname# Run the multi-node setup wizard
chmod +x setup-multi.sh
./setup-multi.shThe wizard will:
- Collect common settings (email, check interval, thresholds, feature toggles)
- Collect server and node details for each machine
- Generate per-host
config.envfiles, a sharednodes.json, and an Ansible inventory - Optionally run the Ansible playbook to deploy everything
If you prefer to run the playbook separately or re-deploy after changes:
# Dry run (shows what would change without modifying anything)
ansible-playbook ansible/playbook.yml -i ansible/inventory.ini --check
# Full deployment
ansible-playbook ansible/playbook.yml -i ansible/inventory.ini
# Deploy to a specific host only
ansible-playbook ansible/playbook.yml -i ansible/inventory.ini --limit node1If the machine you're running the setup from is also a Storj node, the wizard will ask if each server is the current machine. When you answer yes, it sets ansible_connection=local in the inventory so Ansible deploys to that host directly without SSH.
On each target host, the Ansible playbook will:
- Install dependencies (
ssmtp,jq,bc,curl) - Clone or update the StorjMailAlert repository to
~/storjMailAlert - Copy the shared
nodes.json - Copy the host-specific
config.env - Set file permissions
- Install the cron job
- Run a smoke test to verify the installation
To update StorjMailAlert on all nodes after pulling new changes or modifying configuration:
# Re-run the playbook (pulls latest code, re-deploys configs)
ansible-playbook ansible/playbook.yml -i ansible/inventory.iniTo reconfigure from scratch:
./setup-multi.shIf you prefer not to use setup-multi.sh, you can create the files manually:
- Copy
ansible/inventory.ini.exampletoansible/inventory.iniand add your hosts - Create
ansible/generated/nodes.jsonwith your node list - Create
ansible/generated/<hostname>/config.envfor each host - Run the playbook:
ansible-playbook ansible/playbook.yml -i ansible/inventory.ini
- Check ssmtp configuration:
/etc/ssmtp/ssmtp.conf - Test ssmtp:
echo "test" | ssmtp your@email.com - Check logs:
tail logs/storjMailAlert.log
- Verify cron is installed:
crontab -l - Check cron logs:
grep CRON /var/log/syslog - Run manually:
./run_checks.sh
chmod +x run_checks.sh setup.sh
chmod +x checks/*.sh lib/*.shBy default, Docker requires root privileges to access logs. If you are running the script as a non-root user, add your user to the docker group:
sudo usermod -aG docker $USERMIT License
Pull requests welcome! Please ensure your changes:
- Follow the existing code style
- Include appropriate error handling
- Update documentation as needed
- Use descriptive commit messages
- Branch names should follow conventions:
feature/your-feature-namefix/your-bugfix-namedocs/update-readme