diff --git a/README.md b/README.md index 5fd2e02..9aacb76 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,323 @@ -# coolify-migration +# Coolify Migration Script -Conversion of the [popular gist](https://gist.github.com/Geczy/83c1c77389be94ed4709fc283a0d7e23) into a repo to manage PRs and updates. +A comprehensive bash script to backup and migrate your entire Coolify instance from one server to another. This script handles Docker volumes, the Coolify database, SSH keys, and all associated data. -This script will backup your Coolify instance and move everything to a new server. Docker volumes, Coolify database, and ssh keys +## 📋 Table of Contents -1. Script must run on the source server -2. Have all the containers running that you want to migrate +- [Overview](#overview) +- [Features](#features) +- [Prerequisites](#prerequisites) +- [Installation](#installation) +- [Configuration](#configuration) +- [Usage](#usage) +- [What Gets Migrated](#what-gets-migrated) +- [How It Works](#how-it-works) +- [Supported Operating Systems](#supported-operating-systems) +- [Troubleshooting](#troubleshooting) +- [Safety Considerations](#safety-considerations) +- [Contributing](#contributing) +- [License](#license) + +## đŸŽ¯ Overview + +This script automates the complete migration of a Coolify instance, including: +- All Docker volumes from running containers +- Coolify database and configuration files +- SSH authorized keys +- Complete data integrity preservation + +The script runs on the **source server** and transfers everything to the **destination server** via SSH. + +## ✨ Features + +- **Automatic Docker Volume Detection**: Automatically discovers and backs up all volumes from running containers +- **Parallel Compression**: Uses `pigz` (parallel gzip) for faster backups when available, with automatic fallback to `gzip` +- **Auto-Installation**: Can automatically install `pigz` if not present (supports multiple package managers) +- **Interactive Configuration**: Prompts for SSH key and destination host if not pre-configured +- **Comprehensive Error Handling**: Validates prerequisites and provides clear error messages +- **SSH Key Merging**: Safely merges existing SSH keys on destination server +- **Automatic Coolify Installation**: Installs Coolify on the destination server if needed +- **Size Reporting**: Shows total size of data to be migrated before starting +- **Safe Operations**: Includes confirmation prompts for critical operations + +## đŸ“Ļ Prerequisites + +### Source Server Requirements + +- Bash shell +- Docker installed and running +- SSH access to destination server +- Sufficient disk space for backup file +- Root or sudo access (for stopping Docker if needed) + +### Destination Server Requirements + +- Root SSH access +- Sufficient disk space for all migrated data +- Internet connection (for Coolify installation) + +### Network Requirements + +- SSH connectivity from source to destination server +- SSH key-based authentication configured + +## 🚀 Installation + +1. Clone or download this repository: +```bash +git clone https://github.com/rogerb831/coolify-migration.git +cd coolify-migration +``` + +2. Make the script executable: +```bash +chmod +x migrate.sh +``` + +3. Edit the configuration section (optional - you can also be prompted at runtime): +```bash +nano migrate.sh +``` + +## âš™ī¸ Configuration + +The script has two configuration options at the top of the file: + +```bash +sshKeyPath="$HOME/.ssh/your_private_key" # Path to SSH private key +destinationHost="server.example.com" # Destination server hostname/IP +``` + +### Configuration Methods + +**Option 1: Edit the script directly** +- Modify lines 9-10 in `migrate.sh` +- Set your actual SSH key path and destination host + +**Option 2: Interactive prompts** +- Leave the defaults as-is +- The script will prompt you for values at runtime + +### Additional Configuration + +The script uses these default paths (can be modified in the script): +- **Backup source directory**: `/data/coolify/` +- **Backup filename**: `coolify_backup.tar.gz` (created in current directory) + +## 📖 Usage + +### Basic Usage + +1. **Ensure all containers you want to migrate are running** on the source server + +2. **Run the script**: +```bash +./migrate.sh +``` + +3. **Follow the interactive prompts**: + - Configure SSH key and destination (if not pre-configured) + - Choose whether to install `pigz` if not available + - Confirm Docker stop (recommended for data consistency) + - Confirm backup file cleanup after migration + +### Step-by-Step Process + +1. **Configuration Check**: Script verifies or prompts for SSH key and destination host +2. **Pigz Detection**: Checks for `pigz` and offers auto-installation if missing +3. **Prerequisites Validation**: + - Verifies source directory exists + - Checks SSH key file exists + - Tests SSH connectivity to destination +4. **Docker Volume Discovery**: Scans all running containers for volumes +5. **Size Calculation**: Reports total data size to be migrated +6. **Backup Creation**: + - Optionally stops Docker for consistency + - Creates compressed backup archive +7. **Remote Transfer**: + - Transfers backup to destination server + - Extracts files + - Merges SSH keys + - Installs/updates Coolify +8. **Cleanup**: Optionally removes local backup file + +## đŸ“Ļ What Gets Migrated + +The script migrates the following: + +### 1. Coolify Data Directory +- Location: `/data/coolify/` +- Contains: Database, configuration files, application data + +### 2. Docker Volumes +- All volumes attached to running containers +- Location: `/var/lib/docker/volumes/` +- Automatically discovered from running containers + +### 3. SSH Authorized Keys +- Source: `~/.ssh/authorized_keys` +- Safely merged with existing keys on destination server + +## 🔧 How It Works + +### Backup Process + +1. **Volume Discovery**: + - Lists all running Docker containers + - Inspects each container for mounted volumes + - Collects volume paths + +2. **Compression**: + - Uses `pigz` (parallel gzip) if available for faster compression + - Falls back to `gzip` if `pigz` is not available + - Excludes socket files (`*.sock`) from backup + - Suppresses file-changed warnings during compression + +3. **Archive Creation**: + - Creates a tar archive with all data + - Compresses using the selected compressor + - Saves as `coolify_backup.tar.gz` + +### Migration Process + +1. **Transfer**: + - Streams backup file to destination via SSH + - Uses stdin/stdout for efficient transfer + +2. **Extraction**: + - Stops Docker on destination (if running as service) + - Extracts backup archive + - Detects and uses `pigz` for decompression if available + +3. **SSH Key Management**: + - Backs up existing authorized_keys + - Merges with new keys from source + - Removes duplicates + - Sets proper permissions + +4. **Coolify Installation**: + - Installs curl if needed (with OS detection) + - Runs official Coolify installation script + - Ensures Coolify is ready to use + +## đŸ–Ĩī¸ Supported Operating Systems + +The script supports auto-installation of `pigz` on: + +- **Debian/Ubuntu/Raspberry Pi OS**: Uses `apt-get` +- **Red Hat/CentOS/Fedora**: Uses `yum` or `dnf` +- **SUSE/openSUSE**: Uses `zypper` +- **Arch Linux**: Uses `pacman` +- **Alpine Linux**: Uses `apk` + +For other distributions, you can manually install `pigz` or the script will use `gzip` as fallback. + +## 🐛 Troubleshooting + +### Common Issues + +#### "SSH connection failed" +- **Cause**: Network connectivity or authentication issues +- **Solution**: + - Verify destination server is reachable + - Check SSH key permissions: `chmod 600 your_key` + - Test SSH manually: `ssh -i your_key root@destination` + +#### "Source directory does not exist" +- **Cause**: Coolify data directory not at `/data/coolify/` +- **Solution**: Modify `backupSourceDir` variable in the script + +#### "Docker is not installed" +- **Cause**: Docker not in PATH or not installed +- **Solution**: Install Docker or ensure it's in your PATH + +#### "Failed to install pigz" +- **Cause**: Package manager issues or insufficient permissions +- **Solution**: + - Install manually: `sudo apt-get install pigz` (or equivalent) + - Or continue with `gzip` (slower but functional) + +#### "Backup file creation failed" +- **Cause**: Insufficient disk space or permission issues +- **Solution**: + - Check available disk space: `df -h` + - Ensure write permissions in current directory + - Check if backup file already exists and remove if needed + +#### "Container inspection failed" +- **Cause**: Container may have been stopped during migration +- **Solution**: Ensure all containers remain running during volume discovery + +### Getting Help + +If you encounter issues: +1. Check the error messages - they provide specific guidance +2. Verify all prerequisites are met +3. Ensure sufficient disk space on both servers +4. Test SSH connectivity manually before running the script + +## âš ī¸ Safety Considerations + +### Before Migration + +1. **Backup First**: Always have a backup of your data before migration +2. **Test Connectivity**: Verify SSH access works before running the script +3. **Check Disk Space**: Ensure destination has enough space for all data +4. **Stop Services**: Consider stopping non-critical services during migration + +### During Migration + +1. **Don't Interrupt**: Let the script complete - interrupting may leave data in inconsistent state +2. **Monitor Progress**: Watch for error messages +3. **Network Stability**: Ensure stable network connection throughout + +### After Migration + +1. **Verify Data**: Check that all containers and data are present +2. **Test Functionality**: Verify Coolify is working correctly +3. **Clean Up**: Remove backup file after confirming successful migration + +### Important Notes + +- The script **stops Docker** on the destination server during extraction +- Existing SSH keys on destination are **merged**, not replaced +- The script requires **root access** on the destination server +- Socket files (`*.sock`) are **excluded** from backup (they're runtime-only) + +## 🤝 Contributing + +Contributions are welcome! This repository was converted from a [popular gist](https://gist.github.com/Geczy/83c1c77389be94ed4709fc283a0d7e23) to better manage PRs and updates. + +### How to Contribute + +1. Fork the repository +2. Create a feature branch +3. Make your changes +4. Test thoroughly +5. Submit a pull request + +## 📄 License + +This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. + +## 🙏 Acknowledgments + +- Original gist by [Geczy](https://gist.github.com/Geczy/83c1c77389be94ed4709fc283a0d7e23) +- Community contributors who have improved the script + +## 📝 Changelog + +### Recent Improvements + +- Added early `pigz` detection with auto-installation +- Added interactive configuration prompts +- Improved error handling and validation +- Fixed variable quoting issues +- Added comprehensive Docker error handling +- Improved OS detection patterns +- Added fallback for `nproc` command +- Enhanced SSH key merging logic + +--- + +**Note**: Always test the migration process in a non-production environment first to ensure it meets your specific requirements. diff --git a/migrate.sh b/migrate.sh index 457b848..6097a6c 100644 --- a/migrate.sh +++ b/migrate.sh @@ -1,4 +1,5 @@ #!/bin/bash +set -euo pipefail # This script will backup your Coolify instance and move everything to a new server. Docker volumes, Coolify database, and ssh keys @@ -9,9 +10,190 @@ sshKeyPath="$HOME/.ssh/your_private_key" # Key to destination server destinationHost="server.example.com" +# Prompt for configuration if defaults are still set +if [ "$sshKeyPath" = "$HOME/.ssh/your_private_key" ] || [ "$destinationHost" = "server.example.com" ]; then + echo "âš ī¸ Configuration not set. Please provide the following:" + echo "" + + if [ "$sshKeyPath" = "$HOME/.ssh/your_private_key" ]; then + echo "Enter the path to your SSH private key for the destination server:" + read -r sshKeyPath + if [ -z "$sshKeyPath" ]; then + echo "❌ SSH key path cannot be empty" + exit 1 + fi + fi + + if [ "$destinationHost" = "server.example.com" ]; then + echo "Enter the destination server hostname or IP address:" + read -r destinationHost + if [ -z "$destinationHost" ]; then + echo "❌ Destination host cannot be empty" + exit 1 + fi + fi + + echo "" +fi + # -- Shouldn't need to modify anything below -- backupSourceDir="/data/coolify/" backupFileName="coolify_backup.tar.gz" +dockerWasStopped=false + +# Check if pigz (parallel implementation of gzip) is available for faster compression +if ! command -v pigz >/dev/null 2>&1; then + echo "âš ī¸ WARNING: pigz is not installed. The backup will use gzip instead, which may be slower." + echo "" + echo "Do you want to try to auto-install pigz? (y/n)" + read -r install_answer + case "$install_answer" in + [Yy]*) install_pigz=true ;; + *) install_pigz=false ;; + esac + if [ "$install_pigz" = "false" ]; then + # User declined auto-install, ask if they want to continue + echo "" + echo "Do you want to continue without pigz? (y/n)" + read -r answer + case "$answer" in + [Yy]*) + echo "✅ Continuing with gzip..." + ;; + *) + echo "❌ Aborted by user. Please install pigz and try again." + exit 1 + ;; + esac + else + # Try to auto-install pigz + echo "🚸 Attempting to install pigz..." + + # Determine if we need sudo (check if we're root) + if [ "$(id -u)" -eq 0 ]; then + SUDO_CMD="" + else + SUDO_CMD="sudo" + fi + + # Detect OS and install pigz accordingly + if [ -f /etc/debian_version ] || { [ -f /etc/os-release ] && grep -iq "raspbian\|debian\|ubuntu" /etc/os-release; }; then + echo "â„šī¸ Detected Debian-based system" + # shellcheck disable=SC2086 + if $SUDO_CMD apt-get update && $SUDO_CMD apt-get install -y pigz; then + echo "✅ pigz installed successfully" + else + echo "❌ Failed to install pigz on Debian-based system" + echo "" + echo "Do you want to continue without pigz? (y/n)" + read -r answer + case "$answer" in + [Yy]*) + echo "✅ Continuing with gzip..." + ;; + *) + echo "❌ Aborted by user. Please install pigz manually and try again." + exit 1 + ;; + esac + fi + elif [ -f /etc/redhat-release ] || { [ -f /etc/os-release ] && grep -iq "rhel\|centos\|fedora" /etc/os-release; }; then + echo "â„šī¸ Detected Redhat-based system" + # shellcheck disable=SC2086 + if $SUDO_CMD yum install -y pigz 2>/dev/null || $SUDO_CMD dnf install -y pigz; then + echo "✅ pigz installed successfully" + else + echo "❌ Failed to install pigz on Redhat-based system" + echo "" + echo "Do you want to continue without pigz? (y/n)" + read -r answer + case "$answer" in + [Yy]*) + echo "✅ Continuing with gzip..." + ;; + *) + echo "❌ Aborted by user. Please install pigz manually and try again." + exit 1 + ;; + esac + fi + elif [ -f /etc/SuSE-release ] || { [ -f /etc/os-release ] && grep -iq "suse" /etc/os-release; }; then + echo "â„šī¸ Detected SUSE-based system" + # shellcheck disable=SC2086 + if $SUDO_CMD zypper install -y pigz; then + echo "✅ pigz installed successfully" + else + echo "❌ Failed to install pigz on SUSE-based system" + echo "" + echo "Do you want to continue without pigz? (y/n)" + read -r answer + case "$answer" in + [Yy]*) + echo "✅ Continuing with gzip..." + ;; + *) + echo "❌ Aborted by user. Please install pigz manually and try again." + exit 1 + ;; + esac + fi + elif [ -f /etc/arch-release ]; then + echo "â„šī¸ Detected Arch Linux" + # shellcheck disable=SC2086 + if $SUDO_CMD pacman -Sy --noconfirm pigz; then + echo "✅ pigz installed successfully" + else + echo "❌ Failed to install pigz on Arch Linux" + echo "" + echo "Do you want to continue without pigz? (y/n)" + read -r answer + case "$answer" in + [Yy]*) + echo "✅ Continuing with gzip..." + ;; + *) + echo "❌ Aborted by user. Please install pigz manually and try again." + exit 1 + ;; + esac + fi + elif [ -f /etc/alpine-release ]; then + echo "â„šī¸ Detected Alpine Linux" + # shellcheck disable=SC2086 + if $SUDO_CMD apk add --no-cache pigz; then + echo "✅ pigz installed successfully" + else + echo "❌ Failed to install pigz on Alpine Linux" + echo "" + echo "Do you want to continue without pigz? (y/n)" + read -r answer + case "$answer" in + [Yy]*) + echo "✅ Continuing with gzip..." + ;; + *) + echo "❌ Aborted by user. Please install pigz manually and try again." + exit 1 + ;; + esac + fi + else + echo "❌ Unsupported OS. Cannot auto-install pigz." + echo "" + echo "Do you want to continue without pigz? (y/n)" + read -r answer + case "$answer" in + [Yy]*) + echo "✅ Continuing with gzip..." + ;; + *) + echo "❌ Aborted by user. Please install pigz manually and try again." + exit 1 + ;; + esac + fi + fi +fi # Check if the source directory exists if [ ! -d "$backupSourceDir" ]; then @@ -27,79 +209,154 @@ if [ ! -f "$sshKeyPath" ]; then fi echo "✅ SSH key file exists" -# Check if we can SSH to the destination server, ignore "The authenticity of host can't be established." errors -if ! ssh -i "$sshKeyPath" -o "StrictHostKeyChecking no" -o "ConnectTimeout=5" root@$destinationHost "exit"; then +# Check if we can SSH to the destination server (accept-new auto-accepts on first connect, rejects if key changes) +if ! ssh -i "$sshKeyPath" -o "StrictHostKeyChecking accept-new" -o "ConnectTimeout=5" "root@${destinationHost}" "exit"; then echo "❌ SSH connection to $destinationHost failed" exit 1 fi echo "✅ SSH connection successful" -# Get the names of all running Docker containers -containerNames=$(docker ps --format '{{.Names}}') +# Check if the backup file already exists +if [ ! -f "$backupFileName" ]; then + # Get the names of all running Docker containers + if ! command -v docker >/dev/null 2>&1; then + echo "❌ Docker is not installed or not in PATH" + exit 1 + fi -# Initialize an empty string to hold the volume paths -volumePaths="" + if ! containerNames=$(docker ps --format '{{.Names}}' 2>/dev/null); then + echo "❌ Failed to get Docker container list. Is Docker running?" + exit 1 + fi -# Loop over the container names -for containerName in $containerNames; do - # Get the volumes for the current container - volumeNames=$(docker inspect --format '{{range .Mounts}}{{printf "%s\n" .Name}}{{end}}' "$containerName") + # Initialize an empty string to hold the volume paths + volumePaths="" - # Loop over the volume names - for volumeName in $volumeNames; do - # Check if the volume name is not empty - if [ -n "$volumeName" ]; then - # Add the volume path to the volume paths string - volumePaths="$volumePaths /var/lib/docker/volumes/$volumeName" + # Loop over the container names + # shellcheck disable=SC2086 + for containerName in $containerNames; do + # Get the volumes for the current container + if ! volumeNames=$(docker inspect --format '{{range .Mounts}}{{printf "%s\n" .Name}}{{end}}' "$containerName" 2>/dev/null); then + echo "âš ī¸ Warning: Failed to inspect container $containerName, skipping" + continue fi + + # Loop over the volume names + # shellcheck disable=SC2086 + for volumeName in $volumeNames; do + # Check if the volume name is not empty + if [ -n "$volumeName" ]; then + # Add the volume path to the volume paths string + volumePaths="$volumePaths /var/lib/docker/volumes/$volumeName" + fi + done done -done -# Calculate the total size of the volumes -# shellcheck disable=SC2086 -totalSize=$(du -csh $volumePaths 2>/dev/null | grep total | awk '{print $1}') + # Calculate the total size of the volumes + if [ -n "$volumePaths" ]; then + # shellcheck disable=SC2086 + totalSize=$(du -csh $volumePaths 2>/dev/null | grep total | awk '{print $1}') || totalSize="unknown" + else + totalSize="0" + fi -# Print the total size of the volumes -echo "✅ Total size of volumes to migrate: $totalSize" + # Print the total size of the volumes + echo "✅ Total size of volumes to migrate: $totalSize" -# Print size of backupSourceDir -backupSourceDirSize=$(du -csh $backupSourceDir 2>/dev/null | grep total | awk '{print $1}') -echo "✅ Size of the source directory: $backupSourceDirSize" + # Print size of backupSourceDir + backupSourceDirSize=$(du -csh "$backupSourceDir" 2>/dev/null | grep total | awk '{print $1}') || backupSourceDirSize="unknown" + echo "✅ Size of the source directory: $backupSourceDirSize" + + # Check available disk space before creating backup + availableSpace=$(df -P . | awk 'NR==2 {print $4}') + if [ -n "$availableSpace" ] && [ "$availableSpace" -lt 1048576 ] 2>/dev/null; then + availableHuman=$(df -Ph . | awk 'NR==2 {print $4}') + echo "âš ī¸ Low disk space: only ${availableHuman} available in current directory" + echo "Do you want to continue anyway? (y/n)" + read -r answer + case "$answer" in + [Yy]*) echo "🚸 Continuing despite low disk space..." ;; + *) + echo "❌ Aborted. Free up disk space and try again." + exit 1 + ;; + esac + fi -# Check if the backup file already exists -if [ ! -f "$backupFileName" ]; then echo "🚸 Backup file does not exist, creating" # Recommend stopping docker before creating the backup - echo "🚸 It's recommended to stop all Docker containers before creating the backup - Do you want to stop Docker? (y/n)" + echo "🚸 It's recommended to stop all Docker containers before creating the backup" + echo "Do you want to stop Docker? (y/n)" read -r answer - if [ "$answer" != "${answer#[Yy]}" ]; then - if ! systemctl stop docker; then - echo "❌ Docker stop failed" - exit 1 + case "$answer" in + [Yy]*) + if command -v systemctl >/dev/null 2>&1; then + if ! systemctl stop docker; then + echo "❌ Docker stop failed" + exit 1 + fi + dockerWasStopped=true + echo "✅ Docker stopped" + else + echo "âš ī¸ systemctl not found, cannot stop Docker service" + echo "🚸 Continuing with backup (Docker may still be running)" + fi + ;; + *) + echo "🚸 Docker not stopped, continuing with the backup" + ;; + esac + + # Choose compressor + if command -v pigz >/dev/null 2>&1; then + echo "✅ Using pigz for parallel gzip" + # Get number of CPU cores, fallback to 1 if nproc is not available + if command -v nproc >/dev/null 2>&1; then + cores=$(nproc) + else + cores=1 fi - echo "✅ Docker stopped" + compressor="pigz -p${cores}" else - echo "🚸 Docker not stopped, continuing with the backup" + echo "â„šī¸ pigz not found, using gzip" + compressor="gzip" + fi + + # Check if authorized_keys exists locally before attempting to back it up + authKeysPath="$HOME/.ssh/authorized_keys" + authKeysArg="" + if [ -f "$authKeysPath" ]; then + authKeysArg="$authKeysPath" fi + rc=0 # shellcheck disable=SC2086 - if ! tar --exclude='*.sock' -Pczf $backupFileName -C / $backupSourceDir $HOME/.ssh/authorized_keys $volumePaths; then + tar --exclude='*.sock' --warning=no-file-changed -I "$compressor" -Pcf "${backupFileName}" \ + -C / "$backupSourceDir" ${authKeysArg:+"$authKeysArg"} ${volumePaths:+$volumePaths} || rc=$? + if [ "$rc" -gt 1 ]; then echo "❌ Backup file creation failed" exit 1 fi - echo "✅ Backup file created" + echo "✅ Backup file created (with change warnings suppressed)" else echo "🚸 Backup file already exists, skipping creation" + # Check if Docker is stopped from a previous failed run + if command -v systemctl >/dev/null 2>&1 && ! systemctl is-active --quiet docker 2>/dev/null; then + echo "âš ī¸ Docker appears to be stopped on this (source) server." + echo "It may have been stopped by a previous run. Noting for restart prompt later." + dockerWasStopped=true + fi fi # Define the remote commands to be executed remoteCommands=" + set -euo pipefail + # Check if Docker is a service - if systemctl is-active --quiet docker; then + if systemctl is-active --quiet docker /dev/null || dnf install -y curl /dev/null 2>&1; then + if ! systemctl start docker; then + echo "❌ Failed to restart Docker on source server" + else + echo "✅ Docker restarted on source server" + fi + else + echo "âš ī¸ systemctl not found, cannot restart Docker service" + fi + ;; + *) + echo "🚸 Docker left stopped on source server" + ;; + esac +fi + # Clean up - Ask the user for confirmation before removing the local backup file echo "Do you want to remove the local backup file? (y/n)" read -r answer -if [ "$answer" != "${answer#[Yy]}" ]; then - if ! rm -f $backupFileName; then - echo "❌ Failed to remove local backup file" - exit 1 - fi - echo "✅ Local backup file removed" -else - echo "🚸 Local backup file not removed" -fi +case "$answer" in + [Yy]*) + if ! rm -f "${backupFileName}"; then + echo "❌ Failed to remove local backup file" + exit 1 + fi + echo "✅ Local backup file removed" + ;; + *) + echo "🚸 Local backup file not removed" + ;; +esac + +echo "" +echo "✅ Migration complete! Your Coolify instance has been migrated to ${destinationHost}."