How to Engineer Automated 3-2-1 Backups on Netcup: SCP Snapshots, Atomic DB Dumps & Encrypted Restic

Inside the Netcup Server Control Panel (SCP) -> Snapshots tab, you can trigger either an Online Snapshot (while the VM is running) or an Offline Snapshot (which gracefully shuts down the VM, snapshots the block image, and boots it back up). Understanding what happens inside the Linux kernel during an Online Snapshot is critical for database integrity.

Without qemu-guest-agent running inside your VM, an Online Snapshot is purely crash-consistent—equivalent to yanking the power cord on a physical server and booting from the resulting disk state. When qemu-guest-agent is active over /dev/virtio-ports/org.qemu.guest_agent.0, the Netcup KVM hypervisor issues a guest-fsfreeze-freeze command before snapping. The Linux kernel executes FIFREEZE ioctls across mounted filesystems, flushing dirty page-cache buffers to /dev/vda, syncing journaled metadata (ext4/xfs), and pausing new filesystem writes for the sub-second duration of the Copy-on-Write pointer flip before calling guest-fsfreeze-thaw.

Because Netcup KVM storage uses Copy-on-Write (CoW) block tracking, taking a snapshot freezes the current read-only base blocks and writes all subsequent disk modifications into a delta layer within your server’s allocated storage quota. Consequently, if your 256 GB Root Server disk is already 90% full (230 GB used), SCP will refuse to create a snapshot—or worse, heavy database write churn while a snapshot is active will exhaust the remaining 26 GB of CoW delta headroom.

# 1. Install and verify qemu-guest-agent so SCP can issue fsfreeze during online snapshots
sudo apt update && sudo apt install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent
sudo systemctl status qemu-guest-agent --no-pager

# 2. Check filesystem utilization and discard unused blocks (fstrim) so CoW snapshots stay compact
df -h /
sudo fstrim -av

Active: active (running)
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       252G   41G  201G  17% /
/: 198.4 GiB (213031256064 bytes) trimmed on /dev/vda1

Restic is a statically compiled Go backup tool designed with cryptography first: every file is split into variable-length content-defined chunks (512 KiB to 8 MiB average 1 MiB) using sliding-window Rabin fingerprints, compressed with Zstandard (zstd), and encrypted client-side on your Netcup server using AES-256-CTR authenticated with Poly1305-AES MAC tags before a single byte leaves your network interface.

Why does content-defined chunking via Rabin fingerprints matter for daily server backups? Traditional fixed-offset block backup tools fail to deduplicate a 10 GB database dump or tarball if a single 100-byte row is inserted near the beginning of the file, because every subsequent 4 MiB block boundary shifts by 100 bytes. Restic determines chunk boundaries dynamically based on rolling polynomial hashes of the data stream itself. Even if new rows shift offsets inside a daily PostgreSQL dump, Restic recognizes the unchanged downstream table chunks and uploads only the newly mutated megabytes to S3.

On smaller 4 GB RAM Netcup instances (such as a VPS 500 G12.5), Go’s default garbage collector target (GOGC=100) allows Restic’s in-memory deduplication index to grow up to double the live heap size before triggering a sweep. Exporting GOGC=20 and GOMAXPROCS=2 inside /etc/restic/backup.env forces frequent incremental heap sweeps, keeping Restic’s peak RSS memory footprint under 350 MB even across repositories with hundreds of thousands of files.

# 1. Install restic and update to the latest official binary
sudo apt update && sudo apt install -y restic bzip2 postgresql-client
sudo restic self-update

# 2. Create a root-only configuration directory
sudo mkdir -p /etc/restic /var/backups/db-dumps
sudo chmod 0700 /etc/restic /var/backups/db-dumps

# 3. Generate a high-entropy 48-byte repository encryption passphrase
openssl rand -base64 48 | sudo tee /etc/restic/repo.pw > /dev/null
sudo chmod 0400 /etc/restic/repo.pw

# 4. Configure off-site S3-compatible (or SFTP) repository credentials
sudo tee /etc/restic/backup.env > /dev/null <<'EOF'
export RESTIC_REPOSITORY="s3:https://s3.eu-central-003.backblazeb2.com/netcup-prod-backups/rs1000-nue"
export RESTIC_PASSWORD_FILE="/etc/restic/repo.pw"
export AWS_ACCESS_KEY_ID="YOUR_S3_KEY_ID"
export AWS_SECRET_ACCESS_KEY="YOUR_S3_APPLICATION_KEY"
export RESTIC_COMPRESSION="auto"
export GOGC=20
export GOMAXPROCS=2
EOF
sudo chmod 0400 /etc/restic/backup.env

# 5. Initialize the encrypted off-site Restic repository
sudo bash -c 'source /etc/restic/backup.env && restic init'

created restic repository 9c4e18f2a0 at s3:https://s3.eu-central-003.backblazeb2.com/netcup-prod-backups/rs1000-nue

Even with qemu-guest-agent filesystem freeze, copying raw database binary directories (/var/lib/docker/volumes/.../pgdata or /var/lib/mysql) via a file-level scanner like Restic while PostgreSQL or MariaDB is actively mutating B-tree pages and Write-Ahead Log (pg_wal) segments produces torn pages because Restic reads files sequentially over several minutes. Restoring such a backup frequently fails with PANIC: could not locate a valid checkpoint record.

Instead, write a unified backup orchestrator (/usr/local/sbin/netcup-restic-backup) that first creates transactionally consistent logical dumps inside /var/backups/db-dumps. Running pg_dump -Fc opens a single REPEATABLE READ Multi-Version Concurrency Control (MVCC) transaction snapshot in PostgreSQL without locking concurrent web application reads or writes. Notice that we pass --Z=0 (or light compression) to pg_dump when feeding into Restic so that uncompressed table streams align cleanly with Restic’s Rabin fingerprint chunking and native Zstandard (RESTIC_COMPRESSION=auto) engine.

At the end of every nightly run, the script executes restic check --read-data-subset=5%. While a basic restic check only verifies that pack files exist in the S3 bucket listing, adding --read-data-subset=5% downloads a random 5% sample of encrypted remote data packs, verifies their SHA-256 content hashes and Poly1305 authentication tags, and ensures bit-rot or silent S3 object corruption never goes undetected.

sudo tee /usr/local/sbin/netcup-restic-backup > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

# Prevent overlapping backup runs using flock
LOCKFILE="/var/run/netcup-restic-backup.lock"
exec 9>"${LOCKFILE}"
if ! flock -n 9; then
  echo "Another backup process is currently running. Exiting."
  exit 1
fi

source /etc/restic/backup.env
DUMP_DIR="/var/backups/db-dumps"
mkdir -p "${DUMP_DIR}"
chmod 0700 "${DUMP_DIR}"

echo "==> [1/4] Dumping running Docker PostgreSQL & MariaDB databases atomically..."
# Dump any running Postgres container labeled or named with postgres/db
for PG_CONTAINER in $(docker ps --filter "ancestor=postgres:16.4-alpine3.20" --format "{{.Names}}"); do
  echo "  -> Dumping PostgreSQL container: ${PG_CONTAINER}"
  PG_USER=$(docker inspect "${PG_CONTAINER}" --format '{{range .Config.Env}}{{println .}}{{end}}' | awk -F= '/^POSTGRES_USER=/ {print $2}')
  PG_DB=$(docker inspect "${PG_CONTAINER}" --format '{{range .Config.Env}}{{println .}}{{end}}' | awk -F= '/^POSTGRES_DB=/ {print $2}')
  docker exec "${PG_CONTAINER}" pg_dump -U "${PG_USER:-postgres}" -d "${PG_DB:-postgres}" -Fc --Z=0 \
    > "${DUMP_DIR}/${PG_CONTAINER}_${PG_DB}.pgdump.tmp"
  mv "${DUMP_DIR}/${PG_CONTAINER}_${PG_DB}.pgdump.tmp" "${DUMP_DIR}/${PG_CONTAINER}_${PG_DB}.pgdump"
done

echo "==> [2/4] Executing encrypted Restic backup to off-site repository..."
restic backup \
  /etc \
  /opt/stacks \
  /var/backups/db-dumps \
  /home \
  /root \
  --tag "automated-daily" \
  --exclude="/var/lib/docker/overlay2" \
  --exclude="**/node_modules" \
  --exclude="**/.cache" \
  --exclude="**/*.log" \
  --one-file-system

echo "==> [3/4] Applying GFS retention policy (7 daily, 4 weekly, 6 monthly)..."
restic forget \
  --tag "automated-daily" \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 6 \
  --prune

echo "==> [4/4] Verifying repository structure and 5% random data pack subset..."
restic check --read-data-subset=5%

echo "==> Backup & verification completed successfully at $(date -u +"%Y-%m-%dT%H:%M:%SZ")"
EOF

sudo chmod 0700 /usr/local/sbin/netcup-restic-backup

Unlike legacy /etc/crontab entries that silently fail when an environment variable is missing or saturate CPU cores during a traffic spike, a systemd service unit allows us to run Restic under the lowest Linux Completely Fair Scheduler priority (Nice=19) and idle block I/O scheduling class (IOSchedulingClass=idle). Under IOSchedulingClass=idle, the kernel block layer only dispatches Restic read requests to /dev/vda when no normal-priority process (such as PostgreSQL or Caddy) has pending I/O.

Furthermore, adding RandomizedDelaySec=1800 inside restic-backup.timer spreads nightly executions across a 30-minute window after 03:15 AM. If you manage multiple Netcup VPS or Root Servers backing up to the same S3 bucket or Hetzner Storage Box, jitter prevents all nodes from saturating upstream S3 API rate limits simultaneously.

# /etc/systemd/system/restic-backup.service
[Unit]
Description=Encrypted Off-Site Restic Backup & DB Dump for Netcup Server
After=network-online.target docker.service
Wants=network-online.target

[Service]
Type=oneshot
User=root
Nice=19
IOSchedulingClass=idle
IOSchedulingPriority=7
ExecStart=/usr/local/sbin/netcup-restic-backup
StandardOutput=journal
StandardError=journal

sudo tee /etc/systemd/system/restic-backup.timer > /dev/null <<'EOF'
[Unit]
Description=Run Netcup Restic Backup Daily at 03:15 AM with Randomized Jitter

[Timer]
OnCalendar=*-*-* 03:15:00
RandomizedDelaySec=1800
Persistent=true

[Install]
WantedBy=timers.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer

# Trigger an immediate manual run and follow the systemd journal output
sudo systemctl start restic-backup.service
sudo journalctl -u restic-backup.service -n 40 --no-pager
sudo systemctl list-timers restic-backup.timer

NEXT                        LEFT          LAST                        PASSED       UNIT                ACTIVATES
Wed 2026-11-04 03:28:14 CET 6h left       Tue 2026-11-03 20:45:10 CET 12s ago      restic-backup.timer restic-backup.service

An untested backup pipeline is merely a hypothesis. Schedule a quarterly 10-minute restore drill to verify that you can browse snapshots, extract a single accidentally deleted configuration file into a staging directory (/tmp/restore-drill), and inspect or restore a .pgdump archive into PostgreSQL using pg_restore.

Because pg_dump -Fc creates a binary custom-format archive containing both schema DDL and table row data, pg_restore allows you to either restore the entire database cleanly (--clean --if-exists --no-owner) or selectively extract a single accidentally truncated table (pg_restore --table=orders) in seconds without rolling back the rest of your production database.

# 1. List all encrypted snapshots stored in your remote repository
sudo bash -c 'source /etc/restic/backup.env && restic snapshots'

# 2. Restore /opt/stacks and /var/backups/db-dumps from the "latest" snapshot into /tmp/restore-drill
sudo mkdir -p /tmp/restore-drill
sudo bash -c 'source /etc/restic/backup.env && restic restore latest --target /tmp/restore-drill --include /opt/stacks --include /var/backups/db-dumps'

# 3. Verify integrity of the restored PostgreSQL custom dump archive (table of contents check)
pg_restore --list /tmp/restore-drill/var/backups/db-dumps/*.pgdump | head -n 20

# 4. Clean up the temporary restore drill directory
sudo rm -rf /tmp/restore-drill

ID        Time                 Host         Tags             Paths
------------------------------------------------------------------------------------
a81f39c2  2026-11-03 20:45:02  netcup-rs1   automated-daily  /etc, /opt/stacks, /var/backups/db-dumps

; Archive created at 2026-11-03 20:45:00 UTC
;     dbname: prod_app
;     TOC Entries: 48