How to Monitor Netcup Server Resources: CPU Steal (%st), NVMe Latency & Prometheus + Grafana

Understanding the architectural difference between a Netcup VPS (shared vCores scheduled by the host KVM hypervisor) and a Netcup Root Server (dedicated physical AMD EPYC threads pinned to your VM) starts with the %st (steal time) metric vs %iowait. Steal time measures the percentage of time a virtual CPU thread inside your Linux guest was in a runnable state waiting in the host scheduler runqueue while the KVM hypervisor executed another tenant or host kernel thread on the underlying physical CPU core. By contrast, %iowait means the vCPU is idle inside your own VM waiting for a block device read or synchronous fsync() write on /dev/vda to return.

In addition to mpstat and vmstat, modern Linux 6.x kernels expose Pressure Stall Information (PSI) under /proc/pressure/cpu, /proc/pressure/memory, and /proc/pressure/io. Unlike classic 1-minute load averages—which conflate CPU demand with uninterruptible disk sleep (TASK_UNINTERRUPTIBLE) and mask sub-second latency spikes—PSI tracks the exact percentage of wall-clock time over rolling 10-, 60-, and 300-second windows that tasks were stalled waiting for hardware resources.

Both /proc/pressure/memory and /proc/pressure/io report two distinct rows: some and full. The some line indicates the percentage of time at least one thread was delayed waiting for memory page reclaim or NVMe I/O while other threads continued executing on other vCores. The full line indicates the percentage of time every single non-idle task in the system or cgroup was simultaneously stalled waiting for that resource, meaning zero forward computational progress was occurring. Watching full avg10 and full avg60 gives you an early-warning radar several minutes before the Linux kernel OOM killer triggers or database connections pile up.

sudo apt update
sudo apt install -y sysstat iotop ioping vnstat htop btop

# 1. Enable sysstat historical collection every 10 minutes (/var/log/sysstat)
sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat vnstat

# 2. Sample per-vCore CPU utilization and steal time (%steal) every 1 second (5 samples)
mpstat -P ALL 1 5

# 3. Read Linux Kernel Pressure Stall Information (CPU, Memory, and I/O stalls)
head -n 5 /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all    4.12    0.00    1.05    0.08    0.00    0.12    0.00    0.00    0.00   94.63

==> /proc/pressure/memory <==
some avg10=0.00 avg60=0.00 avg300=0.00 total=1420
full avg10=0.00 avg60=0.00 avg300=0.00 total=810

Never run heavy multi-gigabyte fio or dd write loops on a live production server just to check if disk I/O is healthy—doing so saturates the host storage controller queue, triggers hypervisor IOPS rate-limiters, and degrades your own database latencies. Instead, use iostat -xz 1 to inspect live kernel block device counters on /dev/vda and ioping -D (O_DIRECT) to measure non-destructive 4 KiB direct-I/O latency in microseconds.

When reading iostat -xz 1 output on a Netcup KVM server, focus on three columns rather than %util (which can be misleading on multi-queue virtio-blk / virtio-scsi NVMe devices capable of parallel I/O dispatch): r_await (average time in milliseconds for read requests to be serviced, including queue time), w_await (average write completion latency in milliseconds), and aqu-sz (average number of requests waiting in the block queue). Sub-millisecond values (0.15 ms – 0.90 ms) indicate an uncontended enterprise NVMe array, whereas sustained w_await > 8 ms paired with aqu-sz > 4 points to either heavy local checkpoint flushing or host storage pool contention.

To track network bandwidth without relying exclusively on the Netcup Server Control Panel (SCP), enable vnstat. Its background daemon (vnstatd) polls kernel /proc/net/dev byte counters every 30 seconds and flushes aggregates into a compact SQLite database (/var/lib/vnstat/vnstat.db) with near-zero CPU overhead, giving you instant daily (vnstat -d), hourly (vnstat -h), and monthly (vnstat -m) bandwidth audit logs.

# 1. Inspect extended block device statistics (1-second interval, 3 reports)
iostat -xz 1 3

# 2. Measure non-destructive 4 KiB direct I/O latency on the root filesystem (20 requests)
sudo ioping -c 20 -i 0.1 -D /

# 3. Check persistent daily and monthly network traffic totals in vnstat
vnstat -d 5
vnstat -m

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz  aqu-sz  %util
vda              8.50    112.00     0.00   0.00    0.28    13.18   24.20    384.50     4.10  14.49    0.41    15.89    0.01   1.20

--- / (ext4 /dev/vda1) ioping statistics ---
19 requests completed in 4.18 ms, 76 KiB read, 4.54 k iops, 17.7 MiB/s
min/avg/max/mdev = 164.2 us / 220.1 us / 389.4 us / 48.3 us

To retain 30 days of 15-second resolution telemetry across both your Linux host and every running Docker container, deploy a self-contained monitoring stack in /opt/stacks/monitoring. When running this stack on a 4 GB or 8 GB Netcup VPS (such as a VPS 500 G12.5 or VPS 1000 G12.5), two operational optimizations prevent the monitoring agents themselves from becoming a resource bottleneck:

First, cAdvisor defaults to a 1-second housekeeping loop that walks every systemd slice in /sys/fs/cgroup and parses /proc/<pid>/net/tcp for every container, easily consuming 8% to 12% of a vCore on a host running 15+ containers. Passing --docker_only=true, --housekeeping_interval=15s (matching Prometheus’s scrape interval), and --disable_metrics=percpu,sched,tcp,udp,diskIO,hugetlb,referenced_memory drops cAdvisor’s CPU usage below 0.4% while keeping all essential container CPU, RSS memory, working-set memory, and network interface counters.

Second, Prometheus TSDB buffers incoming samples in memory and appends them sequentially to a write-ahead log (wal/ directory) on disk before compacting data into immutable 2-hour block directories every two hours. On Netcup’s NVMe storage, sequential WAL appends take microseconds, but unbounded retention can slowly fill a smaller VPS root partition. Setting both --storage.tsdb.retention.time=30d and --storage.tsdb.retention.size=4GB guarantees deterministic disk usage (~1.2 GB typical for a single host) and keeps WAL checkpoint compaction fast:

• ▪ node-exporter (v1.8): Exports host kernel metrics (/proc, /sys, CPU steal, PSI, disk I/O, filesystem space, network counters).
• ▪ cadvisor (v0.49): Exports per-container cgroupv2 CPU, memory RSS, page cache, and network metrics.
• ▪ prometheus (v2.54): Scrapes both exporters every 15 seconds, evaluates alert rules, and compresses time-series data (--storage.tsdb.retention.time=30d).
• ▪ grafana (v11.2): Provides visual dashboards bound strictly to 127.0.0.1:3000 so no monitoring ports are exposed to public scanners.

name: netcup_monitoring

services:
  prometheus:
    image: prom/prometheus:v2.54.1
    container_name: mon_prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--storage.tsdb.retention.size=4GB"
      - "--web.listen-address=0.0.0.0:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro
      - prom_data:/prometheus
    networks:
      - mon_internal
    deploy:
      resources:
        limits:
          cpus: "0.50"
          memory: 512M
    security_opt:
      - no-new-privileges:true

  node-exporter:
    image: prom/node-exporter:v1.8.2
    container_name: mon_node_exporter
    restart: unless-stopped
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--path.rootfs=/rootfs"
      - "--collector.pressure"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    networks:
      - mon_internal
    deploy:
      resources:
        limits:
          cpus: "0.20"
          memory: 128M
    security_opt:
      - no-new-privileges:true

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: mon_cadvisor
    restart: unless-stopped
    privileged: true
    devices:
      - /dev/kmsg:/dev/kmsg
    command:
      - "--docker_only=true"
      - "--housekeeping_interval=15s"
      - "--disable_metrics=percpu,sched,tcp,udp,diskIO,hugetlb,referenced_memory"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    networks:
      - mon_internal
    deploy:
      resources:
        limits:
          cpus: "0.30"
          memory: 256M

  grafana:
    image: grafana/grafana-oss:11.2.2
    container_name: mon_grafana
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-ChangeMeImmediately_2026!}
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_SERVER_HTTP_ADDR: "0.0.0.0"
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - mon_internal
    depends_on:
      - prometheus
    deploy:
      resources:
        limits:
          cpus: "0.50"
          memory: 384M
    security_opt:
      - no-new-privileges:true

volumes:
  prom_data:
  grafana_data:

networks:
  mon_internal:
    driver: bridge

Create /opt/stacks/monitoring/prometheus/prometheus.yml and /opt/stacks/monitoring/prometheus/alerts.yml. Because all three telemetry containers share the mon_internal bridge network, Prometheus scrapes node-exporter:9100 and cadvisor:8080 directly via Docker’s internal DNS without traversing host iptables port mappings.

Our PromQL alert rules specifically watch for the four conditions that endanger Netcup servers in production: root disk utilization exceeding 82% (which blocks Netcup SCP Copy-on-Write snapshots from allocating delta blocks), sustained KVM CPU steal time above 10%, elevated NVMe disk read/write await latency on /dev/vda, and host available RAM dropping below 10%.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "/etc/prometheus/alerts.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "netcup_host_node"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "netcup_docker_containers"
    static_configs:
      - targets: ["cadvisor:8080"]

groups:
  - name: netcup_infrastructure_alerts
    rules:
      - alert: NetcupHighCpuStealTime
        expr: avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m])) * 100 > 10
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High Hypervisor CPU Steal (>10%) on {{ $labels.instance }}"
          description: "KVM CPU steal time has exceeded 10% for 10 minutes. Consider migrating workload to a dedicated-core Netcup Root Server (RS)."

      - alert: NetcupRootDiskSpaceLowForSnapshots
        expr: (1 - (node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|overlay"})) * 100 > 82
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Root Filesystem >82% Full (SCP Snapshots Risk)"
          description: "Disk usage exceeds 82%. Netcup SCP Copy-on-Write snapshots require free disk headroom to operate."

      - alert: NetcupHighNvmeReadAwaitLatency
        expr: (rate(node_disk_read_time_seconds_total{device="vda"}[5m]) / clamp_min(rate(node_disk_reads_completed_total{device="vda"}[5m]), 1)) * 1000 > 12
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Elevated NVMe Read Latency (>12ms) on /dev/vda"

      - alert: HostMemoryAvailableBelow10Percent
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Host Available RAM <10% on {{ $labels.instance }}"

Start the observability stack and connect to Grafana from your workstation over an encrypted SSH tunnel (-L 3000:127.0.0.1:3000). Once inside Grafana (http://localhost:3000), add http://prometheus:9090 as your default Prometheus data source and import community dashboard ID 1860 (Node Exporter Full—which includes dedicated panels for CPU Steal %st, Pressure Stall Information /proc/pressure/*, and NVMe iostat await times) and 14282 (cAdvisor Docker Insights).

Dashboards only help if someone is looking at them before an outage occurs. Inside Grafana v11’s Alerting -> Contact points menu, configure an outbound notification channel—such as a Telegram Bot API token + Chat ID, a Slack/Discord webhook URL, or an external SMTP relay—and enable Manage alerts from external Alertmanager / Prometheus data sources (or import the PromQL expressions from alerts.yml directly into Grafana-managed alert rules with a 1m evaluation interval and 5m pending window). That way, if a nightly log burst pushes /dev/vda1 past 82% or a container memory leak drains MemAvailable, you receive an immediate push notification with the exact offending metric.

# 1. On your Netcup server: start the monitoring stack
cd /opt/stacks/monitoring
sudo docker compose up -d
sudo docker compose ps

# 2. On your local laptop/workstation: open an SSH tunnel to port 3000
ssh -N -L 3000:127.0.0.1:3000 [email protected]
# Then open http://localhost:3000 in your browser