1c6ab0fa68
Trei lucruri gasite continuand handoff-ul de la incidentul pveelite. 1. Replicarea oracle-backups taia snapshoturile doar pe sursa. Pe destinatie nu curata nimeni, deci din 25.04 se adunasera 11.886 snapshoturi tinand 941G pe pvemini - nodul cu toata productia, ajuns la 93%. Adaugat KEEP_SNAPS_DEST=288 (72h). Dupa curatarea restantei: 93% -> 44%, 1,01T liberi. Verificat ca rularea cron urmatoare pastreaza fix 288. Stergerea pe interval (ds@a%b) prinde si snapshoturile @failback_/@init_ dintre capete, deci scriptul sterge cate unul; intervalul s-a folosit o singura data, manual, dupa ce s-a verificat ca nu exista non-repl_. 2. pvemini-down-alert.sh rula din cron fara PATH, iar ha-manager e in /usr/sbin: sectiunea "HA status" iesea goala tacut la fiecare alerta. 3. Resetarile adaptorului USB LAN nu erau intamplatoare. Un mouse optic defect se re-enumera de ~1000 de ori pe zi pe acelasi controller xHCI (0000:00:14.0) ca adaptorul de retea; tastatura de pe acelasi hub are o singura enumerare. Erorile "xhci_hcd WARN Set TR Deq Ptr" apar lipite de fiecare resetare r8152, la 3 secunde dupa cate o re-enumerare a mouse-ului. Portul mouse-ului dezactivat din sysfs ca experiment reversibil. Punctul cu exportul NFS "inexistent" din planul de preventie era alarma falsa: exportul e viu, doar ca nu e storage Proxmox. Marcat ca atare. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VAEUqrv26jCM5KwAPtkHb8
94 lines
3.9 KiB
Bash
94 lines
3.9 KiB
Bash
#!/bin/bash
|
|
#
|
|
# Replicate rpool/oracle-backups from pveelite (active NFS server) to
|
|
# pvemini (standby) every 15 minutes via incremental zfs send/recv.
|
|
#
|
|
# Why: NFS storage on pveelite is the single point that the DR test and
|
|
# the daily SCP transfers from primary Oracle Windows depend on. With
|
|
# 15-min ZFS replicas, pvemini can take over within minutes if pveelite
|
|
# becomes unreachable (run /opt/scripts/failover-dr-to-pvemini.sh).
|
|
#
|
|
# Why not pvesr or pve-zsync:
|
|
# * pvesr only replicates VM/CT disks, not arbitrary datasets.
|
|
# * pve-zsync would add a package dependency for one job. zfs send
|
|
# over SSH is the simplest mechanism that fits the rest of the
|
|
# cluster's replication patterns.
|
|
#
|
|
# Schedule: */15 * * * * via cron on pveelite.
|
|
# Initial sync:
|
|
# zfs send rpool/oracle-backups@init_<ts> | ssh root@<pvemini> \
|
|
# 'zfs recv -F rpool/oracle-backups && zfs set readonly=on rpool/oracle-backups'
|
|
# ssh root@<pvemini> 'zfs set mountpoint=/mnt/pve/oracle-backups rpool/oracle-backups'
|
|
|
|
set -euo pipefail
|
|
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
|
|
|
DATASET="rpool/oracle-backups"
|
|
TARGET_HOST="10.0.20.201" # pvemini direct IP (avoids tailscale magicDNS detour)
|
|
SNAP_PREFIX="repl"
|
|
KEEP_SNAPS=5 # rolling history on source side
|
|
KEEP_SNAPS_DEST=288 # rolling history on destination (288 x 15 min = 72h)
|
|
LOCK="/var/run/zfs-replicate-oracle-backups.lock"
|
|
LOG="/var/log/oracle-dr/replication.log"
|
|
SSH_OPTS="-o UserKnownHostsFile=/etc/pve/priv/known_hosts -o StrictHostKeyChecking=no -o BatchMode=yes"
|
|
|
|
mkdir -p "$(dirname "$LOG")"
|
|
exec 9>"$LOCK"
|
|
flock -n 9 || { echo "[$(date)] previous run still active, skipping" >>"$LOG"; exit 0; }
|
|
|
|
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >>"$LOG"; }
|
|
|
|
NEW_SNAP="${DATASET}@${SNAP_PREFIX}_$(date +%Y%m%d_%H%M%S)"
|
|
zfs snapshot "$NEW_SNAP"
|
|
|
|
# Find previous replication snapshot (excluding the one we just made)
|
|
PREV_SNAP=$(zfs list -t snapshot -o name -s creation "$DATASET" 2>/dev/null \
|
|
| awk -v p="${DATASET}@${SNAP_PREFIX}_" '$0 ~ p' \
|
|
| grep -v "$NEW_SNAP" \
|
|
| tail -1 || true)
|
|
|
|
if [ -n "$PREV_SNAP" ]; then
|
|
log "Incremental send: $PREV_SNAP -> $NEW_SNAP"
|
|
if ! zfs send -i "$PREV_SNAP" "$NEW_SNAP" | \
|
|
ssh $SSH_OPTS root@${TARGET_HOST} "zfs recv -F $DATASET" 2>>"$LOG"; then
|
|
log "ERROR: incremental send failed"
|
|
zfs destroy "$NEW_SNAP" 2>/dev/null || true
|
|
exit 1
|
|
fi
|
|
else
|
|
log "Full send (no previous snapshot found): $NEW_SNAP"
|
|
if ! zfs send "$NEW_SNAP" | \
|
|
ssh $SSH_OPTS root@${TARGET_HOST} "zfs recv -F $DATASET" 2>>"$LOG"; then
|
|
log "ERROR: full send failed"
|
|
zfs destroy "$NEW_SNAP" 2>/dev/null || true
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
# Prune old snapshots on source (keep last KEEP_SNAPS)
|
|
zfs list -t snapshot -o name -s creation "$DATASET" \
|
|
| awk -v p="${DATASET}@${SNAP_PREFIX}_" '$0 ~ p' \
|
|
| head -n -${KEEP_SNAPS} \
|
|
| xargs -r -n1 zfs destroy 2>>"$LOG" || true
|
|
|
|
# Prune old snapshots on destination (keep last KEEP_SNAPS_DEST).
|
|
#
|
|
# Fara pasul asta destinatia nu taia nimic, niciodata. Intre 25.04 si 28.08.2026
|
|
# s-au adunat 11886 snapshoturi care tineau 941G si urcasera rpool-ul pvemini la
|
|
# 93% - cu productia pe acelasi pool. Sursa se curata singura de la inceput, deci
|
|
# problema a fost invizibila pe pveelite.
|
|
#
|
|
# Se sterge cate un snapshot pe rulare (xargs -n1), NU pe interval (@a%b): un
|
|
# range ar cuprinde si eventualele snapshoturi @failback_/@init_ prinse intre
|
|
# capete, care trebuie sa supravietuiasca. In regim normal e un singur destroy
|
|
# la fiecare rulare, deci costul e neglijabil.
|
|
if ! ssh $SSH_OPTS root@${TARGET_HOST} \
|
|
"zfs list -t snapshot -o name -H -s creation -r '$DATASET' \
|
|
| awk -v p='${DATASET}@${SNAP_PREFIX}_' '\$0 ~ p' \
|
|
| head -n -${KEEP_SNAPS_DEST} \
|
|
| xargs -r -n1 zfs destroy" 2>>"$LOG"; then
|
|
log "WARNING: prune pe destinatie a esuat (replicarea a reusit)"
|
|
fi
|
|
|
|
log "Replication completed successfully"
|