Alerta "OOM x2 on pvemini" (09:36) arata procese mici ucise (dbus-daemon,
oom_score_adj:200), dar mesajul kernel dadea containerul real:
oom_memcg=/lxc/102. OOM local containerului, nu presiune de host - pvemini
avea 24Gi disponibili si zram functional.
Baseline masurat cu ZERO sandbox-uri active: ~1.9GB (sbx 863M + claude 317M +
VS Code Remote 450M + docker/tailscale/portainer 205M). Un sandbox real mai
adauga ~1.9GB (containerd-shim) -> plafonul de 4GB era depasit sistematic.
Contoare cumulate: oom_kill 24, 9279 depasiri memory.high, memory.peak fix pe
limita, swap 510/512 epuizat.
Verificat explicit ca sbx NU are memory leak: RSS urca la ~863M la pornire si
se plafoneaza (esantionat la 10s timp de un minut).
Fix: pct set 102 --memory 12288 --swap 4096 (live, fara restart).
Tabelul de resurse din cluster/README.md era vechi (102 aparea ca "coolify
stopped", lipseau 110 si 171, RAM gresit peste tot) - regenerat din pvesh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LXC 110 (moltbot) a picat dupa un OOM local containerului urmat de reboot:
tailscaled a ramas delogat, iar containerul mostenea resolv.conf-ul Tailscale
de la host-ul pve1 (doar MagicDNS 100.100.100.100) -> rezolutie DNS zero desi
L3 era functional -> echo-core in crash-loop pe telegram.error.TimedOut.
Fixuri aplicate:
- pct set 110 --nameserver '10.0.20.1 1.1.1.1' (elimina dependenta de Tailscale)
- zram-tools pe pve1 (8G zstd, prio 100) - `swap: 4096` din config era fictiv,
host-ul nu avea niciun swap; root pe ZFS deci zram, nu swapfile
- MemoryHigh/MemoryMax pe pocket-tts + supertonic-tts - toate serviciile user
rulau cu limite `infinity`, de unde OOM-uri recurente (Apr 25, May 28 x4)
cu victime aleatorii alese dupa oom_score_adj
Documentatie:
- nou post-mortem in cluster/incidents/, adaugat in indexuri
- README lxc110: host corectat pveelite -> pve1 (+ RAM/CPU/storage reale),
comenzile pct redirectionate spre nodul corect
- README lxc171: tabel cu starea swap pe cele 3 noduri (pveelite are zvol,
nu zram - nu necesita acelasi fix)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Documenteaza incidentul Kingston SNV3S2000G hang la 2026-04-30 (Sensor 2
74°C → emergency mode + restart loop) si masurile aplicate: distantare
temporala backup-uri par/impar, mutare CT 101+110 pe pve1 backup-ssd,
nofail in fstab, hardware watchdog iTCO_wdt, monitoring CSV la 30 min.
Adauga scripturile /opt/scripts/kingston-thermal-{monitor,report}.sh
pentru tracking trend si alertare la depasirea pragurilor termale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The DR test script used set -euo pipefail, so a failing SSH
shutdown command caused the script to exit before qm stop.
On 2026-04-20 this left VM 109 running for 2.5 days and
triggered an OOM cascade when pvemini HA-failed over to
pveelite.
Adds EXIT trap that force-stops VM 109 regardless of exit
path, and makes the Step 7 SSH shutdown tolerant of failure.
Incident details: proxmox/cluster/incidents/2026-04-20-cluster-outage.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
