LXC 103 avea replicare ZFS zilnica (21:02 pve1, 21:03 pveelite) dar niciun
backup. Replicarea oglindeste starea curenta: o stergere sau o coruptie
ajunge pe toate nodurile la urmatorul sync, fara punct de revenire.
Adaugat in job-ul de containere usoare (zile impare 03:00, keep-last=2
keep-weekly=1) — 8.24 GB folositi, 4.3 GB arhiva. Rulat si un backup
initial, fiindca urmatoarea rulare automata era abia pe 13 august.
Ramane descoperit doar LXC 102, singurul `running` fara backup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MoFGxftQw92PX4Sdkn94pv
`backup-impare-adina` (la 2 zile) inlocuit cu `backup-lunar-adina`:
prima sambata din luna la 04:30, keep-monthly=3.
Ora 04:30 evita coliziunea cu job-urile pare/impare de la 03:30 — prima
sambata a lunii poate cadea in orice zi.
Notat in README ca `sat *-1..7` inseamna prima sambata: un job lunar adaugat
la mijlocul lunii nu ruleaza pana luna urmatoare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MoFGxftQw92PX4Sdkn94pv
VM 303 nu era in niciun job de backup dupa transformarea in full clone.
Adaugat `backup-impare-adina` (zile impare 03:30, storage backup,
keep-last=2 keep-weekly=1), pe zile impare ca sa nu se suprapuna cu VM 201
care ruleaza in acelasi slot pe zile pare.
Sectiunea "Backup Job Configuration" din README descria un job unic
(`backup-fbb668c0-726e`, daily 02:00) care nu mai exista de mult. Inlocuita
cu cele 8 job-uri reale.
Semnalate guest-urile fara backup: LXC 102 si 103, ambele `running`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MoFGxftQw92PX4Sdkn94pv
`qm move-disk <vmid> <disk> local-zfs --delete 1` esueaza intotdeauna pentru
zvol-uri: PVE respinge mutarea pe acelasi storage, iar conditia din
Qemu.pm:4688 e mereu adevarata fiindca numele volumului n-are sufix de format.
Nici storage-ul `local` nu e ruta de ocolire — n-are `images` in content.
Documentata metoda care functioneaza (zfs send/recv local + swap de volid),
asa cum a fost aplicata pe VM 303.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MoFGxftQw92PX4Sdkn94pv
VM 303 rula ca linked clone din VM 300, deci template-ul contaminat nu putea
fi sters. Transformat in full clone si VM 300 eliminat din infrastructura.
Metoda: zfs send/recv local + swap de volid in config. `qm move-disk` nu era
o optiune — PVE respinge mutarea pe acelasi storage, iar pentru zvol-uri
conditia din Qemu.pm:4688 e mereu adevarata (numele n-are sufix de format).
- inventarele nu mai listeaza VM 300; VM 310 ramane singurul template
- clone-vm300.sh: comentariul spune de ce sursa e 310 (numele e istoric)
- handoff-ul sters la cererea utilizatorului; faptele care mai conteaza sunt
mutate inline in README-uri, ca sa nu ramana referinte moarte
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MoFGxftQw92PX4Sdkn94pv
VM 300 `Win11-Template` nu a fost sysprep-uit: poarta numele ROACENTRAL si
SID-ul de masina al VM 201 (S-1-5-21-1850128657-3079265004-705332634).
Clonele lui mostenesc acea identitate, iar NTLM intre doua masini cu acelasi
SID de masina esueaza — de aici imposibilitatea accesarii share-urilor SMB
intre VM 303 si VM 201, in ambele sensuri.
VM 300 nu poate fi reparat in loc: e template read-only, iar VM 303 e linked
clone din el. Solutia: clona completa 300 -> 310, sysprep acolo, conversie in
template. Verificat pe o clona de proba: nume si SID noi la fiecare pornire.
- clone-vm300.sh: SOURCE_VMID 300 -> 310
- README-uri: VM 310 marcat ca template de clonare, VM 300 ca "nu clona"
- vm302-oracle-test: corectat exemplul care clona in VM 303 (deja ocupat)
- docs/handoff-smb-vm303-roacentral.md: probele, capcanele de sysprep, stare
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MoFGxftQw92PX4Sdkn94pv
Alerta "OOM x2 on pvemini" (09:36) arata procese mici ucise (dbus-daemon,
oom_score_adj:200), dar mesajul kernel dadea containerul real:
oom_memcg=/lxc/102. OOM local containerului, nu presiune de host - pvemini
avea 24Gi disponibili si zram functional.
Baseline masurat cu ZERO sandbox-uri active: ~1.9GB (sbx 863M + claude 317M +
VS Code Remote 450M + docker/tailscale/portainer 205M). Un sandbox real mai
adauga ~1.9GB (containerd-shim) -> plafonul de 4GB era depasit sistematic.
Contoare cumulate: oom_kill 24, 9279 depasiri memory.high, memory.peak fix pe
limita, swap 510/512 epuizat.
Verificat explicit ca sbx NU are memory leak: RSS urca la ~863M la pornire si
se plafoneaza (esantionat la 10s timp de un minut).
Fix: pct set 102 --memory 12288 --swap 4096 (live, fara restart).
Tabelul de resurse din cluster/README.md era vechi (102 aparea ca "coolify
stopped", lipseau 110 si 171, RAM gresit peste tot) - regenerat din pvesh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
LXC 110 (moltbot) a picat dupa un OOM local containerului urmat de reboot:
tailscaled a ramas delogat, iar containerul mostenea resolv.conf-ul Tailscale
de la host-ul pve1 (doar MagicDNS 100.100.100.100) -> rezolutie DNS zero desi
L3 era functional -> echo-core in crash-loop pe telegram.error.TimedOut.
Fixuri aplicate:
- pct set 110 --nameserver '10.0.20.1 1.1.1.1' (elimina dependenta de Tailscale)
- zram-tools pe pve1 (8G zstd, prio 100) - `swap: 4096` din config era fictiv,
host-ul nu avea niciun swap; root pe ZFS deci zram, nu swapfile
- MemoryHigh/MemoryMax pe pocket-tts + supertonic-tts - toate serviciile user
rulau cu limite `infinity`, de unde OOM-uri recurente (Apr 25, May 28 x4)
cu victime aleatorii alese dupa oom_score_adj
Documentatie:
- nou post-mortem in cluster/incidents/, adaugat in indexuri
- README lxc110: host corectat pveelite -> pve1 (+ RAM/CPU/storage reale),
comenzile pct redirectionate spre nodul corect
- README lxc171: tabel cu starea swap pe cele 3 noduri (pveelite are zvol,
nu zram - nu necesita acelasi fix)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Proxmox default-matcher acum trimite doar la mail-to-root (pve1 smtp
eliminat din matcher → fix emailuri duble pentru backup/vzdump)
- Adaugat tabel cron jobs per nod cu motivul redirect-ului > /dev/null
- Regula: scripturi cu propriile notificari trebuie sa aiba redirect in
crontab, altfel cron genereaza email suplimentar de confirmare
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documenteaza incidentul Kingston SNV3S2000G hang la 2026-04-30 (Sensor 2
74°C → emergency mode + restart loop) si masurile aplicate: distantare
temporala backup-uri par/impar, mutare CT 101+110 pe pve1 backup-ssd,
nofail in fstab, hardware watchdog iTCO_wdt, monitoring CSV la 30 min.
Adauga scripturile /opt/scripts/kingston-thermal-{monitor,report}.sh
pentru tracking trend si alertare la depasirea pragurilor termale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
VM 201 (Windows critical) stays out of HA by design. Added:
- failover-vm201.sh: interactive failover pvemini -> pveelite with ZFS replication state
- recover-vm201-to-pvemini.sh: interactive reverse migration with uptime + split-brain checks
- pvemini-down-alert.sh: cron watchdog on pveelite, emails full runbook after 2min DOWN
Replication RPO tightened: CT 108 + VM 201 to 5min, CT 171 to 15min.
CT 171 added to HA (ha-group-main) for continuous Claude Code access.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Following the 2026-04-20 cluster outage, the cluster README now covers
HA resource limits, corosync token tuning (10s tolerance for USB glitches),
rasdaemon/netconsole/kdump diagnostic stack on pvemini, mail relay via
mail.romfast.ro with SMTP auth, OOM alerting via cron, and swap on pveelite.
VM 109 README now clearly states it was removed from HA and is only
started by the weekly DR test script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The DR test script used set -euo pipefail, so a failing SSH
shutdown command caused the script to exit before qm stop.
On 2026-04-20 this left VM 109 running for 2.5 days and
triggered an OOM cascade when pvemini HA-failed over to
pveelite.
Adds EXIT trap that force-stops VM 109 regardless of exit
path, and makes the Step 7 SSH shutdown tolerant of failure.
Incident details: proxmox/cluster/incidents/2026-04-20-cluster-outage.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Create cluster/ for Proxmox cluster infrastructure (SSH guide, HA monitor, UPS)
- Create lxc108-oracle/ for Oracle Database documentation and scripts
- Create vm201-windows/ for Windows 11 VM docs and SSL certificate scripts
- Add SSL certificate monitoring scripts (check-ssl-certificates.ps1, monitor-ssl-certificates.sh)
- Remove archived VM107 references (decommissioned)
- Update all cross-references between files
- Update main README.md with new structure and navigation
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
