d7b4007af8
Ring1 aplicat si verificat pe cluster live: config_version 17, ring1_addr pe
10.10.10.20x la fiecare nod, interface{linknumber:1}. Validat cu `corosync -t`
inainte de instalare. Testat prin oprirea reala a lui ring0 pe pveelite: link0
disconnected, link1 connected, cvorum 3/3 neatins - exact scenariul care pe
27 august a lasat nodul mort 16 ore.
Test de repornire completa a clusterului, cu Wake-on-LAN (trezire in ~15s).
Insula a urcat singura pe toate trei nodurile; ipoteza enumerarii tarzii a
USB-ului nu s-a materializat. Unealta noua: wake-cluster.ps1.
Testul a scos la iveala o linie ramasa in /etc/fstab pe pve1 si pveelite, care
monta storage-ul NFS de pe IP-ul de productie inaintea lui pvestatd. Backup-ul
trecea tacut pe reteaua gresita, cu storage.cfg corect. cluster-startup verifica
acum asta automat, impreuna cu IP-urile de insula si conectivitatea reala.
Patru bug-uri gasite prin rulare pe cluster live:
- sonda Oracle nu avea timeout: un sqlplus agatat pe o instanta in pornire a
blocat cluster-startup 14 minute, fara mesaj, cu propriul prag de 600s
nefolosit, fiindca bucla n-a apucat o iteratie;
- `bash -c` in loc de `bash -lc`: sqlplus lipsea din PATH, deci baza nu se
oprea si containerul s-ar fi inchis peste ea;
- PowerShell 5.1 pierde ghilimelele duble catre exe-uri native, deci comanda
ajungea rupta pe nod - trecut pe trimitere codificata base64;
- backup-ul de crontab si fisierul de stare se rescriau la o a doua rulare,
lasand monitorizarea oprita permanent si lista de repornit goala.
Documentatia de oprire planificata pornea de la o afirmatie devenita falsa
("nu exista datacenter.cfg") si de la o comanda care ar fi adaugat o a doua
linie `ha:`. Actualizata, impreuna cu inventarul de guest-uri (VM 304 lipsea
din toate listele de ordine si cadea in maturarea de dupa Oracle).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RJFQZcA7uerXRaL1NrsXpS
269 lines
11 KiB
PowerShell
269 lines
11 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Instaleaza reatasarea automata a adaptoarelor USB LAN dupa un reset USB.
|
|
|
|
.DESCRIPTION
|
|
Adaptoarele Realtek RTL8156 (driver r8152) de pe pve1 si pveelite se
|
|
reseteaza singure pe magistrala USB. Kernelul sterge interfata si o
|
|
recreeaza, dar nimeni nu o reconfigureaza: se ridica doar ca efect secundar,
|
|
la boot.
|
|
|
|
Scriptul instaleaza o regula udev care prinde reaparitia interfetei si o
|
|
unitate systemd care reaplica configuratia de retea (ifreload -a).
|
|
Detecteaza singur ce noduri au adaptor r8152 si le sare pe celelalte -
|
|
pvemini are placa Intel pe PCIe si nu are nevoie.
|
|
|
|
CE S-A SCHIMBAT PE 2026-08-29. Cand a fost scris scriptul, dongle-ul USB era
|
|
portul puntii vmbr0, adica placa de retea PRINCIPALA - de-aia un reset USB
|
|
lasa nodul complet invizibil (16 ore de tacere pe pveelite in noaptea de 27
|
|
spre 28 august 2026). Intre timp vmbr0 a fost mutat pe interfata onboard, iar
|
|
dongle-ul a ramas doar cu IP-ul retelei dedicate de cluster
|
|
(10.10.10.200 / 10.10.10.202), declarat acum "auto ... inet static".
|
|
|
|
Regula ramane la fel de utila, dar miza s-a schimbat: un reset USB nu mai ia
|
|
nodul offline, ci opreste tacut replicarea ZFS si storage-ul NFS
|
|
backup-pvemini. In plus, fiind pe ACTION=="add", regula acopera si cazul in
|
|
care USB-ul se enumereaza tarziu la boot, dupa networking.service - singurul
|
|
risc ramas pentru insula. Neconfirmat inca la o repornire reala.
|
|
|
|
Context: ../incidents/2026-08-27-pveelite-down.md
|
|
../docs/switch-2.5g-dedicat-noduri.md
|
|
|
|
RULEAZA DE PE STATIA DE ADMIN. SSH intre nodurile Proxmox trece prin
|
|
Tailscale si cere autentificare interactiva.
|
|
|
|
.PARAMETER Node
|
|
Instaleaza doar pe nodul dat (pve1 / pvemini / pveelite). Implicit: toate.
|
|
|
|
.PARAMETER DryRun
|
|
Arata ce ar face, fara sa modifice nimic.
|
|
|
|
.PARAMETER Test
|
|
Dupa instalare, forteaza un reset USB real ca sa verifice ca regula chiar
|
|
functioneaza. PERICULOS FARA PLASA - vezi nota de mai jos. Scriptul isi pune
|
|
singur o plasa de siguranta: un "ifreload -a" programat peste 90 de secunde,
|
|
care repara nodul chiar daca regula nu s-a declansat. Nu folosi pe pvemini.
|
|
|
|
.EXAMPLE
|
|
.\deploy-usb-lan-hotplug.ps1 -DryRun
|
|
|
|
.EXAMPLE
|
|
.\deploy-usb-lan-hotplug.ps1
|
|
|
|
.EXAMPLE
|
|
.\deploy-usb-lan-hotplug.ps1 -Node pveelite -Test
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[ValidateSet('pve1','pvemini','pveelite')]
|
|
[string]$Node,
|
|
[switch]$DryRun,
|
|
[switch]$Test
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
|
|
$NodeIp = [ordered]@{
|
|
pve1 = '10.0.20.200'
|
|
pvemini = '10.0.20.201'
|
|
pveelite = '10.0.20.202'
|
|
}
|
|
|
|
$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
|
|
$CfgDir = Join-Path (Split-Path -Parent $ScriptDir) 'config\usb-lan-hotplug'
|
|
$RuleSrc = Join-Path $CfgDir '99-usb-lan-hotplug.rules'
|
|
$UnitSrc = Join-Path $CfgDir 'usb-lan-hotplug@.service'
|
|
|
|
$RuleDst = '/etc/udev/rules.d/99-usb-lan-hotplug.rules'
|
|
$UnitDst = '/etc/systemd/system/usb-lan-hotplug@.service'
|
|
|
|
function Write-Step { param([string]$Text) Write-Host ""; Write-Host "== $Text" -ForegroundColor Green }
|
|
function Write-Ok { param([string]$Text) Write-Host " [ok] $Text" -ForegroundColor Green }
|
|
function Write-Warn { param([string]$Text) Write-Host " [!] $Text" -ForegroundColor Yellow }
|
|
function Write-Bad { param([string]$Text) Write-Host " [X] $Text" -ForegroundColor Red }
|
|
function Write-Info { param([string]$Text) Write-Host " $Text" -ForegroundColor DarkGray }
|
|
|
|
function Test-Alive {
|
|
param([string]$Address, [int]$TimeoutMs = 2000)
|
|
try {
|
|
$p = New-Object System.Net.NetworkInformation.Ping
|
|
return ($p.Send($Address, $TimeoutMs).Status -eq 'Success')
|
|
} catch { return $false }
|
|
}
|
|
|
|
function Invoke-Node {
|
|
param([string]$Ip, [string]$Command, [int]$Timeout = 15)
|
|
return ssh -o BatchMode=yes -o ConnectTimeout=$Timeout `
|
|
-o StrictHostKeyChecking=accept-new -o LogLevel=ERROR `
|
|
"root@$Ip" $Command
|
|
}
|
|
|
|
function Invoke-NodeScript {
|
|
# PowerShell inghite ghilimelele cand paseaza argumente catre ssh, iar shell-ul
|
|
# de pe nod primeste [ = r8152 ] in loc de [ "$d" = r8152 ]. In loc sa ne luptam
|
|
# cu nivelurile de citare, trimitem scriptul codificat si il decodam acolo.
|
|
param([string]$Ip, [string]$Script, [int]$Timeout = 20)
|
|
$lf = $Script -replace "`r`n", "`n"
|
|
$b64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($lf))
|
|
return Invoke-Node -Ip $Ip -Command "echo $b64 | base64 -d | sh" -Timeout $Timeout
|
|
}
|
|
|
|
foreach ($f in @($RuleSrc, $UnitSrc)) {
|
|
if (-not (Test-Path $f)) { Write-Bad "lipseste $f"; exit 1 }
|
|
# Un CRLF intr-o regula udev o face sa nu se potriveasca niciodata, tacut.
|
|
# .gitattributes forteaza LF pe directorul asta, dar verificam oricum.
|
|
if ([IO.File]::ReadAllText($f) -match "`r") {
|
|
Write-Bad "$f are CRLF - trebuie LF. Verifica .gitattributes."
|
|
exit 1
|
|
}
|
|
}
|
|
Write-Host ""
|
|
Write-Host "Hotplug pentru adaptoarele USB LAN (r8152)" -ForegroundColor Cyan
|
|
Write-Info "sursa: $CfgDir"
|
|
|
|
$tinte = if ($Node) { @($Node) } else { $NodeIp.Keys }
|
|
$instalate = @()
|
|
|
|
foreach ($n in $tinte) {
|
|
$ip = $NodeIp[$n]
|
|
Write-Step "$n ($ip)"
|
|
|
|
if (-not (Test-Alive $ip)) {
|
|
Write-Warn "nu raspunde la ping - sarit. Reia cu -Node $n cand revine."
|
|
continue
|
|
}
|
|
|
|
# Nodurile fara adaptor USB nu au nevoie de regula. pvemini are Intel igc.
|
|
$ifs = Invoke-NodeScript -Ip $ip -Script @'
|
|
for n in /sys/class/net/*; do
|
|
i=${n##*/}
|
|
d=$(readlink -f "$n/device/driver" 2>/dev/null)
|
|
case "$d" in *r8152) echo "$i" ;; esac
|
|
done
|
|
'@
|
|
if (-not $ifs) {
|
|
Write-Ok "nu are adaptor USB r8152 - nu are nevoie de regula, sarit"
|
|
continue
|
|
}
|
|
Write-Info "adaptoare r8152: $($ifs -join ', ')"
|
|
|
|
# Regula e utila doar daca interfata chiar are ce sa i se reaplice: fie e
|
|
# portul puntii (configuratia dinainte de 2026-08-29), fie are propria strofa
|
|
# in /etc/network/interfaces (azi: IP-ul retelei dedicate de cluster).
|
|
# Fara niciuna, ifreload -a n-ar avea ce configura si regula n-ar folosi la nimic.
|
|
$rol = Invoke-NodeScript -Ip $ip -Script @'
|
|
port=$(sed -n 's/^[[:space:]]*bridge-ports[[:space:]]*//p' /etc/network/interfaces)
|
|
for n in /sys/class/net/*; do
|
|
i=${n##*/}
|
|
d=$(readlink -f "$n/device/driver" 2>/dev/null)
|
|
case "$d" in *r8152) ;; *) continue ;; esac
|
|
case " $port " in *" $i "*) echo "$i: port al puntii vmbr0"; continue ;; esac
|
|
if grep -qE "^[[:space:]]*(auto|allow-hotplug)[[:space:]]+$i([[:space:]]|$)" /etc/network/interfaces; then
|
|
echo "$i: strofa proprie ($(sed -n "s/^[[:space:]]*iface[[:space:]]\+$i[[:space:]]\+//p" /etc/network/interfaces | head -1)), IP: $(ip -4 -o addr show "$i" | awk '{print $4}' | tr '\n' ' ')"
|
|
else
|
|
echo "$i: NECONFIGURAT in /etc/network/interfaces"
|
|
fi
|
|
done
|
|
'@
|
|
$rol | ForEach-Object { Write-Info $_ }
|
|
if ($rol -match 'NECONFIGURAT') {
|
|
Write-Warn "un adaptor r8152 nu e nici port de punte, nici cu strofa proprie -"
|
|
Write-Warn "regula se instaleaza, dar pentru el ifreload -a n-are ce aplica"
|
|
}
|
|
|
|
if ($DryRun) {
|
|
Write-Info "[dry-run] scp $RuleSrc -> ${ip}:$RuleDst"
|
|
Write-Info "[dry-run] scp $UnitSrc -> ${ip}:$UnitDst"
|
|
Write-Info "[dry-run] udevadm control --reload && systemctl daemon-reload"
|
|
continue
|
|
}
|
|
|
|
scp -o BatchMode=yes -o LogLevel=ERROR $RuleSrc "root@${ip}:$RuleDst" | Out-Null
|
|
scp -o BatchMode=yes -o LogLevel=ERROR $UnitSrc "root@${ip}:$UnitDst" | Out-Null
|
|
Invoke-Node -Ip $ip -Command "udevadm control --reload-rules; systemctl daemon-reload" | Out-Null
|
|
|
|
# Verificare reala: fisierele exista, unitatea e vazuta de systemd si regula
|
|
# e sintactic valida. udevadm test nu modifica nimic.
|
|
# Backtick-ul escapeaza variabilele de shell, ca sa nu le interpoleze PowerShell.
|
|
$ver = Invoke-NodeScript -Ip $ip -Script @"
|
|
if [ -f $RuleDst ]; then echo 'regula: prezenta'; else echo 'regula: LIPSA'; fi
|
|
if [ -f $UnitDst ]; then echo 'unitate: prezenta'; else echo 'unitate: LIPSA'; fi
|
|
cr=`$(tr -d '\n' < $RuleDst | tr -d '[:print:]' | wc -c)
|
|
echo "neprintabile in regula: `$cr"
|
|
if systemctl cat usb-lan-hotplug@test.service >/dev/null 2>&1; then echo 'systemd: vede unitatea'; else echo 'systemd: NU vede unitatea'; fi
|
|
"@
|
|
$ver | ForEach-Object { Write-Info $_ }
|
|
|
|
if ($ver -match 'LIPSA' -or $ver -match 'neprintabile in regula: [1-9]' -or $ver -match 'NU vede') {
|
|
Write-Bad "instalare incompleta pe $n"
|
|
} else {
|
|
Write-Ok "instalat"
|
|
$instalate += $n
|
|
}
|
|
}
|
|
|
|
# ------------------------------------------------------- testul cu reset real
|
|
|
|
if ($Test -and -not $DryRun) {
|
|
if (-not $Node) { Write-Bad "-Test cere -Node explicit"; exit 1 }
|
|
if ($Node -eq 'pvemini') { Write-Bad "refuz -Test pe pvemini: acolo ruleaza productia"; exit 1 }
|
|
if ($instalate -notcontains $Node) { Write-Bad "$Node nu a fost instalat cu succes - nu testez"; exit 1 }
|
|
|
|
$ip = $NodeIp[$Node]
|
|
Write-Step "Test cu reset USB real pe $Node"
|
|
Write-Warn "Nodul isi pierde reteaua pentru cateva secunde."
|
|
Write-Info "Plasa de siguranta: 'ifreload -a' programat peste 90 s, care repara"
|
|
Write-Info "nodul si daca regula nu se declanseaza - deci fara drum pana la el."
|
|
|
|
$usbPath = Invoke-NodeScript -Ip $ip -Script @'
|
|
for d in /sys/bus/usb/devices/*; do
|
|
[ -e "$d/idVendor" ] || continue
|
|
[ "$(cat "$d/idVendor")" = "0bda" ] || continue
|
|
ls "$d"/*/net >/dev/null 2>&1 || continue
|
|
echo "${d##*/}"
|
|
break
|
|
done
|
|
'@
|
|
if (-not $usbPath) { Write-Bad "nu gasesc dispozitivul USB Realtek"; exit 1 }
|
|
Write-Info "dispozitiv USB: $usbPath"
|
|
|
|
# Plasa se armeaza INAINTE de reset, cu nohup, ca sa supravietuiasca inchiderii
|
|
# sesiunii SSH odata cu caderea retelei.
|
|
Invoke-NodeScript -Ip $ip -Script @'
|
|
nohup sh -c 'sleep 90; /usr/sbin/ifreload -a' >/dev/null 2>&1 &
|
|
'@ | Out-Null
|
|
Write-Ok "plasa de siguranta armata (90 s)"
|
|
|
|
# Reset real: dezautorizeaza si reautorizeaza dispozitivul, exact ce face
|
|
# controllerul xHCI cand cedeaza singur.
|
|
Invoke-NodeScript -Ip $ip -Script @"
|
|
nohup sh -c 'echo 0 > /sys/bus/usb/devices/$usbPath/authorized; sleep 2; echo 1 > /sys/bus/usb/devices/$usbPath/authorized' >/dev/null 2>&1 &
|
|
"@ | Out-Null
|
|
Write-Info "reset trimis, astept revenirea..."
|
|
|
|
$revenit = $false
|
|
foreach ($i in 1..30) {
|
|
Start-Sleep -Seconds 4
|
|
if (Test-Alive $ip) { $revenit = $true; break }
|
|
}
|
|
|
|
if ($revenit) {
|
|
Write-Ok "nodul a revenit dupa ~$($i * 4) s"
|
|
if ($i * 4 -lt 60) {
|
|
Write-Ok "sub 60 s => regula udev a lucrat, nu plasa de siguranta"
|
|
} else {
|
|
Write-Warn "peste 60 s => posibil sa fi lucrat plasa. Verifica jurnalul:"
|
|
Write-Info " ssh root@$ip 'journalctl -u usb-lan-hotplug@* --since \"-5 min\"'"
|
|
}
|
|
$j = Invoke-Node -Ip $ip -Command "journalctl -u 'usb-lan-hotplug@*' --since '-5 min' --no-pager | tail -12"
|
|
$j | ForEach-Object { Write-Info $_ }
|
|
} else {
|
|
Write-Bad "nodul NU a revenit in 2 minute - verifica la consola fizica"
|
|
exit 1
|
|
}
|
|
}
|
|
|
|
Write-Host ""
|
|
if ($instalate) { Write-Ok "instalat pe: $($instalate -join ', ')" }
|
|
else { Write-Warn "niciun nod modificat" }
|