fix(discord): watchdog de gateway — procesul viu pe un gateway mort nu mai trece neobservat

Puntea a tacut ~7 ore fara ca nimic sa semnaleze: discord.py memoreaza
`resume_gateway_url` primit la ultimul READY si il refoloseste la fiecare
reconectare. Cand gatewayul regional (gateway-us-east-1a) a inceput sa dea
503 la handshake, botul a reincercat la infinit acelasi host mort, cu backoff
pana la ~15 minute. Procesul era viu, deci nici systemd nici dashboardul nu
aveau ce vedea, iar `gateway.discord.gg` raspundea normal tot timpul.

GatewayWatchdog numara de cat timp e legatura jos (on_connect / on_resumed /
on_ready o ridica, on_disconnect o coboara, iar reincercarile esuate nu
reseteaza ceasul). Peste `GATEWAY_WATCHDOG_S` (implicit 300s) alerteaza, iese
cu codul 3 si lasa systemd sa reporneasca — restartul e singurul lucru care
forteaza un IDENTIFY nou pe gateway.discord.gg. Un prag <= 0 il dezactiveaza.

O iesire la 300s ramane sub StartLimitBurst=5/5min, deci bucla de restart nu
poate ajunge sa lase unitul `failed`. `_hard_exit_after` acopera cazul in care
`close()` se blocheaza pe socketul mort.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q4uzvgm7AyJch5WH8QHRhY
This commit is contained in:
Claude Agent
2026-09-01 07:37:38 +00:00
parent 0ace9f97ac
commit ed7eca4fc7
3 changed files with 146 additions and 1 deletions

View File

@@ -421,3 +421,54 @@ async def test_cererea_de_aprobare_posteaza_butoane_in_fir(bridge, allowed):
assert "rm -rf /tmp/x" in ch.texts[0]
if bot.discord is not None:
assert ch.sent[0].kwargs.get("view") is not None
# ---------------------------------------------------------------- watchdog
class FakeClock:
def __init__(self):
self.t = 0.0
def __call__(self):
return self.t
def test_watchdogul_tace_cat_timp_gatewayul_e_sus():
clock = FakeClock()
wd = bot.GatewayWatchdog(300.0, now=clock)
wd.on_up()
clock.t = 10_000.0 # bot linistit ore intregi: niciun eveniment intre timp
assert wd.downtime() == 0.0
assert not wd.expired()
def test_watchdogul_iarta_o_reconectare_scurta():
clock = FakeClock()
wd = bot.GatewayWatchdog(300.0, now=clock)
wd.on_down()
clock.t = 4.0
wd.on_up() # RESUMED
clock.t = 900.0
assert not wd.expired()
def test_watchdogul_expira_dupa_prag_iar_reincercarile_nu_reseteaza_ceasul():
clock = FakeClock()
wd = bot.GatewayWatchdog(300.0, now=clock)
wd.on_down()
for t in (60.0, 120.0, 240.0): # fiecare handshake 503 mai da un on_disconnect
clock.t = t
wd.on_down()
assert not wd.expired()
clock.t = 300.0
assert wd.expired()
assert wd.downtime() == 300.0
def test_watchdogul_se_poate_dezactiva():
clock = FakeClock()
wd = bot.GatewayWatchdog(0.0, now=clock)
wd.on_down()
clock.t = 100_000.0
assert not wd.expired()