feat(embeddings): cache persistent de vectori in SQLite + warmup in fundal
Vectorii corpusului k-NN persista in tabela embedding_cache (PK model+text_hash,
blob float32 LE); la warmup se vectorizeaza doar textele lipsa din cache, deci
restartul cu corpus neschimbat nu mai plateste ~1-2 min de embed (embed=0).
- app/embedding_cache.py: serializare array('f'), load/save/purge chunk 500 cu
BEGIN/COMMIT explicit (conexiuni autocommit), validare dimensiune la scriere
si citire, orchestrare sync_corpus_vectors cu embed_fn injectat
- index_corpus(vectors=): vectori precalculati cu validare aliniere; mismatch
-> fallback embed complet
- ensure_embeddings_corpus: warmup in thread la startup (block=True), calea de
request ne-blocanta (acquire non-blocking pe lock; warmup in curs -> return
imediat); purjare orfane + modele vechi doar dupa indexare reusita
- log warmup: cache=N embed=M in Xs
- 31 teste noi (cold/warm/incremental, model schimbat, concurenta, ranking
exact, echivalenta float32); suita completa 1596 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -48,7 +48,9 @@ Reguli de contract (detalii in `docs/api-rar-contract.md`): `FINALIZATA` e termi
|
||||
> PRD-uri (`docs/prd/prd-X.Y-*.md`), linkate in coloana Detalii. La fiecare livrabila terminata:
|
||||
> schimba statusul + data + linkul PRD si actualizeaza "Ultima actualizare".
|
||||
|
||||
**Ultima actualizare**: 2026-07-06 — **HARDENING 80/20 LIVRAT + PUSH** (`main` la `63b6cbc`; plan /autoplan `main-hardening-8020-plan-20260703.md`, audit securitate 2026-07-03, inainte de expunerea publica autopass.romfast.ro). **P0**: compose fail-safe — `:?` obligatoriu pe `AUTOPASS_REQUIRE_API_KEY`/`AUTOPASS_WORKER_SEND_ENABLED`/`AUTOPASS_RAR_ENV` (split-brain api-prod/worker-test eliminat; post-5.20 variabila e doar ancora de fallback, nu tinta trimiterilor), `FORWARDED_ALLOW_IPS="*"` ca rate-limit-ul sa vada IP-ul real dupa Traefik (invariante in comentariu: api fara `ports:` pe host, Traefik fara `forwardedHeaders.insecure`), `AUTOPASS_SESSION_SECRET` obligatoriu + `SESSION_HTTPS_ONLY` default true; invarianta E1 fail-fast la boot (`validate_prod_invariants`): prod fara cheie API sau session secret -> RuntimeError cu mesaj actionabil. **P0-4 backup criptat SQLite**: `tools/backup_db.sh` (snapshot online stdlib + gpg AES256 + retentie + rclone optional), `tools/restore_check.sh` (integrity_check pe restore), `docs/backup.md`; trigger dur: configurat INAINTE de prima declaratie reala prod. **P1**: headers de securitate in-app (`SecurityHeadersMiddleware`, HSTS doar pe https) + body-cap global 10MB (`BodyCapMiddleware` ASGI pur, 413 inainte de parserul multipart/JSON; verificarea per-endpoint ramane strat 2) + imagine Docker non-root (uid 10001, `HOME=/home/app` pt fastembed, chown /data INAINTE de VOLUME, loguri pe `/data/logs`, port 8010 aliniat EXPOSE/CMD; migrare one-off volum existent: `docker compose run --rm --user root api chown -R app:app /data`). **P2**: print-uri cu email in stdout eliminate (signup + notify degradat din `app/email.py` -> `log_event` fara PII) + fix crestere monotona chei `ratelimit._hits`. Executie multi-agent (4 agenti implementare pe fisiere disjuncte + review dedicat: 0 constatari blocante, 3 nit-uri acceptate). Suita completa: 1557 passed, 1 skipped (3 rulari independente). **Post-deploy HOUR-1 (Dokploy)**: seteaza env-urile obligatorii INAINTE de redeploy (`:?` pica pornirea), creeaza cheia API + tier pro/trial pe contul propriu (altfel 403 PLAN_FARA_API), ruleaza migrarea chown, seteaza passphrase-ul de backup.
|
||||
**Ultima actualizare**: 2026-07-06 — **5.21 CACHE PERSISTENT EMBEDDINGS IN SQLITE — LIVRAT** (PRD: [prd-5.21](prd/prd-5.21-embedding-cache-sqlite.md); executie echipa de agenti Sonnet: cache-core T1+T2, wiring T3+T5, tests T4+T6, reviewer adversarial). Vectorii corpusului k-NN (17.181 exemple SILVER) persista in tabela noua `embedding_cache` (PK `(model, text_hash)`, blob float32 LE 384, `array('f')` end-to-end); la warmup se vectorizeaza DOAR miss-urile (hash sha256 pe `denumire_normalizata` din lista FILTRATA, identica cu ce intra in `index_corpus`), restarturile cu corpus neschimbat = `embed=0`. Modul nou `app/embedding_cache.py` (fara dependinte noi): `sync_corpus_vectors(conn, model, texts, embed_fn)` hash->load->embed(miss)->save; scrieri/stergeri in chunk-uri de 500 cu `BEGIN`/`COMMIT` EXPLICIT (conexiunile sunt autocommit — `conn.commit()` e no-op); validare dimensiune la scriere SI citire (blob corupt = miss re-vectorizat); purjare orfane + modele vechi DOAR dupa indexare reusita; toate erorile SQLite -> `log.warning` + fallback embed complet (degradare gratioasa, nimic propagat). `EmbeddingEngine.index_corpus(vectors=)` valideaza alinierea `len(vectors)==len(items)` + absenta None (mismatch -> fallback embed complet). Secventa protejata de `_embeddings_lock` (mapping.py); pe calea de request (`block=False`) acquire NE-blocant — daca warmup-ul tine lock-ul, return imediat (fix MAJOR din review: altfel primul request in fereastra de cold start ingheta ~1-2 min). Log observabilitate: `embeddings: warmup ok cache=N embed=M in Xs` (eticheta `corpus reindexat` pe calea de request). Semnatura corpusului (`_corpus_signature_silver`) = mecanism de decizie NESCHIMBAT; worker-ul nu atinge cache-ul. Review adversarial: 1 MAJOR + 2 MINOR + 1 NIT, toate reparate. Suita completa: **1596 passed, 1 skipped** (+31 teste noi: 15/15 cai din diagrama de acoperire, toate cu `AUTOPASS_EMBEDDINGS_ENABLED=1` explicit, backend mock). Crestere DB ~26-30MB acceptata; deferate in TODOS.md: CLI embcache, precompute la build, graceful reload, numpy la >50k randuri. **De verificat pe instanta reala (US-006)**: doua restarturi `./start.sh`, al doilea cu `embed=0` si sugestia "similaritate" in <30s.
|
||||
|
||||
> 2026-07-06 — **HARDENING 80/20 LIVRAT + PUSH** (`main` la `63b6cbc`; plan /autoplan `main-hardening-8020-plan-20260703.md`, audit securitate 2026-07-03, inainte de expunerea publica autopass.romfast.ro). **P0**: compose fail-safe — `:?` obligatoriu pe `AUTOPASS_REQUIRE_API_KEY`/`AUTOPASS_WORKER_SEND_ENABLED`/`AUTOPASS_RAR_ENV` (split-brain api-prod/worker-test eliminat; post-5.20 variabila e doar ancora de fallback, nu tinta trimiterilor), `FORWARDED_ALLOW_IPS="*"` ca rate-limit-ul sa vada IP-ul real dupa Traefik (invariante in comentariu: api fara `ports:` pe host, Traefik fara `forwardedHeaders.insecure`), `AUTOPASS_SESSION_SECRET` obligatoriu + `SESSION_HTTPS_ONLY` default true; invarianta E1 fail-fast la boot (`validate_prod_invariants`): prod fara cheie API sau session secret -> RuntimeError cu mesaj actionabil. **P0-4 backup criptat SQLite**: `tools/backup_db.sh` (snapshot online stdlib + gpg AES256 + retentie + rclone optional), `tools/restore_check.sh` (integrity_check pe restore), `docs/backup.md`; trigger dur: configurat INAINTE de prima declaratie reala prod. **P1**: headers de securitate in-app (`SecurityHeadersMiddleware`, HSTS doar pe https) + body-cap global 10MB (`BodyCapMiddleware` ASGI pur, 413 inainte de parserul multipart/JSON; verificarea per-endpoint ramane strat 2) + imagine Docker non-root (uid 10001, `HOME=/home/app` pt fastembed, chown /data INAINTE de VOLUME, loguri pe `/data/logs`, port 8010 aliniat EXPOSE/CMD; migrare one-off volum existent: `docker compose run --rm --user root api chown -R app:app /data`). **P2**: print-uri cu email in stdout eliminate (signup + notify degradat din `app/email.py` -> `log_event` fara PII) + fix crestere monotona chei `ratelimit._hits`. Executie multi-agent (4 agenti implementare pe fisiere disjuncte + review dedicat: 0 constatari blocante, 3 nit-uri acceptate). Suita completa: 1557 passed, 1 skipped (3 rulari independente). **Post-deploy HOUR-1 (Dokploy)**: seteaza env-urile obligatorii INAINTE de redeploy (`:?` pica pornirea), creeaza cheia API + tier pro/trial pe contul propriu (altfel 403 PLAN_FARA_API), ruleaza migrarea chown, seteaza passphrase-ul de backup.
|
||||
|
||||
> 2026-06-18 — 3.4 LIVRAT (interfata web ergonomica: tab-uri + wizard + microcopy). US-001 modul pur `app/web/labels.py` (stari tehnice→text uman + clasa CSS; test parametrizat din CHECK-ul `schema.sql` iese rosu la stare nemapata). US-002 bara status `/_fragments/status` + `_status.html` (etichete umane, defalcare blocate pe motiv, poll 15s, scoped pe cont). US-003 shell 6 tab-uri (Acasa·Import·Coada·Mapari·Cont·Nomenclator) cu deep-link `?tab=`, panou activ randat server-side, fragmente inactive lazy pe click, ARIA real (tablist/tab/tabpanel + aria-selected + navigare cu sageti). US-004 stepper import 4 pasi (PUR vizual, `hx-target="#import-section"` + csrf pastrate). US-005 Acasa onboarding checklist auto-bifat (are_creds/are_trimiteri) + colaps cand totul gata + empty states prietenoase Coada/Mapari. VERIFY lead-driven (TestClient ACs + 434 pytest pass; E2E browser/RAR LIVE neprobat in sesiune — recomandata probare manuala `--send`). Fix izolare teste (reset `ratelimit._hits` in fixturi, 429 la rulare subset). `/code-review` high: regasit avertisment "cont in asteptare de activare" (regresie din scoaterea `/_fragments/banner`) re-introdus in bara status + culori hardcodate→variabile paleta. 434 teste pass. Backend trimitere neatins. PRD: [prd-3.4](prd/prd-3.4-ux-dashboard-web.md). Urmeaza Etapa 4 (4.1 mapare AI/MCP).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user