66 Commits

Author SHA1 Message Date
Claude Agent
70f717d874 feat(import): #11-14 router import Treapta 2 — upload+staging+mapare+preview+commit+export
Implementeaza app/api/v1/import_router.py (router nou, montat in app):

POST /v1/import — upload xlsx/csv, staging in import_batches/import_rows,
  PII criptat Fernet (Issue 5a), BEGIN IMMEDIATE+executemany (Issue 6),
  purge_after 90z (T16), sugestii fuzzy coloane DRY (Issue 5b/Eng#4),
  detectie drift semnatura (T4/D3), multisheet support

GET/POST /v1/import/{id}/column-mapping — mapare coloane per cont cu
  semnatura + drift detection

GET /v1/import/{id}/preview — 6 stari per rand (ok/needs_mapping/needs_data/
  needs_review/already_sent/duplicate_in_file), already_sent batch lookup
  nu N+1 (Eng#5), intra-batch collision EXCLUSIV preview (OV-3/T11)

POST /v1/import/{id}/commit — gate HARD N confirmat (T5/D3), atestare pe
  valori (Voce#1), INSERT ON CONFLICT DO NOTHING TOCTOU (Issue 1/T12),
  import_attestations rows_hash+n_confirmed (Voce#9), batch_id/row_index (T7)

GET /v1/import/{id}/export-failed — CSV randuri esuate cu motiv (T8)

Teste: 36 cazuri noi in tests/test_import_api.py; 243 total, toate verzi.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:41:59 +00:00
Claude Agent
2c8367109c feat(parser): T14 — parser core 2-treceri xlsx/csv + T3 coercion + T10 data col-level + T13 robustete RO
Implementeaza app/import_parse.py (modul nou) cu toate cele 4 taskuri parser:

T14 (task #7): schelet parser + arhitectura 2-treceri (Issue 2 consens cross-model)
- Trecerea 1 read_only=True: dim-check FileTooLarge (>5000 randuri / >5MB) + detectie multi-sheet
- Trecerea 2 normal-mode: header + merged cells + body
- CSV delimiter sniff (csv.Sniffer + proba explicita {; , \t}) — export RO foloseste ;
- Encoding: UTF-8-sig -> UTF-8 -> cp1250 -> latin2 (fallback RO)
- Exceptii custom: FileTooLarge, HeaderError, MultipleSheets
- Coloane duplicate din merge: sufixate _2/_3 (nu HeaderError)

T3 (task #8): coercion guard + needs_review + mesaj formule-None
- VIN numeric (openpyxl citeste ca int/float) -> coercion_flags[row] needs_review
- Odometru float 123456.0 -> tunde .0 inainte de validare (§3.4bis, ordonare critica)
- Rata None > 60% pe coloana obligatorie -> formula_columns (Issue 3 mesaj specific)
- Datetime nativ -> convertit la YYYY-MM-DD (neambiguu)

T10 (task #9): dezambiguizare data la nivel de COLOANA (OV-8)
- Datetime nativ -> "native" (direct, fara ambiguitate)
- String: detectie format din intreaga coloana — daca oricare rand are token[1]>12 -> DD-first
- Daca toti zi<=12 -> "ambiguous" -> needs_review per rand
- parse_date_value() helper pentru preview resolve

T13 (task #10): robustete export RO
- Multi-sheet: >1 sheet non-gol -> MultipleSheets([...]) cu lista; sheet ales -> parse normal
- Merged header: _unmerge_header propaga valoarea topleft la toate coloanele din grup
- Footer trim: randuri trailing cu VIN + data ambele goale -> skip structural (nu needs_data)

Teste: 37 teste verzi in tests/test_import_parse.py (fixture-uri xlsx generate in-memory).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:31:27 +00:00
Claude Agent
ef52dc2823 feat(import): T16 job purjare + purge_after SET la sent (OV-5)
- mark(sent): seteaza purge_after = now + 90 zile (GDPR/L.142)
- purge_expired(conn): sterge submissions sent expirate + import_batches expirate
  (import_rows via ON DELETE CASCADE). NULL purge_after = nu expira.
- run(): tick de purjare odata pe ora (guard _last_purge_time + _PURGE_INTERVAL_S)
  NU mai agresiv, nu blocheaza trimiterea
- 8 teste: purge_after la sent, alte stari fara purge, expirati vs neexpirat,
  queued neatins, cascade import_rows, null purge_after pastrat

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:24:59 +00:00
Claude Agent
8cdfc976e4 feat(import): T7 batch_id scope reresolve_account — R1 INCHIS
- reresolve_account(conn, account_id, batch_id=None):
  - batch_id specificat -> scope la batch-ul exact (import commit explicit)
  - fara batch_id (POST /v1/mapari) -> EXCLUSIV canal API (batch_id IS NULL)
  - salvarea unei mapari NU mai re-queues randuri cross-batch (R1 inchis)
- 6 teste: izolare batch A/B, regresie API canal, batch explicit nu atinge API,
  schema batch_id/row_index, 3 batches izolate

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:23:11 +00:00
Claude Agent
61a7b4ea1c feat(import): T6 gate auto_send pe coduri nou-mapate (OV-1)
- load_mapping_meta: {cod_op_service -> {cod_prestatie, auto_send}}
- has_no_auto_send: verifica daca vreun item rezolvat via mapping are auto_send=0
- reresolve_account: auto_send=0 -> ramane needs_mapping (review_manual stat),
  NU trece pe queued; previne FINALIZATA eronat permanent
- reresolve_account primeste batch_id optional (pregatire T7, urmeaza)
- POST /v1/prezentari: auto_send=0 -> needs_mapping + motiv explicit
- 9 teste: load_mapping_meta, has_no_auto_send, reresolve (zero/unu), POST API

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:21:32 +00:00
Claude Agent
12f0ca3a81 feat(import): T1 accounts.rar_creds_enc durabil + worker fallback + gate purjare
- worker: _creds_from_account(conn, account_id) — fallback la accounts.rar_creds_enc
  cand submission n-are creds (canal web fara re-pusher, restart worker)
- run(): creds = _creds_for(claimed, settings) OR _creds_from_account(conn, account_id)
- gate purjare (Voce#5): comentariu explicit — sterge DOAR submissions.rar_creds_enc,
  NU accounts.rar_creds_enc (inofensiv pt canal web, neatins pt canal API)
- POST /v1/conturi/rar-creds: seteaza creds durabile criptate Fernet per cont
- DELETE /v1/conturi/rar-creds: revenire la modelul efemer Treapta 1
- 7 teste: fallback, restart, coada mixta, endpoint set/delete, gate purjare

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:18:41 +00:00
Claude Agent
4ea21a034e feat(import): T9 canonicalize_row + build_key partajat (idempotency)
- canonicalize_row: VIN upper, odometru strip ".0" (Excel float coercion),
  data strip — INAINTE de validare si cheie (§3.4bis)
- build_key: aplica account_or_default(None->1) inainte de hash (OV-2):
  canal API (None) si canal import (1) produc aceeasi cheie
- build_key_legacy: helper dual-lookup pentru randuri DB vechi (pre-T9)
- router.py: POST /v1/prezentari foloseste build_key(account_id, canonicalize_row(content))
- 14 teste: canonicalizare, cross-canal, dedup float/int odometru, legacy

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:15:59 +00:00
Claude Agent
80897ccbb1 feat(foundation): schema Treapta 2 + migrari aditive + openpyxl pinned (#1)
- accounts.rar_creds_enc TEXT (creds RAR durabile per-cont, D4)
- submissions.batch_id, row_index (T7 scoping R1)
- submissions.purge_after (T16 GDPR)
- Tabele noi: column_mappings, import_batches, import_rows, import_attestations
- _migrate idempotent pe DB veche (ALTER aditiv, pattern existent)
- openpyxl==3.1.5 adaugat in requirements.txt (Issue 4, PINNED)
- 15 teste noi: coloane, tabele, idempotenta, migrare DB veche, openpyxl

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:13:19 +00:00
Claude Agent
6ab22ea0fb feat(T5/dashboard): import DBF idempotent + nomenclator browser + audit CSV + stare RAR
T5 (tools/import_dbf.py): citire prestatii_rar.DBF / mapare_prestatii.DBF cu
dbfread, raport dry-run (randuri valide/duplicate/goale, mapari orfane = cod
necunoscut in nomenclator), --commit cu upsert idempotent in tranzactie.

Dashboard: browser nomenclator, indicator stare RAR (indisponibil? derivat din
ultimul login < 30h, coada arata ultima stare locala), export audit CSV
(/v1/audit/export?status=sent|all&date_from&date_to, b64Image exclus,
coloana purge_after pentru retentia 90z).

Verify: 11 teste noi (test_import_dbf 6, test_dashboard 5), suita 111 pass,
dry-run real pe DBF-urile din repo + smoke live dashboard/CSV.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:32:26 +00:00
Claude Agent
6fb92466cb feat(T6/T7): supervizare worker (healthcheck+autoheal) + backup online + cheie partajata
T6 — worker supravegheat:
- app/worker/healthcheck.py: probe pe heartbeat-ul din DB (beat invechit -> exit 1).
  Prinde worker-ul agatat (proces viu, beat inghetat) pe care restart:always nu-l
  vede. Cablat ca healthcheck pe serviciul worker in compose.
- sidecar autoheal: restarteaza efectiv containerul unhealthy (compose simplu doar
  marcheaza, nu restarteaza la unhealthy).

T7 — deploy:
- tools/backup.py: backup ONLINE via Connection.backup (WAL nu se copiaza sigur cu
  cp); --keep N roteste snapshot-urile.
- .env.example documenteaza env-urile; volum persistent numit deja in compose.

Fix critic (split api/worker in 2 containere): AUTOPASS_CREDS_KEY trebuie PARTAJATA
api<->worker, altfel worker nu decripteaza creds-urile criptate de API -> submission
blocate. Acum impusa in compose (${...:?} -> fail explicit daca lipseste).
.gitignore: exceptie !.env.example.

5 teste noi (tests/test_deploy.py). 100 pass total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:20:18 +00:00
Claude Agent
fbb2695336 feat(creds): livrare creds per-cerere la worker (criptat efemer + sesiuni per-cont)
Plan sect.5: parola RAR vine per-cerere, stocata CRIPTATA in submission pana la
primul login reusit pe cont, apoi stearsa; JWT 30h acopera restul.

- app/crypto.py: Fernet, cheie din AUTOPASS_creds_key (nesetata -> efemera la
  runtime, creds nu supravietuiesc restartului). encrypt/decrypt_creds.
- schema + migrare: submissions.rar_creds_enc (creds criptate).
- ingestie: cripteaza rar_credentials, le lipeste de fiecare submission nou.
  Niciodata in clar in DB.
- worker: AccountSessions (login per-cont cu creds decriptate, cache JWT in
  memorie, sterge creds-urile contului dupa primul login + refresh nomenclator).
  401 creds gresite -> error fara retry; token expirat -> invalidare + requeue;
  fara creds (restart) -> requeue "indisponibile" (ROAAUTO re-trimite).
  claim_one intoarce account_id + creds_enc; recover_orphans filtrabil pe cont.
- requirements: cryptography==46.0.5.

Nota: refresh nomenclator e acum lazy la primul login per-cont (nu la pornire);
seed-ul fallback acopera editorul offline.

10 teste noi (tests/test_creds_delivery.py). 95 pass total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:16:16 +00:00
Claude Agent
c17c1aa4f4 feat(securitate-CORE): redactare creds + auth API-key per cont
Redactare:
- handler RequestValidationError dropeaza input/ctx din 422 (vectorul de
  scurgere a rar_credentials.password pe /v1/prezentari); pastreaza type/loc/msg
- app/security.py: scrub/scrub_text + CredentialRedactingFilter pe root+uvicorn
- models.py: password cu repr=False

Auth API-key:
- app/auth.py: hash SHA-256 in api_keys (cheia in clar emisa o singura data),
  header X-API-Key / Authorization: Bearer, dependency resolve_account_id
- enforcement pe flag AUTOPASS_require_api_key (prod on->401, dev off->cont
  default id=1; cheie prezenta invalida->401 mereu)
- account_id real curge din cheie in ingestie + mapare
- tools/apikey.py: CLI create/rotate/revoke/list (fara endpoint HTTP admin)

16 teste noi (tests/test_security.py). 85 pass total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:02:07 +00:00
Claude Agent
a6df3b636f feat(T5): editor web mapare operatii (hibrid + fuzzy + on-demand needs_mapping)
T5 reinterpretat: nu import DBF, ci editor web al maparii operatie ROAAUTO ->
cod RAR, cu fuzzy lookup si validare de catre utilizator.

- Contract hibrid: item prestatie accepta cod_prestatie (RAR direct, back-compat)
  SAU cod_op_service+denumire (mapat de gateway prin operations_mapping).
- Ingestie: op intern necunoscut -> submission needs_mapping (nu pleaca la RAR);
  codul rezolvat se scrie inapoi in payload_json -> payload builder + worker neatinse.
- Editor HTMX (_mapari.html + GET /_fragments/mapari, POST /mapari): listeaza
  op-urile nemapate, fuzzy preselecteaza codul RAR, save -> re-rezolvare automata
  (queued / needs_data).
- Fuzzy: rapidfuzz.token_sort_ratio pe denumire normalizata (fara diacritice).
- Nomenclator: seed fallback 18 coduri la boot (offline) + refresh live din worker.
- Cont default id=1 cat timp auth API-key (CORE) nu exista (account_id NULL).
- Endpointuri API: GET /v1/mapari/pending, POST /v1/mapari (respinge cod inexistent).
- 15 teste noi (tests/test_mapping.py); 69 pass total.
- Contract actualizat (docs/api-rar-contract.md), rapidfuzz==3.14.5 in requirements.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:25:21 +00:00
Claude Agent
77088daf29 feat(T2): reconciliere anti-duplicat + retry/backoff + recuperare orfane
Inchide bucla de trimitere (plan.md sect. 4 worker, failure registry).

- app/reconcile.py: match_finalizata pe vin+dataPrestatie+odometruFinal (int),
  alege id maxim la duplicate (RAR accepta duplicate, confirmat live)
- app/rar_client.get_finalizate: parseaza data.content (descoperit live ca
  ruta = GET /prezentari/getAllPrezentariFinalizate; filtrele nu merg pe test)
- app/worker rescris:
  - recuperare orfane (rand 'sending' peste lease = worker mort mid-POST)
  - pe eroare tranzitorie/timeout: reconciliere INAINTE de re-send (anti-duplicat);
    daca recordul exista la RAR -> sent fara re-POST
  - retry/backoff exponential; peste worker_max_retries -> error + banner
  - re-login la token expirat (JWT 30h)
- schema: coloana next_attempt_at (backoff) + migrare aditiva in init_db
- config: worker_sending_lease_s, worker_retry_base_s/max_s, worker_max_retries
- contract: documentata ruta+forma getAllPrezentariFinalizate (verificat live)

Verify: pytest 54 passed (15 noi T2) + validare live (reconciliere record 68514).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 18:20:32 +00:00
Claude Agent
36d1b916d5 feat(T4): payload builder finalizat + snapshot test
- app/payload.py rafinat: odometruFinal/odometruInitial string (initial gol -> null),
  evita capcana falsy `or ""` (pastreaza "0"), normalizare vin/nrInm/coduri,
  tipPrestatie niciodata trimis, obs/b64Image omise cand lipsesc
- tests/test_payload.py: 10 teste, inclusiv snapshot vs exemplul oficial din contract

Verify: pytest 39 passed (29 + 10).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 17:28:33 +00:00
Claude Agent
2117ab5c1e feat(T3): validare completa prezentari + 29 teste
- app/validation.py: reguli de continut (VIN ^[A-HJ-NPR-Z0-9]{17}$ fara O/I/Q,
  nrInm ^[A-Z0-9]{1,10}$, dataPrestatie ∈ [2024-12-01, azi] TZ Bucuresti,
  R-ODO/I-ODO -> odometruInitial obligatoriu, odometruInitial<=odometruFinal,
  odometruFinal numeric, prestatii nevide, b64Image base64 valid)
- erori structurate {field, message} (aceeasi forma ca raspunsul RAR), fara exceptii
- modele Pydantic: normalizare strip/upper pe vin/nrInm/coduri
- router /v1/prezentari: validare inainte de enqueue; esec continut -> needs_data
  (tinut, vizibil in dashboard cu motiv), NU 422; JSON malformat -> 422 (shape)
- tests/: 29 teste (per regula + rutare API + idempotenta)

Verify: pytest 29 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:49:20 +00:00