Commit Graph

24 Commits

Author SHA1 Message Date
Claude Agent
854db66abc feat(ui): #15 U5 — web upload import (HTMX) drop→mapare→preview→confirma
Implementare completa U5 din plan-treapta2.md (sectiunea 13):

- _upload.html: drop zone + buton accesibil (a11y: drag nu e la tastatura),
  drag-and-drop JS, mesaj 'NU se trimite nimic pana confirmi',
  selector foi pt multi-sheet xlsx, stari eroare/mesaj
- _mapcoloane.html: formular mapare coloane cu .maprow/.mapcol.grow,
  sugestii fuzzy pre-selectate, etiichete <label> vizibile, sample values,
  format data configurabil
- _preview_import.html: tabel 6 stari, pills rezumat, filtre pe stare,
  .chk per-rand pe needs_review (D11), banner declarant .banner.warn
  direct deasupra input-ului N (D12), bara confirmare sticky,
  text 'dubla cu randul N' pe duplicate_in_file (D10 daltonism),
  link export CSV randuri esuate
- base.html: .s-needs_review (warn), .s-already_sent/.s-duplicate_in_file
  (muted), .drop-zone, .banner.warn, .sticky-bar, .htmx-indicator
- routes.py: rute /_import/upload/mapare-coloane/preview/reset/confirma;
  helper _web_compute_preview refoloseste _resolve_row_for_preview,
  _already_sent_lookup, _signature din import_router (fara a-l edita);
  commit ON CONFLICT DO NOTHING (TOCTOU); log atestare
- tests/test_import_ui.py: 15 teste (dashboard, upload, mapare, preview,
  confirmare N corect/gresit, reset, erori, multi-sheet, a11y D10/D11/D12)

279 teste total, 0 esecuri.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 21:04:56 +00:00
Claude Agent
70f717d874 feat(import): #11-14 router import Treapta 2 — upload+staging+mapare+preview+commit+export
Implementeaza app/api/v1/import_router.py (router nou, montat in app):

POST /v1/import — upload xlsx/csv, staging in import_batches/import_rows,
  PII criptat Fernet (Issue 5a), BEGIN IMMEDIATE+executemany (Issue 6),
  purge_after 90z (T16), sugestii fuzzy coloane DRY (Issue 5b/Eng#4),
  detectie drift semnatura (T4/D3), multisheet support

GET/POST /v1/import/{id}/column-mapping — mapare coloane per cont cu
  semnatura + drift detection

GET /v1/import/{id}/preview — 6 stari per rand (ok/needs_mapping/needs_data/
  needs_review/already_sent/duplicate_in_file), already_sent batch lookup
  nu N+1 (Eng#5), intra-batch collision EXCLUSIV preview (OV-3/T11)

POST /v1/import/{id}/commit — gate HARD N confirmat (T5/D3), atestare pe
  valori (Voce#1), INSERT ON CONFLICT DO NOTHING TOCTOU (Issue 1/T12),
  import_attestations rows_hash+n_confirmed (Voce#9), batch_id/row_index (T7)

GET /v1/import/{id}/export-failed — CSV randuri esuate cu motiv (T8)

Teste: 36 cazuri noi in tests/test_import_api.py; 243 total, toate verzi.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:41:59 +00:00
Claude Agent
2c8367109c feat(parser): T14 — parser core 2-treceri xlsx/csv + T3 coercion + T10 data col-level + T13 robustete RO
Implementeaza app/import_parse.py (modul nou) cu toate cele 4 taskuri parser:

T14 (task #7): schelet parser + arhitectura 2-treceri (Issue 2 consens cross-model)
- Trecerea 1 read_only=True: dim-check FileTooLarge (>5000 randuri / >5MB) + detectie multi-sheet
- Trecerea 2 normal-mode: header + merged cells + body
- CSV delimiter sniff (csv.Sniffer + proba explicita {; , \t}) — export RO foloseste ;
- Encoding: UTF-8-sig -> UTF-8 -> cp1250 -> latin2 (fallback RO)
- Exceptii custom: FileTooLarge, HeaderError, MultipleSheets
- Coloane duplicate din merge: sufixate _2/_3 (nu HeaderError)

T3 (task #8): coercion guard + needs_review + mesaj formule-None
- VIN numeric (openpyxl citeste ca int/float) -> coercion_flags[row] needs_review
- Odometru float 123456.0 -> tunde .0 inainte de validare (§3.4bis, ordonare critica)
- Rata None > 60% pe coloana obligatorie -> formula_columns (Issue 3 mesaj specific)
- Datetime nativ -> convertit la YYYY-MM-DD (neambiguu)

T10 (task #9): dezambiguizare data la nivel de COLOANA (OV-8)
- Datetime nativ -> "native" (direct, fara ambiguitate)
- String: detectie format din intreaga coloana — daca oricare rand are token[1]>12 -> DD-first
- Daca toti zi<=12 -> "ambiguous" -> needs_review per rand
- parse_date_value() helper pentru preview resolve

T13 (task #10): robustete export RO
- Multi-sheet: >1 sheet non-gol -> MultipleSheets([...]) cu lista; sheet ales -> parse normal
- Merged header: _unmerge_header propaga valoarea topleft la toate coloanele din grup
- Footer trim: randuri trailing cu VIN + data ambele goale -> skip structural (nu needs_data)

Teste: 37 teste verzi in tests/test_import_parse.py (fixture-uri xlsx generate in-memory).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:31:27 +00:00
Claude Agent
ef52dc2823 feat(import): T16 job purjare + purge_after SET la sent (OV-5)
- mark(sent): seteaza purge_after = now + 90 zile (GDPR/L.142)
- purge_expired(conn): sterge submissions sent expirate + import_batches expirate
  (import_rows via ON DELETE CASCADE). NULL purge_after = nu expira.
- run(): tick de purjare odata pe ora (guard _last_purge_time + _PURGE_INTERVAL_S)
  NU mai agresiv, nu blocheaza trimiterea
- 8 teste: purge_after la sent, alte stari fara purge, expirati vs neexpirat,
  queued neatins, cascade import_rows, null purge_after pastrat

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:24:59 +00:00
Claude Agent
8cdfc976e4 feat(import): T7 batch_id scope reresolve_account — R1 INCHIS
- reresolve_account(conn, account_id, batch_id=None):
  - batch_id specificat -> scope la batch-ul exact (import commit explicit)
  - fara batch_id (POST /v1/mapari) -> EXCLUSIV canal API (batch_id IS NULL)
  - salvarea unei mapari NU mai re-queues randuri cross-batch (R1 inchis)
- 6 teste: izolare batch A/B, regresie API canal, batch explicit nu atinge API,
  schema batch_id/row_index, 3 batches izolate

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:23:11 +00:00
Claude Agent
61a7b4ea1c feat(import): T6 gate auto_send pe coduri nou-mapate (OV-1)
- load_mapping_meta: {cod_op_service -> {cod_prestatie, auto_send}}
- has_no_auto_send: verifica daca vreun item rezolvat via mapping are auto_send=0
- reresolve_account: auto_send=0 -> ramane needs_mapping (review_manual stat),
  NU trece pe queued; previne FINALIZATA eronat permanent
- reresolve_account primeste batch_id optional (pregatire T7, urmeaza)
- POST /v1/prezentari: auto_send=0 -> needs_mapping + motiv explicit
- 9 teste: load_mapping_meta, has_no_auto_send, reresolve (zero/unu), POST API

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:21:32 +00:00
Claude Agent
12f0ca3a81 feat(import): T1 accounts.rar_creds_enc durabil + worker fallback + gate purjare
- worker: _creds_from_account(conn, account_id) — fallback la accounts.rar_creds_enc
  cand submission n-are creds (canal web fara re-pusher, restart worker)
- run(): creds = _creds_for(claimed, settings) OR _creds_from_account(conn, account_id)
- gate purjare (Voce#5): comentariu explicit — sterge DOAR submissions.rar_creds_enc,
  NU accounts.rar_creds_enc (inofensiv pt canal web, neatins pt canal API)
- POST /v1/conturi/rar-creds: seteaza creds durabile criptate Fernet per cont
- DELETE /v1/conturi/rar-creds: revenire la modelul efemer Treapta 1
- 7 teste: fallback, restart, coada mixta, endpoint set/delete, gate purjare

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:18:41 +00:00
Claude Agent
4ea21a034e feat(import): T9 canonicalize_row + build_key partajat (idempotency)
- canonicalize_row: VIN upper, odometru strip ".0" (Excel float coercion),
  data strip — INAINTE de validare si cheie (§3.4bis)
- build_key: aplica account_or_default(None->1) inainte de hash (OV-2):
  canal API (None) si canal import (1) produc aceeasi cheie
- build_key_legacy: helper dual-lookup pentru randuri DB vechi (pre-T9)
- router.py: POST /v1/prezentari foloseste build_key(account_id, canonicalize_row(content))
- 14 teste: canonicalizare, cross-canal, dedup float/int odometru, legacy

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:15:59 +00:00
Claude Agent
80897ccbb1 feat(foundation): schema Treapta 2 + migrari aditive + openpyxl pinned (#1)
- accounts.rar_creds_enc TEXT (creds RAR durabile per-cont, D4)
- submissions.batch_id, row_index (T7 scoping R1)
- submissions.purge_after (T16 GDPR)
- Tabele noi: column_mappings, import_batches, import_rows, import_attestations
- _migrate idempotent pe DB veche (ALTER aditiv, pattern existent)
- openpyxl==3.1.5 adaugat in requirements.txt (Issue 4, PINNED)
- 15 teste noi: coloane, tabele, idempotenta, migrare DB veche, openpyxl

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:13:19 +00:00
Claude Agent
b12be3d26c style(design): FINDING-004 — htmx vendorizat local, nu din CDN (offline-first)
Dashboard-ul incarca htmx din unpkg.com. Gateway-ul ruleaza offline (LXC/VPS
+ Cloudflare Tunnel); cand CDN-ul e blocat/inaccesibil, htmx nu se incarca si
dashboard-ul ramane static: zero polling (banner alerta blocate, coada,
mapari), butoanele de mapare nu trimit. Aceeasi decizie offline-first ca fontul
UI (FINDING-002, fara dependinta CDN).

Fix: vendorizez htmx.org@1.9.12 minificat in app/web/static/, montez StaticFiles
la /static in main.py, base.html refera /static/htmx.min.js. Verificat live:
window.htmx.version=1.9.12, fragment polls 200, zero regresie vizuala.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:58:24 +00:00
Claude Agent
bba5b31540 style(design): FINDING-003 — link-uri actiune card mai usor de atins (>=36px) + antet se rupe curat pe mobil 2026-06-15 20:50:14 +00:00
Claude Agent
facb1ca8b4 style(design): FINDING-002 — font stack UI intentionat (cross-platform, fara dependinta CDN; gateway ruleaza offline) 2026-06-15 20:49:39 +00:00
Claude Agent
39c0e16248 style(design): FINDING-001 — ierarhie titluri (H1 dominant 20px, sectiuni 15px) 2026-06-15 20:49:11 +00:00
Claude Agent
db64972c1d style(design): FINDING-002 — clarifica link-uri export CSV (tot -> toate)
Link-ul 'tot' era criptic. 'export CSV: trimise' / 'toate' spune ce descarca
fiecare (don't-make-me-think).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:39:21 +00:00
Claude Agent
8d3bc6bea5 style(design): FINDING-001 — tabele scroll in card pe mobil (fix overflow orizontal pagina)
Tabelul submissions (7 coloane, 591px) iesea din card pe 390px -> toata pagina
scrolla orizontal (docScrollW 636 > 390). Wrap in .tablewrap{overflow-x:auto} +
white-space:nowrap pe celule -> scroll IN card, pagina nu mai deborda. tabular-nums
pe coloanele numerice.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:39:04 +00:00
Claude Agent
6ab22ea0fb feat(T5/dashboard): import DBF idempotent + nomenclator browser + audit CSV + stare RAR
T5 (tools/import_dbf.py): citire prestatii_rar.DBF / mapare_prestatii.DBF cu
dbfread, raport dry-run (randuri valide/duplicate/goale, mapari orfane = cod
necunoscut in nomenclator), --commit cu upsert idempotent in tranzactie.

Dashboard: browser nomenclator, indicator stare RAR (indisponibil? derivat din
ultimul login < 30h, coada arata ultima stare locala), export audit CSV
(/v1/audit/export?status=sent|all&date_from&date_to, b64Image exclus,
coloana purge_after pentru retentia 90z).

Verify: 11 teste noi (test_import_dbf 6, test_dashboard 5), suita 111 pass,
dry-run real pe DBF-urile din repo + smoke live dashboard/CSV.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:32:26 +00:00
Claude Agent
6fb92466cb feat(T6/T7): supervizare worker (healthcheck+autoheal) + backup online + cheie partajata
T6 — worker supravegheat:
- app/worker/healthcheck.py: probe pe heartbeat-ul din DB (beat invechit -> exit 1).
  Prinde worker-ul agatat (proces viu, beat inghetat) pe care restart:always nu-l
  vede. Cablat ca healthcheck pe serviciul worker in compose.
- sidecar autoheal: restarteaza efectiv containerul unhealthy (compose simplu doar
  marcheaza, nu restarteaza la unhealthy).

T7 — deploy:
- tools/backup.py: backup ONLINE via Connection.backup (WAL nu se copiaza sigur cu
  cp); --keep N roteste snapshot-urile.
- .env.example documenteaza env-urile; volum persistent numit deja in compose.

Fix critic (split api/worker in 2 containere): AUTOPASS_CREDS_KEY trebuie PARTAJATA
api<->worker, altfel worker nu decripteaza creds-urile criptate de API -> submission
blocate. Acum impusa in compose (${...:?} -> fail explicit daca lipseste).
.gitignore: exceptie !.env.example.

5 teste noi (tests/test_deploy.py). 100 pass total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:20:18 +00:00
Claude Agent
fbb2695336 feat(creds): livrare creds per-cerere la worker (criptat efemer + sesiuni per-cont)
Plan sect.5: parola RAR vine per-cerere, stocata CRIPTATA in submission pana la
primul login reusit pe cont, apoi stearsa; JWT 30h acopera restul.

- app/crypto.py: Fernet, cheie din AUTOPASS_creds_key (nesetata -> efemera la
  runtime, creds nu supravietuiesc restartului). encrypt/decrypt_creds.
- schema + migrare: submissions.rar_creds_enc (creds criptate).
- ingestie: cripteaza rar_credentials, le lipeste de fiecare submission nou.
  Niciodata in clar in DB.
- worker: AccountSessions (login per-cont cu creds decriptate, cache JWT in
  memorie, sterge creds-urile contului dupa primul login + refresh nomenclator).
  401 creds gresite -> error fara retry; token expirat -> invalidare + requeue;
  fara creds (restart) -> requeue "indisponibile" (ROAAUTO re-trimite).
  claim_one intoarce account_id + creds_enc; recover_orphans filtrabil pe cont.
- requirements: cryptography==46.0.5.

Nota: refresh nomenclator e acum lazy la primul login per-cont (nu la pornire);
seed-ul fallback acopera editorul offline.

10 teste noi (tests/test_creds_delivery.py). 95 pass total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:16:16 +00:00
Claude Agent
c17c1aa4f4 feat(securitate-CORE): redactare creds + auth API-key per cont
Redactare:
- handler RequestValidationError dropeaza input/ctx din 422 (vectorul de
  scurgere a rar_credentials.password pe /v1/prezentari); pastreaza type/loc/msg
- app/security.py: scrub/scrub_text + CredentialRedactingFilter pe root+uvicorn
- models.py: password cu repr=False

Auth API-key:
- app/auth.py: hash SHA-256 in api_keys (cheia in clar emisa o singura data),
  header X-API-Key / Authorization: Bearer, dependency resolve_account_id
- enforcement pe flag AUTOPASS_require_api_key (prod on->401, dev off->cont
  default id=1; cheie prezenta invalida->401 mereu)
- account_id real curge din cheie in ingestie + mapare
- tools/apikey.py: CLI create/rotate/revoke/list (fara endpoint HTTP admin)

16 teste noi (tests/test_security.py). 85 pass total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 20:02:07 +00:00
Claude Agent
a6df3b636f feat(T5): editor web mapare operatii (hibrid + fuzzy + on-demand needs_mapping)
T5 reinterpretat: nu import DBF, ci editor web al maparii operatie ROAAUTO ->
cod RAR, cu fuzzy lookup si validare de catre utilizator.

- Contract hibrid: item prestatie accepta cod_prestatie (RAR direct, back-compat)
  SAU cod_op_service+denumire (mapat de gateway prin operations_mapping).
- Ingestie: op intern necunoscut -> submission needs_mapping (nu pleaca la RAR);
  codul rezolvat se scrie inapoi in payload_json -> payload builder + worker neatinse.
- Editor HTMX (_mapari.html + GET /_fragments/mapari, POST /mapari): listeaza
  op-urile nemapate, fuzzy preselecteaza codul RAR, save -> re-rezolvare automata
  (queued / needs_data).
- Fuzzy: rapidfuzz.token_sort_ratio pe denumire normalizata (fara diacritice).
- Nomenclator: seed fallback 18 coduri la boot (offline) + refresh live din worker.
- Cont default id=1 cat timp auth API-key (CORE) nu exista (account_id NULL).
- Endpointuri API: GET /v1/mapari/pending, POST /v1/mapari (respinge cod inexistent).
- 15 teste noi (tests/test_mapping.py); 69 pass total.
- Contract actualizat (docs/api-rar-contract.md), rapidfuzz==3.14.5 in requirements.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 19:25:21 +00:00
Claude Agent
77088daf29 feat(T2): reconciliere anti-duplicat + retry/backoff + recuperare orfane
Inchide bucla de trimitere (plan.md sect. 4 worker, failure registry).

- app/reconcile.py: match_finalizata pe vin+dataPrestatie+odometruFinal (int),
  alege id maxim la duplicate (RAR accepta duplicate, confirmat live)
- app/rar_client.get_finalizate: parseaza data.content (descoperit live ca
  ruta = GET /prezentari/getAllPrezentariFinalizate; filtrele nu merg pe test)
- app/worker rescris:
  - recuperare orfane (rand 'sending' peste lease = worker mort mid-POST)
  - pe eroare tranzitorie/timeout: reconciliere INAINTE de re-send (anti-duplicat);
    daca recordul exista la RAR -> sent fara re-POST
  - retry/backoff exponential; peste worker_max_retries -> error + banner
  - re-login la token expirat (JWT 30h)
- schema: coloana next_attempt_at (backoff) + migrare aditiva in init_db
- config: worker_sending_lease_s, worker_retry_base_s/max_s, worker_max_retries
- contract: documentata ruta+forma getAllPrezentariFinalizate (verificat live)

Verify: pytest 54 passed (15 noi T2) + validare live (reconciliere record 68514).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 18:20:32 +00:00
Claude Agent
36d1b916d5 feat(T4): payload builder finalizat + snapshot test
- app/payload.py rafinat: odometruFinal/odometruInitial string (initial gol -> null),
  evita capcana falsy `or ""` (pastreaza "0"), normalizare vin/nrInm/coduri,
  tipPrestatie niciodata trimis, obs/b64Image omise cand lipsesc
- tests/test_payload.py: 10 teste, inclusiv snapshot vs exemplul oficial din contract

Verify: pytest 39 passed (29 + 10).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 17:28:33 +00:00
Claude Agent
2117ab5c1e feat(T3): validare completa prezentari + 29 teste
- app/validation.py: reguli de continut (VIN ^[A-HJ-NPR-Z0-9]{17}$ fara O/I/Q,
  nrInm ^[A-Z0-9]{1,10}$, dataPrestatie ∈ [2024-12-01, azi] TZ Bucuresti,
  R-ODO/I-ODO -> odometruInitial obligatoriu, odometruInitial<=odometruFinal,
  odometruFinal numeric, prestatii nevide, b64Image base64 valid)
- erori structurate {field, message} (aceeasi forma ca raspunsul RAR), fara exceptii
- modele Pydantic: normalizare strip/upper pe vin/nrInm/coduri
- router /v1/prezentari: validare inainte de enqueue; esec continut -> needs_data
  (tinut, vizibil in dashboard cu motiv), NU 422; JSON malformat -> 422 (shape)
- tests/: 29 teste (per regula + rutare API + idempotenta)

Verify: pytest 29 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:49:20 +00:00
Claude Agent
f1b5f1f80f feat: schelet gateway FastAPI (API v1 + worker + dashboard + SQLite WAL)
Structura repo conform plan.md sect. 4, booteaza cu /healthz verde:
- app/main.py: FastAPI (lifespan init_db), /healthz (worker viu + last login + queue), /metrics
- app/api/v1: POST /v1/prezentari (enqueue + dedup idempotency UNIQUE), GET prezentari/{id}, nomenclator, mapari
- app/rar_client.py: client RAR real (login/JWT, nomenclator, postPrezentare, getFinalizate) cu User-Agent obligatoriu (fix WAF 403)
- app/worker: proces separat, claim atomic BEGIN IMMEDIATE, heartbeat, login+send (send dezactivat by default)
- app/web: dashboard Jinja2+HTMX (coada, banner alerta blocate, worker viu/mort, stari empty)
- app/db.py + schema.sql: SQLite WAL, tabele accounts/api_keys/operations_mapping/nomenclator_rar/submissions/worker_heartbeat
- app/idempotency.py + payload.py: hash continut canonic + builder payload (status FINALIZATA, fara tipPrestatie)
- Dockerfile + docker-compose.yml (api+worker, volum SQLite persistent, restart:always)
- tools/import_dbf.py: stub T5

Verificat live: login prin rar_client OK (token 259), nomenclator 18 coduri, worker heartbeat -> /healthz worker_alive=True.
Ramas: T3 validare Pydantic, T4 snapshot payload, T2 reconciliere/retry worker, T5 import DBF, auth API-key, middleware redactare creds, criptare PII.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 12:04:11 +00:00