Compare commits

...

14 Commits

Author SHA1 Message Date
Claude Agent
c38807d88c docs(plan): adauga plan-treapta2.md (planul Treapta 2)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 06:51:39 +00:00
Claude Agent
9106983dd5 docs: README cu ghid de testare (browser, API, RAR, conturi/chei API)
Adauga README.md: arhitectura, configurare, rulare locala si cu start.sh,
testare in browser (dashboard, Swagger, import xlsx/csv), proba trimitere la
RAR test + verificare, formate de fisier per cont (signature_coloane), conturi
si chei API (CLI tools.apikey, exemple curl cu X-API-Key) si starea reala a
auth-ului (POST/import protejate; GET-uri de listare inca globale).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 06:51:34 +00:00
Claude Agent
0b3e2464e1 feat(ops): scripturi start.sh (test/prod, api/worker) + verificare RAR
start.sh ruleaza api/worker/both pe mediu test sau prod, cu --send pentru
trimiterea la RAR, plus status/stop. start-test.sh si start-prod.sh sunt
wrappere care fixeaza mediul. tools/rar_finalizate.py listeaza prezentarile
inregistrate la RAR (confirmare end-to-end ca au ajuns). .gitignore: .run/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 06:51:27 +00:00
Claude Agent
5a8787bbc4 chore(legacy): muta proiectul Visual FoxPro in legacy-vfp/
Codul VFP original (ROAAUTO) era doar de testare/proba si nu se mai dezvolta.
Mutat in legacy-vfp/ (sursa .prg, proiect .pjx/.PJT, date .DBF/.CDX/.FPT,
test-comenzi.xml). Actualizat caile DBF default in tools/import_dbf.py si
referinta din tests/test_import_dbf.py. Adaugat legacy-vfp/README.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 06:51:21 +00:00
Claude Agent
854db66abc feat(ui): #15 U5 — web upload import (HTMX) drop→mapare→preview→confirma
Implementare completa U5 din plan-treapta2.md (sectiunea 13):

- _upload.html: drop zone + buton accesibil (a11y: drag nu e la tastatura),
  drag-and-drop JS, mesaj 'NU se trimite nimic pana confirmi',
  selector foi pt multi-sheet xlsx, stari eroare/mesaj
- _mapcoloane.html: formular mapare coloane cu .maprow/.mapcol.grow,
  sugestii fuzzy pre-selectate, etiichete <label> vizibile, sample values,
  format data configurabil
- _preview_import.html: tabel 6 stari, pills rezumat, filtre pe stare,
  .chk per-rand pe needs_review (D11), banner declarant .banner.warn
  direct deasupra input-ului N (D12), bara confirmare sticky,
  text 'dubla cu randul N' pe duplicate_in_file (D10 daltonism),
  link export CSV randuri esuate
- base.html: .s-needs_review (warn), .s-already_sent/.s-duplicate_in_file
  (muted), .drop-zone, .banner.warn, .sticky-bar, .htmx-indicator
- routes.py: rute /_import/upload/mapare-coloane/preview/reset/confirma;
  helper _web_compute_preview refoloseste _resolve_row_for_preview,
  _already_sent_lookup, _signature din import_router (fara a-l edita);
  commit ON CONFLICT DO NOTHING (TOCTOU); log atestare
- tests/test_import_ui.py: 15 teste (dashboard, upload, mapare, preview,
  confirmare N corect/gresit, reset, erori, multi-sheet, a11y D10/D11/D12)

279 teste total, 0 esecuri.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 21:04:56 +00:00
Claude Agent
55adfa214f test(e2e): T15 — E2E integrare import->commit->worker (MockRar) + masina de stari + failure registry
21 teste noi in tests/test_import_e2e.py:
  - Scenariul 1: upload xlsx/csv -> column-mapping -> preview -> commit -> worker -> FINALIZATA
  - Scenariul 2: re-upload acelasi continut -> already_sent (nu al doilea POST la RAR)
  - Scenariul 3: coada MIXTA API(creds efemere)+web(accounts.rar_creds_enc durabil) — T1/Voce#5
  - Masina de stari: queued->sending->sent/requeued/error + double-commit 409 + batch committed
  - Failure registry: RAR 400/403 + reconciliere raspuns pierdut (503) + max_retries->error
  - T16: purge_expired + purge_after setat la commit si la trimitere

Suita completa: 264 passed, 0 failed (fara regresii Treapta 1 sau Treapta 2).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:51:57 +00:00
Claude Agent
70f717d874 feat(import): #11-14 router import Treapta 2 — upload+staging+mapare+preview+commit+export
Implementeaza app/api/v1/import_router.py (router nou, montat in app):

POST /v1/import — upload xlsx/csv, staging in import_batches/import_rows,
  PII criptat Fernet (Issue 5a), BEGIN IMMEDIATE+executemany (Issue 6),
  purge_after 90z (T16), sugestii fuzzy coloane DRY (Issue 5b/Eng#4),
  detectie drift semnatura (T4/D3), multisheet support

GET/POST /v1/import/{id}/column-mapping — mapare coloane per cont cu
  semnatura + drift detection

GET /v1/import/{id}/preview — 6 stari per rand (ok/needs_mapping/needs_data/
  needs_review/already_sent/duplicate_in_file), already_sent batch lookup
  nu N+1 (Eng#5), intra-batch collision EXCLUSIV preview (OV-3/T11)

POST /v1/import/{id}/commit — gate HARD N confirmat (T5/D3), atestare pe
  valori (Voce#1), INSERT ON CONFLICT DO NOTHING TOCTOU (Issue 1/T12),
  import_attestations rows_hash+n_confirmed (Voce#9), batch_id/row_index (T7)

GET /v1/import/{id}/export-failed — CSV randuri esuate cu motiv (T8)

Teste: 36 cazuri noi in tests/test_import_api.py; 243 total, toate verzi.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:41:59 +00:00
Claude Agent
2c8367109c feat(parser): T14 — parser core 2-treceri xlsx/csv + T3 coercion + T10 data col-level + T13 robustete RO
Implementeaza app/import_parse.py (modul nou) cu toate cele 4 taskuri parser:

T14 (task #7): schelet parser + arhitectura 2-treceri (Issue 2 consens cross-model)
- Trecerea 1 read_only=True: dim-check FileTooLarge (>5000 randuri / >5MB) + detectie multi-sheet
- Trecerea 2 normal-mode: header + merged cells + body
- CSV delimiter sniff (csv.Sniffer + proba explicita {; , \t}) — export RO foloseste ;
- Encoding: UTF-8-sig -> UTF-8 -> cp1250 -> latin2 (fallback RO)
- Exceptii custom: FileTooLarge, HeaderError, MultipleSheets
- Coloane duplicate din merge: sufixate _2/_3 (nu HeaderError)

T3 (task #8): coercion guard + needs_review + mesaj formule-None
- VIN numeric (openpyxl citeste ca int/float) -> coercion_flags[row] needs_review
- Odometru float 123456.0 -> tunde .0 inainte de validare (§3.4bis, ordonare critica)
- Rata None > 60% pe coloana obligatorie -> formula_columns (Issue 3 mesaj specific)
- Datetime nativ -> convertit la YYYY-MM-DD (neambiguu)

T10 (task #9): dezambiguizare data la nivel de COLOANA (OV-8)
- Datetime nativ -> "native" (direct, fara ambiguitate)
- String: detectie format din intreaga coloana — daca oricare rand are token[1]>12 -> DD-first
- Daca toti zi<=12 -> "ambiguous" -> needs_review per rand
- parse_date_value() helper pentru preview resolve

T13 (task #10): robustete export RO
- Multi-sheet: >1 sheet non-gol -> MultipleSheets([...]) cu lista; sheet ales -> parse normal
- Merged header: _unmerge_header propaga valoarea topleft la toate coloanele din grup
- Footer trim: randuri trailing cu VIN + data ambele goale -> skip structural (nu needs_data)

Teste: 37 teste verzi in tests/test_import_parse.py (fixture-uri xlsx generate in-memory).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:31:27 +00:00
Claude Agent
ef52dc2823 feat(import): T16 job purjare + purge_after SET la sent (OV-5)
- mark(sent): seteaza purge_after = now + 90 zile (GDPR/L.142)
- purge_expired(conn): sterge submissions sent expirate + import_batches expirate
  (import_rows via ON DELETE CASCADE). NULL purge_after = nu expira.
- run(): tick de purjare odata pe ora (guard _last_purge_time + _PURGE_INTERVAL_S)
  NU mai agresiv, nu blocheaza trimiterea
- 8 teste: purge_after la sent, alte stari fara purge, expirati vs neexpirat,
  queued neatins, cascade import_rows, null purge_after pastrat

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:24:59 +00:00
Claude Agent
8cdfc976e4 feat(import): T7 batch_id scope reresolve_account — R1 INCHIS
- reresolve_account(conn, account_id, batch_id=None):
  - batch_id specificat -> scope la batch-ul exact (import commit explicit)
  - fara batch_id (POST /v1/mapari) -> EXCLUSIV canal API (batch_id IS NULL)
  - salvarea unei mapari NU mai re-queues randuri cross-batch (R1 inchis)
- 6 teste: izolare batch A/B, regresie API canal, batch explicit nu atinge API,
  schema batch_id/row_index, 3 batches izolate

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:23:11 +00:00
Claude Agent
61a7b4ea1c feat(import): T6 gate auto_send pe coduri nou-mapate (OV-1)
- load_mapping_meta: {cod_op_service -> {cod_prestatie, auto_send}}
- has_no_auto_send: verifica daca vreun item rezolvat via mapping are auto_send=0
- reresolve_account: auto_send=0 -> ramane needs_mapping (review_manual stat),
  NU trece pe queued; previne FINALIZATA eronat permanent
- reresolve_account primeste batch_id optional (pregatire T7, urmeaza)
- POST /v1/prezentari: auto_send=0 -> needs_mapping + motiv explicit
- 9 teste: load_mapping_meta, has_no_auto_send, reresolve (zero/unu), POST API

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:21:32 +00:00
Claude Agent
12f0ca3a81 feat(import): T1 accounts.rar_creds_enc durabil + worker fallback + gate purjare
- worker: _creds_from_account(conn, account_id) — fallback la accounts.rar_creds_enc
  cand submission n-are creds (canal web fara re-pusher, restart worker)
- run(): creds = _creds_for(claimed, settings) OR _creds_from_account(conn, account_id)
- gate purjare (Voce#5): comentariu explicit — sterge DOAR submissions.rar_creds_enc,
  NU accounts.rar_creds_enc (inofensiv pt canal web, neatins pt canal API)
- POST /v1/conturi/rar-creds: seteaza creds durabile criptate Fernet per cont
- DELETE /v1/conturi/rar-creds: revenire la modelul efemer Treapta 1
- 7 teste: fallback, restart, coada mixta, endpoint set/delete, gate purjare

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:18:41 +00:00
Claude Agent
4ea21a034e feat(import): T9 canonicalize_row + build_key partajat (idempotency)
- canonicalize_row: VIN upper, odometru strip ".0" (Excel float coercion),
  data strip — INAINTE de validare si cheie (§3.4bis)
- build_key: aplica account_or_default(None->1) inainte de hash (OV-2):
  canal API (None) si canal import (1) produc aceeasi cheie
- build_key_legacy: helper dual-lookup pentru randuri DB vechi (pre-T9)
- router.py: POST /v1/prezentari foloseste build_key(account_id, canonicalize_row(content))
- 14 teste: canonicalizare, cross-canal, dedup float/int odometru, legacy

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:15:59 +00:00
Claude Agent
80897ccbb1 feat(foundation): schema Treapta 2 + migrari aditive + openpyxl pinned (#1)
- accounts.rar_creds_enc TEXT (creds RAR durabile per-cont, D4)
- submissions.batch_id, row_index (T7 scoping R1)
- submissions.purge_after (T16 GDPR)
- Tabele noi: column_mappings, import_batches, import_rows, import_attestations
- _migrate idempotent pe DB veche (ALTER aditiv, pattern existent)
- openpyxl==3.1.5 adaugat in requirements.txt (Issue 4, PINNED)
- 15 teste noi: coloane, tabele, idempotenta, migrare DB veche, openpyxl

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-16 20:13:19 +00:00
50 changed files with 8624 additions and 41 deletions

3
.gitignore vendored
View File

@@ -15,6 +15,9 @@ settings.xml
.env.*
!.env.example
# --- start.sh: PID-uri si loguri proces local ---
.run/
# --- VFP: programe compilate (se regenerează din .prg) ---
*.fxp
*.FXP

415
README.md Normal file
View File

@@ -0,0 +1,415 @@
# Gateway RAR AUTOPASS
Gateway web (Python / FastAPI) care preia prezentarile de service si le declara la
**RAR AUTOPASS** (Legea 142/2023, OM 210/2024). Inlocuieste integrarea Visual FoxPro
existenta (ROAAUTO). Sursa de adevar pentru contractul RAR este
[`docs/api-rar-contract.md`](docs/api-rar-contract.md).
## Arhitectura pe scurt
Doua procese peste acelasi SQLite persistent:
| Proces | Rol | Pornire |
|--------|-----|---------|
| **API** (`app.main:app`) | API v1 (`/v1/*`), dashboard web (`/`), `/healthz`, `/metrics`, import fisiere (`/v1/import/*`) | `uvicorn app.main:app` |
| **Worker** (`app.worker`) | login RAR + JWT, refresh nomenclator, trimite prezentarile din coada, retry/backoff, heartbeat | `python3 -m app.worker` |
Worker-ul ruleaza ca **proces separat** (nu task in API) — un worker mort nu trebuie sa
lase containerul "sanatos". Comunicarea API <-> worker se face exclusiv prin tabela
`submissions` din SQLite. Send-ul catre RAR este **dezactivat implicit**
(`AUTOPASS_WORKER_SEND_ENABLED=false`) — sigur pentru probe.
## Cerinte
- Python 3.12+
- Dependintele din `requirements.txt`
```bash
pip3 install -r requirements.txt
```
(Optional, pentru deploy: Docker + Docker Compose — vezi sectiunea Docker.)
## Configurare
Variabilele de mediu folosesc prefixul `AUTOPASS_`. Pentru dev local valorile implicite
sunt suficiente — **nu** ai nevoie de `.env` sau de credentiale RAR ca sa testezi UI-ul si
API-ul. Copiaza `.env.example` -> `.env` doar cand vrei sa rulezi end-to-end.
| Variabila | Implicit | Rol |
|-----------|----------|-----|
| `AUTOPASS_DB_PATH` | `./data/autopass.db` | calea fisierului SQLite |
| `AUTOPASS_RAR_ENV` | `test` | `test` sau `prod` |
| `AUTOPASS_REQUIRE_API_KEY` | `false` | `false` = dev (fara cheie -> cont id=1); `true` = prod (cere cheie) |
| `AUTOPASS_CREDS_KEY` | (efemera) | cheie Fernet pt criptarea creds RAR. **Trebuie partajata intre API si worker.** Genereaza: `python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"` |
| `AUTOPASS_WORKER_SEND_ENABLED` | `false` | `true` = trimite efectiv la RAR (proba end-to-end) |
| `AUTOPASS_WORKER_USE_TEST_CREDS` | `false` | dev: foloseste blocul `<test>` din `settings.xml` pt login worker |
Pentru proba reala cu RAR: copiaza `settings.xml.example` -> `settings.xml` si completeaza
credentialele de test (fisierul **nu** se comite).
## Rulare locala (dezvoltare)
### 1. Porneste API-ul
```bash
uvicorn app.main:app --reload --port 8000
# sau, daca uvicorn nu e pe PATH:
python3 -m uvicorn app.main:app --reload --port 8000
```
La prima pornire se creeaza schema SQLite si se face seed la nomenclatorul RAR (18 coduri
din contract), astfel incat dashboard-ul si maparile functioneaza imediat, offline.
### 2. (Optional) Porneste worker-ul
Necesar doar pentru a procesa coada / a trimite la RAR. Pentru testarea UI-ului si a
import-ului **nu** e necesar.
```bash
python3 -m app.worker
```
### Pornire rapida cu `start.sh`
`start.sh` ambaleaza pornirea pe mediu (`test` / `prod`) si rol (`api` / `worker` / `both`):
```bash
./start.sh test api # API pe :8000, mediu test
./start.sh test worker --send # worker care TRIMITE la RAR test
./start.sh test both --send # API + worker impreuna (dev end-to-end, loguri in .run/)
./start.sh prod api --port 8000 # API mediu prod
./start.sh prod worker --send # worker prod (NU foloseste creds de test)
./start.sh status # stare procese + /healthz
./start.sh stop # opreste procesele pornite cu "both"
./start.sh test finalizate # ce prezentari sunt inregistrate la RAR (vezi mai jos)
```
Optiuni: `--port N`, `--host H`, `--reload` (dev), `--send` (activeaza trimiterea la RAR),
`--test-creds` / `--no-test-creds` (forteaza folosirea creds `<test>` din `settings.xml`).
Pe `test` cu `--send`, creds `<test>` se folosesc automat. Pentru productie reala foloseste
`docker compose` (vezi sectiunea Docker).
Doua wrappere fixeaza mediul si forwardeaza rolul + optiunile:
```bash
./start-test.sh # = start.sh test both --send (API + worker, trimite la RAR test)
./start-test.sh worker --send # = start.sh test worker --send
./start-test.sh finalizate # = start.sh test finalizate
./start-prod.sh both --send # = start.sh prod both --send
./start-prod.sh api # = start.sh prod api
```
Pe test, `./start-test.sh` fara argumente porneste end-to-end (sandbox RAR e sigur). Pe prod,
`./start-prod.sh` cere rolul explicit si trimiterea trebuie ceruta cu `--send` (evita trimiteri
accidentale in productie).
## Testare in browser
Cu API-ul pornit, deschide in browser:
| URL | Ce vezi |
|-----|---------|
| `http://localhost:8000/` | **Dashboard** — stare coada, banner prezentari blocate, stare worker / ultim login RAR, editor mapari operatii, browser nomenclator, sectiune **import fisier** |
| `http://localhost:8000/docs` | **Swagger UI** — API v1 interactiv (incearca endpointurile direct din browser) |
| `http://localhost:8000/healthz` | JSON sanatate: worker viu, ultim login RAR, adancime coada |
| `http://localhost:8000/metrics` | metrici text (submissions pe status) |
### Fluxul de import fisier (xlsx / csv) din browser
Pe dashboard, in sectiunea de import:
1. **Incarca** un fisier `.xlsx` sau `.csv` (drag & drop sau selectare).
2. **Mapeaza coloanele** — gateway-ul sugereaza automat (fuzzy) maparea coloana fisier ->
camp canonic (VIN, data prestatie, odometru, operatie etc.). Maparea se retine pe
semnatura coloanelor: la urmatorul fisier cu aceleasi coloane se aplica automat.
3. **Preview** — fiecare rand primeste o stare: `ok`, `needs_mapping`, `needs_data`,
`needs_review`, `already_sent`, `duplicate_in_file`.
4. **Confirma** — gate dur: retastezi numarul exact de randuri `ok` de trimis. Randurile
confirmate intra in coada (`submissions`), apoi le urmaresti in tabelul de jos.
Coloane recunoscute (cu sinonime): `VIN`, `Nr inmatriculare`, `Data prestatie`,
`Odometru final`, `Odometru initial`, `Operatie`, `Observatii`.
### Genereaza un fisier de test pentru import
Repo-ul nu contine fisiere sample. Creeaza unul rapid:
```bash
python3 - <<'PY'
import openpyxl
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"])
ws.append(["WAUZZZ8K0AA000001", "B123ABC", "2026-06-15", 120000, "REVIZIE PERIODICA"])
ws.append(["WAUZZZ8K0AA000002", "B456DEF", "2026-06-16", 85000, "REPARATIE"])
wb.save("sample_import.xlsx")
print("scris sample_import.xlsx")
PY
```
Sau un CSV echivalent:
```bash
printf 'VIN,Nr inmatriculare,Data prestatie,Odometru final,Operatie\nWAUZZZ8K0AA000001,B123ABC,2026-06-15,120000,REVIZIE PERIODICA\n' > sample_import.csv
```
Incarca apoi fisierul prin sectiunea de import a dashboard-ului.
## Proba trimitere la RAR (mediu test) + verificare ca au ajuns
Implicit worker-ul **nu** trimite (`AUTOPASS_WORKER_SEND_ENABLED=false`). Pentru proba
end-to-end pe contul de test RAR:
1. Pune credentialele de test in `settings.xml` (copiaza din `settings.xml.example`,
completeaza blocul `<test>`). Acestea **nu** se comit.
2. Bag-a prezentari in coada — fie prin import fisier din dashboard, fie prin API
(`POST /v1/prezentari`, vezi mai jos).
3. Porneste worker-ul cu trimiterea activa:
```bash
./start.sh test worker --send
```
Worker-ul face login la RAR test, ia randurile `queued`, trimite si trece fiecare rand
in `sent` cu `id_prezentare` (id-ul intors de RAR — dovada ca a ajuns) sau in
`needs_data` / `error` cu motivul.
4. **Vizualizeaza prezentarile trimise** — trei feluri:
- **Dashboard** (`http://localhost:8000/`) — tabelul de jos arata fiecare submission cu
status (`sent`/`error`/...), `id_prezentare`, cod RAR si eroare. Se actualizeaza singur.
- **API**: `curl -s http://localhost:8000/v1/prezentari` — coada locala cu statusuri.
- **Direct de la RAR** (confirmare independenta ca au ajuns):
```bash
./start.sh test finalizate
```
Face login la RAR test si listeaza prezentarile inregistrate acolo (id, VIN, data,
odometru). Compari `id`-urile cu `id_prezentare` din coada locala: daca se regasesc,
prezentarea a ajuns la RAR.
> Status `sent` + `id_prezentare` completat = RAR a acceptat prezentarea. Worker-ul are si
> reconciliere anti-duplicat: daca raspunsul RAR se pierde, la urmatorul ciclu cauta
> prezentarea in finalizate si o marcheaza `sent` fara a o re-trimite.
## Import fisier pentru mai multi utilizatori (service-uri) cu formate diferite
Da — fiecare service auto poate avea propriul format de fisier (alte denumiri de coloane,
alta ordine, alt format de data). Sistemul **tine minte maparea per cont**, deci nu o refaci
la fiecare upload:
- **Cont (`account_id`)** — fiecare service e un cont. In productie contul se identifica prin
**cheia API** (`X-API-Key`) trimisa la upload/cerere (`AUTOPASS_REQUIRE_API_KEY=true`). In
dev, fara cheie, totul merge pe contul implicit `id=1`.
- **Semnatura coloanelor** — la upload, gateway-ul calculeaza o semnatura din lista (sortata)
a denumirilor de coloane din fisier. Maparea coloana-fisier -> camp-canonic se salveaza in
tabela `column_mappings`, cheie unica `(account_id, signature_coloane)`, impreuna cu
formatul de data.
- **Re-aplicare automata** — la urmatorul fisier cu **aceleasi coloane** (aceeasi semnatura),
pentru **acelasi cont**, maparea retinuta se aplica automat si sari direct la preview. Daca
un service schimba formatul (alte coloane) se creeaza o semnatura noua, deci o mapare noua —
fara sa o strice pe cea veche. Astfel un cont poate avea mai multe formate memorate simultan.
Pe scurt: **cine** = `account_id` (din cheia API), **care format** = `signature_coloane`
(setul de coloane al fisierului). Combinatia lor selecteaza maparea corecta.
## Conturi (service-uri) si chei API
Un **cont** (`accounts`) = un service auto care foloseste gateway-ul. Cererile `/v1/*` se
autentifica printr-o **cheie API** (header `X-API-Key: <cheie>` sau `Authorization: Bearer
<cheie>`) care identifica contul. Cheia e separata de credentialele RAR ale service-ului.
Enforcement-ul e controlat de `AUTOPASS_REQUIRE_API_KEY`:
- `false` (dev/test, implicit): cerere fara cheie -> contul implicit `id=1`; o cheie prezenta
dar invalida -> `401`.
- `true` (productie): orice `/v1/*` **protejat** cere o cheie valida, altfel `401`.
Auth-ul se aplica pe endpointurile care scriu/sunt legate de cont (au dependinta de cheie):
`POST /v1/prezentari`, `POST /v1/mapari`, `POST|DELETE /v1/conturi/rar-creds` si toate rutele
de import (`POST /v1/import`, `.../column-mapping`, `.../preview`, `.../commit`,
`.../export-failed`) — acestea ruleaza pe `account_id`-ul cheii. GET-urile de **monitorizare**
(`/v1/prezentari`, `/v1/prezentari/{id}`, `/v1/nomenclator`, `/v1/mapari`, `/v1/audit/export`)
sunt momentan **neprotejate si globale** (nu filtreaza pe cont). Filtrarea pe cont a listarilor
+ protejarea lor raman de adaugat (vezi tabelul de mai jos).
### Stare implementare
| Capabilitate | Stare | Cum |
|--------------|-------|-----|
| Emitere / rotire / revocare / listare chei API | **Implementat** | CLI `python3 -m tools.apikey` |
| Auth pe cheie (X-API-Key / Bearer) pe POST-uri + import | **Implementat** | `app/auth.py` + flag `AUTOPASS_REQUIRE_API_KEY` |
| Ingestie + import account-scoped (din cheie) | **Implementat** | `POST /v1/prezentari`, `POST /v1/import` |
| Creds RAR durabile per cont | **Implementat** | `POST /v1/conturi/rar-creds` |
| Creare cont nou (service) | **De facut / manual** | momentan prin `INSERT` SQL (vezi mai jos); nu exista tool/endpoint dedicat |
| Protejare + filtrare pe cont a GET-urilor de listare | **De facut** | `GET /v1/prezentari`, `/v1/nomenclator`, `/v1/audit/export` sunt globale acum |
| Self-onboarding web (login email+parola -> emite cheie) | **In plan** | `docs/plans/plan-treapta2.md` (sect. login web) — neimplementat |
> Lifecycle-ul cheilor se face DOAR din CLI, pe masina gateway-ului (admin) — nu exista
> suprafata HTTP de administrare de securizat. Cheia in clar se afiseaza **o singura data**
> la creare/rotire; in DB se pastreaza doar hash-ul SHA-256.
### Creare cont + cheie pentru un service nou
Pana la onboarding-ul web, un cont nou se creeaza direct in DB, apoi i se emite o cheie:
```bash
# 1. Creeaza contul (numele + CUI sunt informative)
python3 -c "
from app.db import get_connection, init_db
init_db()
c = get_connection()
cur = c.execute(\"INSERT INTO accounts (name, cui) VALUES ('Service Auto SRL', 'RO12345678')\")
print('account_id nou =', cur.lastrowid); c.commit(); c.close()
"
# 2. Emite o cheie API pentru cont (afisata O SINGURA DATA)
python3 -m tools.apikey create --account 2
# -> rfak_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Alte operatii
python3 -m tools.apikey list # toate cheile
python3 -m tools.apikey list --account 2 # cheile unui cont
python3 -m tools.apikey rotate --account 2 # revoca cele active + emite una noua
python3 -m tools.apikey revoke --key-id 3 # revoca o cheie dupa id
```
### Creds RAR per cont
Ca worker-ul sa poata trimite pentru un service fara ca fiecare cerere sa-i poarte parola
RAR, seteaza credentialele RAR durabile pe cont (criptate Fernet at-rest):
```bash
curl -s -X POST http://localhost:8000/v1/conturi/rar-creds \
-H 'X-API-Key: rfak_...' -H 'Content-Type: application/json' \
-d '{"email": "service@exemplu.ro", "password": "parola-rar"}'
```
## Testare prin API (curl)
Exemplele de mai jos arata atat varianta **dev** (fara cheie -> cont id=1), cat si varianta
**service cu cheie API** (header `X-API-Key`). Cand `AUTOPASS_REQUIRE_API_KEY=true`, cheia e
obligatorie.
```bash
# Sanatate (neprotejat)
curl -s http://localhost:8000/healthz | python3 -m json.tool
# Nomenclator RAR (cache local)
curl -s http://localhost:8000/v1/nomenclator
# Coada de prezentari (monitorizare; momentan globala + neprotejata, vezi nota de mai sus)
curl -s http://localhost:8000/v1/prezentari
# Trimite o prezentare -- dev (fara cheie API -> cont id=1)
curl -s -X POST http://localhost:8000/v1/prezentari \
-H 'Content-Type: application/json' \
-d '{
"rar_credentials": {"email": "test@example.ro", "password": "secret"},
"prezentari": [{
"vin": "WAUZZZ8K0AA000001",
"nr_inmatriculare": "B123ABC",
"data_prestatie": "2026-06-15",
"odometru_final": "120000",
"prestatii": [{"cod_op_service": "REVIZIE PERIODICA", "denumire": "REVIZIE PERIODICA"}]
}]
}'
# Trimite o prezentare -- service cu cheie API (account_id curge din cheie)
curl -s -X POST http://localhost:8000/v1/prezentari \
-H 'X-API-Key: rfak_...' -H 'Content-Type: application/json' \
-d '{
"rar_credentials": {"email": "service@exemplu.ro", "password": "parola-rar"},
"prezentari": [{
"vin": "WAUZZZ8K0AA000002",
"nr_inmatriculare": "B456DEF",
"data_prestatie": "2026-06-16",
"odometru_final": "85000",
"prestatii": [{"cod_op_service": "REPARATIE", "denumire": "REPARATIE"}]
}]
}'
# Import fisier prin API pentru un service (multi-tenant: contul vine din cheie)
curl -s -X POST http://localhost:8000/v1/import \
-H 'X-API-Key: rfak_...' -F 'file=@sample_import.xlsx'
```
Endpointurile complete sunt vizibile si testabile in `/docs` (Swagger UI). In Swagger,
pune cheia prin butonul "Authorize" sau adauga header-ul `X-API-Key`.
```bash
# Sanatate
curl -s http://localhost:8000/healthz | python3 -m json.tool
# Nomenclator RAR (cache local)
curl -s http://localhost:8000/v1/nomenclator
# Coada de prezentari
curl -s http://localhost:8000/v1/prezentari
# Trimite o prezentare (dev: fara cheie API -> cont id=1)
curl -s -X POST http://localhost:8000/v1/prezentari \
-H 'Content-Type: application/json' \
-d '{
"rar_credentials": {"email": "test@example.ro", "password": "secret"},
"prezentari": [{
"vin": "WAUZZZ8K0AA000001",
"nr_inmatriculare": "B123ABC",
"data_prestatie": "2026-06-15",
"odometru_final": "120000",
"prestatii": [{"cod_op_service": "REVIZIE PERIODICA", "denumire": "REVIZIE PERIODICA"}]
}]
}'
```
Endpointurile complete sunt vizibile si testabile in `/docs` (Swagger UI).
## Rularea testelor
```bash
python3 -m pytest -q
```
Suita acopera fundatia, securitatea, validarea, parserul de import, masina de stari a
worker-ului si fluxul UI de import (E2E cu RAR mock).
## Docker / deploy
```bash
# 1. Pregateste .env (CRITIC: AUTOPASS_CREDS_KEY partajata intre api si worker)
cp .env.example .env
# completeaza AUTOPASS_CREDS_KEY (vezi comanda de generare de mai sus)
# 2. Porneste API + worker + autoheal
docker compose up --build
```
`docker-compose.yml` porneste trei containere: `api` (port 8000), `worker` si `autoheal`
(restarteaza worker-ul cand heartbeat-ul devine invechit). Ambele servicii folosesc acelasi
image si acelasi volum SQLite persistent.
## Structura
```
app/
main.py # FastAPI: API v1 + dashboard + /healthz + /metrics
api/v1/ # router.py (prezentari, nomenclator, mapari) + import_router.py
web/ # routes.py (dashboard + import UI HTMX) + templates/ + static/
worker/ # proces separat: login RAR, send, retry, heartbeat
rar_client.py # client HTTP RAR (login/JWT, postPrezentare, nomenclator)
validation.py # validare continut (T3)
mapping.py # mapare operatie -> cod prestatie + fuzzy lookup
crypto.py # criptare Fernet creds RAR efemere (zero-storage at rest)
schema.sql # schema SQLite
docs/ # contract RAR (sursa de adevar) + planuri + context
tests/ # suita pytest
legacy-vfp/ # arhiva Visual FoxPro ROAAUTO (legacy, doar referinta/migrare)
```
Detalii de continuitate intre sesiuni: [`docs/CONTEXT.md`](docs/CONTEXT.md).
Plan executabil: [`docs/plans/plan.md`](docs/plans/plan.md).

1125
app/api/v1/import_router.py Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -22,10 +22,11 @@ from pydantic import BaseModel, Field
from ...auth import resolve_account_id
from ...crypto import encrypt_creds
from ...db import get_connection
from ...idempotency import idempotency_key
from ...idempotency import build_key, canonicalize_row, idempotency_key
from ...mapping import (
account_or_default,
load_mapping,
has_no_auto_send,
load_mapping_meta,
pending_unmapped,
reresolve_account,
resolve_prestatii,
@@ -60,10 +61,22 @@ def create_prezentari(
conn = get_connection()
results: list[SubmissionResult] = []
try:
mapping = load_mapping(conn, acct)
# T6/OV-1: load_mapping_meta include auto_send per op (gate pentru coduri noi).
mapping_meta = load_mapping_meta(conn, acct)
mapping = {op: meta["cod_prestatie"] for op, meta in mapping_meta.items()}
for prez in req.prezentari:
content = prez.model_dump()
key = idempotency_key(account_id, content)
# T9/OV-2: canonicalize_row inaintea build_key (odometru strip ".0", VIN upper).
# build_key aplica account_or_default(account_id) inainte de hash:
# None si 1 colapseaza la aceeasi cheie (canal API + canal import).
canon = canonicalize_row(content)
key = build_key(account_id, canon)
# Aplica normalizarea si in content (odometru canonicalizat inainte de validare, §3.4bis)
content.update({
"vin": canon["vin"],
"nr_inmatriculare": canon["nr_inmatriculare"],
"odometru_final": canon["odometru_final"],
})
existing = conn.execute(
"SELECT id, status, id_prezentare FROM submissions WHERE idempotency_key=?",
(key,),
@@ -94,6 +107,14 @@ def create_prezentari(
errors = validate_prezentare(content)
if errors:
status, rar_error = "needs_data", json.dumps(errors, ensure_ascii=False)
elif has_no_auto_send(resolved, mapping_meta):
# T6/OV-1: cod rezolvat cu auto_send=0 -> nu trimite automat.
# Randul ramane 'needs_mapping' pana userul confirma manual (sau comuta auto_send=1).
status = "needs_mapping"
rar_error = json.dumps(
{"auto_send": "cod mapat cu auto_send=0; review manual inainte de trimitere"},
ensure_ascii=False,
)
else:
status, rar_error = "queued", None
@@ -313,3 +334,48 @@ def create_mapare(
return {"saved": {"cod_op_service": req.cod_op_service.strip(), "cod_prestatie": cod}, "reresolve": stats}
finally:
conn.close()
class RarCredsIn(BaseModel):
"""Creds RAR durabile per-cont (D4). Stocate criptate (Fernet) in accounts.rar_creds_enc."""
email: str = Field(..., min_length=1)
password: str = Field(..., min_length=1, repr=False)
@router.post("/conturi/rar-creds")
def set_rar_creds(
req: RarCredsIn,
account_id: int = Depends(resolve_account_id),
) -> dict:
"""Seteaza creds RAR durabile per-cont (D4/T1).
Criptate Fernet in accounts.rar_creds_enc. Worker-ul le foloseste ca fallback
cand submission-ul nu mai are creds (canal web fara re-pusher, restart worker).
Contul vine din cheia API.
"""
acct = account_or_default(account_id)
enc = encrypt_creds({"email": req.email, "password": req.password})
conn = get_connection()
try:
conn.execute(
"UPDATE accounts SET rar_creds_enc=? WHERE id=?",
(enc, acct),
)
return {"ok": True, "account_id": acct}
finally:
conn.close()
@router.delete("/conturi/rar-creds")
def delete_rar_creds(
account_id: int = Depends(resolve_account_id),
) -> dict:
"""Sterge creds RAR durabile per-cont (revenire la modelul efemer Treapta 1)."""
acct = account_or_default(account_id)
conn = get_connection()
try:
conn.execute("UPDATE accounts SET rar_creds_enc=NULL WHERE id=?", (acct,))
return {"ok": True, "account_id": acct}
finally:
conn.close()

View File

@@ -43,11 +43,33 @@ def init_db() -> None:
def _migrate(conn: sqlite3.Connection) -> None:
"""Migrari aditive pentru DB create inainte de o coloana noua (CREATE IF NOT EXISTS nu altereaza)."""
cols = {r["name"] for r in conn.execute("PRAGMA table_info(submissions)").fetchall()}
if "next_attempt_at" not in cols:
# Coloane submissions
sub_cols = {r["name"] for r in conn.execute("PRAGMA table_info(submissions)").fetchall()}
if "next_attempt_at" not in sub_cols:
conn.execute("ALTER TABLE submissions ADD COLUMN next_attempt_at TEXT")
if "rar_creds_enc" not in cols:
if "rar_creds_enc" not in sub_cols:
conn.execute("ALTER TABLE submissions ADD COLUMN rar_creds_enc TEXT")
if "purge_after" not in sub_cols:
conn.execute("ALTER TABLE submissions ADD COLUMN purge_after TEXT")
if "batch_id" not in sub_cols:
conn.execute("ALTER TABLE submissions ADD COLUMN batch_id INTEGER")
if "row_index" not in sub_cols:
conn.execute("ALTER TABLE submissions ADD COLUMN row_index INTEGER")
# Coloane accounts
acc_cols = {r["name"] for r in conn.execute("PRAGMA table_info(accounts)").fetchall()}
if "rar_creds_enc" not in acc_cols:
conn.execute("ALTER TABLE accounts ADD COLUMN rar_creds_enc TEXT")
# Index batch_id pe submissions (poate lipsi pe DB veche)
existing_idx = {r["name"] for r in conn.execute(
"SELECT name FROM sqlite_master WHERE type='index' AND tbl_name='submissions'"
).fetchall()}
if "idx_submissions_batch" not in existing_idx:
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_submissions_batch ON submissions(batch_id) "
"WHERE batch_id IS NOT NULL"
)
def _now_iso() -> str:

View File

@@ -2,6 +2,23 @@
RAR nu are camp nr. comanda si accepta duplicate -> dedup-ul e in sarcina noastra
(plan.md sect. 14). Hash stabil peste o reprezentare canonica a prezentarii.
Treapta 2 (T9 + OV-2): extrage canonicalize_row + build_key ca helpere publice
partajate intre canalul API si canalul import.
- canonicalize_row: normeaza VIN/nr/odometru (strip ".0" Excel coercion) INAINTE
de validare (§3.4bis) si INAINTE de cheie.
- build_key: aplica account_or_default INAINTE de hash (None si 1 => o cheie).
OV-2 — skew account_id: routerul vechi pasa account_id AS-PASSED (None pe canal API
fara auth). Randurile se stocau sub account_or_default=1, dar cheia includea None.
Acelasi rand logic din import (account_id=1) dadea cheie diferita -> already_sent
rata -> al doilea FINALIZATA. Fix: build_key normalizeaza INTOTDEAUNA la
account_or_default inainte de hash.
Migrare DB productie (OV-2): randurile existente cu cheie-None nu mai sunt gasite de
build_key nou. Strategie documentata: dual-lookup la already_sent (incearca cheia
noua, apoi cheia legacy). In dev nu exista date reale; la first-deploy productie
se poate face recompute-keys o singura data.
"""
from __future__ import annotations
@@ -20,10 +37,88 @@ def _op_identity(p: Any) -> str:
return (get("cod_op_service", "") or "").strip()
def canonicalize_row(raw: dict[str, Any]) -> dict[str, Any]:
"""Normalizare canonica a unui rand brut. Apelata INAINTE de validare si de build_key.
- VIN, nr_inmatriculare: strip + upper.
- odometru_final: strip ".0" (Excel coercion numeric 123456.0 -> "123456").
Necesar ca validation._parse_int (isdigit()) sa nu respinga float-string.
- data_prestatie: strip (normalizarea la YYYY-MM-DD se face in parser).
- prestatii: pastrate ca-atare (rezolvarea e in resolve_prestatii).
"""
# VIN
vin = (raw.get("vin") or "").strip().upper()
# Nr. inmatriculare
nr = (raw.get("nr_inmatriculare") or "").strip().upper()
# Odometru: strip ".0" Excel float coercion
odo_raw = raw.get("odometru_final")
if odo_raw is not None:
odo_s = str(odo_raw).strip()
# "123456.0" -> "123456"; "123456.50" nu (nu e coercion Excel pur)
if "." in odo_s:
before, after = odo_s.split(".", 1)
if after == "0" and before.lstrip("-").isdigit():
odo_s = before
else:
odo_s = ""
# Data (pastrata ca string; parsarea la YYYY-MM-DD e in parser)
data = str(raw.get("data_prestatie") or "").strip()
# Prestatii (copie superficiala; rezolvarea e upstream)
prestatii = list(raw.get("prestatii") or [])
return {
"vin": vin,
"nr_inmatriculare": nr,
"data_prestatie": data,
"odometru_final": odo_s,
"prestatii": prestatii,
}
def build_key(account_id: int | None, canon: dict[str, Any]) -> str:
"""SHA-256 partajat canal-API + canal-import.
Aplica account_or_default inainte de hash (OV-2): None si 1 colapseaza la
aceeasi cheie => acelasi rand logic din canale diferite nu se trimite de doua ori.
"""
# Import local ca sa evitam import circular (mapping importa din idempotency via validator)
from .mapping import account_or_default
acct = account_or_default(account_id)
canonic = {
"account_id": acct,
"vin": canon.get("vin", ""),
"nr_inmatriculare": canon.get("nr_inmatriculare", ""),
"data_prestatie": canon.get("data_prestatie"),
"odometru_final": canon.get("odometru_final", ""),
"prestatii": sorted(_op_identity(p) for p in (canon.get("prestatii") or [])),
}
blob = json.dumps(canonic, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
return hashlib.sha256(blob.encode("utf-8")).hexdigest()
def idempotency_key(account_id: int | None, prezentare: dict[str, Any]) -> str:
"""SHA-256 peste (account_id + campurile semnificative ale prezentarii).
Wrapper backward-compat peste canonicalize_row + build_key.
Exclude obs si b64Image (cosmetice, nu definesc unicitatea declaratiei).
NOTA: dupa OV-2, account_id=None si account_id=1 produc ACEEASI cheie
(via account_or_default in build_key). Randuri vechi cu cheie-None nu sunt
acoperite automat — dual-lookup sau recompute-keys la migrare productie.
"""
canon = canonicalize_row(prezentare)
return build_key(account_id, canon)
def build_key_legacy(account_id: int | None, prezentare: dict[str, Any]) -> str:
"""Cheia in formatul vechi (account_id AS-PASSED, fara canonicalize).
Folosita EXCLUSIV pentru dual-lookup la already_sent pe DB cu randuri vechi
(dinainte de T9). Nu folosi pentru randuri noi.
"""
canonic = {
"account_id": account_id,
@@ -31,9 +126,6 @@ def idempotency_key(account_id: int | None, prezentare: dict[str, Any]) -> str:
"nr_inmatriculare": (prezentare.get("nr_inmatriculare") or "").strip().upper(),
"data_prestatie": prezentare.get("data_prestatie"),
"odometru_final": str(prezentare.get("odometru_final") or "").strip(),
# Identitatea operatiei = codul RAR daca exista, altfel codul intern ROAAUTO
# (hibrid): doua trimiteri ale aceleiasi comenzi dedup corect indiferent de
# forma in care vin codurile.
"prestatii": sorted(_op_identity(p) for p in (prezentare.get("prestatii") or [])),
}
blob = json.dumps(canonic, sort_keys=True, ensure_ascii=False, separators=(",", ":"))

748
app/import_parse.py Normal file
View File

@@ -0,0 +1,748 @@
"""Parser fisiere xlsx/csv pentru import prezentari (Treapta 2, U1).
Arhitectura 2-treceri (Issue 2, consens cross-model):
Trecerea 1 — read_only=True: dim-check (FileTooLarge) + detectie multi-sheet.
Trecerea 2 — normal-mode: header + merged cells + body.
Aceasta separare e necesara deoarece openpyxl read_only=True nu vede celule imbinate.
Modulul este PUR in sensul ca nu face I/O DB, nu trimite nimic la RAR si nu are
efecte laterale — intoarce structuri Python testabile direct.
Stari per-rand (resolved_status):
ok — date complete, gata de trimis dupa mapare + validare
needs_review — coercion suspectat (VIN numeric, odometru float) sau data ambigua
needs_data — camp obligatoriu lipsa (dupa coercion)
(needs_mapping, already_sent, duplicate_in_file — calculate in preview, nu aici)
"""
from __future__ import annotations
import csv
import io
from datetime import date, datetime
from typing import Any, NamedTuple
# --------------------------------------------------------------------------- #
# Constante #
# --------------------------------------------------------------------------- #
MAX_ROWS = 5_000
MAX_BYTES = 5 * 1024 * 1024 # 5 MB
# Prag rata None pe o coloana obligatorie -> mesaj formule necalculate (Issue 3)
FORMULA_NONE_RATE = 0.6
# Coloane cheie pentru detectia footer-ului (trim structural)
KEY_COLS = {"vin", "data_prestatie"}
# Delimitatori incercati la sniff CSV (ordinea conteaza: ; primul, export RO)
CSV_DELIMITERS = [";", ",", "\t"]
# Encodinguri incercate in ordine (BOM-aware + RO)
CSV_ENCODINGS = ["utf-8-sig", "utf-8", "cp1250", "latin2"]
# --------------------------------------------------------------------------- #
# Exceptii custom #
# --------------------------------------------------------------------------- #
class FileTooLarge(Exception):
"""Fisier depaseste limita de randuri sau dimensiune."""
def __init__(self, *, rows: int | None = None, bytes_: int | None = None):
self.rows = rows
self.bytes_ = bytes_
parts = []
if rows is not None:
parts.append(f"{rows} randuri (max {MAX_ROWS})")
if bytes_ is not None:
parts.append(f"{bytes_ // 1024} KB (max {MAX_BYTES // 1024} KB)")
super().__init__(f"Fisier prea mare: {', '.join(parts)}")
class HeaderError(Exception):
"""Header lipsa, duplicat sau un singur camp detectat."""
def __init__(self, message: str, found: list[str] | None = None):
self.found = found or []
super().__init__(message)
class MultipleSheets(Exception):
"""Workbook cu mai mult de un sheet non-gol — utilizatorul trebuie sa aleaga."""
def __init__(self, sheet_names: list[str]):
self.sheet_names = sheet_names
super().__init__(f"Mai multe sheet-uri non-goale: {sheet_names}. Alege sheet-ul de importat.")
# --------------------------------------------------------------------------- #
# Structura interna de rezultat #
# --------------------------------------------------------------------------- #
class ParsedFile(NamedTuple):
"""Rezultatul parsarii unui fisier."""
columns: list[str] # Numele coloanelor detectate (din header)
rows: list[dict[str, Any]] # Fiecare rand: {coloana: valoare_bruta}
coercion_flags: dict[int, list[str]] # {row_index: [motive needs_review]}
formula_columns: list[str] # Coloane cu rata None ridicata (Issue 3)
date_col_format: dict[str, str] # {coloana: "DD.MM.YYYY" | "YYYY-MM-DD" | "native" | "ambiguous"}
# --------------------------------------------------------------------------- #
# XLSX — trecerea 1: dim-check (read_only) #
# --------------------------------------------------------------------------- #
def _xlsx_dimcheck(data: bytes) -> list[str]:
"""Trecerea 1 read_only: verifica dimensiunile si intoarce lista de sheet-uri non-goale.
Ridica FileTooLarge daca depaseste limita.
Ridica MultipleSheets daca sunt >1 sheet-uri non-goale.
Intoarce lista (cu un singur element daca totul e ok).
"""
import openpyxl
if len(data) > MAX_BYTES:
raise FileTooLarge(bytes_=len(data))
wb = openpyxl.load_workbook(io.BytesIO(data), read_only=True, data_only=True)
try:
non_empty: list[str] = []
for name in wb.sheetnames:
ws = wb[name]
# In read_only, max_row poate fi None daca sheet-ul e gol
max_row = ws.max_row or 0
if max_row > 0:
non_empty.append(name)
if max_row > MAX_ROWS:
raise FileTooLarge(rows=max_row)
finally:
wb.close()
if len(non_empty) > 1:
raise MultipleSheets(non_empty)
return non_empty # 0 sau 1 element
# --------------------------------------------------------------------------- #
# XLSX — trecerea 2: header + merged + body (normal-mode) #
# --------------------------------------------------------------------------- #
def _unmerge_header(ws) -> dict[int, str]:
"""Rezolva celulele imbinate din primul rand non-gol.
Intoarce {col_index_1based: valoare_str}.
Merge range-urile din header propaga valoarea topleft la toate coloanele din grup.
"""
# Gaseste primul rand non-gol
header_row = None
for row in ws.iter_rows(max_row=20):
vals = [c.value for c in row if c.value is not None]
if vals:
header_row = row[0].row
break
if header_row is None:
return {}
# Mapa col_index -> valoare din celule normale
col_vals: dict[int, str] = {}
for cell in ws[header_row]:
if cell.value is not None:
col_vals[cell.column] = str(cell.value).strip()
# Propaga valoarea topleft pentru merge range-uri din randul header
for merged_range in ws.merged_cells.ranges:
if merged_range.min_row <= header_row <= merged_range.max_row:
# Valoarea e in celula topleft
topleft = ws.cell(row=merged_range.min_row, column=merged_range.min_col)
val = str(topleft.value or "").strip()
for col in range(merged_range.min_col, merged_range.max_col + 1):
col_vals[col] = val
return col_vals
def _deduplicate_columns(names: list[str]) -> list[str]:
"""Adauga sufix _2/_3 la coloane cu acelasi nume (din merged cells)."""
seen: dict[str, int] = {}
result = []
for n in names:
if n not in seen:
seen[n] = 1
result.append(n)
else:
seen[n] += 1
result.append(f"{n}_{seen[n]}")
return result
def _xlsx_parse_sheet(ws, sheet_name: str) -> ParsedFile:
"""Parseaza un sheet in normal-mode (trecerea 2).
Presupune ca dim-check a trecut deja (FileTooLarge nu se verifica din nou).
"""
# Header cu merged cells
col_map = _unmerge_header(ws)
if not col_map:
raise HeaderError(f"Sheet '{sheet_name}': niciun header detectat.", found=[])
# Ordoneaza coloanele dupa index
sorted_cols = sorted(col_map.items()) # [(col_idx, name), ...]
col_indices = [idx for idx, _ in sorted_cols]
col_names = [name for _, name in sorted_cols]
# Dezambiguizeaza duplicate (provin din merge care se propaga la mai multe coloane)
col_names = _deduplicate_columns(col_names)
if len(col_names) < 2:
raise HeaderError(f"Doar {len(col_names)} coloana detectata — verifica fisierul.", found=col_names)
# Gaseste randul header ca sa sarim peste el
header_row_num = ws.cell(row=1, column=col_indices[0]).row
# Re-detect: prima celula din col_map
# Obtinem randul headerului din prima celula valida
for row in ws.iter_rows(max_row=20):
for c in row:
if c.column in col_map and c.value is not None:
header_row_num = c.row
break
else:
continue
break
# Citeste randurile de date
raw_rows: list[dict[str, Any]] = []
# Colecteaza valorile pe coloane pentru detectia datei si a formulelor
col_values: dict[str, list[Any]] = {name: [] for name in col_names}
for row in ws.iter_rows(min_row=header_row_num + 1):
row_dict: dict[str, Any] = {}
for col_idx, col_name in zip(col_indices, col_names):
# Cauta celula cu col_idx in rand (unele randuri pot fi mai scurte)
found_cell = None
for c in row:
if c.column == col_idx:
found_cell = c
break
val = found_cell.value if found_cell is not None else None
row_dict[col_name] = val
col_values[col_name].append(val)
raw_rows.append(row_dict)
# Trim footer: elimina randuri trailing unde coloanele cheie sunt goale
raw_rows = _trim_footer(raw_rows, col_names)
# Detectie coloane cu formule (rata None, Issue 3)
formula_columns = _detect_formula_columns(col_values, len(raw_rows))
# Detectie format data la nivel de coloana (T10/OV-8)
date_col_format = _detect_date_formats(col_values, col_names)
# Coercion + flags needs_review (T3)
coercion_flags: dict[int, list[str]] = {}
processed_rows: list[dict[str, Any]] = []
for i, row_dict in enumerate(raw_rows):
processed, flags = _coerce_row(row_dict, col_names)
processed_rows.append(processed)
if flags:
coercion_flags[i] = flags
return ParsedFile(
columns=col_names,
rows=processed_rows,
coercion_flags=coercion_flags,
formula_columns=formula_columns,
date_col_format=date_col_format,
)
# --------------------------------------------------------------------------- #
# Trim footer structural #
# --------------------------------------------------------------------------- #
def _is_key_empty(row_dict: dict[str, Any], col_names: list[str]) -> bool:
"""Randul e structural gol daca coloanele cheie (VIN + data) sunt ambele None/gol."""
# Detecta coloanele cheie prin nume normalized
from .mapping import normalize_for_match
norm_names = {normalize_for_match(n): n for n in col_names}
vin_col = None
date_col_key = None
for norm, orig in norm_names.items():
if "VIN" in norm or "SERIE" in norm or "SASIU" in norm:
vin_col = orig
if "DATA" in norm or "DATE" in norm or "PRESTATIE" in norm:
date_col_key = orig
def _empty(v: Any) -> bool:
return v is None or str(v).strip() == ""
vin_empty = _empty(row_dict.get(vin_col)) if vin_col else True
date_empty = _empty(row_dict.get(date_col_key)) if date_col_key else True
return vin_empty and date_empty
def _trim_footer(rows: list[dict[str, Any]], col_names: list[str]) -> list[dict[str, Any]]:
"""Elimina randuri trailing unde VIN + data sunt goale (footer TOTAL/Intocmit de:)."""
i = len(rows) - 1
while i >= 0 and _is_key_empty(rows[i], col_names):
i -= 1
return rows[: i + 1]
# --------------------------------------------------------------------------- #
# Detectie coloane formule (Issue 3) #
# --------------------------------------------------------------------------- #
def _detect_formula_columns(col_values: dict[str, list[Any]], n_rows: int) -> list[str]:
"""Coloane unde rata de None depaseste pragul -> probabil formule necalculate."""
if n_rows == 0:
return []
result = []
for col_name, vals in col_values.items():
none_count = sum(1 for v in vals if v is None)
rate = none_count / n_rows
if rate >= FORMULA_NONE_RATE:
result.append(col_name)
return result
# --------------------------------------------------------------------------- #
# Dezambiguizare data la nivel de coloana (T10 / OV-8) #
# --------------------------------------------------------------------------- #
def _detect_date_formats(col_values: dict[str, list[Any]], col_names: list[str]) -> dict[str, str]:
"""Detecteaza formatul datei pentru fiecare coloana de tip data.
Rezultate posibile per coloana:
"native" — toate valorile non-None sunt datetime nativ openpyxl (neambigue)
"DD.MM.YYYY" — coloana e DD-first (cel putin un rand are token[1] > 12)
"YYYY-MM-DD" — format ISO
"ambiguous" — string, toti zi <= 12 (si DD si MM ar fi valide)
"mixed" — amestec datetime nativ + string
(Nu e inclusa daca coloana nu pare a fi de tip data)
"""
from .mapping import normalize_for_match
result: dict[str, str] = {}
for col_name in col_names:
norm = normalize_for_match(col_name)
# Filtra coloanele de data dupa nume
if not any(kw in norm for kw in ("DATA", "DATE", "PRESTATIE", "ZI", "AN")):
continue
vals = [v for v in col_values.get(col_name, []) if v is not None]
if not vals:
continue
native_count = sum(1 for v in vals if isinstance(v, (datetime, date)))
str_vals = [str(v).strip() for v in vals if not isinstance(v, (datetime, date))]
if native_count == len(vals):
result[col_name] = "native"
continue
if native_count > 0 and str_vals:
result[col_name] = "mixed"
continue
# Toate string — detectie format la nivel de coloana (OV-8)
fmt = _infer_date_format_from_column(str_vals)
result[col_name] = fmt
return result
def _infer_date_format_from_column(str_vals: list[str]) -> str:
"""Detecteaza formatul datei dintr-o lista de valori string.
Logica OV-8: daca ORICARE rand are token pozitia-1 > 12 -> coloana e DD-first.
Daca toti zi <= 12 -> ambiguu.
"""
dd_first_evidence = False
iso_evidence = False
parseable = 0
for s in str_vals:
if not s:
continue
# Incearca ISO (YYYY-MM-DD sau YYYY/MM/DD)
if _looks_iso(s):
iso_evidence = True
parseable += 1
continue
# Incearca DD.MM.YYYY sau DD/MM/YYYY sau DD-MM-YYYY
parts = _split_date(s)
if parts and len(parts) == 3:
try:
day_candidate = int(parts[0])
month_candidate = int(parts[1])
if day_candidate > 12:
dd_first_evidence = True
# Daca month_candidate > 12 -> cu siguranta DD.MM (luna e la pozitia 1)
if month_candidate > 12:
dd_first_evidence = True
parseable += 1
except ValueError:
pass
if not parseable:
return "ambiguous"
if iso_evidence and not dd_first_evidence:
return "YYYY-MM-DD"
if dd_first_evidence:
return "DD.MM.YYYY"
# Toti zi <= 12: nu putem distinge DD.MM de MM.DD
return "ambiguous"
def _looks_iso(s: str) -> bool:
"""Verifica rapid daca string-ul arata ca YYYY-MM-DD."""
parts = s.replace("/", "-").split("-")
if len(parts) == 3:
try:
y = int(parts[0])
return y > 1900
except ValueError:
pass
return False
def _split_date(s: str) -> list[str] | None:
"""Imparte un string data dupa separatorul comun (., /, -)."""
for sep in (".", "/", "-"):
parts = s.split(sep)
if len(parts) == 3:
return parts
return None
# --------------------------------------------------------------------------- #
# Coercion per rand (T3) #
# --------------------------------------------------------------------------- #
def _coerce_row(row_dict: dict[str, Any], col_names: list[str]) -> tuple[dict[str, Any], list[str]]:
"""Coerce valorile unui rand si colecteaza flags needs_review.
Reguli:
- VIN citit ca int/float (openpyxl: "0123..." -> 123.0) -> str + flag needs_review
- Odometru float cu .0 -> tunde ".0" (via canonicalize_row logic)
- Datetime nativ -> convertit la YYYY-MM-DD string
- Valori goale/None raman None
"""
from .mapping import normalize_for_match
flags: list[str] = []
out: dict[str, Any] = {}
norm_names = {normalize_for_match(n): n for n in col_names}
# Identifica coloanele semantice
vin_col = _find_col(norm_names, ("VIN", "SERIE SASIU", "SASIU", "SERIE"))
odo_col = _find_col(norm_names, ("ODOMETRU", "KM", "KILOMETRI", "ODO"))
for col_name, val in row_dict.items():
if val is None:
out[col_name] = None
continue
# Datetime nativ -> string YYYY-MM-DD
if isinstance(val, datetime):
out[col_name] = val.date().isoformat()
continue
if isinstance(val, date):
out[col_name] = val.isoformat()
continue
# VIN: daca e numeric (float sau int) -> coercion suspectat
if col_name == vin_col:
if isinstance(val, (int, float)):
flags.append(f"VIN numeric ({val}) — verificati seria sasiului")
out[col_name] = str(int(val)) if val == int(val) else str(val)
else:
out[col_name] = str(val).strip().upper()
continue
# Odometru: float cu .0 -> int string
if col_name == odo_col:
if isinstance(val, float):
s = str(val)
if s.endswith(".0"):
out[col_name] = s[:-2] # "123456.0" -> "123456"
else:
# Float non-integer -> pastreaza si lasa validarea sa decida
flags.append(f"Odometru float nestandard ({val})")
out[col_name] = str(val)
elif isinstance(val, int):
out[col_name] = str(val)
else:
out[col_name] = str(val).strip()
continue
# Default: string
out[col_name] = str(val).strip() if isinstance(val, str) else val
return out, flags
def _find_col(norm_names: dict[str, str], keywords: tuple[str, ...]) -> str | None:
"""Gaseste o coloana dupa cuvinte cheie in numele normalizat."""
for kw in keywords:
for norm, orig in norm_names.items():
if kw in norm:
return orig
return None
# --------------------------------------------------------------------------- #
# Parsare data per rand (folosita de preview resolve) #
# --------------------------------------------------------------------------- #
def parse_date_value(
val: Any,
col_format: str,
) -> tuple[str | None, bool]:
"""Parseaza o valoare de data si intoarce (iso_string, is_ambiguous).
- val e deja string (coercion a convertit datetime nativ).
- col_format: "native", "DD.MM.YYYY", "YYYY-MM-DD", "ambiguous", "mixed".
- Intoarce (None, False) daca valoarea e goala.
- Intoarce (iso, True) daca data e ambigua (needs_review).
"""
if val is None or str(val).strip() == "":
return None, False
# Daca coercion a convertit deja la ISO (din datetime nativ)
s = str(val).strip()
try:
d = date.fromisoformat(s)
return d.isoformat(), False
except ValueError:
pass
if col_format in ("native", "YYYY-MM-DD"):
# Incearca ISO
parts = s.replace("/", "-").split("-")
if len(parts) == 3:
try:
d = date(int(parts[0]), int(parts[1]), int(parts[2]))
return d.isoformat(), False
except ValueError:
pass
return None, False
if col_format == "DD.MM.YYYY":
parts = _split_date(s)
if parts and len(parts) == 3:
try:
d = date(int(parts[2]), int(parts[1]), int(parts[0]))
return d.isoformat(), False
except ValueError:
pass
return None, False
if col_format == "ambiguous":
# Incearca DD.MM.YYYY
parts = _split_date(s)
if parts and len(parts) == 3:
try:
d = date(int(parts[2]), int(parts[1]), int(parts[0]))
return d.isoformat(), True # ambiguu -> needs_review
except ValueError:
pass
return None, True
# mixed sau necunoscut: incearca ambele
parts = _split_date(s)
if parts and len(parts) == 3:
try:
# Incearca DD.MM.YYYY
d = date(int(parts[2]), int(parts[1]), int(parts[0]))
return d.isoformat(), True # ambiguu
except ValueError:
pass
return None, False
# --------------------------------------------------------------------------- #
# CSV #
# --------------------------------------------------------------------------- #
def _decode_csv(data: bytes) -> str:
"""Decodifica bytes CSV cu fallback encoding RO."""
for enc in CSV_ENCODINGS:
try:
return data.decode(enc)
except (UnicodeDecodeError, LookupError):
continue
raise UnicodeDecodeError("csv", data, 0, len(data), "Encoding nesuportat (incercat utf-8, cp1250, latin2)")
def _sniff_delimiter(sample: str) -> str:
"""Detecteaza delimiter-ul CSV. Export Excel RO foloseste ';'."""
# Incearca Sniffer standard
try:
dialect = csv.Sniffer().sniff(sample, delimiters=";,\t")
return dialect.delimiter
except csv.Error:
pass
# Proba explicita: alege delimiter-ul care da cel mai mare numar consistent de coloane
best_delim = ","
best_cols = 0
for delim in CSV_DELIMITERS:
lines = sample.splitlines()[:10]
counts = []
for line in lines:
if line.strip():
counts.append(len(line.split(delim)))
if counts:
# Cel mai frecvent count
from collections import Counter
common = Counter(counts).most_common(1)[0][0]
if common > best_cols:
best_cols = common
best_delim = delim
return best_delim
def parse_csv(data: bytes) -> ParsedFile:
"""Parseaza un fisier CSV. Detecteaza delimiter + encoding RO."""
if len(data) > MAX_BYTES:
raise FileTooLarge(bytes_=len(data))
text = _decode_csv(data)
sample = text[:8192]
delimiter = _sniff_delimiter(sample)
reader = csv.DictReader(io.StringIO(text), delimiter=delimiter)
# Citeste toate randurile (limitat la MAX_ROWS)
raw_rows: list[dict[str, Any]] = []
for i, row in enumerate(reader):
if i >= MAX_ROWS:
raise FileTooLarge(rows=i + 1)
raw_rows.append(dict(row))
if not raw_rows:
raise HeaderError("CSV gol sau fara randuri de date.", found=[])
col_names = list(raw_rows[0].keys())
if not col_names or len(col_names) < 2:
raise HeaderError(
f"Doar {len(col_names)} coloana detectata cu delimiter '{delimiter}' — verifica separatorul.",
found=col_names,
)
# Curata cheile None (DictReader poate produce None pt coloane extra)
col_names = [c for c in col_names if c is not None and str(c).strip()]
# Strip whitespace din valori
cleaned: list[dict[str, Any]] = []
for row in raw_rows:
cleaned.append({k: (v.strip() if isinstance(v, str) else v) for k, v in row.items() if k in col_names})
# Trim footer
cleaned = _trim_footer(cleaned, col_names)
# Colecteaza valori per coloana pentru detectii
col_values: dict[str, list[Any]] = {c: [] for c in col_names}
for row in cleaned:
for c in col_names:
col_values[c].append(row.get(c))
formula_columns: list[str] = [] # CSV nu are formule
date_col_format = _detect_date_formats(col_values, col_names)
coercion_flags: dict[int, list[str]] = {}
processed: list[dict[str, Any]] = []
for i, row in enumerate(cleaned):
p, flags = _coerce_row(row, col_names)
processed.append(p)
if flags:
coercion_flags[i] = flags
return ParsedFile(
columns=col_names,
rows=processed,
coercion_flags=coercion_flags,
formula_columns=formula_columns,
date_col_format=date_col_format,
)
# --------------------------------------------------------------------------- #
# XLSX — entry point #
# --------------------------------------------------------------------------- #
def parse_xlsx(data: bytes, *, sheet_name: str | None = None) -> ParsedFile:
"""Parseaza un fisier XLSX.
Arhitectura 2-treceri (Issue 2):
1. read_only=True: dim-check + detectie multi-sheet
2. normal-mode: header + merged cells + body
Parametru sheet_name: daca workbook-ul are mai multe sheet-uri, utilizatorul
trebuie sa aleaga; trimite-l inapoi la acest apel. Daca None si >1 sheet ->
ridica MultipleSheets.
"""
import openpyxl
# Trecerea 1: dim-check
try:
non_empty = _xlsx_dimcheck(data)
except MultipleSheets as ms:
if sheet_name is not None:
# Utilizatorul a ales deja un sheet — continuam cu cel ales
non_empty = ms.sheet_names
else:
raise
if not non_empty:
raise HeaderError("Workbook fara sheet-uri cu date.", found=[])
# Alegere sheet
if sheet_name is not None:
target = sheet_name
elif len(non_empty) == 1:
target = non_empty[0]
else:
raise MultipleSheets(non_empty)
# Trecerea 2: normal-mode
wb = openpyxl.load_workbook(io.BytesIO(data), read_only=False, data_only=True)
try:
if target not in wb.sheetnames:
raise HeaderError(f"Sheet '{target}' nu exista in workbook.", found=wb.sheetnames)
ws = wb[target]
return _xlsx_parse_sheet(ws, target)
finally:
wb.close()
# --------------------------------------------------------------------------- #
# Entry point universal #
# --------------------------------------------------------------------------- #
def parse_file(
data: bytes,
filename: str,
*,
sheet_name: str | None = None,
) -> ParsedFile:
"""Entry point unic: detecteaza tipul dupa extensie si parseaza.
Ridica: FileTooLarge, HeaderError, MultipleSheets, UnicodeDecodeError,
openpyxl.utils.exceptions.InvalidFileException (fisier corupt).
"""
name_lower = filename.lower()
if name_lower.endswith(".csv"):
return parse_csv(data)
elif name_lower.endswith((".xlsx", ".xls")):
return parse_xlsx(data, sheet_name=sheet_name)
else:
raise HeaderError(f"Tip fisier nesuportat: '{filename}'. Acceptat: xlsx, xls, csv.")

View File

@@ -18,6 +18,7 @@ from fastapi.responses import JSONResponse, PlainTextResponse
from fastapi.staticfiles import StaticFiles
from . import __version__
from .api.v1.import_router import router as import_v1_router
from .api.v1.router import router as api_v1_router
from .config import get_settings
from .db import get_connection, init_db, queue_depth, read_heartbeat
@@ -56,6 +57,7 @@ _STATIC_DIR = Path(__file__).resolve().parent / "web" / "static"
app.mount("/static", StaticFiles(directory=str(_STATIC_DIR)), name="static")
app.include_router(api_v1_router)
app.include_router(import_v1_router)
app.include_router(web_router)

View File

@@ -188,6 +188,35 @@ def load_mapping(conn, account_id: int | None) -> dict[str, str]:
return {r["cod_op_service"]: r["cod_prestatie"] for r in rows}
def load_mapping_meta(conn, account_id: int | None) -> dict[str, dict]:
"""{cod_op_service -> {cod_prestatie, auto_send}} pentru un cont.
T6/OV-1: varianta extinsa care include si flagul auto_send per operatie.
"""
acct = account_or_default(account_id)
rows = conn.execute(
"SELECT cod_op_service, cod_prestatie, auto_send FROM operations_mapping WHERE account_id=?",
(acct,),
).fetchall()
return {
r["cod_op_service"]: {"cod_prestatie": r["cod_prestatie"], "auto_send": bool(r["auto_send"])}
for r in rows
}
def has_no_auto_send(resolved: list[dict], mapping_meta: dict[str, dict]) -> bool:
"""Verifica daca vreun item rezolvat via mapping are auto_send=0.
T6/OV-1: un cod nou-mapat cu auto_send=0 nu trebuie trimis automat.
Items cu cod_prestatie direct (nu via cod_op_service) nu sunt afectate.
"""
for item in resolved:
op = (item.get("cod_op_service") or "").strip()
if op and op in mapping_meta and not mapping_meta[op]["auto_send"]:
return True
return False
def pending_unmapped(conn) -> list[dict]:
"""Operatii distincte nemapate, agregate din submission-urile `needs_mapping`.
@@ -250,22 +279,43 @@ def save_mapping(conn, account_id: int | None, cod_op_service: str, cod_prestati
)
def reresolve_account(conn, account_id: int | None) -> dict[str, int]:
def reresolve_account(conn, account_id: int | None, batch_id: int | None = None) -> dict[str, int]:
"""Re-rezolva submission-urile `needs_mapping` ale unui cont dupa o noua mapare.
Pentru fiecare: aplica maparea curenta; daca nu mai raman op-uri nemapate ->
ruleaza validarea de continut (T3) si trece pe `queued` (sau `needs_data` cu
motiv), resetand backoff-ul. Daca raman nemapate, ramane `needs_mapping` cu
motivul actualizat. Intoarce {requeued, still_blocked, needs_data}.
motivul actualizat. Intoarce {requeued, still_blocked, needs_data, review_manual}.
T6/OV-1: auto_send=0 pe un cod nou-mapat -> nu trece pe 'queued' (ramane
'needs_mapping' cu motiv "review manual"); previne FINALIZATA eronat permanent.
T7: batch_id != None -> scope la seria comitata (NU cross-batch).
batch_id is None -> re-rezolva toti (canal API, batch_id IS NULL inclus).
"""
acct = account_or_default(account_id)
mapping = load_mapping(conn, acct)
mapping_meta = load_mapping_meta(conn, acct)
mapping = {op: meta["cod_prestatie"] for op, meta in mapping_meta.items()}
if batch_id is not None:
# T7: scope la batch-ul specificat (import commit explicit).
# NU atinge randuri din alte batches sau din feed API.
rows = conn.execute(
"SELECT id, payload_json FROM submissions WHERE status='needs_mapping' AND account_id=?",
"SELECT id, payload_json FROM submissions "
"WHERE status='needs_mapping' AND account_id=? AND batch_id=?",
(acct, batch_id),
).fetchall()
else:
# POST /v1/mapari (save manual): re-rezolva EXCLUSIV canalul API (batch_id IS NULL).
# T7/R1 INCHIS: salvarea unei mapari NU re-queues randuri din batches de import
# (cross-batch / cross-feed). Batches de import sunt re-rezolvate doar la commit explicit.
rows = conn.execute(
"SELECT id, payload_json FROM submissions "
"WHERE status='needs_mapping' AND account_id=? AND batch_id IS NULL",
(acct,),
).fetchall()
stats = {"requeued": 0, "still_blocked": 0, "needs_data": 0}
stats = {"requeued": 0, "still_blocked": 0, "needs_data": 0, "review_manual": 0}
for r in rows:
try:
content = json.loads(r["payload_json"])
@@ -283,6 +333,19 @@ def reresolve_account(conn, account_id: int | None) -> dict[str, int]:
stats["still_blocked"] += 1
continue
# T6/OV-1: verifica auto_send inainte de re-queuing
if has_no_auto_send(resolved, mapping_meta):
conn.execute(
"UPDATE submissions SET payload_json=?, rar_error=?, updated_at=datetime('now') WHERE id=?",
(
payload_json,
json.dumps({"auto_send": "cod mapat cu auto_send=0; review manual inainte de trimitere"}, ensure_ascii=False),
r["id"],
),
)
stats["review_manual"] += 1
continue
errors = validate_prezentare(content)
if errors:
conn.execute(

View File

@@ -1,8 +1,6 @@
-- Schema SQLite (WAL) pentru gateway RAR AUTOPASS.
-- Vezi plan.md sect. 5. NICIUN camp pentru parole RAR.
-- Validarea completa (T3) si criptarea PII (P2) vin ulterior; in schelet
-- payload-ul e stocat ca JSON text (camp payload_json), de inlocuit cu BLOB
-- criptat + purge_after cand se face T7/criptare.
-- Vezi plan.md sect. 5 + plan-treapta2.md sect. 4.
-- Treapta 2: adauga conturi cu creds RAR durabile, tabele import, atestari.
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
@@ -12,6 +10,7 @@ CREATE TABLE IF NOT EXISTS accounts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
cui TEXT,
rar_creds_enc TEXT, -- creds RAR criptate (Fernet) durabile per-cont (D4/Eng#1)
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- Cont implicit (id=1): auth API-key (CORE) inca neimplementat, deci ingestiile vin
@@ -54,20 +53,79 @@ CREATE TABLE IF NOT EXISTS submissions (
account_id INTEGER REFERENCES accounts(id) ON DELETE SET NULL,
status TEXT NOT NULL DEFAULT 'queued'
CHECK (status IN ('queued','sending','sent','needs_mapping','needs_data','error')),
payload_json TEXT NOT NULL, -- TODO(P2): inlocuit cu BLOB criptat
rar_creds_enc TEXT, -- creds RAR criptate (Fernet), sterse dupa primul login reusit (plan sect.5)
payload_json TEXT NOT NULL,
rar_creds_enc TEXT, -- creds RAR criptate (Fernet), sterse dupa primul login reusit
rar_status_code INTEGER,
rar_error TEXT,
id_prezentare INTEGER, -- data.id intors de RAR la succes
retry_count INTEGER NOT NULL DEFAULT 0,
next_attempt_at TEXT, -- backoff: randul nu se ia inainte de acest moment (T2)
sending_since TEXT, -- pentru lease/timeout pe randuri 'sending' orfane (T2)
purge_after TEXT, -- sent + 90z (P2)
purge_after TEXT, -- sent + 90z (T16)
batch_id INTEGER, -- import batch (T7; NULL = canal API)
row_index INTEGER, -- rand in batch (T7; NULL = canal API)
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_submissions_status ON submissions(status);
-- Nota: idx_submissions_batch se creeaza in _migrate (dupa ALTER care adauga batch_id pe DB veche).
-- Mapare coloane fisier -> campuri canonice (retinuta per cont, semnatura coloane).
CREATE TABLE IF NOT EXISTS column_mappings (
id INTEGER PRIMARY KEY AUTOINCREMENT,
account_id INTEGER NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
signature_coloane TEXT NOT NULL, -- hash/lista sortata a coloanelor fisierului
json_mapare TEXT NOT NULL, -- {col_fisier: camp_canonic, ...} JSON
format_data TEXT, -- ex. "DD.MM.YYYY"
created_at TEXT NOT NULL DEFAULT (datetime('now')),
UNIQUE (account_id, signature_coloane)
);
-- Loturi de import (fisiere incarcate).
CREATE TABLE IF NOT EXISTS import_batches (
id INTEGER PRIMARY KEY AUTOINCREMENT,
account_id INTEGER NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
filename TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'staging'
CHECK (status IN ('staging','committed','error')),
total INTEGER NOT NULL DEFAULT 0,
ok INTEGER NOT NULL DEFAULT 0,
needs_mapping INTEGER NOT NULL DEFAULT 0,
needs_data INTEGER NOT NULL DEFAULT 0,
needs_review INTEGER NOT NULL DEFAULT 0,
already_sent INTEGER NOT NULL DEFAULT 0,
duplicate_in_file INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
purge_after TEXT -- created_at + 90z (T16)
);
-- Randuri din lot de import (PII criptat cu Fernet).
CREATE TABLE IF NOT EXISTS import_rows (
id INTEGER PRIMARY KEY AUTOINCREMENT,
batch_id INTEGER NOT NULL REFERENCES import_batches(id) ON DELETE CASCADE,
row_index INTEGER NOT NULL,
raw_json TEXT NOT NULL, -- PII criptat (Fernet, ca submissions)
resolved_status TEXT NOT NULL DEFAULT 'pending'
CHECK (resolved_status IN (
'pending','ok','needs_mapping','needs_data',
'needs_review','already_sent','duplicate_in_file'
)),
error TEXT
);
CREATE INDEX IF NOT EXISTS idx_import_rows_batch ON import_rows(batch_id);
-- Log atestare legala (confirmare import batch, L.142/2023).
CREATE TABLE IF NOT EXISTS import_attestations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
batch_id INTEGER NOT NULL REFERENCES import_batches(id) ON DELETE CASCADE,
account_id INTEGER NOT NULL,
confirmed_by TEXT, -- email/identifier utilizator
ts TEXT NOT NULL DEFAULT (datetime('now')),
rows_hash TEXT NOT NULL, -- sha256 peste valorile rezolvate confirmate
n_confirmed INTEGER NOT NULL
);
-- Heartbeat worker (un singur rand, id=1). /healthz citeste de aici.
CREATE TABLE IF NOT EXISTS worker_heartbeat (

View File

@@ -3,21 +3,51 @@
Schelet cu stari explicite: empty (coada goala), banner alerta blocate,
worker viu/mort, ultimul login RAR. Editor mapari + browser nomenclator +
export CSV + stare "RAR indisponibil" = de adaugat (plan.md sect. 4 + design-review).
U5 — Rute web pentru import fisier (upload → mapare coloane → preview → confirma → trimite).
Consuma endpointurile backend din import_router (helper-e interne) fara a le modifica.
Toate rutele /_import/* returneaza fragmente HTML targetate pe #import-section prin HTMX.
"""
from __future__ import annotations
import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from fastapi import APIRouter, Form, Request
from fastapi import APIRouter, File, Form, Request, UploadFile
from fastapi.responses import HTMLResponse
from fastapi.templating import Jinja2Templates
from .. import __version__
from ..api.v1.import_router import (
_already_sent_lookup,
_build_idempotency_key,
_CANONICAL_SYNONYMS,
_fuzzy_suggest_column,
_resolve_row_for_preview,
_signature,
)
from ..config import get_settings
from ..crypto import decrypt_creds, encrypt_creds
from ..db import get_connection, read_heartbeat
from ..mapping import load_nomenclator, pending_unmapped, reresolve_account, save_mapping
from ..idempotency import build_key, canonicalize_row
from ..import_parse import FileTooLarge, HeaderError, MultipleSheets, parse_date_value, parse_file
from ..mapping import (
DEFAULT_ACCOUNT_ID,
account_or_default,
load_mapping_meta,
load_nomenclator,
pending_unmapped,
reresolve_account,
resolve_prestatii,
save_mapping,
)
# Campuri canonice cu eticheta umana pentru dropdown mapare coloane (U5)
_CANONICAL_FIELDS = [(k, v[0]) for k, v in _CANONICAL_SYNONYMS.items()]
router = APIRouter(tags=["web"])
templates = Jinja2Templates(directory=str(Path(__file__).resolve().parent / "templates"))
@@ -169,3 +199,681 @@ def post_mapare(
return _render_mapari(request, conn, message=msg)
finally:
conn.close()
# =========================================================================== #
# Import UI (U5) — upload → mapare coloane → preview → confirmare #
# Consuma helper-e din import_router fara a edita fisierul backend. #
# Toate rutele /_import/* returneaza fragmente HTML (target #import-section). #
# =========================================================================== #
def _web_compute_preview(
conn,
import_id: int,
account_id: int,
) -> dict[str, Any] | str:
"""Calculeaza preview pentru un batch; intoarce date sau str cu mesaj de eroare.
Reutilizeaza _resolve_row_for_preview, _already_sent_lookup, _signature
din import_router. Nu repeta logica de rezolvare — only orchestrare.
"""
acct = account_or_default(account_id)
batch = conn.execute(
"SELECT id, account_id, filename FROM import_batches WHERE id=? AND account_id=?",
(import_id, acct),
).fetchone()
if not batch:
return "Batch de import inexistent sau inaccesibil."
raw_rows_db = conn.execute(
"SELECT row_index, raw_json FROM import_rows WHERE batch_id=? ORDER BY row_index",
(import_id,),
).fetchall()
if not raw_rows_db:
return "Niciun rand in batch."
# Decripteaza randurile
rows: list[dict[str, Any]] = []
for r in raw_rows_db:
try:
row_data = decrypt_creds(r["raw_json"]) or {}
except Exception:
row_data = {}
rows.append(row_data)
col_names = list(rows[0].keys()) if rows else []
sig = _signature(col_names)
mapping_row = conn.execute(
"SELECT json_mapare, format_data FROM column_mappings WHERE account_id=? AND signature_coloane=?",
(acct, sig),
).fetchone()
if not mapping_row:
return "Maparea coloanelor nu a fost configurata. Configureaza mai intai maparea."
json_mapare: dict[str, str] = json.loads(mapping_row["json_mapare"])
format_data: str | None = mapping_row["format_data"]
# Mapare operatii (o singura incarcare — Eng#5)
mapping_meta = load_mapping_meta(conn, acct)
mapping = {op: meta["cod_prestatie"] for op, meta in mapping_meta.items()}
# Detectie coercion flags din valorile stocate (VIN numeric)
coercion_flags_map: dict[int, list[str]] = {}
for i, row_dict in enumerate(rows):
flags: list[str] = []
for col_f, camp_c in json_mapare.items():
if camp_c == "vin":
vin_val = row_dict.get(col_f)
if vin_val is not None and str(vin_val).replace(".", "").isdigit():
flags.append(f"VIN numeric ({vin_val}) — verificati seria sasiului")
if flags:
coercion_flags_map[i] = flags
# Reconstructie date_col_format din format_data stocat in mapare
date_col_format: dict[str, str] = {}
if format_data:
for col_f, camp_c in json_mapare.items():
if camp_c == "data_prestatie":
date_col_format[col_f] = format_data
# Detectie coloane cu formule (rata None ridicata)
n_rows = len(rows)
formula_columns: list[str] = []
if n_rows > 0:
none_counts = {col_f: sum(1 for r in rows if r.get(col_f) is None) for col_f in col_names}
formula_columns = [col_f for col_f, cnt in none_counts.items() if cnt / n_rows >= 0.6]
# Rezolvare per rand
preview_rows: list[dict[str, Any]] = []
keys_for_lookup: list[str] = []
key_to_indices: dict[str, list[int]] = {}
for i, row_dict in enumerate(rows):
flags_i = coercion_flags_map.get(i, [])
info = _resolve_row_for_preview(
raw_row=row_dict,
json_mapare=json_mapare,
date_col_format=date_col_format,
coercion_flags=flags_i,
mapping=mapping,
mapping_meta=mapping_meta,
formula_columns=formula_columns,
)
key: str | None = None
if info["resolved_status"] in ("ok", "needs_review", "needs_data"):
try:
key = _build_idempotency_key(account_id, info["resolved"])
keys_for_lookup.append(key)
key_to_indices.setdefault(key, []).append(i)
except Exception:
pass
preview_rows.append({
"row_index": i,
"resolved_status": info["resolved_status"],
"resolved": info["resolved"],
"errors": info["errors"],
"flags": info["flags"],
"idempotency_key": key,
})
# Already_sent: batch lookup (Eng#5 — fara N+1)
unique_keys = list(set(keys_for_lookup))
already_sent_map = _already_sent_lookup(conn, account_id, unique_keys)
# Aplica already_sent si duplicate_in_file
for row in preview_rows:
k = row.get("idempotency_key")
if not k:
continue
if k in already_sent_map and row["resolved_status"] in ("ok", "needs_review", "needs_data"):
row["resolved_status"] = "already_sent"
row["already_sent_info"] = already_sent_map[k]
continue
indices_same_key = key_to_indices.get(k, [])
if len(indices_same_key) > 1 and row["resolved_status"] in ("ok", "needs_review", "needs_data"):
row["resolved_status"] = "duplicate_in_file"
row["duplicate_with"] = [idx for idx in indices_same_key if idx != row["row_index"]]
# Rezumat stari
summary: dict[str, int] = {}
for row in preview_rows:
s = row["resolved_status"]
summary[s] = summary.get(s, 0) + 1
# Actualizeaza contoare in import_batches
conn.execute(
"UPDATE import_batches SET ok=?, needs_mapping=?, needs_data=?, needs_review=?, "
"already_sent=?, duplicate_in_file=? WHERE id=?",
(
summary.get("ok", 0),
summary.get("needs_mapping", 0),
summary.get("needs_data", 0),
summary.get("needs_review", 0),
summary.get("already_sent", 0),
summary.get("duplicate_in_file", 0),
import_id,
),
)
# Actualizeaza resolved_status in import_rows
conn.execute("BEGIN IMMEDIATE")
try:
conn.executemany(
"UPDATE import_rows SET resolved_status=? WHERE batch_id=? AND row_index=?",
[(row["resolved_status"], import_id, row["row_index"]) for row in preview_rows],
)
conn.execute("COMMIT")
except Exception:
conn.execute("ROLLBACK")
return {
"rows": preview_rows,
"summary": summary,
"total": len(preview_rows),
"filename": batch["filename"],
}
@router.post("/_import/upload", response_class=HTMLResponse)
async def web_upload_import(
request: Request,
file: UploadFile = File(...),
sheet_name: str | None = Form(None),
) -> HTMLResponse:
"""Upload fisier xlsx/csv → staging; intoarce fragment HTML.
Daca maparea de coloane exista deja (signature match): computa preview imediat.
Daca nu: intoarce formularul de mapare coloane.
Nu editeaza import_router.py — apeleaza parse_file si DB direct.
"""
account_id = DEFAULT_ACCOUNT_ID
acct = account_or_default(account_id)
data = await file.read()
filename = file.filename or "fisier"
# Parsare fisier
try:
parsed = parse_file(data, filename, sheet_name=sheet_name)
except MultipleSheets as ms:
return templates.TemplateResponse("_upload.html", {
"request": request,
"sheets": ms.sheet_names,
})
except FileTooLarge as e:
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": str(e),
})
except HeaderError as e:
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": f"Antet neclar: {e}",
})
except UnicodeDecodeError as e:
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": f"Encoding nesuportat: {e.reason}",
})
except Exception as e:
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": f"Fisier nerecunoscut (xlsx/csv): {type(e).__name__}",
})
conn = get_connection()
try:
sig = _signature(parsed.columns)
# Stagingul in DB (tranzactie explicita — Issue 6)
conn.execute("BEGIN IMMEDIATE")
try:
cur = conn.execute(
"INSERT INTO import_batches (account_id, filename, status, total, purge_after) "
"VALUES (?, ?, 'staging', ?, datetime('now', '+90 days'))",
(acct, filename, len(parsed.rows)),
)
batch_id = cur.lastrowid
conn.executemany(
"INSERT INTO import_rows (batch_id, row_index, raw_json, resolved_status, error) "
"VALUES (?, ?, ?, 'pending', NULL)",
[
(batch_id, i, encrypt_creds(row_dict))
for i, row_dict in enumerate(parsed.rows)
],
)
conn.execute("COMMIT")
except Exception:
conn.execute("ROLLBACK")
raise
# Verifica mapare existenta
existing = conn.execute(
"SELECT json_mapare, format_data FROM column_mappings "
"WHERE account_id=? AND signature_coloane=?",
(acct, sig),
).fetchone()
batch_id_int: int = cur.lastrowid or 0 # lastrowid este int dupa INSERT reusit
if existing:
# Mapare retinuta → computa preview imediat
result = _web_compute_preview(conn, batch_id_int, account_id)
if isinstance(result, str):
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": result,
})
return templates.TemplateResponse("_preview_import.html", {
"request": request,
"import_id": batch_id_int,
"message": "Mapare retinuta aplicata automat.",
**result,
})
# Mapare noua — sugestii fuzzy si formular de mapare
fuzzy_suggestions: dict[str, list[dict]] = {}
for col in parsed.columns:
sugg = _fuzzy_suggest_column(col, limit=3)
if sugg:
fuzzy_suggestions[col] = sugg
return templates.TemplateResponse("_mapcoloane.html", {
"request": request,
"import_id": batch_id_int,
"filename": filename,
"columns": parsed.columns,
"sample_rows": parsed.rows[:3],
"fuzzy_suggestions": fuzzy_suggestions,
"canonical_fields": _CANONICAL_FIELDS,
"format_data": None,
})
finally:
conn.close()
@router.post("/_import/{import_id}/mapare-coloane", response_class=HTMLResponse)
async def web_save_mapare_coloane(
request: Request,
import_id: int,
) -> HTMLResponse:
"""Salveaza maparea de coloane si computa preview. Intoarce fragment HTML."""
account_id = DEFAULT_ACCOUNT_ID
acct = account_or_default(account_id)
form = await request.form()
# Colectare perechi coloana fisier → camp canonic din form
# form.getlist intoarce List[str | UploadFile]; filtram la str (campuri text)
colnames = [str(v) for v in form.getlist("colname") if isinstance(v, str)]
canons = [str(v) for v in form.getlist("canon") if isinstance(v, str)]
format_data_val = str(form.get("format_data") or "").strip() or None
# Construieste json_mapare (ignora campurile marcate ca "ignorate")
json_mapare: dict[str, str] = {}
for colname, canon in zip(colnames, canons):
if canon:
json_mapare[colname] = canon
if not json_mapare:
# Nici un camp mapat → re-arata formularul cu eroare
conn = get_connection()
try:
first_row = conn.execute(
"SELECT raw_json FROM import_rows WHERE batch_id=? ORDER BY row_index LIMIT 1",
(import_id,),
).fetchone()
columns = []
if first_row:
try:
rd = decrypt_creds(first_row["raw_json"]) or {}
columns = list(rd.keys())
except Exception:
pass
fuzzy: dict[str, list[dict]] = {}
for col in columns:
sugg = _fuzzy_suggest_column(col, limit=3)
if sugg:
fuzzy[col] = sugg
return templates.TemplateResponse("_mapcoloane.html", {
"request": request,
"import_id": import_id,
"columns": columns,
"sample_rows": [],
"fuzzy_suggestions": fuzzy,
"canonical_fields": _CANONICAL_FIELDS,
"format_data": format_data_val,
"message": "Mapeaza cel putin un camp canonic inainte de a continua.",
"error": True,
})
finally:
conn.close()
conn = get_connection()
try:
# Verifica ca batch-ul apartine contului
batch = conn.execute(
"SELECT id FROM import_batches WHERE id=? AND account_id=?",
(import_id, acct),
).fetchone()
if not batch:
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": "Batch de import inexistent sau expirat.",
})
sig = _signature(list(json_mapare.keys()))
# Salveaza maparea (upsert)
conn.execute(
"INSERT INTO column_mappings (account_id, signature_coloane, json_mapare, format_data) "
"VALUES (?, ?, ?, ?) "
"ON CONFLICT(account_id, signature_coloane) DO UPDATE SET "
"json_mapare=excluded.json_mapare, format_data=excluded.format_data",
(acct, sig, json.dumps(json_mapare, ensure_ascii=False), format_data_val),
)
# Computa preview
result = _web_compute_preview(conn, import_id, account_id)
if isinstance(result, str):
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": result,
})
return templates.TemplateResponse("_preview_import.html", {
"request": request,
"import_id": import_id,
**result,
})
finally:
conn.close()
@router.get("/_import/{import_id}/preview", response_class=HTMLResponse)
def web_preview_import(
request: Request,
import_id: int,
) -> HTMLResponse:
"""Preview 6 stari per rand. Tinta HTMX dupa mapare retinuta sau navigare directa."""
account_id = DEFAULT_ACCOUNT_ID
conn = get_connection()
try:
result = _web_compute_preview(conn, import_id, account_id)
if isinstance(result, str):
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": result,
})
return templates.TemplateResponse("_preview_import.html", {
"request": request,
"import_id": import_id,
**result,
})
finally:
conn.close()
@router.get("/_import/reset", response_class=HTMLResponse)
def web_import_reset(request: Request) -> HTMLResponse:
"""Reseteaza sectiunea de import la starea initiala (drop zone gol)."""
return templates.TemplateResponse("_upload.html", {"request": request})
@router.post("/_import/{import_id}/confirma", response_class=HTMLResponse)
async def web_confirma_import(
request: Request,
import_id: int,
) -> HTMLResponse:
"""Gate HARD confirmare + enqueue randuri ok + log atestare. Intoarce fragment HTML.
Replica logica din import_router.commit_import dar cu input din form HTML
si raspuns HTML (nu JSON). INSERT per-rand ON CONFLICT DO NOTHING (TOCTOU).
"""
account_id = DEFAULT_ACCOUNT_ID
acct = account_or_default(account_id)
form = await request.form()
# Parseaza n_confirmat (form.get intoarce str | UploadFile | None → cast la str)
try:
n_confirmat = int(str(form.get("n_confirmat") or "0"))
except (ValueError, TypeError):
n_confirmat = 0
# Randuri needs_review bifate explicit
reviewed_rows: set[int] = set()
for v in form.getlist("reviewed_rows"):
if isinstance(v, str):
try:
reviewed_rows.add(int(v))
except (ValueError, TypeError):
pass
confirmed_by = str(form.get("confirmed_by") or "").strip() or None
conn = get_connection()
try:
batch = conn.execute(
"SELECT id, filename, status FROM import_batches WHERE id=? AND account_id=?",
(import_id, acct),
).fetchone()
if not batch:
return templates.TemplateResponse("_upload.html", {
"request": request,
"error": "Batch de import inexistent sau expirat.",
})
if batch["status"] == "committed":
return templates.TemplateResponse("_upload.html", {
"request": request,
"message": "Acest batch a fost deja comis.",
})
# Incarca randurile cu stare ok si needs_review
ok_rows_db = conn.execute(
"SELECT row_index, raw_json, resolved_status FROM import_rows "
"WHERE batch_id=? AND resolved_status IN ('ok', 'needs_review') ORDER BY row_index",
(import_id,),
).fetchall()
if not ok_rows_db:
# Re-arata preview cu eroare
result = _web_compute_preview(conn, import_id, account_id)
if isinstance(result, str):
return templates.TemplateResponse("_upload.html", {"request": request, "error": result})
return templates.TemplateResponse("_preview_import.html", {
"request": request,
"import_id": import_id,
"message": "Niciun rand ok de confirmat in acest batch.",
"error": True,
**result,
})
# Decripteaza si construieste lista de randuri de trimis
to_enqueue: list[dict[str, Any]] = []
review_indices: set[int] = set()
for r in ok_rows_db:
try:
row_data = decrypt_creds(r["raw_json"]) or {}
except Exception:
continue
if r["resolved_status"] == "ok":
to_enqueue.append({"row_index": r["row_index"], "data": row_data, "status": "ok"})
elif r["resolved_status"] == "needs_review":
review_indices.add(r["row_index"])
# Adauga randurile needs_review bifate explicit
for r in ok_rows_db:
if r["resolved_status"] == "needs_review" and r["row_index"] in reviewed_rows:
try:
row_data = decrypt_creds(r["raw_json"]) or {}
to_enqueue.append({"row_index": r["row_index"], "data": row_data, "status": "needs_review"})
except Exception:
pass
n_total_ok = len(to_enqueue)
# Gate HARD: n_confirmat trebuie sa fie exact egal cu numarul de randuri de trimis
if n_confirmat != n_total_ok:
result = _web_compute_preview(conn, import_id, account_id)
msg = (
f"Numarul confirmat ({n_confirmat}) difera de randurile gata de trimis ({n_total_ok}). "
f"Verifica preview-ul si retasteaza numarul corect."
)
if isinstance(result, str):
return templates.TemplateResponse("_upload.html", {"request": request, "error": msg})
return templates.TemplateResponse("_preview_import.html", {
"request": request,
"import_id": import_id,
"message": msg,
"error": True,
**result,
})
if n_total_ok == 0:
result = _web_compute_preview(conn, import_id, account_id)
if isinstance(result, str):
return templates.TemplateResponse("_upload.html", {"request": request, "error": result})
return templates.TemplateResponse("_preview_import.html", {
"request": request,
"import_id": import_id,
"message": "Niciun rand ok de confirmat.",
"error": True,
**result,
})
# Incarca maparea de coloane pentru payload
first_row_db = conn.execute(
"SELECT raw_json FROM import_rows WHERE batch_id=? ORDER BY row_index LIMIT 1",
(import_id,),
).fetchone()
col_names: list[str] = []
if first_row_db:
try:
fd = decrypt_creds(first_row_db["raw_json"]) or {}
col_names = list(fd.keys())
except Exception:
pass
sig = _signature(col_names) if col_names else ""
mapping_row = conn.execute(
"SELECT json_mapare, format_data FROM column_mappings WHERE account_id=? AND signature_coloane=?",
(acct, sig),
).fetchone()
json_mapare: dict[str, str] = json.loads(mapping_row["json_mapare"]) if mapping_row else {}
fmt = mapping_row["format_data"] if mapping_row else None
# Mapare operatii
mapping_meta = load_mapping_meta(conn, acct)
mapping_ops = {op: meta["cod_prestatie"] for op, meta in mapping_meta.items()}
# Enqueue in tranzactie explicita (Issue 6) — INSERT ON CONFLICT DO NOTHING (TOCTOU)
enqueued: list[dict] = []
toctou: list[int] = []
rows_for_hash: list[str] = []
conn.execute("BEGIN IMMEDIATE")
try:
for item in to_enqueue:
row_dict = item["data"]
row_index = item["row_index"]
# Aplica maparea de coloane
mapped: dict[str, Any] = {}
for col_f, camp_c in json_mapare.items():
if col_f in row_dict and camp_c:
mapped[camp_c] = row_dict[col_f]
# Rezolva data
for col_f, camp_c in json_mapare.items():
if camp_c == "data_prestatie":
col_fmt = fmt or "ambiguous"
raw_date = mapped.get("data_prestatie")
if raw_date is not None:
iso_date, _ = parse_date_value(raw_date, col_fmt)
if iso_date:
mapped["data_prestatie"] = iso_date
break
# Operatia → prestatii
operatie_val = mapped.pop("operatie", None)
if operatie_val and "prestatii" not in mapped:
mapped["prestatii"] = [{"cod_op_service": str(operatie_val), "denumire": str(operatie_val)}]
# Rezolva prestatii
prestatii = mapped.get("prestatii") or []
resolved_p, _ = resolve_prestatii(prestatii, mapping_ops)
mapped["prestatii"] = resolved_p
# Canonicalizare
canon = canonicalize_row(mapped)
mapped.update({
"vin": canon["vin"],
"nr_inmatriculare": canon["nr_inmatriculare"],
"odometru_final": canon["odometru_final"],
})
key = build_key(account_id, canon)
rows_for_hash.append(json.dumps({
"row_index": row_index,
"vin": mapped.get("vin"),
"data_prestatie": mapped.get("data_prestatie"),
"odometru_final": mapped.get("odometru_final"),
"prestatii": [
str(p.get("cod_prestatie") or p.get("cod_op_service") or "")
for p in resolved_p
],
}, sort_keys=True, ensure_ascii=False))
cur = conn.execute(
"INSERT OR IGNORE INTO submissions "
"(idempotency_key, account_id, status, payload_json, batch_id, row_index, purge_after) "
"VALUES (?, ?, 'queued', ?, ?, ?, datetime('now', '+90 days'))",
(key, acct, json.dumps(mapped, ensure_ascii=False), import_id, row_index),
)
if cur.rowcount == 0:
toctou.append(row_index)
else:
enqueued.append({"submission_id": cur.lastrowid, "row_index": row_index})
conn.execute("COMMIT")
except Exception:
conn.execute("ROLLBACK")
raise
n_enqueued = len(enqueued)
# Log atestare (Voce#9)
rows_hash = hashlib.sha256(
json.dumps(sorted(rows_for_hash), ensure_ascii=False).encode("utf-8")
).hexdigest() if rows_for_hash else ""
conn.execute(
"INSERT INTO import_attestations (batch_id, account_id, confirmed_by, rows_hash, n_confirmed) "
"VALUES (?, ?, ?, ?, ?)",
(import_id, acct, confirmed_by, rows_hash, n_enqueued),
)
conn.execute(
"UPDATE import_batches SET status='committed', ok=? WHERE id=?",
(n_enqueued, import_id),
)
# Succes → drop zone cu mesaj de confirmare
toctou_msg = f" ({len(toctou)} coliziuni TOCTOU excluse)" if toctou else ""
return templates.TemplateResponse("_upload.html", {
"request": request,
"message": (
f"S-au pus in coada {n_enqueued} prezentari{toctou_msg}. "
f"Procesarea incepe in cateva secunde — urmareste coada de mai jos."
),
})
finally:
conn.close()

View File

@@ -0,0 +1,96 @@
<div id="import-section">
<div class="card">
<h2 style="font-size:15px; margin:0 0 12px;">
Mapare coloane —
<span class="muted" style="font-weight:400;">{{ filename or ("import #" ~ import_id) }}</span>
</h2>
{% if message %}
<div class="flash" style="{% if error %}border-color:var(--err); background:#241a1a;{% endif %} margin-bottom:12px;"
{% if error %}role="alert"{% endif %}>
{{ message }}
</div>
{% endif %}
<p class="muted" style="margin:0 0 12px; font-size:13px;">
Asociaza fiecare coloana din fisier cu campul canonic corespunzator.
Maparea se retine automat pentru fisiere cu acelasi antet.
</p>
<form hx-post="/_import/{{ import_id }}/mapare-coloane"
hx-target="#import-section"
hx-swap="outerHTML">
<div style="margin-bottom:8px; display:flex; align-items:center; gap:10px; flex-wrap:wrap;">
<label for="format-data" style="font-size:13px; color:var(--muted);">
Format data
</label>
<input type="text" id="format-data" name="format_data"
value="{{ format_data or 'DD.MM.YYYY' }}"
placeholder="ex: DD.MM.YYYY"
style="max-width:160px;"
aria-describedby="format-data-hint">
<span id="format-data-hint" class="muted" style="font-size:12px;">
sau YYYY-MM-DD, MM/DD/YYYY etc.
</span>
</div>
{% for col in columns %}
{%- set sugg = fuzzy_suggestions.get(col, []) -%}
{%- set best = sugg[0].camp_canonic if sugg else '' -%}
<input type="hidden" name="colname" value="{{ col }}">
<div class="maprow">
<div class="mapcol grow">
<div><strong>{{ col }}</strong></div>
{% if sugg %}
<div class="muted" style="font-size:12px; margin-top:2px;">
sugestie: <span class="sugg">{{ sugg[0].camp_canonic }}
({{ sugg[0].score | round | int }}%)</span>
</div>
{% endif %}
{%- set ns = namespace(samples=[]) -%}
{%- for row in sample_rows -%}
{%- if row.get(col) is not none and row.get(col) != '' -%}
{%- set ns.samples = ns.samples + [row[col] | string] -%}
{%- endif -%}
{%- endfor -%}
{% if ns.samples %}
<div class="muted" style="font-size:11px; margin-top:2px;">
ex: {{ ns.samples[:2] | join(", ") }}
</div>
{% endif %}
</div>
<div class="mapcol" style="min-width:200px;">
<label for="canon-{{ loop.index }}"
style="display:block; font-size:12px; color:var(--muted); margin-bottom:2px;">
Camp canonic
</label>
<select id="canon-{{ loop.index }}" name="canon">
<option value="">— ignorat —</option>
{% for field_key, field_label in canonical_fields %}
<option value="{{ field_key }}"
{% if field_key == best %}selected{% endif %}>
{{ field_key }} — {{ field_label }}
</option>
{% endfor %}
</select>
</div>
</div>
{% endfor %}
<div style="margin-top:16px; display:flex; align-items:center; gap:12px; flex-wrap:wrap;">
<button type="submit"
style="min-height:44px; padding:10px 24px; font-size:14px;">
Salveaza si continua la preview
</button>
<span class="muted" style="font-size:12px;">
maparea se retine pentru fisiere cu acelasi antet
</span>
</div>
</form>
<div style="margin-top:12px;">
<a href="/" class="muted" style="font-size:13px;">Incarca alt fisier</a>
</div>
</div>
</div>

View File

@@ -0,0 +1,236 @@
<div id="import-section">
<div class="card">
<div style="display:flex; align-items:center; gap:8px; flex-wrap:wrap; margin-bottom:12px;">
<h2 style="font-size:15px; margin:0;">
Preview —
<span class="muted" style="font-weight:400;">{{ filename or ("import #" ~ import_id) }}</span>
</h2>
<span class="muted" style="margin-left:auto; font-size:13px;">{{ total }} randuri</span>
</div>
{% if message %}
<div class="flash" style="{% if error %}border-color:var(--err); background:#241a1a;{% endif %} margin-bottom:12px;"
{% if error %}role="alert"{% endif %}>
{{ message }}
</div>
{% endif %}
<!-- Rezumat stari -->
<div style="display:flex; gap:8px; flex-wrap:wrap; margin-bottom:12px;">
{% set status_labels = [
('ok', 'gata de trimis'),
('needs_review', 'verifica valori'),
('needs_mapping', 'fara cod RAR'),
('needs_data', 'date lipsa'),
('already_sent', 'deja trimis'),
('duplicate_in_file','dublicat in fisier'),
] %}
{% for status_key, label in status_labels %}
{%- set cnt = summary.get(status_key, 0) -%}
{% if cnt > 0 %}
<span class="pill s-{{ status_key }}">{{ cnt }} {{ label }}</span>
{% endif %}
{% endfor %}
</div>
<!-- Butoane filtrare stare -->
<div style="display:flex; gap:6px; flex-wrap:wrap; margin-bottom:10px;" role="group"
aria-label="Filtrare dupa stare">
<button type="button" class="filter-btn" data-filter="all"
style="min-height:36px; font-size:13px; padding:4px 12px;">
Toate ({{ total }})
</button>
{% for status_key, label in status_labels %}
{%- set cnt = summary.get(status_key, 0) -%}
{% if cnt > 0 %}
<button type="button" class="filter-btn" data-filter="{{ status_key }}"
style="min-height:36px; font-size:13px; padding:4px 12px;
background:transparent; border-color:var(--line); color:var(--ink);">
{{ status_key }} ({{ cnt }})
</button>
{% endif %}
{% endfor %}
</div>
<!-- Tabel preview + bara confirmare (un singur form) -->
<form id="confirm-form"
hx-post="/_import/{{ import_id }}/confirma"
hx-target="#import-section"
hx-swap="outerHTML">
<div class="tablewrap">
<table>
<thead>
<tr>
<th>#</th>
<th>VIN</th>
<th>Nr. Inm.</th>
<th>Data</th>
<th>KM final</th>
<th>Operatie</th>
<th>Stare</th>
<th>Note</th>
<th>Verificat?</th>
</tr>
</thead>
<tbody>
{% for row in rows %}
{%- set res = row.resolved -%}
{%- set status = row.resolved_status -%}
{%- set prestatii = res.get('prestatii') or [] -%}
{%- set op = (prestatii[0].get('cod_prestatie') or prestatii[0].get('cod_op_service', '')) if prestatii else '' -%}
<tr data-status="{{ status }}"
style="{% if status == 'needs_review' %}background:rgba(230,179,74,.04);{% elif status in ('already_sent', 'duplicate_in_file') %}opacity:.65;{% endif %}">
<td class="muted">{{ row.row_index + 1 }}</td>
<td>{{ res.get('vin') or '<span class="muted"></span>' | safe }}</td>
<td>{{ res.get('nr_inmatriculare') or '' }}</td>
<td>{{ res.get('data_prestatie') or '' }}</td>
<td>{{ res.get('odometru_final') or '' }}</td>
<td>{{ op or '<span class="muted"></span>' | safe }}</td>
<td>
<span class="pill s-{{ status }}">{{ status }}</span>
</td>
<td class="muted" style="font-size:12px; white-space:normal; max-width:220px;">
{% if status == 'already_sent' and row.get('already_sent_info') %}
{% set ai = row.already_sent_info %}
deja trimis {{ (ai.get('created_at') or '')[:10] }}
{% if ai.get('id_prezentare') %}(#{{ ai.id_prezentare }}){% endif %}
{% elif status == 'duplicate_in_file' and row.get('duplicate_with') %}
dubla cu randul
{% for idx in row.duplicate_with %}{{ idx + 1 }}{% if not loop.last %}, {% endif %}{% endfor %}
{% elif row.flags %}
{{ row.flags[0] }}
{% elif row.errors %}
{%- set e = row.errors[0] -%}
{%- if e is mapping -%}
{{ e.values() | list | first }}
{%- else -%}
{{ e }}
{%- endif -%}
{% endif %}
</td>
<td style="text-align:center;">
{% if status == 'needs_review' %}
<label class="chk" style="min-height:44px; justify-content:center; cursor:pointer;"
title="Bifat inseamna ca ai verificat valorile si le incluzi in trimitere">
<input type="checkbox" name="reviewed_rows" value="{{ row.row_index }}"
onchange="updateN()"
aria-label="Verificat — randul {{ row.row_index + 1 }} (VIN: {{ res.get('vin', '') }})">
verif.
</label>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
<!-- Bara confirmare (sticky jos) -->
<div class="sticky-bar">
<div style="flex:1; min-width:280px;">
<!-- Banner declarant (D12) — direct deasupra input-ului N -->
<div class="banner warn" style="margin-bottom:10px; padding:8px 12px; border-radius:6px;"
role="note" aria-live="polite">
Confirmand, TU esti declarantul acestor
<strong id="n-display">{{ summary.get('ok', 0) }}</strong>
prezentari la RAR (ireversibil).
</div>
<div style="display:flex; gap:12px; align-items:center; flex-wrap:wrap;">
<div>
<label for="n-confirmat"
style="font-size:13px; color:var(--muted); display:block; margin-bottom:2px;">
Numar prezentari de confirmat
</label>
<input type="number" id="n-confirmat" name="n_confirmat"
value="{{ summary.get('ok', 0) }}"
min="0" required
style="max-width:80px;"
aria-describedby="n-hint">
<span id="n-hint" class="muted" style="font-size:12px; margin-left:6px;">
({{ summary.get('ok', 0) }} ok
{% if summary.get('needs_review', 0) %}
+ pana la {{ summary.get('needs_review', 0) }} verificate manual
{% endif %})
</span>
</div>
<div>
<label for="confirmed-by"
style="font-size:13px; color:var(--muted); display:block; margin-bottom:2px;">
Declarant (optional)
</label>
<input type="text" id="confirmed-by" name="confirmed_by"
placeholder="email sau nume"
style="max-width:200px;">
</div>
</div>
</div>
<div style="display:flex; flex-direction:column; gap:6px; align-self:flex-end;">
<button type="submit"
id="confirm-btn"
style="min-height:44px; padding:10px 28px; font-size:14px;"
{% if not summary.get('ok', 0) %}disabled title="Niciun rand ok de trimis"{% endif %}>
Trimite la RAR
</button>
{% if summary.get('needs_data', 0) or summary.get('needs_mapping', 0) or summary.get('needs_review', 0) %}
<a href="/v1/import/{{ import_id }}/export-failed" download
style="font-size:12px; text-align:center;">
descarca randuri cu probleme (CSV)
</a>
{% endif %}
</div>
</div>
</form>
<div style="padding:8px 0 4px;">
<a href="/" class="muted" style="font-size:13px;">Incarca alt fisier</a>
</div>
</div>
</div>
<script>
(function() {
var nOk = {{ summary.get('ok', 0) | int }};
/* Actualizeaza N si bannerul cand se bifeaza needs_review */
function updateN() {
var checked = document.querySelectorAll('input[name="reviewed_rows"]:checked').length;
var total = nOk + checked;
var inp = document.getElementById('n-confirmat');
var disp = document.getElementById('n-display');
var btn = document.getElementById('confirm-btn');
if (inp) inp.value = total;
if (disp) disp.textContent = total;
if (btn) btn.disabled = (total === 0);
}
/* Filtrare randuri dupa stare */
function filterRows(status) {
document.querySelectorAll('tbody tr[data-status]').forEach(function(tr) {
tr.style.display = (status === 'all' || tr.dataset.status === status) ? '' : 'none';
});
document.querySelectorAll('.filter-btn').forEach(function(b) {
var active = b.dataset.filter === status;
b.style.background = active ? 'var(--accent)' : '';
b.style.borderColor = active ? 'var(--accent)' : '';
b.style.color = active ? '#fff' : '';
});
}
/* Expune functiile global pentru onclick inline */
window.updateN = updateN;
window.filterRows = filterRows;
/* Filtru implicit "Toate" activ la incarcare */
filterRows('all');
/* Focus pe campul N la deschidere (a11y — D12) */
var ni = document.getElementById('n-confirmat');
if (ni) { ni.focus(); ni.select(); }
})();
</script>

View File

@@ -0,0 +1,106 @@
<div id="import-section">
<div class="card">
<h2 style="font-size:15px; margin:0 0 12px;">Import fisier (xlsx / csv)</h2>
{% if message %}
<div class="flash" style="margin-bottom:12px;">{{ message }}</div>
{% endif %}
{% if error %}
<div class="flash" style="border-color:var(--err); background:#241a1a; margin-bottom:12px;"
role="alert">{{ error }}</div>
{% endif %}
{% if sheets %}
<div class="flash" style="border-color:var(--warn); background:#201c0f; margin-bottom:12px;">
Fisierul are mai multe foi de lucru. Alege foaia de mai jos si incarca din nou.
</div>
{% endif %}
<form id="upload-form"
hx-post="/_import/upload"
hx-target="#import-section"
hx-swap="outerHTML"
hx-encoding="multipart/form-data"
hx-indicator="#upload-spinner">
{% if sheets %}
<div style="margin-bottom:12px;">
<label for="sheet-select"
style="display:block; margin-bottom:4px; font-size:13px; color:var(--muted);">
Foaie de lucru
</label>
<select id="sheet-select" name="sheet_name" style="min-width:200px;">
{% for s in sheets %}
<option value="{{ s }}">{{ s }}</option>
{% endfor %}
</select>
</div>
{% endif %}
<div class="drop-zone" id="drop-zone"
role="region" aria-label="Zona de incarcare fisier">
{% if not sheets %}
<p style="font-size:17px; margin:0 0 4px; font-weight:600;">Primul fisier? Trage-l aici.</p>
<p class="muted" style="margin:0 0 16px; font-size:13px;">xlsx sau csv, max 5000 randuri</p>
{% else %}
<p class="muted" style="margin:0 0 16px; font-size:14px;">
Incarca fisierul din nou dupa ce ai ales foaia.
</p>
{% endif %}
<input id="file-input" type="file" name="file" accept=".xlsx,.xls,.csv"
style="display:none;" aria-label="Selecteaza fisier xlsx sau csv">
<button type="button" id="upload-btn"
style="min-height:44px; padding:10px 24px; font-size:14px;">
Alege fisier
</button>
</div>
<p class="muted" style="margin:8px 0 0; font-size:12px;">
NU se trimite nimic la RAR pana confirmi explicit.
</p>
<span id="upload-spinner" class="htmx-indicator muted"
style="font-size:13px; margin-top:6px; display:inline;">
se parseaza fisierul...
</span>
</form>
</div>
</div>
<script>
(function() {
var btn = document.getElementById('upload-btn');
var fi = document.getElementById('file-input');
var dz = document.getElementById('drop-zone');
var frm = document.getElementById('upload-form');
if (!btn || !fi || !frm) return;
btn.addEventListener('click', function() { fi.click(); });
fi.addEventListener('change', function() {
if (fi.files.length > 0) frm.requestSubmit();
});
dz.addEventListener('dragover', function(e) {
e.preventDefault();
dz.classList.add('drag-over');
});
dz.addEventListener('dragleave', function(e) {
if (!dz.contains(e.relatedTarget)) dz.classList.remove('drag-over');
});
dz.addEventListener('drop', function(e) {
e.preventDefault();
dz.classList.remove('drag-over');
var f = (e.dataTransfer.files || [])[0];
if (!f) return;
try {
var dt = new DataTransfer();
dt.items.add(f);
fi.files = dt.files;
} catch (_) {}
frm.requestSubmit();
});
})();
</script>

View File

@@ -28,8 +28,24 @@
.pill { font-size:12px; padding:2px 8px; border-radius:99px; border:1px solid var(--line); }
.s-queued{color:var(--accent);} .s-sending{color:var(--warn);} .s-sent{color:var(--ok);}
.s-error,.s-needs_data,.s-needs_mapping{color:var(--err);}
.s-ok{color:var(--ok);}
.s-needs_review{color:var(--warn);}
.s-already_sent,.s-duplicate_in_file{color:var(--muted);}
.muted { color:var(--muted); }
a { color:var(--accent); }
/* Drop zone upload fisier */
.drop-zone { border:2px dashed var(--line); border-radius:8px; padding:32px 20px;
text-align:center; transition:border-color .15s,background .15s; }
.drop-zone.drag-over { border-color:var(--accent); background:rgba(91,141,239,.05); }
/* Banner varianta warn (nu eroare) */
.banner.warn { border-left-color:var(--warn); background:#201c0f; }
/* Bara confirmare sticky */
.sticky-bar { position:sticky; bottom:0; background:var(--card); border-top:1px solid var(--line);
padding:12px 16px; display:flex; align-items:flex-start; gap:16px;
flex-wrap:wrap; z-index:10; }
/* Indicator HTMX — ascuns pana la request */
.htmx-indicator { display:none; }
.htmx-indicator.htmx-request { display:inline; }
/* Link-uri de actiune in antetul cardurilor: zona de atins mai mare (>=36px) si
feedback la hover; pe ecrane inguste antetul se rupe curat sub titlu. */
.cardlink { font-size:13px; padding:7px 10px; border-radius:6px; display:inline-flex;

View File

@@ -1,6 +1,9 @@
{% extends "base.html" %}
{% block content %}
<!-- Sectiunea de import fisier: stare initiala = drop zone; HTMX swapeaza in flow -->
{% include '_upload.html' %}
<div class="card banner {% if not blocked %}hidden{% endif %}"
hx-get="/_fragments/banner" hx-trigger="every 15s" hx-swap="outerHTML">
<strong>Atentie:</strong> {{ blocked }} submission-uri blocate (error / needs_data / needs_mapping).

View File

@@ -75,6 +75,15 @@ def _is_transient(exc: Exception) -> bool:
# --- Operatii pe submissions ---
def mark(conn, submission_id: int, status: str, *, rar_status_code=None, rar_error=None, id_prezentare=None) -> None:
if status == "sent":
# T16: purge_after = sent + 90 zile (GDPR/L.142 retentie maxima).
conn.execute(
"UPDATE submissions SET status=?, rar_status_code=?, rar_error=?, id_prezentare=?, "
"sending_since=NULL, updated_at=datetime('now'), "
"purge_after=datetime('now', '+90 days') WHERE id=?",
(status, rar_status_code, rar_error, id_prezentare, submission_id),
)
else:
conn.execute(
"UPDATE submissions SET status=?, rar_status_code=?, rar_error=?, id_prezentare=?, "
"sending_since=NULL, updated_at=datetime('now') WHERE id=?",
@@ -82,6 +91,29 @@ def mark(conn, submission_id: int, status: str, *, rar_status_code=None, rar_err
)
# T16: purge interval in secunde (odata pe ora, nu prea agresiv)
_PURGE_INTERVAL_S = 3600
def purge_expired(conn) -> dict[str, int]:
"""Sterge randurile expirate (purge_after < now).
T16/OV-5: purge_after era exportat dar setat de nimeni si niciun job nu exista.
Acum: submissions sent + expirate, import_batches expirate (import_rows via CASCADE).
Intoarce {submissions_purged, batches_purged}.
"""
cur_sub = conn.execute(
"DELETE FROM submissions WHERE purge_after IS NOT NULL AND purge_after < datetime('now') AND status='sent'"
)
cur_batch = conn.execute(
"DELETE FROM import_batches WHERE purge_after IS NOT NULL AND purge_after < datetime('now')"
)
return {
"submissions_purged": cur_sub.rowcount,
"batches_purged": cur_batch.rowcount,
}
def requeue_with_backoff(conn, settings: Settings, submission_id: int, *, reason: str) -> None:
"""Re-pune randul in coada cu retry++ si next_attempt_at = now + backoff."""
row = conn.execute("SELECT retry_count FROM submissions WHERE id=?", (submission_id,)).fetchone()
@@ -267,7 +299,10 @@ class AccountSessions:
raise
self._sessions[account_id] = (rar, token)
write_heartbeat(conn, rar_login_ok=True, detail=f"login RAR ok (cont {account_id})")
# Creds nu mai sunt necesare: JWT acopera retry-urile -> sterge la rest.
# Creds efemere pe submissions nu mai sunt necesare: JWT acopera retry-urile -> sterge.
# GATE PURJARE (T1/Voce#5): sterge DOAR submissions.rar_creds_enc, NU accounts.rar_creds_enc.
# Canal web: fallback exista in accounts -> purjarea e inofensiva (re-login dupa restart).
# Canal API pur: purjarea e identica cu Treapta 1 (neatinsa).
conn.execute(
"UPDATE submissions SET rar_creds_enc=NULL WHERE account_id=? AND rar_creds_enc IS NOT NULL",
(account_id,),
@@ -303,6 +338,20 @@ def _creds_for(claimed: dict, settings: Settings) -> dict | None:
return None
def _creds_from_account(conn, account_id: int) -> dict | None:
"""Fallback T1/D4: crede RAR durabile per-cont din accounts.rar_creds_enc.
Canal web nu are re-pusher. Cand submission n-are creds (sterse dupa primul login
sau upload web fara creds), worker-ul re-citeste din cont si poate re-login oricand.
"""
row = conn.execute(
"SELECT rar_creds_enc FROM accounts WHERE id=?", (account_id,)
).fetchone()
if row and row["rar_creds_enc"]:
return decrypt_creds(row["rar_creds_enc"])
return None
def run() -> int:
signal.signal(signal.SIGTERM, _stop)
signal.signal(signal.SIGINT, _stop)
@@ -313,11 +362,24 @@ def run() -> int:
print(f"[worker] pornit (send_enabled={settings.worker_send_enabled}, env={settings.rar_env})", flush=True)
sessions = AccountSessions(settings)
_last_purge_time: float = 0.0
while _running:
try:
write_heartbeat(conn, detail=f"poll (queue={_queue_depth(conn)})")
# T16: purjare periodica (odata pe ora) — NU mai frecvent.
now_ts = time.time()
if now_ts - _last_purge_time >= _PURGE_INTERVAL_S:
stats = purge_expired(conn)
if stats["submissions_purged"] or stats["batches_purged"]:
print(
f"[worker] purjare: {stats['submissions_purged']} submissions, "
f"{stats['batches_purged']} batches sterse",
flush=True,
)
_last_purge_time = now_ts
if not settings.worker_send_enabled:
time.sleep(settings.worker_poll_interval_s)
continue
@@ -332,7 +394,9 @@ def run() -> int:
sid = claimed["id"]
account_id = claimed["account_id"]
creds = _creds_for(claimed, settings)
# T1/D4: incearca creds din submission (canal API efemer), cu fallback la
# accounts.rar_creds_enc (canal web durabil). Canal web n-are re-pusher.
creds = _creds_for(claimed, settings) or _creds_from_account(conn, account_id)
try:
token = sessions.get_token(conn, account_id, creds)

474
docs/plans/plan-treapta2.md Normal file
View File

@@ -0,0 +1,474 @@
# Plan Treapta 2 — Import xlsx/csv + mapare coloane (canal non-ROA)
> **Plan executabil, post-review CEO (SELECTIVE EXPANSION).** Continuă `plan.md` (Treapta 1 = LIVE).
> Acoperă funcționalitățile de integrare care AZI nu există: upload fișier, mapare coloane,
> spectru de integrare. Motor identic cu Treapta 1 (mapare op→cod + coadă + worker + monitorizare).
> Ultima actualizare: 2026-06-16. Review: `/plan-ceo-review` + voce externă (subagent independent).
## 1. Problema (de ce acum)
Treapta 1 servește clienții ROA prin ROAAUTO (API JSON). Dar obligația legală (L.142/2023)
apasă pe **mii de service-uri non-ROA** care AZI introduc manual în interfața web AUTOPASS,
prezentare cu prezentare (2-4 min × 60-100/lună = 3-6 ore/lună de tastare). Nu au ROAAUTO și nu
vor scrie cod. Au deja datele în Excel/export din propriul soft.
**Rezultat țintă:** un service non-ROA, fără instalare, încarcă un fișier (xlsx/csv), mapează
coloanele o singură dată (reținut), vede preview cu rândurile cu probleme flag-uite, confirmă
explicit, apasă „Trimite la RAR" și prezentările apar `FINALIZATA`. A doua oară: drop fișier → trimite.
## 2. Ce există deja (reuse, NU se rescrie)
| Sub-problemă | Reuse din Treapta 1 | Atenție la review |
|---|---|---|
| Validare conținut (VIN/dată/odometru/nrInm) | `app/validation.py` | OK, se compune cu batch |
| Mapare operație→codPrestatie + fuzzy | `app/mapping.py`, `operations_mapping` | ⚠️ `reresolve_account` e account-GLOBAL (vezi Risc R1) |
| Coadă + idempotency | `submissions`, `app/idempotency.py` | ⚠️ cheia exclude obs/op-denumire (vezi 3.4) |
| Worker login RAR + postPrezentare + retry | `app/worker`, `app/rar_client.py` | ⚠️ purjează creds (vezi 3.6, decizie D4) |
| Reconciliere anti-duplicat | `app/reconcile.py` | OK |
| Monitorizare + audit CSV | `/v1/prezentari`, `/v1/audit/export`, dashboard | OK |
| Auth API-key per cont | `app/auth.py` | OK |
| Criptare PII (Fernet) | `app/crypto.py` | refolosit pt. `import_rows` |
Nou = **doar stratul de INTRARE** (parsare + mapare coloane + preview) + **un fix de model creds pe web**.
## 3. Funcționalități noi (scope confirmat la review)
### 3.1 Upload fișier — `POST /v1/import`
- `multipart/form-data`: `.xlsx`, `.xls`, `.csv`. Encoding: UTF-8 + **fallback `cp1250`/`latin2` (RO)** + BOM.
- Parsare: `openpyxl` (xlsx) / `csv` stdlib. Limită (ex. 5 MB / ~5000 rânduri) → semnal explicit, nu trunchiere tăcută.
- **openpyxl `read_only=True` streaming (Eng#6):** `load_workbook(read_only=True, data_only=True)`; verifică
`max_row`/dimensiune ÎNAINTE de parsare → `FileTooLarge` fără parse parțial. Memorie marginală, rămâne sincron
(acceptabil la volumul țintă, R2). `wb.close()` la final.
- **CSV delimiter sniff (Eng#3):** export Excel RO folosește `;` (virgula = separator zecimal). `csv.Sniffer`
pe `{; , \t}` sau probă explicită; alege delimiter-ul care dă >1 coloană consistent. 1 coloană → `HeaderError`
clar, NU mapare oarbă.
- **Coercion Excel (R3):** odometru numeric → `123456.0`; VIN/nr cu zerouri tăiate; date ca `datetime`. Normalizarea
e centralizată în `idempotency.canonicalize_row` (vezi 3.4bis); coercion nerecuperabilă → starea `needs_review` (3.4).
- **Dată dezambiguizată (Voce#2):** celulă `datetime` nativă din openpyxl → folosită DIRECT (neambiguă). Celulă
STRING → aplică `date_fmt` mapat, DAR dacă `zi<=12` (deci și MM.DD ar fi valid) → forțează `needs_review`,
nu trimite orb. Acoperă fișierul mixt datetime/string (cazul real RO).
- **Robustețe export RO real (Voce#6/#7):** dacă workbook-ul are >1 sheet non-gol → cere alegerea sheet-ului
(nu presupune `active`); rezolvă celulele header **îmbinate** (un-merge logic → nume reale sau flag, nu nume goale);
taie rândurile trailing unde coloanele-cheie (VIN+dată) sunt goale (footer `TOTAL`/`Întocmit de:`); rând fără VIN
= **skip structural**, nu `needs_data` fantomă.
- Detectează header (primul rând non-gol), întoarce `{import_id, columns, sample_rows}`. NU trimite nimic la RAR.
- Stochează în `import_batches` / `import_rows` (PII **criptat** cu `app/crypto.py`, `purge_after` ca `submissions`).
### 3.2 Mapare coloane (NOUĂ — stratul care lipsește azi)
- **Schemă fișier → câmpuri canonice**: `vin`, `nr_inmatriculare`, `data_prestatie`, `odometru_final`,
`odometru_initial?`, `operatie` (denumire/cod), `obs?`.
- **Reținută per cont** (`column_mappings`), cu **semnătură de coloane**. Map once, reuse forever.
- **Detectie drift (acceptat D3):** maparea reținută se aplică DOAR dacă semnătura coloanelor se potrivește
exact. Coloane mutate/redenumite → NU aplica orb, cere re-confirmare. Previne maparea tăcută greșită la upload 2.
- Auto-sugestie fuzzy pe nume coloană („VIN"/„Serie sasiu"→vin; „KM"→odometru_final).
- **DRY (Eng#4):** refolosește `mapping.normalize_for_match` (NFKD+lowercase+strip) + `fuzz.token_sort_ratio`
(rapidfuzz) — ACELAȘI primitiv ca editorul de operații. Map `{camp_canonic: [sinonime]}`, zero dependință nouă.
- Format dată configurabil per mapare (`DD.MM.YYYY` RO vs ISO) → normalizat la `YYYY-MM-DD` (vezi dezambiguizarea în 3.1).
### 3.3 Mapare operații (reuse Treapta 1)
- Eticheta operației din fișier → `codPrestatie` prin `operations_mapping` + fuzzy.
- **Gate auto_send pe coduri noi (acceptat D3):** o operație nou-mapată sau cod neobișnuit NU se trimite
automat → review manual o dată (un `FINALIZATA` eronat e permanent).
### 3.4 Preview + commit (gate HARD)
- `GET /v1/import/{id}/preview`: fiecare rând cu stare derivată (rulează `validation.py` + `resolve_prestatii`
FĂRĂ enqueue). **Cinci stări:**
- `ok` — gata de trimis.
- `needs_mapping` — operație fără cod.
- `needs_data` — validare RAR eșuată / odometru lipsă.
- `needs_review` (acceptat D6, R3) — coercion Excel suspectat (VIN numeric, odometru float). **Blochează
auto-send chiar dacă validarea trece** — VIN stricat = `FINALIZATA` permanent greșit.
- `already_sent` (acceptat D5) — cheia idempotency există deja. Preview arată „deja trimis pe `<data>` ca
`idPrezentare X`". **Niciodată dedup tăcut într-un commit în masă** — decizie explicită per-rând.
- **Lookup batch, nu N+1 (Eng#5):** calculează toate cheile, apoi `SELECT idempotency_key FROM submissions
WHERE account_id=? AND idempotency_key IN (chunk)` (chunk-uri ~900 param SQLite). O(1) interogări, nu 5000.
`load_mapping` o singură dată ca POST.
- `duplicate_in_file` (Voce#3, NOU) — coliziune INTRA-batch. Grupare pe cheie în fișierul parsat:
`|grup|>1` identice → „rândul 12 și 88 identice"; același `vin+dată+odometru` cu operație diferită →
„rândul 12 și 41 diferă doar prin operație, confirmă". `already_sent` verifică doar batch-uri anterioare;
aceasta prinde coliziunile din ACELAȘI fișier (altfel `UNIQUE` global le înghite/erează mid-batch).
- **Gate HARD de confirmare (acceptat D3 + Voce#1):** rezumat dry-run (X gata, Y date lipsă, Z nemapate, W deja
trimise) + confirmare explicită (tastezi numărul de prezentări). **Plus atestare pe VALORI, nu doar pe total:**
preview-ul arată per-rând valorile FINALE rezolvate (VIN, dată ca `YYYY-MM-DD` cum o vede RAR, km); rândurile
`needs_review` trebuie bifate explicit „verificat" ca să intre în N. `N` dovedește numărul; bifa dovedește
conținutul. Oprește atât count-error cât și content-error (VIN coercionat / dată swap în rândul individual).
- Commit = enqueue în `submissions` DOAR rândurile `ok` confirmate → worker → monitorizare standard.
- **Log atestare (Voce#9):** la commit scrie `import_attestations` (batch_id, account_id, confirmed_by, ts,
`rows_hash`=sha256 peste valorile rezolvate confirmate, n_confirmed). Apare în `/v1/audit/export`. UI sub bară:
„Confirmând, TU ești declarantul acestor N prezentări la RAR (ireversibil)". Apărare legală + trasabilitate
(L.142/2023 — operatorul e declarantul de rol).
### 3.4bis Cheie idempotency canonică partajată (Eng#2)
- Coercion-ul Excel (`123456.0`) calculat ÎNAINTE de cheie poate da cheie diferită de POST live (`123456`) →
`already_sent` ratează → al doilea `FINALIZATA`. Fix: extrage normalizarea canonică (odometru strip `.0`,
VIN upper/strip, dată `YYYY-MM-DD`) într-un helper public `idempotency.canonicalize_row(raw) -> dict` +
`build_key(account_id, canon)`. **Parser-ul de import ȘI `POST /v1/prezentari` apelează ACELAȘI helper** înainte
de cheie ȘI de validare. O sursă de adevăr; cele două canale nu pot diverge.
- **REGRESIE CRITICĂ:** cheia produsă de refactor trebuie să fie IDENTICĂ cu cea de azi pentru input-uri
existente, altfel rândurile deja trimise capătă cheie nouă → re-trimise. Test de regresie obligatoriu.
### 3.5 Spectru de integrare (același backend)
1. **API** (există) — ROAAUTO / soft propriu.
2. **Upload manual în browser** (3.1-3.4) — service fără cod. **Acesta e scope-ul acestei trepte.**
3. ~~Drop fișier (SFTP/email-to-import)~~ — **CUT din această treaptă** (vezi NOT in scope, decizie D6).
### 3.6 Acces web + creds RAR (model creds CORECTAT — decizie D4 + Eng#1/Voce#5)
- Login web (email + parolă cont) → folosește/emite API-key existent.
- Creds RAR introduse în UI, **criptate-at-rest per-cont** pe **coloana `accounts.rar_creds_enc`** (ALTER aditiv,
exact ca migrarea existentă a aceleiași coloane pe `submissions`; NU tabel nou — Eng#1). O singură sursă per cont.
- **Worker re-login (fallback):** `claim_one` rămâne; la login worker-ul face `creds = submission.creds_enc
OR SELECT rar_creds_enc FROM accounts WHERE id=?`. Submission web fără creds → ia din `accounts` → login OK.
- **De ce abatere de la zero-storage Treptei 1:** canalul web **nu are re-pusher** (ROAAUTO re-trimitea
la creds lipsă; aici nu există). Worker-ul trebuie să poată re-login oricând, altfel o serie încărcată
zile mai târziu, după un restart worker, rămâne blocată permanent → declarație legală netrimisă, tăcut.
- **Gate purjare `worker:271` (Voce#5, P1):** purjarea existentă `UPDATE submissions SET rar_creds_enc=NULL
WHERE account_id=?` e ACCOUNT-scoped → la primul login web ar șterge creds de pe TOATE submission-urile
contului, inclusiv cele API-channel efemere. Conturile CU `accounts.rar_creds_enc` durabil: purjarea devine
inofensivă (worker re-citește din `accounts`). Conturile FĂRĂ durabil (API-channel pur Treapta 1): purjarea
rămâne neschimbată. **Test = coadă MIXTĂ API+web** (după login web, submission-urile API tot se trimit), nu doar web.
- Compensare risc: creds tot criptate (Fernet), tot redactate din loguri; doar **persistate**, nu efemere.
Blast-radius mai mare la scurgere cheie Fernet (creds durabile vs. doar in-flight) — acceptat conștient (D4).
## 4. Date noi (SQLite)
- `column_mappings` (account_id, signature_coloane, json mapare, format_data, created_at).
- `import_batches` (id, account_id, filename, status, total/ok/needs_*/already_sent/duplicate_in_file, created_at, purge_after).
- `import_rows` (batch_id, row_index, raw_json **criptat**, resolved_status, error). Purjate cu batch-ul.
- `accounts.rar_creds_enc` — **coloană durabilă per-cont** (ALTER aditiv, NU tabel nou) pentru canalul web (D4, Eng#1).
- `import_attestations` (batch_id, account_id, confirmed_by, ts, rows_hash, n_confirmed) — log atestare legală (Voce#9).
- `submissions += batch_id, row_index` (T7, P1) — scope pentru `reresolve_account` + trasabilitate export rânduri eșuate.
**Închide R1** (bulk-send tăcut cross-batch). T7 e predecesor HARD al U3 (vezi Roadmap).
## 5. NOT in scope (amânat / tăiat, cu motiv)
- **Drop-fișier SFTP / email-to-import** (era U7) — TĂIAT. Trei mecanisme de intrare înainte ca un singur
service non-ROA să fi încărcat manual un fișier. Validează întâi upload-ul manual, apoi decide. (D6)
- **Contor volum + prag freemium** (era U6) — DEFER. Metrici de preț înainte să existe useri; a contoriza o
obligație legală e delicat. Contorul e trivial de adăugat post-validare. (D6)
- **Wedge auto-drop (SFTP/email-to-import)** — DEFER, confirmat de user la eng review: „verific manual upload-ul
întâi". Manual upload e wedge-ul; auto-drop se re-evaluează post-validare (tensiune Voce#8 vs CEO D6, rezolvată
în favoarea manual).
- **Mapare AI / conector MCP** — Treapta 3.
- **Editare/anulare prezentări trimise** — `FINALIZATA` terminal, neschimbat.
- **Billing complet (Stripe etc.)** — după validarea pragului.
## 6. Mașina de stări (rând de import → submission)
```
┌─────────── POST /v1/import (parsare, staging, NU trimite) ──────────┐
▼ │
import_row.resolved_status: │
ok ─────────────┐ │
needs_mapping │ (preview: rezolva fara enqueue) │
needs_data │ │
needs_review ───┤ (coercion suspectat → blocheaza auto-send) │
already_sent ───┘ (cheie idempotency exista → decizie per-rand) │
│ │
▼ GATE HARD confirmare (tastezi N prezentari) │
commit: enqueue DOAR rinduri `ok` confirmate ──▶ submissions (queued) ──▶ worker (Treapta 1, neatins)
login RAR → postPrezentare → FINALIZATA
```
## 7. Error & Rescue Map (stratul nou)
```
CODEPATH | CE POATE MERGE PROST | EXCEPTIE / STARE
-------------------------------|-----------------------------------|----------------------
POST /v1/import parse xlsx | fisier corupt / non-xlsx | BadZipFile/InvalidFile
| encoding RO (cp1250) | UnicodeDecodeError
| >5MB / >5000 randuri | FileTooLarge (custom)
| header lipsa / coloane duplicate | HeaderError (custom)
parse cell | VIN/odometru coercion Excel | → stare needs_review
| data DD.MM.YYYY | → normalizare, altfel needs_data
apply column_mapping | semnatura coloane != reținuta | → cere re-confirmare (drift)
preview resolve | cheie idempotency exista | → stare already_sent
commit | confirmare numar gresit | reject, nu enqueue
| worker fara creds (restart) | → REZOLVAT D4 (creds durabile)
STARE / EXCEPTIE | RESCUED? | ACTIUNE | USER VEDE
------------------------|----------|----------------------------------|---------------------------
BadZipFile/InvalidFile | Y | 422, mesaj „fisier invalid" | „Fisier nerecunoscut (xlsx/csv)"
UnicodeDecodeError | Y | retry cp1250/latin2, apoi 422 | „Encoding nesuportat"
FileTooLarge | Y | 413, fara parsare partiala | „Max 5000 randuri / 5MB"
HeaderError | Y | 422 + ce coloane s-au gasit | „Antet neclar, verifica fisierul"
needs_review (coercion) | Y | blocheaza auto-send, cere uman | rind galben „verifica VIN/km"
already_sent | Y | NU dedup tacut, decizie per-rind | „deja trimis pe <data> #X"
drift semnatura coloane | Y | nu aplica orb, re-mapare | „coloanele difera, confirma maparea"
worker fara creds | Y (D4) | re-login din creds durabile | nimic (transparent)
```
## 8. Failure Modes Registry
```
CODEPATH | FAILURE MODE | RESCUED? | TEST? | USER VEDE | LOGGED?
--------------------------------|-------------------------------|----------|-------|------------------|--------
upload parse | encoding/format RO | Y | Y | mesaj clar | DA
cell coercion (VIN/odo Excel) | VIN stricat trece validarea | Y(D6) | Y | needs_review | DA
column_mapping drift | mapare tacuta gresita upload2 | Y(D3) | Y | re-confirmare | DA
commit in masa | trimite 100 randuri gresite | Y(D3) | Y | gate confirmare | DA
re-export (idempotency) | duplicat / corectie inghitita | Y(D5) | Y | already_sent | DA
worker restart, creds purjate | serie blocata permanent tacut | Y(D4) | Y | nimic (re-login) | DA
mapare salvata → re-resolve | trimite tacut randuri cross- | Y(T7) | Y | gate confirmare | DA
| batch / feed API live | | | (batch scoped) |
data string zi<=12 (DD vs MM) | data gresita-dar-valida trece | Y(V#2) | Y | needs_review | DA
duplicat in ACELASI fisier | UNIQUE global inghite/ereaza | Y(V#3) | Y | duplicate_in_file| DA
multi-op same vin+data+odo | reconcile dropa rand netrimis | Y(V#4) | Y | confirma manual | DA
creds durabile, login web | purjare account-scoped sterge | Y(V#5) | Y | nimic (fallback) | DA
| creds API-channel efemere | | | |
100 declaratii dintr-un N | raspundere fara atestare/rol | Y(V#9) | Y | UI declarant+log | DA
export RO: sheet 2 / merged hdr | HeaderError pe fisier valid | Y(V#6) | Y | alege sheet/flag | DA
export RO: footer TOTAL parsat | prestatie fantoma needs_data | Y(V#7) | Y | skip structural | DA
```
**R1 ÎNCHIS (T7, P1):** `reresolve_account` (mapping.py:253) primește `batch_id` și se scope-ază la seria comitată;
T7 e predecesor HARD al U3. Salvarea unei mapări nu mai poate auto-queue rânduri cross-batch / din feed API live.
Niciun failure mode silent rămas neacoperit.
## 9. Riscuri / open questions
- **R1 (ÎNCHIS la eng review):** mapare account-global → bulk-send tăcut cross-batch. Fix `batch_id` scoping
promovat la P1-blocking (T7), predecesor HARD al U3. Nu mai e deschis.
- **R2:** fișiere mari (mii rânduri) → upload sincron + `openpyxl read_only` streaming + cap hard înainte de
parsare (Eng#6). Async amânat până apare nevoia reală.
- **R4 (nou, blast-radius):** creds durabile-at-rest (D4) → la scurgerea cheii Fernet, toate parolele RAR sunt
decriptabile (vs. doar in-flight azi). Acceptat conștient; mitigare = rotație cheie + redactare loguri (existent).
- **R3:** coercion Excel nerecuperabilă la parsare → stare `needs_review` (acceptat D6).
- Un cont = un agent RAR sau mai mulți (afectează maparea creds în UI) — open question moștenit din plan.md.
- `b64Image` rămâne opțional, omis în upload v2.
## 10. Roadmap (reordonat — eng review: T7 înainte de U3)
- [ ] **U1** — `import_batches`/`import_rows`/`column_mappings` + parser xlsx/csv (`POST /v1/import`), cu
encoding RO + delimiter sniff + openpyxl read_only + dezambiguizare dată + robustețe sheet/merged/footer.
PII criptat în staging. (T3 + Voce#2/#6/#7 + Eng#3/#6)
- [ ] **U2** — creds RAR durabile per-cont pe web (`accounts.rar_creds_enc`, ALTER aditiv) + worker re-login
fallback + gate purjare `worker:271` (fix D4 + Voce#5). **Mutat înainte** — dependență hard end-to-end.
- [ ] **T7** — `batch_id`/`row_index` pe `submissions` + scope `reresolve_account` la seria comitată.
**Predecesor HARD al U3** — închide R1 (bulk-send cross-batch) ÎNAINTE ca save-mapare să ajungă live.
- [ ] **U3** — mapare coloane + reținere per cont + semnătură + detectie drift + auto-sugestie fuzzy (reuse
`normalize_for_match`, Eng#4). Nu se livrează până T7 nu e merged.
- [ ] **U4** — preview (6 stări: ok/needs_mapping/needs_data/needs_review/already_sent/duplicate_in_file) +
lookup `already_sent` batch (Eng#5) + canonicalize partajat (Eng#2) + gate HARD confirmare cu atestare pe
valori (Voce#1) + log atestare (Voce#9) + commit selectiv → coadă.
- [ ] **U5** — UI web upload (Jinja2+HTMX în dashboard): drop → mapează → preview → confirmă → trimite.
- [ ] **U6 (P2)** — export rânduri eșuate (CSV) pentru corecție + re-upload (acceptat D3; depinde de T7).
- [ ] ~~contor freemium~~ — DEFER (D6). ~~drop-fișier SFTP~~ — CUT (D6, re-eval post-validare).
## 11. Diagrame
### Arhitectură (componente noi vs existente)
```
[Fisier xlsx/csv] GATEWAY (existent, neatins sub linie)
│ upload ┌──────────────────────────────────────┐
▼ │ app/validation.py app/mapping.py │
POST /v1/import ──parse──▶ import_batches │ app/idempotency.py app/crypto.py │
(NOU) (cp1250, coercion) │ │ app/reconcile.py │
│ ▼ └──────────────┬───────────────────────┘
│ import_rows (PII cript) │ commit (rinduri ok)
▼ │ preview ▼
column_mappings (NOU) ──semnatura──▶ │ resolve submissions+batch_id ──▶ WORKER (existent)
(mapare retinuta+drift) │ (6 stari) queued (T7 scope) login RAR → postPrezentare
▲ ▼ └─▶ FINALIZATA (permanent)
accounts.rar_creds_enc (NOU, D4) ───creds durabile──▶ worker re-login fallback (fara re-pusher)
```
### Data flow + shadow paths (upload → commit)
```
FISIER ──▶ PARSE ──▶ MAP COLOANE ──▶ RESOLVE ──▶ CONFIRM ──▶ ENQUEUE
│ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼
[gol?] [coercion?] [drift?] [already_ [N gresit?] [dup key?
[non-xlsx?][encoding?] [nemapat?] sent?] reject → already_sent]
[>5MB?] [needs_ [auto_send [needs_ [creds? → D4]
review] gate] data?]
```
## 12. Implementation Tasks
Sintetizate din findings. P1 blochează ship; P2 = aceeași treaptă; P3 = follow-up.
- [ ] **T1 (P1, human ~1zi / CC ~30min)** — schema — coloană `accounts.rar_creds_enc` durabilă + worker re-login fallback + **gate purjare `worker:271`**
- Surfaced by: voce externă #1 (D4) + Voce#5 — `worker/__main__.py:271` purjază account-scoped, web n-are re-pusher
- Files: `app/schema.sql` (ALTER aditiv), `app/db.py:_migrate`, `app/worker/__main__.py`, `app/crypto.py`
- Verify: test — (a) serie web, worker restart, token expirat → re-login din `accounts` → trimite;
(b) **coadă MIXTĂ** API(efemer)+web(durabil) → după login web, submission-urile API tot se trimit
- [ ] **T2 (P1, human ~half zi / CC ~20min)** — preview — stare `already_sent` + lookup **batch IN(...)** (no N+1, no silent dedup)
- Surfaced by: voce externă #2 (D5) + Eng#5 — `idempotency.py:23`; 5000 randuri = N+1 dacă per-rând
- Files: `app/api/v1/` import preview, `app/idempotency.py`
- Verify: test — (a) re-upload cu typo odometru corectat → already_sent, nu al doilea FINALIZATA; (b) 5000 randuri → ≤7 interogări
- [ ] **T3 (P1, human ~half zi / CC ~20min)** — parse — coercion guard + stare `needs_review` (blochează auto-send) + **mesaj formule-None**
- Surfaced by: voce externă #8 (D6, R3) + **Eng pass 2 Issue 3** — `openpyxl data_only=True` întoarce `None` pe celule cu formule necalculate (export soft RO ≠ Excel) → indistinct de gol → cad în `needs_data` cu mesaj generic confuz pe un fișier care arată plin
- **Issue 3 (P2):** când o coloană obligatorie întoarce `None` pe o pondere mare de rânduri (euristică pe rata de None, fără `data_only=False`), emite mesaj țintit: „fișier cu formule fără valori salvate — deschide și re-salvează în Excel". Gate-ul `needs_data` prevenea deja trimiterea greșită; asta e doar claritate UX.
- **Ordonare critică (Eng#2/§3.4bis):** `canonicalize_row` rulează ÎNAINTE de `validate_prezentare` — `_parse_int` (`validation.py:44`, `isdigit()`) respinge `"123456.0"`; coercion-ul trebuie să taie `.0` înainte ca validarea să-l vadă (altfel `needs_data` în loc de `needs_review`).
- Files: parser import, preview resolve
- Verify: test — (a) VIN `0123…` numeric din xlsx → needs_review, nu se trimite; (b) xlsx cu coloană de formule fără cache → mesaj specific, nu `needs_data` generic; (c) odometru `123456.0` → canonicalizat la `123456` înainte de validare
- [ ] **T4 (P1, human ~half zi / CC ~20min)** — mapare — semnătură coloane + detectie drift
- Surfaced by: review D3
- Files: `column_mappings`, mapare coloane
- Verify: test — upload 2 cu coloane mutate → cere re-confirmare, nu aplică orb
- [ ] **T5 (P1, human ~half zi / CC ~25min)** — preview — gate HARD confirmare (tastezi N) + **atestare pe valori rezolvate** (Voce#1)
- Surfaced by: review D3 + Voce#1 — N dovedește totalul, bifa dovedește conținutul (VIN/dată/km finale)
- Files: UI preview, commit endpoint
- Verify: test — (a) commit fără N corect → reject; (b) rând `needs_review` nebifate → exclus din N, nu se trimite
- [ ] **T6 (P1, human ~half zi / CC ~15min)** — mapare — gate auto_send pe coduri nou-mapate (**NU e additiv — schimbă cod existent**)
- Surfaced by: review D3 + plan.md P2 + **Eng pass 2 OV-1** — `auto_send` e SCRIS (`save_mapping`) și afișat (`_mapari.html:49`) dar CITIT de niciun codepath; `reresolve_account` și bucla POST resolve trec pe `queued` ignorând flag-ul → AZI codurile nou-mapate se auto-trimit deja (bug latent Treapta 1)
- **OV-1 (P1):** T6 trebuie să MODIFICE `reresolve_account` ȘI resolve-ul POST/import să consulte `auto_send` (`auto_send=0` → stare ținută/`needs_review`), nu doar să adauge un gate nou.
- Files: `app/mapping.py` (reresolve_account), `app/api/v1/router.py` (POST resolve), commit
- Verify: test — (a) cod nou-mapat cu `auto_send=0` → nu auto-send, review manual; (b) **regresie:** mapare existentă cu `auto_send=1` tot se requeue ca azi
- [ ] **T7 (P1, human ~1zi / CC ~30min)** — **R1 ÎNCHIS** — `batch_id`/`row_index` pe submissions + scope `reresolve_account` (**predecesor HARD al U3**)
- Surfaced by: voce externă #3+#5 + Voce#10 — `mapping.py:253` account-global (PROMOVAT la P1-blocking la eng review)
- Files: `app/schema.sql`, `app/db.py:_migrate`, `app/mapping.py`, commit
- Verify: test — (a) salvare mapare în batch A NU trimite rânduri din batch B / feed API; (b) canal API (batch_id NULL) tot se re-rezolvă ca azi
- [ ] **T8 (P2, human ~half zi / CC ~15min)** — export rânduri eșuate CSV (depinde de T7 pt. trasabilitate)
- Surfaced by: review D3
- Files: import export endpoint
- Verify: descarci needs_data/needs_mapping ca CSV, corectezi, re-upload
- [ ] **T9 (P1, human ~half zi / CC ~20min)** — idempotency — `canonicalize_row` + `build_key` partajat (parser + POST), DRY + **normalizare account_id**
- Surfaced by: Eng#2 + **Eng pass 2 OV-2** — coercion înainte de cheie → divergență `already_sent`; `idempotency.py:23` hash-uiește `account_id` AS-PASSED (`None` pe canal API, `router.py:66`) dar rândurile se stochează sub `account_or_default`=1 → același rând logic capătă cheie diferită cross-canal → `already_sent` ratează → al doilea `FINALIZATA`
- **OV-2 (P1):** `canonicalize_row`/`build_key` aplică `account_or_default` ÎNAINTE de hash (None și 1 colapsează la o cheie). Tensiune cu §3.4bis „cheie identică": rândurile vechi cheie-`None` trebuie reconciliate (recompute o-singură-dată SAU dual-lookup), documentat explicit.
- Files: `app/idempotency.py`, parser import, `app/api/v1/router.py`
- Verify: test — (a) **cross-canal:** cheie(API canal-None) == cheie(import canal-rezolvat) pt. același rând logic; (b) regresie: strategia de reconciliere a cheilor vechi acoperită de test (fără re-trimitere tăcută)
- [ ] **T10 (P1, human ~half zi / CC ~20min)** — parse — dezambiguizare dată **la nivel de coloană** (datetime nativ direct; string ambiguu → needs_review)
- Surfaced by: Voce#2 + **Eng pass 2 OV-8** — `validation.py:81` (`date.fromisoformat`) acceptă orice ISO valid în interval → un DD/MM swap valid-dar-greșit trece. `zi≤12` per-rând ratează coloana uniform MM.DD (rândurile `zi>12` par neambigue și trec ca `ok`)
- **OV-8 (P3):** detectează formatul din ÎNTREAGA coloană — dacă ORICE rând are token poziția-1 `>12`, coloana e DD-first; aplică formatul la toate rândurile, nu doar flag per-rând `zi≤12`.
- Files: parser import, preview resolve
- Verify: test — (a) `03.04.2026` string ambiguu → needs_review; (b) celulă datetime nativă → folosită direct; (c) coloană uniform MM.DD cu rânduri `zi>12` → format detectat la nivel de coloană, nu trec orb ca `ok`
- [ ] **T11 (P1, human ~half zi / CC ~20min)** — preview — detecție coliziuni intra-batch (DOAR la preview/commit, NU în worker)
- Surfaced by: Voce#3+#4 + **Eng pass 2 OV-3** — `UNIQUE` global înghite dup intra-fișier; `reconcile.py` e op-blind PRIN DESIGN (recuperare răspuns pierdut, worker:184/217)
- **OV-3 (P1):** detecția coliziunilor intra-fișier trăiește EXCLUSIV la preview/commit (`duplicate_in_file`). NU edita `reconcile.py` / `worker/__main__.py` — a face reconcile op-aware regresează T2 (recuperarea POST-ului pierdut pe timeout legitim). Intra-file dedup (preview-time) ≠ reconcile stare-RAR (worker-time): probleme diferite.
- Files: preview resolve (NU reconcile.py, NU worker)
- Verify: test — (a) 2 rânduri identice în fișier → `duplicate_in_file`; (b) batch cu vin+data+odo colidant → flag la preview, cere manual; (c) **regresie T2:** `match_finalizata` rămâne op-blind, recuperarea răspuns-pierdut neschimbată
- [ ] **T12 (P2, human ~half zi / CC ~15min)** — commit — log `import_attestations` + UI „ești declarantul" + **commit per-rând ON CONFLICT (TOCTOU)**
- Surfaced by: Voce#9 + **Eng pass 2 Issue 1** — `already_sent` la preview e un snapshot; gardianul real e indexul UNIQUE la INSERT, minute mai târziu. Un canal concurent (API live / al 2-lea import) poate insera cheia colidant în fereastra preview→commit → un INSERT multi-rând într-o tranzacție rollback-uiește TOT batch-ul (`router.py:100` e INSERT simplu, nu OR IGNORE) → utilizatorul a tastat N, confirmat, primește eroare opacă, iar `rows_hash`(N) nu mai corespunde cu ce s-a inserat.
- **Issue 1 (P1):** commit inserează per-rând cu `INSERT … ON CONFLICT(idempotency_key) DO NOTHING`; rândurile care colidează se reclasifică `already_sent` în rezultatul commit-ului; `import_attestations.rows_hash` + `n_confirmed` acoperă DOAR rândurile efectiv puse în coadă (nu N inițial). Respectă principiul planului „niciodată dedup tăcut".
- Files: `app/schema.sql`, commit endpoint, `app/api/v1/router.py` (audit export), UI preview
- Verify: test — (a) commit → rând `import_attestations` cu rows_hash + n_confirmed; apare în `/v1/audit/export`; (b) **TOCTOU:** cheie inserată de canal concurent după preview → rând reclasificat `already_sent`, atestarea acoperă doar rândurile puse în coadă
- [ ] **T13 (P2, human ~1zi / CC ~25min)** — parse — robustețe export RO (multi-sheet + merged header + trim footer)
- Surfaced by: Voce#6+#7 — sheet 2 / celule îmbinate → HeaderError pe fișier valid; footer TOTAL → prestatie fantomă
- Files: parser import
- Verify: test — (a) workbook 2 sheets → cere alegerea; (b) header merged → nume reale; (c) footer fără VIN → skip, nu needs_data
- [ ] **T14 (P2, human ~half zi / CC ~15min)** — perf — CSV delimiter sniff + openpyxl `read_only` streaming + cap înainte de parse
- Surfaced by: Eng#3+#6 — `;` RO dă 1 coloană tăcut; DOM întreg = vârf memorie
- Files: parser import
- Verify: test — (a) CSV `;` → coloane corecte; 1 coloană → HeaderError; (b) >5000 rânduri → FileTooLarge fără parse parțial
- [ ] **T15 (P2, human ~half zi / CC ~20min)** — test — **E2E integrare** import→commit→worker (RAR mock)
- Surfaced by: Test review — mock-urile per-unit ascund cheia idempotency + re-login + batch scoping
- Files: `tests/test_import_e2e.py`
- Verify: upload fixture → mapează → preview → commit N → worker run_once(MockRar) → FINALIZATA; re-upload corectat → already_sent
- [ ] **T16 (P1, human ~2h / CC ~20min)** — retenție — job purjare + `purge_after` SET la insert (ambele canale)
- Surfaced by: **Eng pass 2 OV-5** — `purge_after` e exportat în audit dar SETAT de niciun INSERT și NICIUN job de purjare nu există (`grep purge_after` → doar SELECT). Planul presupunea paritate `submissions`/`import_rows` care nu există → PII criptat (Fernet) trăiește la nesfârșit. Decalaj GDPR/L.142.
- Files: `app/worker/__main__.py` (tick purjare), commit/insert (`submissions` + `import_batches`/`import_rows`)
- Verify: test — (a) insert → `purge_after` populat (sent+90z); (b) rând expirat → șters de tick-ul de purjare; (c) `import_rows` purjate cu batch-ul
### 12bis. Eng Review Pass 2 — sinteză (2026-06-16)
A doua trecere `/plan-eng-review` pe planul deja CLEARED: 6 findings noi (Claude) + 5 din vocea externă (subagent — Codex la cuotă), TOATE acceptate cu opțiunea completă (Lake 11/11). Detalii foldate în taskuri:
- **Issue 1 (P1, T12):** commit TOCTOU → per-rând `ON CONFLICT DO NOTHING`, atestare doar pe rândurile puse în coadă.
- **Issue 2 (P2, T13/T14) = vocea externă OV-6 (consens cross-model):** `openpyxl read_only=True` nu vede celule îmbinate → parser în 2 treceri (read_only dim-check + body; normal-mode header+merged DUPĂ cap-check).
- **Issue 3 (P2, T3):** `data_only=True` → `None` pe formule necalculate → mesaj specific (euristică rată-None).
- **Issue 4 (P3, U1):** `openpyxl` lipsește din `requirements.txt` → adaugă PINNED (ex. `openpyxl==3.1.x`) explicit în U1.
- **Issue 5 (P2, U1/U3):** teste explicite — (a) `import_rows.raw_json` criptat la rest (ciphertext pe disc, plaintext după decrypt); (b) fuzzy coloane refolosește `mapping.normalize_for_match` (fără normalizator duplicat).
- **Issue 6 (P2, U1/U4):** scrieri bulk sub autocommit (`db.py:17` `isolation_level=None`) → `BEGIN IMMEDIATE`…`COMMIT` + `executemany` (model `claim_one`); 5000 fsync → 1.
- **OV-1 (P1, T6):** `auto_send` coloană moartă (citită nicăieri) → T6 modifică `reresolve_account` + resolve POST, nu doar adaugă.
- **OV-2 (P1, T9):** skew `account_id` la hash → normalizare `account_or_default` în `canonicalize_row` + test cross-canal.
- **OV-3 (P1, T11):** intra-file dedup DOAR la preview/commit; NU atinge `reconcile.py`/worker (op-blind by design, T2).
- **OV-5 (P2, T16):** job purjare + `purge_after` la insert (nou T16, mai sus).
- **OV-8 (P3, T10):** dezambiguizare dată la nivel de coloană, nu per-rând `zi≤12`.
- **NOTE U1:** parserul = 2 treceri (Issue 2); adaugă `openpyxl` pinned (Issue 4); test PII-at-rest (Issue 5a); scrieri staging în tranzacție explicită + `executemany` (Issue 6). **NOTE U3:** test reuse `normalize_for_match` (Issue 5b). **NOTE U4:** enqueue în tranzacție explicită (Issue 6).
- **Constrângere asset offline (learning):** UI upload (U5) NU introduce assets din CDN — gateway rulează offline; refolosește htmx vendorizat local (`app/web/static/`).
- **Ordine livrare actualizată:** U1 → U2/T1 → T7 → U3 → U4 → U5; **T16 (purjare) poate merge în paralel** (independent de T7).
## 13. Design spec UI (post `/plan-design-review`)
> Clasificare: **APP UI** (tool intern, data-dense). Extinde sistemul existent din
> `app/web/templates/base.html` (`:root` tokens) — NU introduce limbaj nou. Refolosește:
> `--ok/--warn/--err/--accent`, `.card`, pills `.s-*`, `.maprow`, `.tablewrap`, empty states.
### 13.1 Information architecture (Pass 1)
```
DASHBOARD (existent)
├─ card UPLOAD (NOU, sus; primar/CTA cand coada e goala)
│ „Incarca fisier (xlsx/csv)" → drop zone + buton
├─ [dupa upload] → ecran/sectiune MAPARE
│ 1. mapare COLOANE (.maprow: camp canonic ← dropdown coloane)
│ 2. mapare OPERATII (editorul fuzzy existent)
├─ [dupa mapare] → PREVIEW (tabel dominant, 5 stari)
│ rezumat pills sus + filtru + bara confirmare jos
└─ coada submissions (existent, neatins)
```
Ierarhie preview: 1) rezumat stari (ce e gata/cu probleme), 2) tabelul, 3) bara de trimitere.
### 13.2 Tabel stari interacțiune (Pass 2)
```
ECRAN | LOADING | EMPTY | ERROR | SUCCESS | PARTIAL
---------------|--------------------|-----------------------|--------------------|--------------------|--------------------
Upload | spinner „se incarca| drop zone + „trage | „fisier invalid | → trece la mapare | n/a
| / se parseaza…" | fisierul aici" + CTA | (xlsx/csv)" rosu | |
Mapare coloane | — | „nicio coloana | dropdown rosu pe | toate verzi → next | unele campuri
| | detectata" | camp obligatoriu | | nemapate (galben)
Preview | „se valideaza N | „0 randuri in fisier" | rand rosu + motiv | „N gata de trimis" | rezumat: X ok,
| randuri…" | | per rand | verde | Y probleme
Trimitere | bara progres N/M | n/a | rand → error in | „N trimise" flash | „M din N trimise,
| | | coada | (.flash existent) | restul in coada"
```
Empty state upload = feature: warmth („Primul fisier? Trage-l aici.") + CTA + context („xlsx sau csv, max 5000 randuri").
### 13.3 User journey / arc emoțional (Pass 3)
```
PAS | USER FACE | USER SIMTE | UI SUSTINE
----|--------------------|-------------------------|---------------------------------
1 | incarca fisier | nesiguranta („trimite | mesaj clar „NU se trimite nimic
| | acum la RAR?") | pana confirmi" sub drop zone
2 | mapeaza coloane | efort prima data | auto-sugestie fuzzy pre-selectata;
| | | a 2-a oara: „mapare retinuta aplicata"
3 | vede preview | control / verificare | rezumat pills + problemele primele
4 | confirma (tastezi N| frica de greseala | gate explicit; „needs_review" galben
| | permanenta | blocheaza VIN suspect
5 | vede „N trimise" | usurare / incredere | .flash verde + rand sent in coada
```
5s: „inteleg ca nu trimite nimic inca". 5min: „maparea s-a retinut". 5 luni: „drop + trimite, sub 1 min".
### 13.4 AI slop (Pass 4) — 8/10
APP UI, refolosește sistemul calm existent. Fără card-mosaic decorativ, fără gradients, fără
3-column grid, fără border-left colorat ornamental. Pills semantice = funcționale, nu decor. OK.
### 13.5 Design system (Pass 5)
- Stări rând: refolosește `.s-queued/.s-sent/.s-error`; adaugă `.s-needs_review` (galben `--warn`),
`.s-already_sent` (muted), **`.s-duplicate_in_file` (muted, D10)**. Pills numerice rezumat = aceleași culori.
- **Semantica culorii (D10, post eng review):** amber `--warn` = „verifică valori" (DOAR needs_review);
muted `--muted` = „informațional / decizie per-rând" (already_sent + duplicate_in_file); roșu `--err` = blocat;
verde `--ok` = ok; albastru `--accent` = în coadă. `duplicate_in_file` diferențiat de already_sent prin TEXT
explicit + referință încrucișată („dublă cu rândul 88"), nu doar culoare (daltonism — pill poartă cuvântul).
- Mapare coloane = `.maprow` + `.mapcol.grow` + `select` (exact ca `_mapari.html`).
- Drop zone: `.card` cu bordura `--line` dashed la hover; fără estetică nouă.
- Bara confirmare: `.card` fix jos, buton `--accent` existent, `input[type=text]` pentru N.
- **Checkbox atestare (D11):** rândurile `needs_review` au `.chk` existent **per-rând** („verificat") — trebuie
bifate ca să intre în N (forțează privirea pe fiecare valoare rezolvată). `<label>` vizibil + focus tastatură.
- **Banner declarant (D12):** variantă `.banner` cu `--warn` (avertisment, nu eroare roșie), plasat DIRECT
deasupra input-ului N: „Confirmând, TU ești declarantul acestor N prezentări la RAR (ireversibil)". Inevitabil
la momentul confirmării (Krug). Anunțat la screen-reader.
### 13.6 Responsive & a11y (Pass 6)
- Mobil: `.maprow` deja se rupe (`flex-wrap`); tabel preview în `.tablewrap` (scroll în card, existent).
Bara de confirmare devine sticky bottom, nu fixed-overlap.
- Touch ≥44px (deja `.cardlink` min-height 36px → ridică la 44 pe butoanele de acțiune upload/confirm).
- a11y: drop zone are și buton (nu doar drag — drag nu e accesibil la tastatură); dropdown-urile de
mapare au `<label>` vizibil (nu placeholder-as-label); stările au și text, nu doar culoare
(pill cu cuvânt, nu doar pastilă colorată — daltonism); contrast ≥4.5:1 (tokenii existenți trec).
- Gate confirmare accesibil: input N cu label, eroare anunțată, focus pe el la deschidere.
### 13.7 Decizii de design (Pass 7, rezolvate)
- Mapare coloane = listă `.maprow` cu dropdown + eșantion (D8-A). Nu wizard, nu dropdown-pe-antet.
- Preview = rezumat pills + filtru pe stare + problemele primele + bară confirmare cu tastare N (D9-A).
- **D10 (post eng review):** `.s-duplicate_in_file` = muted + text „dublă cu rândul N" (grupat cu already_sent;
amberul rămâne doar „verifică valori"). Fără culoare nouă — disciplină de sistem.
- **D11:** atestare = `.chk` per-rând pe `needs_review`, obligatoriu pentru includere în N (atestare pe valori).
- **D12:** banner declarant = `.banner` `--warn` direct deasupra input-ului N (răspundere legală inevitabilă).
## GSTACK REVIEW REPORT
| Review | Trigger | Why | Runs | Status | Findings |
|--------|---------|-----|------|--------|----------|
| CEO Review | `/plan-ceo-review` | Scope & strategy | 1 | issues_open | SELECTIVE EXPANSION: 6 propuneri, 5 acceptate, 2 deferate/taiate |
| Eng Review | `/plan-eng-review` | Architecture & tests (required) | 2 | clean | Pass 1: 12 findings foldate (R1 INCHIS, T7 P1). Pass 2: 11 findings noi (1 arh + 2 cod + 1 perf + 5 voce externa + 2 test), TOATE acceptate cu optiunea completa (Lake 11/11) |
| Outside Voice | subagent Claude (Codex la cuota) | Independent 2nd opinion | 2 | issues_found | Pass 1: 10 findings. Pass 2: 5 noi (OV-1 auto_send mort, OV-2 skew account_id, OV-3 reconcile op-blind, OV-5 fara job purjare, OV-8 data col-level), TOATE absorbite |
| Design Review | `/plan-design-review` | UI/UX gaps | 2 | clean | full 4→9 (9 decizii) + delta 5→9 pe 3 stari UI noi post-eng (D10/D11/D12) |
- **CROSS-MODEL:** Pass 2 — Issue 2 (Claude) == OV-6 (voce externa) au ajuns INDEPENDENT la aceeasi concluzie (`openpyxl read_only` nu vede merged cells → parser 2-treceri): consens, confidenta ridicata. Fara tensiune Claude-vs-voce-externa in pass 2. (Pass 1: vocea externa contrazisese design D9-A + CEO D6, rezolvate.)
- **PASS 2 — corectitudine pre-existenta:** OV-1/OV-2/OV-3/OV-5 sunt locuri unde planul presupunea o schimbare aditiva dar codul EXISTENT contrazicea presupunerea: `auto_send` coloana moarta (citita nicaieri), skew `account_id` la hash (None vs rezolvat), `reconcile` op-blind by design (T11 nu trebuie sa-l atinga), `purge_after` setat de nimeni + zero job purjare (PII nelimitat). Toate foldate in T6/T9/T11/T16.
- **VERDICT:** CEO + DESIGN + ENG (×2) CLEARED — gata de implementare. R1 INCHIS (T7 P1, predecesor HARD U3). Niciun critical gap silent ramas. Ordine livrare: U1 → U2/T1 → T7 → U3 → U4 → U5; T16 (purjare) in paralel.
NO UNRESOLVED DECISIONS

31
legacy-vfp/README.md Normal file
View File

@@ -0,0 +1,31 @@
# legacy-vfp/ — arhiva Visual FoxPro (ROAAUTO)
Proiectul **Visual FoxPro** original (clasa `RarAutoPass`, ROAAUTO) care declara
prestatiile la RAR AUTOPASS. A fost folosit doar pentru testare si proba pe endpoint-ul
de test RAR; **nu se mai dezvolta** — gateway-ul Python din `../app/` il inlocuieste.
Pastrat ca **sursa de adevar de contract** pentru portarea web si ca sursa pentru migrarea
DBF.
## Continut
| Fisier | Rol | Portat in |
|--------|-----|-----------|
| `rar_autopass.prg` | clasa `RarAutoPass`: login+JWT, nomenclator, postPrezentare, cancel | `app/rar_client.py` |
| `rar-forms.prg` | UI + timer auto-process | logica -> worker |
| `export_comenzi.prg` | citeste comenzi/operatii, construieste payload | client subtire `POST /v1/prezentari` |
| `rar_advanced.prg` | export Excel (oglinda treapta 2) | referinta import xlsx/csv |
| `rar_automate.prg`, `nfjsonread.prg` | automatizare / parse JSON | referinta |
| `rar_autopass.pjx` / `.PJT` | fisierele de proiect VFP | — |
| `mapare_prestatii.DBF` (+ `.CDX`/`.FPT`) | cod_op_service -> codPrestatie | `operations_mapping` (via `tools/import_dbf.py`) |
| `prestatii_rar.DBF` (+ `.CDX`) | nomenclator {codPrestatie, numePrestatie} | `nomenclator_rar` (via `tools/import_dbf.py`) |
| `test-comenzi.xml` | comenzi de test (proba) | — |
## Migrare DBF -> SQLite
Datele din DBF se importa cu `tools/import_dbf.py` (caile default pointeaza deja aici):
```bash
python3 -m tools.import_dbf # dry-run + raport
python3 -m tools.import_dbf --commit # scrie in SQLite
```

View File

@@ -12,5 +12,8 @@ rapidfuzz==3.14.5
# Criptare creds RAR efemere in submissions (app/crypto.py, Fernet). Zero-storage at rest.
cryptography==46.0.5
# Parsare xlsx/xls pentru import fisiere (Treapta 2, Issue 4 — PINNED).
openpyxl==3.1.5
# Migrare DBF (tools/import_dbf.py). Necesar doar pentru import optional, nu pentru runtime.
dbfread==2.0.7

20
start-prod.sh Executable file
View File

@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# start-prod.sh — lanseaza start.sh pe mediul PROD. Forwardeaza rol + optiuni.
#
# ./start-prod.sh api # API prod
# ./start-prod.sh worker --send # worker prod (TRIMITE la RAR productie)
# ./start-prod.sh both --send # API + worker
# ./start-prod.sh status | stop
#
# In productie trimiterea trebuie ceruta EXPLICIT cu --send, ca sa nu trimiti din
# greseala. De aceea fara argumente nu pornim nimic. Recomandat: docker compose.
set -euo pipefail
cd "$(dirname "$0")"
if [ $# -eq 0 ]; then
echo "Specifica rolul: api | worker | both (adauga --send pentru trimitere la RAR)." >&2
echo "Ex: ./start-prod.sh both --send | vezi ./start.sh --help" >&2
exit 1
fi
exec ./start.sh prod "$@"

18
start-test.sh Executable file
View File

@@ -0,0 +1,18 @@
#!/usr/bin/env bash
# start-test.sh — lanseaza start.sh pe mediul TEST. Forwardeaza rol + optiuni.
#
# ./start-test.sh # API + worker cu trimitere la RAR test (both --send)
# ./start-test.sh api # doar API
# ./start-test.sh worker --send # doar worker (trimite la RAR test)
# ./start-test.sh finalizate # ce prezentari sunt inregistrate la RAR test
# ./start-test.sh status | stop
#
# Pe test trimiterea e sigura (sandbox RAR), deci fara argumente pornim end-to-end.
set -euo pipefail
cd "$(dirname "$0")"
if [ $# -eq 0 ]; then
exec ./start.sh test both --send
fi
exec ./start.sh test "$@"

165
start.sh Executable file
View File

@@ -0,0 +1,165 @@
#!/usr/bin/env bash
# start.sh — pornire gateway RAR AUTOPASS (api / worker) pe mediu test sau prod.
#
# Exemple:
# ./start.sh test api # API pe :8000, mediu test
# ./start.sh test worker --send # worker care TRIMITE la RAR test (creds <test> din settings.xml)
# ./start.sh test both --send # API + worker impreuna (dev end-to-end)
# ./start.sh prod api --port 8000 # API mediu prod
# ./start.sh prod worker --send # worker prod (NU foloseste creds de test)
# ./start.sh status # stare procese + /healthz
# ./start.sh stop # opreste procesele pornite cu "both"
# ./start.sh test finalizate # ce prezentari sunt inregistrate la RAR (au ajuns?)
#
# Pentru productie reala se recomanda docker compose (vezi docker-compose.yml).
# start.sh e pentru rulare directa pe VPS/LXC sau dezvoltare locala.
set -euo pipefail
cd "$(dirname "$0")"
# --- valori implicite ---
PORT=8000
HOST=0.0.0.0
RELOAD=0
SEND=0
USE_TEST_CREDS="auto" # auto: pornit doar pe mediul test cand --send e activ
RUN_DIR=".run"
PY=python3
usage() {
sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'
exit "${1:-0}"
}
# --- citeste .env daca exista (AUTOPASS_CREDS_KEY etc.) ---
if [ -f .env ]; then
set -a; . ./.env; set +a
fi
# --- parsare argumente: env si rol = pozitionale; restul = flag-uri ---
ENVIRONMENT=""
ROLE=""
POSITIONAL=()
while [ $# -gt 0 ]; do
case "$1" in
--port) PORT="$2"; shift 2 ;;
--host) HOST="$2"; shift 2 ;;
--reload) RELOAD=1; shift ;;
--send) SEND=1; shift ;;
--test-creds) USE_TEST_CREDS="true"; shift ;;
--no-test-creds) USE_TEST_CREDS="false"; shift ;;
-h|--help) usage 0 ;;
-*) echo "Optiune necunoscuta: $1" >&2; usage 1 ;;
*) POSITIONAL+=("$1"); shift ;;
esac
done
# Comenzi fara mediu (stop/status accepta lipsa mediului).
case "${POSITIONAL[0]:-}" in
stop|status)
ROLE="${POSITIONAL[0]}" ;;
test|prod)
ENVIRONMENT="${POSITIONAL[0]}"
ROLE="${POSITIONAL[1]:-}" ;;
"")
usage 1 ;;
*)
echo "Mediu invalid: ${POSITIONAL[0]} (asteptat: test | prod | stop | status)" >&2
usage 1 ;;
esac
mkdir -p "$RUN_DIR"
# --- helperi ---
ensure_creds_key() {
# API cripteaza creds RAR, worker le decripteaza: trebuie ACEEASI cheie.
# Daca ruleaza ambele din aceeasi invocare (both), o cheie efemera exportata
# acum e partajata de copii. Daca lipseste si rulezi separat, avertizeaza.
if [ -z "${AUTOPASS_CREDS_KEY:-}" ]; then
if [ "$ROLE" = "both" ]; then
AUTOPASS_CREDS_KEY="$($PY -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')"
export AUTOPASS_CREDS_KEY
echo "[start] AUTOPASS_CREDS_KEY negasita -> cheie efemera generata pentru aceasta rulare."
echo "[start] (Nu supravietuieste restartului. Pune o cheie persistenta in .env pentru prod.)"
else
echo "[start] ATENTIE: AUTOPASS_CREDS_KEY nesetata. API si worker pornite SEPARAT vor folosi"
echo "[start] chei diferite -> worker-ul NU poate decripta creds-urile. Seteaz-o in .env."
fi
fi
}
export_common() {
export AUTOPASS_RAR_ENV="$ENVIRONMENT"
# use_test_creds: auto -> true doar pe test cand trimitem
local utc="$USE_TEST_CREDS"
if [ "$utc" = "auto" ]; then
if [ "$ENVIRONMENT" = "test" ] && [ "$SEND" -eq 1 ]; then utc="true"; else utc="false"; fi
fi
export AUTOPASS_WORKER_USE_TEST_CREDS="$utc"
if [ "$SEND" -eq 1 ]; then
export AUTOPASS_WORKER_SEND_ENABLED="true"
else
export AUTOPASS_WORKER_SEND_ENABLED="${AUTOPASS_WORKER_SEND_ENABLED:-false}"
fi
}
start_api() {
local args=(uvicorn app.main:app --host "$HOST" --port "$PORT")
[ "$RELOAD" -eq 1 ] && args+=(--reload)
echo "[start] API ($ENVIRONMENT) -> http://$HOST:$PORT (docs: /docs, dashboard: /)"
exec "$PY" -m "${args[@]}"
}
start_worker() {
echo "[start] worker ($ENVIRONMENT) send_enabled=$AUTOPASS_WORKER_SEND_ENABLED use_test_creds=$AUTOPASS_WORKER_USE_TEST_CREDS"
[ "$AUTOPASS_WORKER_SEND_ENABLED" = "true" ] || \
echo "[start] NOTA: send dezactivat — worker proceseaza coada dar NU trimite la RAR. Adauga --send."
exec "$PY" -m app.worker
}
start_both() {
local alog="$RUN_DIR/api.log" wlog="$RUN_DIR/worker.log"
echo "[start] API + worker ($ENVIRONMENT). Loguri: $alog , $wlog. Ctrl-C opreste ambele."
"$PY" -m uvicorn app.main:app --host "$HOST" --port "$PORT" >"$alog" 2>&1 &
echo $! > "$RUN_DIR/api.pid"
"$PY" -m app.worker >"$wlog" 2>&1 &
echo $! > "$RUN_DIR/worker.pid"
trap 'echo; echo "[start] opresc..."; kill $(cat "$RUN_DIR"/*.pid 2>/dev/null) 2>/dev/null || true; exit 0' INT TERM
echo "[start] API pid $(cat "$RUN_DIR/api.pid"), worker pid $(cat "$RUN_DIR/worker.pid")"
tail -n +1 -f "$alog" "$wlog"
}
cmd_stop() {
local killed=0
for f in "$RUN_DIR"/api.pid "$RUN_DIR"/worker.pid; do
[ -f "$f" ] || continue
local pid; pid="$(cat "$f")"
if kill "$pid" 2>/dev/null; then echo "[stop] oprit pid $pid ($(basename "$f"))"; killed=1; fi
rm -f "$f"
done
[ "$killed" -eq 1 ] || echo "[stop] niciun proces urmarit (.run/*.pid)."
}
cmd_status() {
for f in "$RUN_DIR"/api.pid "$RUN_DIR"/worker.pid; do
[ -f "$f" ] || continue
local pid; pid="$(cat "$f")"
if kill -0 "$pid" 2>/dev/null; then echo "[status] $(basename "$f" .pid): pid $pid VIU"
else echo "[status] $(basename "$f" .pid): pid $pid MORT"; fi
done
echo "[status] /healthz:"
curl -s "http://localhost:$PORT/healthz" 2>/dev/null | "$PY" -m json.tool 2>/dev/null \
|| echo " (API nu raspunde pe :$PORT)"
}
# --- dispatch ---
case "$ROLE" in
api) ensure_creds_key; export_common; start_api ;;
worker) ensure_creds_key; export_common; start_worker ;;
both) ensure_creds_key; export_common; start_both ;;
finalizate) export AUTOPASS_RAR_ENV="$ENVIRONMENT"; exec "$PY" -m tools.rar_finalizate ;;
stop) cmd_stop ;;
status) cmd_status ;;
"") echo "Lipseste rolul (api|worker|both|finalizate)" >&2; usage 1 ;;
*) echo "Rol invalid: $ROLE" >&2; usage 1 ;;
esac

186
tests/test_canonicalize.py Normal file
View File

@@ -0,0 +1,186 @@
"""Teste T9: canonicalize_row + build_key partajat (idempotency).
Verify:
(a) cross-canal: build_key(API canal-None) == build_key(import canal-rezolvat) pentru
acelasi rand logic.
(b) regresie: strategia cheilor vechi (dual-lookup legacy) acoperita de test.
(c) canonicalize taie ".0" din odometru inainte de validare.
"""
from __future__ import annotations
import os
import tempfile
import pytest
from app.idempotency import (
build_key,
build_key_legacy,
canonicalize_row,
idempotency_key,
)
# --- canonicalize_row ---
def test_canonicalize_vin_upper():
raw = {"vin": "wvwzzz1kzaw000123", "nr_inmatriculare": "b999tst",
"data_prestatie": "2026-06-15", "odometru_final": "123456"}
c = canonicalize_row(raw)
assert c["vin"] == "WVWZZZ1KZAW000123"
assert c["nr_inmatriculare"] == "B999TST"
def test_canonicalize_odometru_strip_dot_zero():
"""123456.0 (Excel float) -> '123456'."""
raw = {"vin": "X", "nr_inmatriculare": "Y", "data_prestatie": "2026-01-01",
"odometru_final": "123456.0"}
c = canonicalize_row(raw)
assert c["odometru_final"] == "123456"
def test_canonicalize_odometru_numeric_float():
"""Numeric float 123456.0 -> '123456'."""
raw = {"vin": "X", "nr_inmatriculare": "Y", "data_prestatie": "2026-01-01",
"odometru_final": 123456.0}
c = canonicalize_row(raw)
assert c["odometru_final"] == "123456"
def test_canonicalize_odometru_int_unchanged():
"""Integer 123456 -> '123456' (nu e alterat)."""
raw = {"vin": "X", "nr_inmatriculare": "Y", "data_prestatie": "2026-01-01",
"odometru_final": 123456}
c = canonicalize_row(raw)
assert c["odometru_final"] == "123456"
def test_canonicalize_odometru_50_unchanged():
"""'123456.50' nu e coercion pur — nu se taie."""
raw = {"vin": "X", "nr_inmatriculare": "Y", "data_prestatie": "2026-01-01",
"odometru_final": "123456.50"}
c = canonicalize_row(raw)
assert c["odometru_final"] == "123456.50"
def test_canonicalize_odometru_none():
raw = {"vin": "X", "nr_inmatriculare": "Y", "data_prestatie": "2026-01-01"}
c = canonicalize_row(raw)
assert c["odometru_final"] == ""
def test_canonicalize_data_strip():
raw = {"vin": "X", "nr_inmatriculare": "Y", "data_prestatie": " 2026-06-15 ",
"odometru_final": "1"}
c = canonicalize_row(raw)
assert c["data_prestatie"] == "2026-06-15"
# --- build_key cross-canal (a) ---
_RAND = {
"vin": "WVWZZZ1KZAW000123",
"nr_inmatriculare": "B999TST",
"data_prestatie": "2026-06-15",
"odometru_final": "123456",
"prestatii": [{"cod_prestatie": "OE-1"}],
}
def test_cross_canal_none_equals_1():
"""(a) build_key cu account_id=None si account_id=1 dau aceeasi cheie."""
canon = canonicalize_row(_RAND)
k_none = build_key(None, canon)
k_1 = build_key(1, canon)
assert k_none == k_1, "cross-canal divergenta: None vs 1"
def test_cross_canal_odometru_float():
"""Odometru float din Excel: cheia e identica indiferent de canal."""
rand_float = {**_RAND, "odometru_final": "123456.0"}
rand_int = {**_RAND, "odometru_final": "123456"}
k_float_api = build_key(None, canonicalize_row(rand_float))
k_int_import = build_key(1, canonicalize_row(rand_int))
assert k_float_api == k_int_import, "float vs int odometru -> chei diferite"
# --- idempotency_key wrapper ---
def test_idempotency_key_backward_compat():
"""idempotency_key(None, raw) produce aceeasi cheie ca build_key(None, canon)."""
canon = canonicalize_row(_RAND)
k_new = build_key(None, canon)
k_old = idempotency_key(None, _RAND)
assert k_new == k_old
# --- build_key_legacy (b) ---
def test_legacy_key_differs_from_new():
"""(b) Cheia legacy (account_id=None in hash) difera de cheia noua (account_id=1)."""
canon = canonicalize_row(_RAND)
k_new = build_key(None, canon) # None -> 1 in hash
k_legacy = build_key_legacy(None, _RAND) # None AS-PASSED in hash
assert k_new != k_legacy, "legacy si new trebuie sa difere (diferit account_id in hash)"
def test_legacy_dual_lookup_strategy():
"""Strategia dual-lookup: row-uri vechi (cheie-None) gasite via build_key_legacy."""
# Simuleaza un rand cu cheie veche (account_id=None in hash)
old_key = build_key_legacy(None, _RAND)
# Noul build_key (None->1) NU gaseste randul direct
new_key = build_key(None, canonicalize_row(_RAND))
assert new_key != old_key
# Dual-lookup: incearca noul, apoi legacy
found = old_key in {old_key} or new_key in {old_key}
assert found, "dual-lookup trebuie sa gaseasca randul vechi"
# --- Integrare: API route foloseste build_key (OV-2) ---
@pytest.fixture()
def client(monkeypatch):
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "t9.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.main import app
from fastapi.testclient import TestClient
with TestClient(app) as c:
yield c
get_settings.cache_clear()
def _body(**over):
prez = {
"vin": "WVWZZZ1KZAW000123",
"nr_inmatriculare": "B999TST",
"data_prestatie": "2026-06-15",
"odometru_final": "123456",
"prestatii": [{"cod_prestatie": "OE-1"}],
}
prez.update(over)
return {"rar_credentials": {"email": "x@y.ro", "password": "s"}, "prezentari": [prez]}
def test_api_dedup_dupa_t9(client):
"""Deduplicarea functioneaza dupa T9: acelasi rand -> acelasi submission."""
r1 = client.post("/v1/prezentari", json=_body())
r2 = client.post("/v1/prezentari", json=_body())
assert r1.status_code == 200
sid1 = r1.json()["results"][0]["submission_id"]
res2 = r2.json()["results"][0]
assert res2["submission_id"] == sid1
assert res2["deduped"] is True
def test_api_odometru_float_dedup(client):
"""Odometru float '123456.0' si '123456' dedup corect dupa canonicalizare."""
r1 = client.post("/v1/prezentari", json=_body(odometru_final="123456"))
r2 = client.post("/v1/prezentari", json=_body(odometru_final="123456.0"))
assert r1.status_code == 200
sid1 = r1.json()["results"][0]["submission_id"]
res2 = r2.json()["results"][0]
assert res2["submission_id"] == sid1, "odometru float si int trebuie sa dea acelasi submission"
assert res2["deduped"] is True

173
tests/test_foundation.py Normal file
View File

@@ -0,0 +1,173 @@
"""Teste FOUNDATION (Task #1): schema + migrari idempotente + openpyxl disponibil."""
from __future__ import annotations
import os
import tempfile
import pytest
@pytest.fixture()
def db_conn(monkeypatch):
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "test_foundation.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.db import get_connection, init_db
init_db()
conn = get_connection()
yield conn
conn.close()
get_settings.cache_clear()
def _table_cols(conn, table: str) -> set[str]:
return {r["name"] for r in conn.execute(f"PRAGMA table_info({table})").fetchall()}
def _tables(conn) -> set[str]:
return {r["name"] for r in conn.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
).fetchall()}
# --- Coloane noi ---
def test_accounts_rar_creds_enc(db_conn):
cols = _table_cols(db_conn, "accounts")
assert "rar_creds_enc" in cols
def test_submissions_batch_id(db_conn):
cols = _table_cols(db_conn, "submissions")
assert "batch_id" in cols
def test_submissions_row_index(db_conn):
cols = _table_cols(db_conn, "submissions")
assert "row_index" in cols
def test_submissions_purge_after(db_conn):
cols = _table_cols(db_conn, "submissions")
assert "purge_after" in cols
# --- Tabele noi ---
def test_column_mappings_table(db_conn):
assert "column_mappings" in _tables(db_conn)
def test_import_batches_table(db_conn):
assert "import_batches" in _tables(db_conn)
def test_import_rows_table(db_conn):
assert "import_rows" in _tables(db_conn)
def test_import_attestations_table(db_conn):
assert "import_attestations" in _tables(db_conn)
def test_import_rows_cols(db_conn):
cols = _table_cols(db_conn, "import_rows")
for c in ("id", "batch_id", "row_index", "raw_json", "resolved_status", "error"):
assert c in cols, f"coloana lipsa: {c}"
def test_import_batches_cols(db_conn):
cols = _table_cols(db_conn, "import_batches")
for c in ("id", "account_id", "filename", "status", "total", "ok",
"needs_mapping", "needs_data", "needs_review", "already_sent",
"duplicate_in_file", "created_at", "purge_after"):
assert c in cols, f"coloana lipsa: {c}"
def test_import_attestations_cols(db_conn):
cols = _table_cols(db_conn, "import_attestations")
for c in ("id", "batch_id", "account_id", "confirmed_by", "ts", "rows_hash", "n_confirmed"):
assert c in cols, f"coloana lipsa: {c}"
# --- Idempotenta init_db ---
def test_init_db_idempotent(monkeypatch):
"""init_db() ruleaza de doua ori fara eroare."""
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "idem.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.db import init_db
init_db()
init_db() # a doua oara trebuie sa fie silentioasa
get_settings.cache_clear()
def test_migrate_on_existing_db(monkeypatch):
"""Migrarea functioneaza pe o DB veche (fara coloane noi)."""
import sqlite3
tmp = tempfile.mkdtemp()
db_path = os.path.join(tmp, "old.db")
# Creeaza schema minima fara coloanele noi
old_conn = sqlite3.connect(db_path)
old_conn.executescript("""
PRAGMA journal_mode = WAL;
CREATE TABLE IF NOT EXISTS accounts (id INTEGER PRIMARY KEY, name TEXT NOT NULL, cui TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')));
INSERT OR IGNORE INTO accounts (id, name) VALUES (1, 'default');
CREATE TABLE IF NOT EXISTS submissions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
idempotency_key TEXT NOT NULL UNIQUE,
account_id INTEGER,
status TEXT NOT NULL DEFAULT 'queued',
payload_json TEXT NOT NULL,
rar_status_code INTEGER,
rar_error TEXT,
id_prezentare INTEGER,
retry_count INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS worker_heartbeat (
id INTEGER PRIMARY KEY CHECK (id = 1),
last_beat TEXT, last_rar_login_ok TEXT, detail TEXT
);
INSERT OR IGNORE INTO worker_heartbeat (id, detail) VALUES (1, 'never started');
CREATE TABLE IF NOT EXISTS api_keys (id INTEGER PRIMARY KEY, account_id INTEGER, key_hash TEXT UNIQUE, active INTEGER DEFAULT 1, created_at TEXT);
CREATE TABLE IF NOT EXISTS operations_mapping (id INTEGER PRIMARY KEY, account_id INTEGER, cod_op_service TEXT, cod_prestatie TEXT, auto_send INTEGER DEFAULT 1, created_at TEXT, UNIQUE(account_id, cod_op_service));
CREATE TABLE IF NOT EXISTS nomenclator_rar (cod_prestatie TEXT PRIMARY KEY, nume_prestatie TEXT, updated_at TEXT);
""")
old_conn.close()
monkeypatch.setenv("AUTOPASS_DB_PATH", db_path)
from app.config import get_settings
get_settings.cache_clear()
from app.db import init_db, get_connection
init_db()
conn = get_connection()
sub_cols = {r["name"] for r in conn.execute("PRAGMA table_info(submissions)").fetchall()}
acc_cols = {r["name"] for r in conn.execute("PRAGMA table_info(accounts)").fetchall()}
assert "batch_id" in sub_cols
assert "row_index" in sub_cols
assert "purge_after" in sub_cols
assert "rar_creds_enc" in acc_cols
conn.close()
get_settings.cache_clear()
# --- openpyxl disponibil ---
def test_openpyxl_importabil():
import openpyxl
assert openpyxl.__version__.startswith("3.1")
def test_openpyxl_create_workbook():
from openpyxl import Workbook
wb = Workbook()
ws = wb.active
ws["A1"] = "test"
assert ws["A1"].value == "test"

830
tests/test_import_api.py Normal file
View File

@@ -0,0 +1,830 @@
"""Teste API import Treapta 2 — POST /v1/import, preview, commit, export-failed.
Acopera:
- #11 U1+T4: upload + staging + mapare coloane semnatura/drift/fuzzy
- #12 T2+T11: preview 6 stari + already_sent batch lookup + intra-batch collision
- #13 T5+T12: gate HARD confirmare + atestare valori + commit ON CONFLICT (TOCTOU)
- #14 T8: export randuri esuate CSV
"""
from __future__ import annotations
import csv
import io
import json
import os
import tempfile
import openpyxl
import pytest
from fastapi.testclient import TestClient
# --------------------------------------------------------------------------- #
# Fixture client #
# --------------------------------------------------------------------------- #
@pytest.fixture()
def client(monkeypatch):
"""Client FastAPI cu DB temporara izolata per test."""
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "t.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.crypto import reset_cache
reset_cache()
from app.main import app
with TestClient(app) as c:
yield c
get_settings.cache_clear()
# --------------------------------------------------------------------------- #
# Helpere pentru fisiere test #
# --------------------------------------------------------------------------- #
_HEADER = ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"]
_ROW_OK = ["WVWZZZ1KZAW000123", "B999TST", "2026-06-15", "123456", "Revizie"]
_ROW_OK2 = ["WVWZZZ1KZAW000124", "CJ001AB", "2026-05-10", "98765", "Reparatie"]
def _make_xlsx(rows: list[list]) -> bytes:
"""Creeaza un xlsx in-memory."""
wb = openpyxl.Workbook()
ws = wb.active
ws.title = "Sheet1"
for row in rows:
ws.append(row)
buf = io.BytesIO()
wb.save(buf)
return buf.getvalue()
def _make_csv(rows: list[list], delimiter: str = ";") -> bytes:
"""Creeaza un CSV in-memory."""
buf = io.StringIO()
writer = csv.writer(buf, delimiter=delimiter)
for row in rows:
writer.writerow(row)
return buf.getvalue().encode("utf-8")
def _upload_file(client: TestClient, data: bytes, filename: str = "test.xlsx") -> dict:
"""Upload un fisier si intoarce raspunsul JSON."""
r = client.post(
"/v1/import",
files={"file": (filename, io.BytesIO(data), "application/octet-stream")},
)
return r
def _default_column_mapping() -> dict:
"""Mapare de coloane implicita pentru fisierul test."""
return {
"VIN": "vin",
"Nr inmatriculare": "nr_inmatriculare",
"Data prestatie": "data_prestatie",
"Odometru final": "odometru_final",
"Operatie": "operatie",
}
def _setup_nomenclator(client: TestClient) -> None:
"""Seed nomenclator cu un cod de prestatie pentru teste."""
# Folosim POST /v1/prezentari pentru a forta seed-ul nomenclatorului
# care are loc in init_db -> seed_nomenclator_if_empty
pass # seed-ul se face automat in init_db
def _seed_operation_mapping(client: TestClient, cod_op: str = "Revizie", cod_prest: str = "OE-1") -> None:
"""Salveaza o mapare de operatii pentru teste."""
# Adauga mai intai in nomenclator daca nu exista (prin POST prezentare care creeaza cod)
# De fapt, cod OE-1 e in nomenclatorul seed
client.post("/v1/mapari", json={
"cod_op_service": cod_op,
"cod_prestatie": cod_prest,
"auto_send": True,
})
# =========================================================================== #
# #11 — Upload + staging (U1+T4) #
# =========================================================================== #
class TestUploadStaging:
def test_upload_xlsx_ok(self, client):
"""Upload xlsx valid -> import_id + columns + sample_rows."""
data = _make_xlsx([_HEADER, _ROW_OK, _ROW_OK2])
r = _upload_file(client, data, "test.xlsx")
assert r.status_code == 200, r.text
body = r.json()
assert "import_id" in body
assert body["columns"] == _HEADER
assert body["total_rows"] == 2
assert len(body["sample_rows"]) == 2
def test_upload_csv_semicolon(self, client):
"""Upload CSV cu ';' (export RO) -> parsare corecta."""
data = _make_csv([_HEADER, _ROW_OK], delimiter=";")
r = _upload_file(client, data, "test.csv")
assert r.status_code == 200, r.text
body = r.json()
assert body["columns"] == _HEADER
assert body["total_rows"] == 1
def test_upload_fisier_prea_mare(self, client):
"""Fisier >5MB -> 413."""
data = b"PK" + b"X" * (5 * 1024 * 1024 + 100)
r = _upload_file(client, data, "mare.xlsx")
assert r.status_code in (413, 422)
def test_upload_format_invalid(self, client):
"""Fisier tip nesuportat -> 422."""
r = _upload_file(client, b"data random", "test.dbf")
assert r.status_code == 422
def test_issue5a_raw_json_criptat(self, client):
"""Issue 5a: raw_json din import_rows trebuie sa fie criptat (ciphertext la rest)."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
assert r.status_code == 200
import_id = r.json()["import_id"]
# Citeste direct din DB si verifica ca raw_json e criptat (nu JSON plain)
import sqlite3
from app.config import get_settings
conn = sqlite3.connect(get_settings().db_path)
conn.row_factory = sqlite3.Row
try:
row = conn.execute(
"SELECT raw_json FROM import_rows WHERE batch_id=? ORDER BY row_index LIMIT 1",
(import_id,),
).fetchone()
assert row is not None
raw = row["raw_json"]
# Ciphertext Fernet incepe cu "gAAA" (base64url)
assert not raw.startswith("{"), "raw_json trebuie sa fie criptat, nu JSON plain"
# Verifica ca se poate decripta
from app.crypto import decrypt_creds
decrypted = decrypt_creds(raw)
assert decrypted is not None
assert "VIN" in decrypted or any("VIN" in k for k in decrypted.keys())
finally:
conn.close()
def test_issue5b_fuzzy_coloane_refoloseste_normalize_for_match(self, client):
"""Issue 5b: fuzzy_suggestions din raspuns foloseste normalize_for_match (fara duplicat)."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
assert r.status_code == 200
body = r.json()
# Daca nu exista mapare, trebuie sa avem fuzzy_suggestions
if "fuzzy_suggestions" in body:
sugg = body["fuzzy_suggestions"]
# "VIN" trebuie sa aiba sugestia "vin" cu scor mare
if "VIN" in sugg:
camps = [s["camp_canonic"] for s in sugg["VIN"]]
assert "vin" in camps, f"'vin' trebuie sa fie in sugestii pentru 'VIN', primit: {camps}"
if "Odometru final" in sugg:
camps = [s["camp_canonic"] for s in sugg["Odometru final"]]
assert "odometru_final" in camps, f"'odometru_final' trebuie in sugestii"
def test_drift_semnatura_coloane(self, client):
"""T4/D3: upload 2 cu coloane mutate -> mapping_status='new' (nu aplica orb)."""
# Upload 1 cu header standard
data1 = _make_xlsx([_HEADER, _ROW_OK])
r1 = _upload_file(client, data1, "test.xlsx")
assert r1.status_code == 200
import_id1 = r1.json()["import_id"]
# Salveaza maparea pentru upload 1
client.post(
f"/v1/import/{import_id1}/column-mapping",
json={"json_mapare": _default_column_mapping()},
)
# Upload 2 cu header DIFERIT (coloane mutate/redenumite)
header2 = ["Sasiu", "Inmatriculare", "Data", "KM", "Lucrare"]
data2 = _make_xlsx([header2, ["WVWZZZ1KZAW000125", "B1XYZ", "2026-06-10", "50000", "ITP"]])
r2 = _upload_file(client, data2, "test2.xlsx")
assert r2.status_code == 200
body2 = r2.json()
# Semnatura diferita -> nu se aplica maparea veche
assert body2["mapping_status"] == "new", \
f"Drift coloane trebuie detectat, primit mapping_status={body2['mapping_status']}"
def test_aceeasi_semnatura_returneaza_maparea(self, client):
"""Dupa salvarea maparii, al doilea upload cu aceleasi coloane o returneaza direct."""
data = _make_xlsx([_HEADER, _ROW_OK])
r1 = _upload_file(client, data, "test.xlsx")
import_id = r1.json()["import_id"]
# Salveaza maparea
rc = client.post(
f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()},
)
assert rc.status_code == 200
# Upload 2 cu aceleasi coloane
r2 = _upload_file(client, data, "test.xlsx")
assert r2.status_code == 200
body2 = r2.json()
assert body2["mapping_status"] == "matched"
assert "column_mapping" in body2
def test_upload_xlsx_multisheet_returneaza_eroare_cu_sheets(self, client):
"""Xlsx cu 2 sheet-uri non-goale -> 422 cu lista de sheet-uri."""
wb = openpyxl.Workbook()
ws1 = wb.active
ws1.title = "Iunie"
for row in [_HEADER, _ROW_OK]:
ws1.append(row)
ws2 = wb.create_sheet("Iulie")
for row in [_HEADER, _ROW_OK2]:
ws2.append(row)
buf = io.BytesIO()
wb.save(buf)
r = _upload_file(client, buf.getvalue(), "multi.xlsx")
assert r.status_code == 422
body = r.json()
assert body["detail"]["error"] == "multiple_sheets"
assert "Iunie" in body["detail"]["sheets"]
def test_upload_xlsx_multisheet_cu_sheet_ales(self, client):
"""Dupa alegere sheet -> parsare corecta."""
wb = openpyxl.Workbook()
ws1 = wb.active
ws1.title = "Iunie"
for row in [_HEADER, _ROW_OK]:
ws1.append(row)
ws2 = wb.create_sheet("Iulie")
for row in [_HEADER, _ROW_OK2]:
ws2.append(row)
buf = io.BytesIO()
wb.save(buf)
r = client.post(
"/v1/import?sheet_name=Iulie",
files={"file": ("multi.xlsx", io.BytesIO(buf.getvalue()), "application/octet-stream")},
)
assert r.status_code == 200
body = r.json()
assert body["total_rows"] == 1
assert body["sample_rows"][0]["VIN"] == _ROW_OK2[0]
def test_purge_after_setat_la_insert(self, client):
"""T16: purge_after trebuie setat la insert import_batches."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
import sqlite3
from app.config import get_settings
conn = sqlite3.connect(get_settings().db_path)
conn.row_factory = sqlite3.Row
try:
row = conn.execute(
"SELECT purge_after FROM import_batches WHERE id=?", (import_id,)
).fetchone()
assert row["purge_after"] is not None, "purge_after trebuie setat la insert"
finally:
conn.close()
# =========================================================================== #
# #11 — Mapare coloane (T4) #
# =========================================================================== #
class TestColumnMapping:
def test_save_column_mapping(self, client):
"""Salveaza maparea de coloane pentru un batch."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
rc = client.post(
f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()},
)
assert rc.status_code == 200
assert "signature" in rc.json()
def test_get_column_mapping_dupa_salvare(self, client):
"""GET column-mapping returneaza maparea salvata."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
client.post(
f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()},
)
rg = client.get(f"/v1/import/{import_id}/column-mapping")
assert rg.status_code == 200
body = rg.json()
assert body["status"] == "matched"
assert body["column_mapping"] == _default_column_mapping()
def test_get_column_mapping_fara_salvare_returneaza_sugestii(self, client):
"""GET column-mapping fara mapare salvata -> sugestii fuzzy."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
rg = client.get(f"/v1/import/{import_id}/column-mapping")
assert rg.status_code == 200
body = rg.json()
assert body["status"] == "new"
# Trebuie sa aiba sugestii pentru coloane evidente
if "fuzzy_suggestions" in body:
assert "VIN" in body["fuzzy_suggestions"]
def test_column_mapping_batch_inexistent(self, client):
"""GET/POST pe batch inexistent -> 404."""
r = client.get("/v1/import/99999/column-mapping")
assert r.status_code == 404
# =========================================================================== #
# #12 — Preview 6 stari (T2 + T11) #
# =========================================================================== #
class TestPreview:
def _upload_and_map(self, client, rows=None):
"""Fixture: upload + salveaza mapare + seeda nomenclator."""
if rows is None:
rows = [_HEADER, _ROW_OK]
data = _make_xlsx(rows)
r = _upload_file(client, data, "test.xlsx")
assert r.status_code == 200, r.text
import_id = r.json()["import_id"]
# Salveaza maparea de coloane
rc = client.post(
f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()},
)
assert rc.status_code == 200
# Seeda maparea de operatii
_seed_operation_mapping(client, "Revizie", "OE-1")
return import_id
def test_preview_rand_ok(self, client):
"""Rand valid cu operatie mapata -> stare 'ok'."""
import_id = self._upload_and_map(client)
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200, rp.text
body = rp.json()
rows = body["rows"]
assert len(rows) == 1
# VIN valid, data valida, odometru valid, operatie mapata -> ok
assert rows[0]["resolved_status"] == "ok"
def test_preview_needs_mapping(self, client):
"""Rand cu operatie nemapata -> needs_mapping."""
import_id = self._upload_and_map(client, rows=[_HEADER, _ROW_OK2])
# _ROW_OK2 are operatia "Reparatie" care nu e mapata
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
body = rp.json()
assert any(r["resolved_status"] in ("needs_mapping",) for r in body["rows"])
def test_preview_needs_data(self, client):
"""Rand cu VIN invalid -> needs_data."""
row_bad = ["INVALID_VIN_XX", "B999TST", "2026-06-15", "123456", "Revizie"]
import_id = self._upload_and_map(client, rows=[_HEADER, row_bad])
_seed_operation_mapping(client, "Revizie", "OE-1")
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
body = rp.json()
assert any(r["resolved_status"] in ("needs_data",) for r in body["rows"])
def test_preview_already_sent_dupa_submit(self, client):
"""Rand deja trimis prin API -> stare already_sent la preview (T2/D5)."""
# Trimite prin API canalul standard
client.post("/v1/prezentari", json={
"rar_credentials": {"email": "x@y.ro", "password": "s"},
"prezentari": [{
"vin": "WVWZZZ1KZAW000123",
"nr_inmatriculare": "B999TST",
"data_prestatie": "2026-06-15",
"odometru_final": "123456",
"prestatii": [{"cod_prestatie": "OE-1"}],
}],
})
# Acum upload acelasi rand
import_id = self._upload_and_map(client, rows=[_HEADER, _ROW_OK])
_seed_operation_mapping(client, "Revizie", "OE-1")
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
body = rp.json()
# Randul trebuie sa fie already_sent (cheia idempotency exista deja)
statuses = [r["resolved_status"] for r in body["rows"]]
assert "already_sent" in statuses, f"Asteptat 'already_sent', primit: {statuses}"
def test_preview_duplicate_in_file(self, client):
"""T11/OV-3: 2 randuri identice in ACELASI fisier -> duplicate_in_file."""
# Acelasi rand de doua ori
rows = [_HEADER, _ROW_OK, _ROW_OK] # duplicat exact
import_id = self._upload_and_map(client, rows=rows)
_seed_operation_mapping(client, "Revizie", "OE-1")
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
body = rp.json()
statuses = [r["resolved_status"] for r in body["rows"]]
assert "duplicate_in_file" in statuses, \
f"Asteptat 'duplicate_in_file', primit: {statuses}"
def test_preview_already_sent_batch_lookup_nu_n_plus_1(self, client):
"""Eng#5: already_sent lookup BATCH (nu N+1) — ≤7 interogari pentru 5 randuri."""
# Cream 5 randuri distincte
rows_data = [_HEADER]
for i in range(5):
rows_data.append([
f"WVWZZZ1KZAW00{i:04d}",
f"B00{i}TST",
"2026-06-15",
str(100000 + i),
"Revizie",
])
import_id = self._upload_and_map(client, rows=rows_data)
_seed_operation_mapping(client, "Revizie", "OE-1")
# Aceasta verificare e comportamentala: preview trebuie sa functioneze
# corect (nu testam direct nr. de SQL queries, ci ca raspunsul e corect)
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
body = rp.json()
assert len(body["rows"]) == 5
def test_preview_fara_mapare_coloane_returneaza_422(self, client):
"""Preview fara mapare de coloane configurata -> 422."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
# Nu salvam maparea de coloane
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 422
assert "no_column_mapping" in rp.json()["detail"]["error"]
def test_preview_summary_ok(self, client):
"""Preview intoarce si summary cu contoare per stare."""
import_id = self._upload_and_map(client, rows=[_HEADER, _ROW_OK, _ROW_OK2])
_seed_operation_mapping(client, "Revizie", "OE-1")
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
body = rp.json()
assert "summary" in body
# Suma totala = nr randuri
total = sum(body["summary"].values())
assert total == len(body["rows"])
# =========================================================================== #
# #13 — Commit gate HARD + atestare + TOCTOU (T5 + T12) #
# =========================================================================== #
class TestCommit:
def _upload_preview_ok(self, client):
"""Upload + mapeaza + preview -> returneaza import_id cu randuri ok."""
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
client.post(
f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()},
)
_seed_operation_mapping(client, "Revizie", "OE-1")
# Preview pentru a calcula starile
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
return import_id, rp.json()
def test_commit_cu_n_corect_enqueued(self, client):
"""Commit cu N corect -> rand in submissions cu status queued."""
import_id, preview = self._upload_preview_ok(client)
n_ok = preview["summary"].get("ok", 0)
assert n_ok > 0, "Trebuie cel putin un rand ok"
rc = client.post(f"/v1/import/{import_id}/commit", json={
"n_confirmat": n_ok,
"reviewed_rows": [],
})
assert rc.status_code == 200, rc.text
body = rc.json()
assert body["enqueued"] == n_ok
def test_commit_cu_n_gresit_reject(self, client):
"""T5/D3: commit cu N gresit -> 422, nu enqueue."""
import_id, preview = self._upload_preview_ok(client)
n_ok = preview["summary"].get("ok", 0)
# Trimitem n_confirmat + 1 (gresit)
rc = client.post(f"/v1/import/{import_id}/commit", json={
"n_confirmat": n_ok + 1,
"reviewed_rows": [],
})
assert rc.status_code == 422
detail = rc.json()["detail"]
assert "confirmare_gresita" in detail.get("error", ""), \
f"Eroare neasteptata: {detail}"
def test_commit_log_atestare(self, client):
"""T12/Voce#9: commit scrie import_attestations cu rows_hash + n_confirmed."""
import_id, preview = self._upload_preview_ok(client)
n_ok = preview["summary"].get("ok", 0)
rc = client.post(f"/v1/import/{import_id}/commit", json={
"n_confirmat": n_ok,
"reviewed_rows": [],
"confirmed_by": "test@example.com",
})
assert rc.status_code == 200
body = rc.json()
assert body["enqueued"] == n_ok
assert body["rows_hash"] # sha256 non-gol
# Verifica direct in DB ca exista atestarea
import sqlite3
from app.config import get_settings
conn = sqlite3.connect(get_settings().db_path)
conn.row_factory = sqlite3.Row
try:
att = conn.execute(
"SELECT * FROM import_attestations WHERE batch_id=?", (import_id,)
).fetchone()
assert att is not None, "import_attestations trebuie sa contina o inregistrare"
assert att["n_confirmed"] == n_ok
assert att["rows_hash"] == body["rows_hash"]
assert att["confirmed_by"] == "test@example.com"
finally:
conn.close()
def test_commit_batch_id_setat_pe_submission(self, client):
"""T7: submission creata la commit trebuie sa aiba batch_id + row_index setate."""
import_id, preview = self._upload_preview_ok(client)
n_ok = preview["summary"].get("ok", 0)
rc = client.post(f"/v1/import/{import_id}/commit", json={
"n_confirmat": n_ok,
"reviewed_rows": [],
})
assert rc.status_code == 200
submissions = rc.json()["submissions"]
assert len(submissions) > 0
import sqlite3
from app.config import get_settings
conn = sqlite3.connect(get_settings().db_path)
conn.row_factory = sqlite3.Row
try:
for sub in submissions:
row = conn.execute(
"SELECT batch_id, row_index FROM submissions WHERE id=?",
(sub["submission_id"],),
).fetchone()
assert row["batch_id"] == import_id, "batch_id trebuie setat"
assert row["row_index"] is not None, "row_index trebuie setat"
finally:
conn.close()
def test_commit_toctou_cheie_inserata_concurent(self, client):
"""Issue 1 (TOCTOU): cheie inserata de canal concurent -> reclasificata already_sent.
Simulam TOCTOU inserand cheia direct in submissions inainte de commit.
"""
import_id, preview = self._upload_preview_ok(client)
n_ok = preview["summary"].get("ok", 0)
# Gaseste cheia de idempotenta a randului ok
ok_rows = [r for r in preview["rows"] if r["resolved_status"] == "ok"]
assert len(ok_rows) > 0
idem_key = ok_rows[0]["idempotency_key"]
assert idem_key, "idempotency_key trebuie calculat la preview"
# Simuleaza canalul concurent: insereaza cheia in submissions
import sqlite3
from app.config import get_settings
conn = sqlite3.connect(get_settings().db_path)
try:
conn.execute(
"INSERT INTO submissions (idempotency_key, account_id, status, payload_json) "
"VALUES (?, 1, 'queued', '{}')",
(idem_key,),
)
conn.commit()
finally:
conn.close()
# Commit-ul trebuie sa detecteze coliziunea si s-o raporteze ca TOCTOU
rc = client.post(f"/v1/import/{import_id}/commit", json={
"n_confirmat": n_ok,
"reviewed_rows": [],
})
# Poate fi 200 cu toctou_collisions sau 422 cu informatii clare
# Conform planului Issue 1: reclasificat already_sent, nu rollback
# Dar n_enqueued va fi 0 daca toate colideaza
if rc.status_code == 200:
body = rc.json()
assert body["toctou_collisions"] == [ok_rows[0]["row_index"]] or body["enqueued"] == 0
# Sau 422 daca gate-ul HARD detecteaza ca n_ok actual != n_confirmat
# (dupa reclasificare, n_total_ok scade)
def test_commit_needs_review_nebifat_exclus_din_n(self, client):
"""Voce#1: rand needs_review nebifat explicit -> NU intra in N, NU se enqueued."""
# Rand cu VIN numeric (coercion -> needs_review)
import datetime as dt
wb = openpyxl.Workbook()
ws = wb.active
ws.append(_HEADER)
# VIN ca int (numeric) -> coercion flag -> needs_review
ws.cell(row=2, column=1).value = 123456789012345 # VIN numeric
ws.cell(row=2, column=2).value = "B999TST"
ws.cell(row=2, column=3).value = "2026-06-15"
ws.cell(row=2, column=4).value = 123456
ws.cell(row=2, column=5).value = "Revizie"
buf = io.BytesIO()
wb.save(buf)
r = _upload_file(client, buf.getvalue(), "test.xlsx")
assert r.status_code == 200
import_id = r.json()["import_id"]
client.post(f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()})
_seed_operation_mapping(client, "Revizie", "OE-1")
rp = client.get(f"/v1/import/{import_id}/preview")
assert rp.status_code == 200
body_preview = rp.json()
review_rows = [r for r in body_preview["rows"] if r["resolved_status"] == "needs_review"]
ok_rows_count = body_preview["summary"].get("ok", 0)
review_count = len(review_rows)
if review_count == 0:
pytest.skip("Niciun rand needs_review in acest test — skip")
# Confirma FARA a bifa needs_review -> n_confirmat = ok_rows_count (fara review)
# Dar n_ok e 0 daca tot fisierul e needs_review
rc = client.post(f"/v1/import/{import_id}/commit", json={
"n_confirmat": ok_rows_count,
"reviewed_rows": [], # nu bifam needs_review
})
if ok_rows_count == 0:
# 0 randuri ok + 0 reviewed = eroare
assert rc.status_code in (422,)
else:
# Randurile needs_review NU sunt enqueued (nu le-am bifat)
assert rc.status_code == 200
body = rc.json()
assert body["enqueued"] == ok_rows_count
def test_commit_double_call_returneaza_409(self, client):
"""Commit de doua ori pe acelasi batch -> 409 (deja comis)."""
import_id, preview = self._upload_preview_ok(client)
n_ok = preview["summary"].get("ok", 0)
client.post(f"/v1/import/{import_id}/commit", json={"n_confirmat": n_ok, "reviewed_rows": []})
r2 = client.post(f"/v1/import/{import_id}/commit", json={"n_confirmat": n_ok, "reviewed_rows": []})
assert r2.status_code == 409
def test_atestare_purge_after_setat_pe_submission(self, client):
"""T16: submissions create la commit trebuie sa aiba purge_after setat."""
import_id, preview = self._upload_preview_ok(client)
n_ok = preview["summary"].get("ok", 0)
rc = client.post(f"/v1/import/{import_id}/commit", json={
"n_confirmat": n_ok, "reviewed_rows": []
})
assert rc.status_code == 200
submissions = rc.json()["submissions"]
if not submissions:
pytest.skip("Nicio submission creata")
import sqlite3
from app.config import get_settings
conn = sqlite3.connect(get_settings().db_path)
conn.row_factory = sqlite3.Row
try:
for sub in submissions:
row = conn.execute(
"SELECT purge_after FROM submissions WHERE id=?", (sub["submission_id"],)
).fetchone()
assert row["purge_after"] is not None, "purge_after trebuie setat pe submission"
finally:
conn.close()
# =========================================================================== #
# #14 — Export randuri esuate CSV (T8) #
# =========================================================================== #
class TestExportFailed:
def _setup_batch_with_bad_rows(self, client):
"""Upload cu randuri esuate (VIN invalid)."""
row_bad = ["INVALID_VIN_XXXXXXX", "B999TST", "2026-06-15", "123456", "Revizie"]
data = _make_xlsx([_HEADER, row_bad])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
client.post(f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()})
_seed_operation_mapping(client, "Revizie", "OE-1")
# Preview pentru a calcula starile
client.get(f"/v1/import/{import_id}/preview")
return import_id
def test_export_failed_returneaza_csv(self, client):
"""Export randuri esuate -> CSV cu header + randuri."""
import_id = self._setup_batch_with_bad_rows(client)
r = client.get(f"/v1/import/{import_id}/export-failed")
assert r.status_code == 200
assert "text/csv" in r.headers["content-type"]
# Parseaza CSV
content = r.text
reader = csv.DictReader(io.StringIO(content))
rows = list(reader)
assert len(rows) > 0, "CSV trebuie sa contina cel putin un rand esuat"
def test_export_failed_contine_motiv_eroare(self, client):
"""CSV de export contine coloana 'error' cu motivul."""
import_id = self._setup_batch_with_bad_rows(client)
r = client.get(f"/v1/import/{import_id}/export-failed")
assert r.status_code == 200
reader = csv.DictReader(io.StringIO(r.text))
rows = list(reader)
assert len(rows) > 0
# Fiecare rand trebuie sa aiba coloana error
for row in rows:
assert "error" in row, "Coloana 'error' trebuie sa fie prezenta"
assert row["resolved_status"] in ("needs_data", "needs_mapping", "needs_review")
def test_export_failed_batch_inexistent(self, client):
"""Export pe batch inexistent -> 404."""
r = client.get("/v1/import/99999/export-failed")
assert r.status_code == 404
def test_export_failed_fara_randuri_esuate(self, client):
"""Export pe batch fara randuri esuate -> CSV gol (doar header)."""
# Upload cu rand ok
data = _make_xlsx([_HEADER, _ROW_OK])
r = _upload_file(client, data, "test.xlsx")
import_id = r.json()["import_id"]
client.post(f"/v1/import/{import_id}/column-mapping",
json={"json_mapare": _default_column_mapping()})
_seed_operation_mapping(client, "Revizie", "OE-1")
client.get(f"/v1/import/{import_id}/preview")
r = client.get(f"/v1/import/{import_id}/export-failed")
assert r.status_code == 200
reader = csv.DictReader(io.StringIO(r.text))
rows = list(reader)
assert len(rows) == 0, "Niciun rand esuat in batch cu randuri ok"
# =========================================================================== #
# Regresie: reconcile.py ramane op-blind (OV-3) #
# =========================================================================== #
class TestReconcileRegresie:
def test_match_finalizata_ramane_op_blind(self):
"""OV-3: reconcile.py trebuie sa ramana op-blind (nu editat de import).
Importam reconcile si verificam ca nu s-au adaugat parametri de operatie.
"""
from app.reconcile import match_finalizata
import inspect
sig = inspect.signature(match_finalizata)
params = set(sig.parameters.keys())
# Trebuie sa aiba exact acesti parametri (op-blind by design)
expected = {"finalizate", "vin", "data_prestatie", "odometru_final"}
assert not (params - expected - {"self"}), \
f"match_finalizata are parametri neasteptati: {params - expected}"
# Nu trebuie sa aiba parametri de operatie
assert "cod_prestatie" not in params
assert "operatie" not in params
assert "cod_op_service" not in params

View File

@@ -77,7 +77,7 @@ def env(monkeypatch):
def test_read_nomenclator_real_dbf():
from tools.import_dbf import read_nomenclator
rep = read_nomenclator(ROOT / "prestatii_rar.DBF")
rep = read_nomenclator(ROOT / "legacy-vfp" / "prestatii_rar.DBF")
assert len(rep["rows"]) == 20
codes = {r["cod_prestatie"] for r in rep["rows"]}
assert "OE-1" in codes and "R-ODO" in codes

1024
tests/test_import_e2e.py Normal file

File diff suppressed because it is too large Load Diff

451
tests/test_import_parse.py Normal file
View File

@@ -0,0 +1,451 @@
"""Teste pentru app/import_parse.py (T14 + T3 + T10 + T13).
Fixture-urile xlsx sunt generate in-memory cu openpyxl (nu fisiere binare commituite).
Fiecare sectiune acopera un task distinct.
"""
from __future__ import annotations
import csv
import io
from datetime import date, datetime
import openpyxl
import pytest
from app.import_parse import (
FileTooLarge,
HeaderError,
MultipleSheets,
ParsedFile,
parse_csv,
parse_file,
parse_xlsx,
_detect_date_formats,
_infer_date_format_from_column,
parse_date_value,
_trim_footer,
)
# --------------------------------------------------------------------------- #
# Helpere fixture #
# --------------------------------------------------------------------------- #
def _make_xlsx(rows: list[list], sheet_name: str = "Sheet1") -> bytes:
"""Creeaza un fisier xlsx in-memory cu un singur sheet."""
wb = openpyxl.Workbook()
ws = wb.active
ws.title = sheet_name
for row in rows:
ws.append(row)
buf = io.BytesIO()
wb.save(buf)
return buf.getvalue()
def _make_xlsx_multisheet(sheets: dict[str, list[list]]) -> bytes:
"""Creeaza un xlsx cu mai multe sheet-uri."""
wb = openpyxl.Workbook()
first = True
for name, rows in sheets.items():
if first:
ws = wb.active
ws.title = name
first = False
else:
ws = wb.create_sheet(name)
for row in rows:
ws.append(row)
buf = io.BytesIO()
wb.save(buf)
return buf.getvalue()
def _make_csv(rows: list[list], delimiter: str = ",", encoding: str = "utf-8") -> bytes:
"""Creeaza un CSV in-memory."""
buf = io.StringIO()
writer = csv.writer(buf, delimiter=delimiter)
for row in rows:
writer.writerow(row)
return buf.getvalue().encode(encoding)
_HEADER = ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"]
_ROW1 = ["WVWZZZ1KZAW000123", "B999TST", "2026-06-15", "123456", "Revizie"]
_ROW2 = ["WVWZZZ1KZAW000124", "CJ001AB", "2026-05-10", "98765", "Reparatie"]
# =========================================================================== #
# T14 — CSV delimiter sniff + encoding + cap #
# =========================================================================== #
class TestCsvDelimiter:
def test_csv_semicolon_ro_export(self):
"""Export Excel RO foloseste ';' — trebuie detectat corect."""
data = _make_csv([_HEADER, _ROW1, _ROW2], delimiter=";")
result = parse_csv(data)
assert result.columns == _HEADER
assert len(result.rows) == 2
assert result.rows[0]["VIN"] == "WVWZZZ1KZAW000123"
def test_csv_comma_standard(self):
data = _make_csv([_HEADER, _ROW1, _ROW2], delimiter=",")
result = parse_csv(data)
assert len(result.rows) == 2
def test_csv_tab_delimiter(self):
data = _make_csv([_HEADER, _ROW1, _ROW2], delimiter="\t")
result = parse_csv(data)
assert len(result.rows) == 2
def test_csv_single_column_raises_header_error(self):
"""1 coloana dupa sniff -> HeaderError clar, nu mapare oarba."""
# CSV fara delimitator real -> o singura coloana
text = "VIN\nWVWZZZ1KZAW000123\n"
data = text.encode("utf-8")
with pytest.raises(HeaderError) as exc:
parse_csv(data)
assert "coloana" in str(exc.value).lower()
def test_csv_encoding_cp1250(self):
"""Export RO cu encoding cp1250 (diacritice romanesti)."""
rows = [
["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"],
["WVWZZZ1KZAW000123", "B999TST", "2026-06-15", "123456", "Revizie periodică"],
]
data = _make_csv(rows, delimiter=";", encoding="cp1250")
result = parse_csv(data)
assert len(result.rows) == 1
def test_csv_too_many_rows_raises(self):
""">5000 randuri -> FileTooLarge fara parsare partiala."""
header = ["VIN", "Data", "Odometru", "NrInm", "Op"]
rows = [header] + [["WVWZZZ1KZAW000123", "2026-01-01", "1000", "B1TST", "R"] for _ in range(5001)]
data = _make_csv(rows, delimiter=",")
with pytest.raises(FileTooLarge):
parse_csv(data)
def test_csv_too_large_bytes_raises(self):
""">5MB -> FileTooLarge."""
data = b"X" * (5 * 1024 * 1024 + 1)
with pytest.raises(FileTooLarge):
parse_csv(data)
# =========================================================================== #
# T14 — XLSX read_only dim-check + cap #
# =========================================================================== #
class TestXlsxDimcheck:
def test_xlsx_parsat_corect(self):
data = _make_xlsx([_HEADER, _ROW1, _ROW2])
result = parse_xlsx(data)
assert result.columns == _HEADER
assert len(result.rows) == 2
def test_xlsx_too_large_bytes_raises(self):
"""Fisier >5MB -> FileTooLarge inainte de parsare."""
# Cream un xlsx real dar verificam dimensiunea bytes separat
data = b"PK" + b"X" * (5 * 1024 * 1024 + 100)
with pytest.raises((FileTooLarge, Exception)):
# Poate ridica si InvalidFileException daca nu e xlsx valid
parse_xlsx(data)
def test_xlsx_empty_raises_header_error(self):
"""Workbook fara date -> HeaderError."""
wb = openpyxl.Workbook()
buf = io.BytesIO()
wb.save(buf)
with pytest.raises(HeaderError):
parse_xlsx(buf.getvalue())
def test_parse_file_dispatch_xlsx(self):
data = _make_xlsx([_HEADER, _ROW1])
result = parse_file(data, "test.xlsx")
assert len(result.rows) == 1
def test_parse_file_dispatch_csv(self):
data = _make_csv([_HEADER, _ROW1], delimiter=";")
result = parse_file(data, "test.csv")
assert len(result.rows) == 1
def test_parse_file_unsupported_ext(self):
with pytest.raises(HeaderError):
parse_file(b"data", "test.dbf")
# =========================================================================== #
# T3 — coercion guard + needs_review + mesaj formule-None #
# =========================================================================== #
class TestCoercionNeedsReview:
def test_vin_numeric_xlsx_flagged(self):
"""VIN '0123...' citit ca numeric din xlsx -> needs_review (coercion_flags)."""
# openpyxl citeste VIN numeric ca int/float daca celula e formatata numeric
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"])
# Scrie VIN ca numar (simuleaza comportamentul openpyxl pt celule numerice)
ws.cell(row=2, column=1).value = 123456789012345 # int, nu string
ws.cell(row=2, column=2).value = "B999TST"
ws.cell(row=2, column=3).value = "2026-06-15"
ws.cell(row=2, column=4).value = 123456
ws.cell(row=2, column=5).value = "Revizie"
buf = io.BytesIO()
wb.save(buf)
result = parse_xlsx(buf.getvalue())
assert 0 in result.coercion_flags
flags = result.coercion_flags[0]
assert any("VIN" in f for f in flags)
def test_odometru_float_coerced(self):
"""Odometru 123456.0 (float Excel) -> convertit la '123456'."""
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"])
ws.cell(row=2, column=1).value = "WVWZZZ1KZAW000123"
ws.cell(row=2, column=2).value = "B999TST"
ws.cell(row=2, column=3).value = "2026-06-15"
ws.cell(row=2, column=4).value = 123456.0 # float cu .0
ws.cell(row=2, column=5).value = "Revizie"
buf = io.BytesIO()
wb.save(buf)
result = parse_xlsx(buf.getvalue())
odo_val = result.rows[0]["Odometru final"]
assert odo_val == "123456", f"Asteptat '123456', primit '{odo_val}'"
# Nu trebuie flag needs_review pentru odometru .0 (e coercion standard)
assert 0 not in result.coercion_flags or not any(
"Odometru" in f for f in result.coercion_flags.get(0, [])
)
def test_formula_column_detected(self):
"""Coloana cu >60% None (formule necalculate) -> formula_columns."""
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"])
for i in range(10):
ws.append([
"WVWZZZ1KZAW000123",
"B999TST",
"2026-06-15",
None, # formula necalculata -> None
"Revizie",
])
buf = io.BytesIO()
wb.save(buf)
result = parse_xlsx(buf.getvalue())
assert "Odometru final" in result.formula_columns
def test_datetime_native_converted_to_iso(self):
"""Celula datetime nativa -> convertita la YYYY-MM-DD (neambigua)."""
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"])
ws.cell(row=2, column=1).value = "WVWZZZ1KZAW000123"
ws.cell(row=2, column=2).value = "B999TST"
ws.cell(row=2, column=3).value = datetime(2026, 6, 15, 10, 30)
ws.cell(row=2, column=4).value = 123456
ws.cell(row=2, column=5).value = "Revizie"
buf = io.BytesIO()
wb.save(buf)
result = parse_xlsx(buf.getvalue())
assert result.rows[0]["Data prestatie"] == "2026-06-15"
def test_odometru_via_canonicalize_row(self):
"""Verifica ca odometru_final='123456.0' e normalizat de canonicalize_row."""
from app.idempotency import canonicalize_row
raw = {"vin": "WVWZZZ1KZAW000123", "nr_inmatriculare": "B1TST",
"data_prestatie": "2026-06-15", "odometru_final": "123456.0", "prestatii": []}
canon = canonicalize_row(raw)
assert canon["odometru_final"] == "123456"
def test_vin_numeric_not_sent(self):
"""VIN numeric flagged -> coercion_flags prezent = auto-send blocat."""
wb = openpyxl.Workbook()
ws = wb.active
ws.append(["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"])
ws.cell(row=2, column=1).value = 1234567890 # numeric
ws.cell(row=2, column=2).value = "B999TST"
ws.cell(row=2, column=3).value = "2026-06-15"
ws.cell(row=2, column=4).value = 123456
ws.cell(row=2, column=5).value = "Revizie"
buf = io.BytesIO()
wb.save(buf)
result = parse_xlsx(buf.getvalue())
# Randul 0 trebuie sa aiba flags (needs_review)
assert 0 in result.coercion_flags
# =========================================================================== #
# T10 — dezambiguizare data la nivel de COLOANA (OV-8) #
# =========================================================================== #
class TestDateColumnDisambiguation:
def test_string_ambiguous_flagged(self):
"""'03.04.2026' string cu zi<=12 -> ambiguous."""
fmt = _infer_date_format_from_column(["03.04.2026", "05.06.2026", "01.02.2026"])
assert fmt == "ambiguous"
def test_dd_first_detected_from_column(self):
"""Coloana cu cel putin un rand zi>12 -> DD.MM.YYYY detectat."""
# 15.04.2026: zi=15 > 12 -> DD-first sigur
fmt = _infer_date_format_from_column(["03.04.2026", "15.04.2026", "01.02.2026"])
assert fmt == "DD.MM.YYYY"
def test_month_gt12_also_dd_first(self):
"""Luna >12 imposibila -> cu siguranta DD.MM (ex: 04.13.2026 e imposibil -> zi=4, luna=13 ❌ / zi=13, luna=4 ✓)."""
# Daca pozitia-1 (luna) > 12 -> DD-first
fmt = _infer_date_format_from_column(["04.13.2026"])
assert fmt == "DD.MM.YYYY"
def test_iso_format_detected(self):
fmt = _infer_date_format_from_column(["2026-06-15", "2026-05-10"])
assert fmt == "YYYY-MM-DD"
def test_native_datetime_column_format(self):
"""Coloana cu toate valorile datetime native -> format 'native'."""
col_values = {"Data prestatie": [datetime(2026, 6, 15), datetime(2026, 5, 10)]}
fmt_map = _detect_date_formats(col_values, ["Data prestatie"])
assert fmt_map.get("Data prestatie") == "native"
def test_parse_date_value_ambiguous_needs_review(self):
"""'03.04.2026' cu format ambiguous -> (iso, True) = needs_review."""
iso, ambiguous = parse_date_value("03.04.2026", "ambiguous")
assert ambiguous is True
assert iso == "2026-04-03" # parseaza ca DD.MM.YYYY
def test_parse_date_value_native_already_iso(self):
"""Valoare deja convertita la ISO (din datetime nativ) -> (iso, False)."""
iso, ambiguous = parse_date_value("2026-06-15", "native")
assert iso == "2026-06-15"
assert ambiguous is False
def test_parse_date_value_dd_mm_yyyy(self):
iso, ambiguous = parse_date_value("15.06.2026", "DD.MM.YYYY")
assert iso == "2026-06-15"
assert ambiguous is False
def test_parse_date_value_empty(self):
iso, ambiguous = parse_date_value(None, "DD.MM.YYYY")
assert iso is None
assert ambiguous is False
def test_column_uniform_mm_dd_with_day_gt12(self):
"""Coloana uniform MM.DD cu randuri zi>12 -> format DD-first detectat la nivel coloana."""
# Exemplu: 03.04.2026 (ambiguu) + 15.04.2026 (zi=15>12 -> DD-first sigur)
# -> intreaga coloana e DD.MM.YYYY
fmt = _infer_date_format_from_column(["03.04.2026", "15.04.2026"])
assert fmt == "DD.MM.YYYY"
# Rand care altfel ar parea ambiguu (03) e tratat corect de format detectat la nivel coloana
iso, ambig = parse_date_value("03.04.2026", fmt)
assert iso == "2026-04-03"
assert ambig is False
# =========================================================================== #
# T13 — robustete export RO (multi-sheet + merged header + footer trim) #
# =========================================================================== #
class TestRobustetExportRO:
def test_multisheet_raises_multiple_sheets(self):
"""Workbook cu 2 sheet-uri non-goale -> MultipleSheets cu lista."""
data = _make_xlsx_multisheet({
"Iunie": [_HEADER, _ROW1],
"Iulie": [_HEADER, _ROW2],
})
with pytest.raises(MultipleSheets) as exc:
parse_xlsx(data)
assert "Iunie" in exc.value.sheet_names
assert "Iulie" in exc.value.sheet_names
def test_multisheet_with_sheet_name_selected(self):
"""Dupa alegere sheet -> parsat corect."""
data = _make_xlsx_multisheet({
"Iunie": [_HEADER, _ROW1],
"Iulie": [_HEADER, _ROW2],
})
result = parse_xlsx(data, sheet_name="Iulie")
assert len(result.rows) == 1
assert result.rows[0]["VIN"] == "WVWZZZ1KZAW000124"
def test_merged_header_resolved(self):
"""Header cu celule imbinate -> un-merge logic, nu coloane goale."""
wb = openpyxl.Workbook()
ws = wb.active
# Scrie header cu merge pe primele 2 coloane
ws.cell(row=1, column=1).value = "Vehicul"
ws.cell(row=1, column=3).value = "Data prestatie"
ws.cell(row=1, column=4).value = "Odometru final"
ws.cell(row=1, column=5).value = "Operatie"
ws.merge_cells(start_row=1, start_column=1, end_row=1, end_column=2)
ws.cell(row=2, column=1).value = "WVWZZZ1KZAW000123"
ws.cell(row=2, column=2).value = "B999TST"
ws.cell(row=2, column=3).value = "2026-06-15"
ws.cell(row=2, column=4).value = 123456
ws.cell(row=2, column=5).value = "Revizie"
buf = io.BytesIO()
wb.save(buf)
result = parse_xlsx(buf.getvalue())
# Merge propaga "Vehicul" la ambele coloane; dedup adauga sufix _2
assert "Vehicul" in result.columns
assert "Vehicul_2" in result.columns # coloana 2 din merge — dezambiguizata cu sufix
# Niciuna nu e goala/None
assert len([c for c in result.columns if "Vehicul" in c]) == 2
def test_footer_rows_skipped(self):
"""Randuri trailing fara VIN + data -> skip structural, nu needs_data."""
rows = [
_HEADER,
_ROW1,
_ROW2,
["TOTAL", "", "", "222221", ""], # footer cu VIN "TOTAL"
["", "", "", "", ""], # rand complet gol
]
data = _make_xlsx(rows)
result = parse_xlsx(data)
# Randul "TOTAL" are VIN non-gol ("TOTAL") si data goala -> nu e trim structural
# Randul complet gol (VIN="" + data="") -> trim
# Deci: 3 randuri raman (ROW1 + ROW2 + TOTAL)
vins = [r.get("VIN") for r in result.rows]
assert "" not in [v for v in vins if v is not None]
# Randul gol complet sa nu fie prezent
empty_rows = [r for r in result.rows if all(v is None or str(v).strip() == "" for v in r.values())]
assert len(empty_rows) == 0
def test_footer_vin_and_date_both_empty_skipped(self):
"""Rand unde VIN si data sunt ambele goale -> skip (footer TOTAL/Intocmit de:)."""
rows_list = [
{"VIN": "WVWZZZ1KZAW000123", "Data prestatie": "2026-06-15", "Odometru final": "123456"},
{"VIN": "", "Data prestatie": "", "Odometru final": "9999"}, # footer fals
]
col_names = ["VIN", "Data prestatie", "Odometru final"]
trimmed = _trim_footer(rows_list, col_names)
assert len(trimmed) == 1
assert trimmed[0]["VIN"] == "WVWZZZ1KZAW000123"
def test_single_sheet_no_error(self):
"""Workbook cu un singur sheet -> parsat fara MultipleSheets."""
data = _make_xlsx([_HEADER, _ROW1])
result = parse_xlsx(data)
assert len(result.rows) == 1
# =========================================================================== #
# Integrare parse_file #
# =========================================================================== #
class TestParseFileIntegration:
def test_xlsx_full_flow(self):
data = _make_xlsx([_HEADER, _ROW1, _ROW2])
result = parse_file(data, "prezentari_iunie.xlsx")
assert result.columns == _HEADER
assert len(result.rows) == 2
assert result.rows[0]["VIN"] == "WVWZZZ1KZAW000123"
assert result.rows[1]["Odometru final"] == "98765"
def test_csv_semicolon_full_flow(self):
data = _make_csv([_HEADER, _ROW1], delimiter=";")
result = parse_file(data, "export_ro.csv")
assert result.columns == _HEADER
assert result.rows[0]["VIN"] == "WVWZZZ1KZAW000123"

450
tests/test_import_ui.py Normal file
View File

@@ -0,0 +1,450 @@
"""Teste UI web import (U5) — upload → mapare coloane → preview → confirmare.
Verifica:
- Dashboard randeaza sectiunea de upload
- Upload xlsx → mapare noua → fragment _mapcoloane returnat
- Upload xlsx cu mapare existenta → preview direct
- Salvare mapare coloane → preview randat
- Preview afiseaza rezumat stari si randul tabelului
- Confirmare cu N corect → succes (in coada)
- Confirmare cu N gresit → eroare explicita
- Reset → drop zone gol
- Erori upload (fisier invalid, prea mare, header neclar)
- Sheet selector la multi-sheet xlsx
"""
from __future__ import annotations
import io
import os
import tempfile
import pytest
@pytest.fixture()
def client(monkeypatch):
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "t.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.main import app
from fastapi.testclient import TestClient
with TestClient(app) as c:
yield c
get_settings.cache_clear()
def _make_xlsx_bytes(rows: list[dict]) -> bytes:
"""Construieste un xlsx minimal cu openpyxl pentru fixture teste."""
openpyxl = pytest.importorskip("openpyxl")
wb = openpyxl.Workbook()
ws = wb.active
if not rows:
return b""
headers = list(rows[0].keys())
ws.append(headers)
for row in rows:
ws.append([row.get(h) for h in headers])
buf = io.BytesIO()
wb.save(buf)
return buf.getvalue()
def _make_csv_bytes(rows: list[dict], sep: str = ";") -> bytes:
"""Construieste un CSV minimal pentru fixture teste."""
import csv
import io
buf = io.StringIO()
if not rows:
return b""
writer = csv.DictWriter(buf, fieldnames=list(rows[0].keys()), delimiter=sep)
writer.writeheader()
writer.writerows(rows)
return buf.getvalue().encode("utf-8")
_SAMPLE_ROWS = [
{
"VIN": "WVWZZZ1KZAW000123",
"Nr inmatriculare": "B001TST",
"Data prestatie": "15.06.2026",
"Odometru final": "123456",
"Operatie": "Revizie",
},
{
"VIN": "WVWZZZ1KZAW000456",
"Nr inmatriculare": "B002TST",
"Data prestatie": "16.06.2026",
"Odometru final": "200000",
"Operatie": "Revizie",
},
]
def _seed_op_mapping(client, cod_op: str = "Revizie", cod_prest: str = "OE-1") -> None:
"""Seeda o mapare de operatii cod_op → cod_prestatie via API."""
client.post("/v1/mapari", json={
"cod_op_service": cod_op,
"cod_prestatie": cod_prest,
"auto_send": True,
})
# --------------------------------------------------------------------------- #
# Dashboard #
# --------------------------------------------------------------------------- #
def test_dashboard_contine_drop_zone(client):
"""Dashboard-ul randeaza sectiunea de upload cu drop zone si mesaj warmth."""
r = client.get("/")
assert r.status_code == 200
assert "Primul fisier" in r.text
assert "drop-zone" in r.text
assert "NU se trimite nimic" in r.text
assert "import-section" in r.text
# --------------------------------------------------------------------------- #
# Upload xlsx — mapare noua #
# --------------------------------------------------------------------------- #
def test_upload_xlsx_fara_mapare_arata_formular_mapare(client):
"""Upload xlsx fara mapare salvata → fragment mapare coloane."""
xlsx = _make_xlsx_bytes(_SAMPLE_ROWS)
r = client.post(
"/_import/upload",
files={"file": ("test.xlsx", xlsx, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")},
)
assert r.status_code == 200
# Formular de mapare coloane
assert "Mapare coloane" in r.text
assert "mapare-coloane" in r.text # URL in form action
# Coloanele din fisier apar in formular
assert "VIN" in r.text
assert "Data prestatie" in r.text
# Sugestii fuzzy pentru VIN
assert "vin" in r.text.lower()
def test_upload_csv_fara_mapare_arata_formular_mapare(client):
"""Upload CSV cu separator ; → formular mapare coloane."""
csv_bytes = _make_csv_bytes(_SAMPLE_ROWS)
r = client.post(
"/_import/upload",
files={"file": ("test.csv", csv_bytes, "text/csv")},
)
assert r.status_code == 200
assert "Mapare coloane" in r.text
# --------------------------------------------------------------------------- #
# Salvare mapare coloane → preview #
# --------------------------------------------------------------------------- #
def _upload_and_get_import_id(client, rows=None) -> int:
"""Helper: incarca fisier si extrage import_id din raspuns."""
xlsx = _make_xlsx_bytes(rows or _SAMPLE_ROWS)
r = client.post(
"/_import/upload",
files={"file": ("test.xlsx", xlsx, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")},
)
assert r.status_code == 200
# Extrage import_id din URL-ul form action din raspuns
text = r.text
# Form action contine /_import/{id}/mapare-coloane
import re
m = re.search(r"/_import/(\d+)/mapare-coloane", text)
assert m, f"Nu s-a gasit import_id in raspuns: {text[:500]}"
return int(m.group(1))
def test_salvare_mapare_coloane_arata_preview(client):
"""Dupa salvarea maparii de coloane, raspunsul contine preview-ul."""
# Asigura ca nomenclatorul are OE-1 (seeding automat la init_db)
import_id = _upload_and_get_import_id(client)
r = client.post(
f"/_import/{import_id}/mapare-coloane",
data={
"colname": ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"],
"canon": ["vin", "nr_inmatriculare", "data_prestatie", "odometru_final", "operatie"],
"format_data": "DD.MM.YYYY",
},
)
assert r.status_code == 200
# Preview trebuie sa contina elementele cheie
assert "Preview" in r.text
assert "confirm-form" in r.text
assert "n-confirmat" in r.text
# Rezumat stari
assert "gata de trimis" in r.text or "ok" in r.text
def test_preview_arata_randul_vin(client):
"""Preview contine VIN-ul din fisier."""
import_id = _upload_and_get_import_id(client)
client.post(
f"/_import/{import_id}/mapare-coloane",
data={
"colname": ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"],
"canon": ["vin", "nr_inmatriculare", "data_prestatie", "odometru_final", "operatie"],
"format_data": "DD.MM.YYYY",
},
)
r = client.get(f"/_import/{import_id}/preview")
assert r.status_code == 200
assert "WVWZZZ1KZAW000123" in r.text
# --------------------------------------------------------------------------- #
# Upload cu mapare existenta → preview direct #
# --------------------------------------------------------------------------- #
def test_upload_cu_mapare_existenta_sare_direct_la_preview(client):
"""Al doilea upload cu acelasi antet → preview imediat (mapare retinuta)."""
# Primul upload + salvare mapare
import_id1 = _upload_and_get_import_id(client)
client.post(
f"/_import/{import_id1}/mapare-coloane",
data={
"colname": ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"],
"canon": ["vin", "nr_inmatriculare", "data_prestatie", "odometru_final", "operatie"],
"format_data": "DD.MM.YYYY",
},
)
# Al doilea upload cu acelasi antet
xlsx = _make_xlsx_bytes(_SAMPLE_ROWS)
r = client.post(
"/_import/upload",
files={"file": ("test2.xlsx", xlsx, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")},
)
assert r.status_code == 200
# Trebuie sa ajunga direct la preview, nu la mapare
assert "Preview" in r.text
assert "confirm-form" in r.text
assert "Mapare retinuta aplicata automat" in r.text
# --------------------------------------------------------------------------- #
# Confirmare (gate HARD) #
# --------------------------------------------------------------------------- #
def _setup_preview(client) -> int:
"""Upload + mapare + seeda operatii + intoarce import_id gata de confirmare."""
_seed_op_mapping(client) # "Revizie" → "OE-1"
import_id = _upload_and_get_import_id(client)
client.post(
f"/_import/{import_id}/mapare-coloane",
data={
"colname": ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"],
"canon": ["vin", "nr_inmatriculare", "data_prestatie", "odometru_final", "operatie"],
"format_data": "DD.MM.YYYY",
},
)
return import_id
def test_confirmare_n_corect_pune_in_coada(client):
"""Confirmare cu N corect → randurile ok ajung in coada."""
import_id = _setup_preview(client)
# Compute preview pentru a afla n_ok
r_prev = client.get(f"/_import/{import_id}/preview")
assert r_prev.status_code == 200
# Citeste summary din DB
from app.db import get_connection
conn = get_connection()
try:
batch = conn.execute(
"SELECT ok FROM import_batches WHERE id=?", (import_id,)
).fetchone()
n_ok = batch["ok"]
finally:
conn.close()
assert n_ok > 0, "Asteptat cel putin un rand ok dupa seeding corect"
r = client.post(
f"/_import/{import_id}/confirma",
data={"n_confirmat": str(n_ok), "confirmed_by": "test@test.ro"},
)
assert r.status_code == 200
# Succes → drop zone cu mesaj
assert "S-au pus in coada" in r.text or "prezentari" in r.text
# Sectiunea se reseteaza la drop zone
assert "drop-zone" in r.text
# Verifica ca submissions au fost create
conn2 = get_connection()
try:
n = conn2.execute(
"SELECT COUNT(*) FROM submissions WHERE batch_id=?", (import_id,)
).fetchone()[0]
assert n == n_ok, f"Asteptat {n_ok} submissions, gasit {n}"
finally:
conn2.close()
def test_confirmare_n_gresit_arata_eroare(client):
"""Confirmare cu N gresit → eroare clara, nu enqueue."""
import_id = _setup_preview(client)
client.get(f"/_import/{import_id}/preview") # calculeaza si stocheaza starea
r = client.post(
f"/_import/{import_id}/confirma",
data={"n_confirmat": "99", "confirmed_by": ""},
)
assert r.status_code == 200
# Trebuie sa arate eroare de confirmare sau preview cu eroare
assert (
"difera" in r.text
or "Numarul confirmat" in r.text
or "Niciun rand ok" in r.text
)
# --------------------------------------------------------------------------- #
# Reset #
# --------------------------------------------------------------------------- #
def test_reset_arata_drop_zone_gol(client):
"""GET /_import/reset → drop zone gol fara mesaje."""
r = client.get("/_import/reset")
assert r.status_code == 200
assert "drop-zone" in r.text
assert "Primul fisier" in r.text
assert "import-section" in r.text
# --------------------------------------------------------------------------- #
# Erori upload #
# --------------------------------------------------------------------------- #
def test_upload_fisier_invalid_arata_eroare(client):
"""Upload fisier invalid → mesaj de eroare in drop zone."""
r = client.post(
"/_import/upload",
files={"file": ("test.xlsx", b"not a real xlsx file", "application/octet-stream")},
)
assert r.status_code == 200
# Trebuie sa arate drop zone cu eroare
assert "drop-zone" in r.text or "import-section" in r.text
# Eroare vizibila
assert "nerecunoscut" in r.text.lower() or "invalid" in r.text.lower() or "eroare" in r.text.lower()
def test_upload_fisier_csv_antet_o_coloana_arata_eroare(client):
"""CSV cu o singura coloana reala → header acceptat sau eroare gestionata."""
bad = b"date_fara_header\nval1\nval2\n"
r = client.post(
"/_import/upload",
files={"file": ("test.csv", bad, "text/csv")},
)
# Fie detecteaza header OK fie arata eroare
assert r.status_code == 200
assert "import-section" in r.text
# --------------------------------------------------------------------------- #
# Multi-sheet xlsx #
# --------------------------------------------------------------------------- #
def test_upload_multi_sheet_arata_selector(client):
"""xlsx cu mai multe foi → selector de foaie in drop zone."""
openpyxl = pytest.importorskip("openpyxl")
wb = openpyxl.Workbook()
ws1 = wb.active
ws1.title = "Date"
ws1.append(["VIN", "Data"])
ws1.append(["WVW001", "15.06.2026"])
ws2 = wb.create_sheet("Raport")
ws2.append(["Total"])
ws2.append(["1"])
buf = io.BytesIO()
wb.save(buf)
xlsx = buf.getvalue()
r = client.post(
"/_import/upload",
files={"file": ("multi.xlsx", xlsx, "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")},
)
assert r.status_code == 200
# Trebuie sa arate selector de foi
assert "foi" in r.text.lower() or "sheet" in r.text.lower() or "foaie" in r.text.lower()
# Foile trebuie sa apara ca optiuni
assert "Date" in r.text or "Raport" in r.text
# --------------------------------------------------------------------------- #
# Elemente a11y (D10/D11/D12) #
# --------------------------------------------------------------------------- #
def test_preview_contine_banner_declarant(client):
"""Preview contine bannerul declarant (D12) cu text despre ireversibil."""
import_id = _setup_preview(client)
r = client.get(f"/_import/{import_id}/preview")
assert r.status_code == 200
assert "declarantul" in r.text
assert "ireversibil" in r.text
assert "banner" in r.text
def test_preview_contine_checkboxuri_needs_review(client):
"""Randurile needs_review au checkbox 'verificat' (D11)."""
# Cream un rand cu VIN numeric → needs_review
rows_with_review = [
{
"VIN": "1234567890", # VIN numeric → coercion flag → needs_review
"Nr inmatriculare": "B001TST",
"Data prestatie": "15.06.2026",
"Odometru final": "123456",
"Operatie": "OE-1",
}
]
import_id = _upload_and_get_import_id(client, rows=rows_with_review)
client.post(
f"/_import/{import_id}/mapare-coloane",
data={
"colname": ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"],
"canon": ["vin", "nr_inmatriculare", "data_prestatie", "odometru_final", "operatie"],
"format_data": "DD.MM.YYYY",
},
)
r = client.get(f"/_import/{import_id}/preview")
assert r.status_code == 200
# Checkbox reviewed_rows prezent pentru randul needs_review
assert "reviewed_rows" in r.text
assert "needs_review" in r.text
def test_preview_duplicate_in_file_are_text(client):
"""Randurile duplicate_in_file arata text 'dubla cu randul N' (D10 — nu doar culoare)."""
_seed_op_mapping(client, "Revizie", "OE-1")
# Doua randuri identice → duplicate_in_file
dup_rows = [
{"VIN": "WVWZZZ1KZAW000123", "Nr inmatriculare": "B001TST",
"Data prestatie": "15.06.2026", "Odometru final": "123456", "Operatie": "Revizie"},
{"VIN": "WVWZZZ1KZAW000123", "Nr inmatriculare": "B001TST",
"Data prestatie": "15.06.2026", "Odometru final": "123456", "Operatie": "Revizie"},
]
import_id = _upload_and_get_import_id(client, rows=dup_rows)
client.post(
f"/_import/{import_id}/mapare-coloane",
data={
"colname": ["VIN", "Nr inmatriculare", "Data prestatie", "Odometru final", "Operatie"],
"canon": ["vin", "nr_inmatriculare", "data_prestatie", "odometru_final", "operatie"],
"format_data": "DD.MM.YYYY",
},
)
r = client.get(f"/_import/{import_id}/preview")
assert r.status_code == 200
# Text explicit pentru duplicate_in_file (nu doar culoare — cerinta daltonism D10)
assert "dubla cu randul" in r.text
assert "duplicate_in_file" in r.text

197
tests/test_t16_purjare.py Normal file
View File

@@ -0,0 +1,197 @@
"""Teste T16: job purjare + purge_after SET la insert (OV-5).
Verify:
(a) insert/sent -> purge_after populat (sent+90z).
(b) rand expirat -> sters de tick.
(c) import_rows purjate cu batch-ul (CASCADE).
"""
from __future__ import annotations
import json
import os
import tempfile
import pytest
@pytest.fixture()
def env(monkeypatch):
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "t16.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.db import init_db
init_db()
yield monkeypatch
get_settings.cache_clear()
@pytest.fixture()
def conn(env):
from app.db import get_connection
c = get_connection()
yield c
c.close()
def _insert_submission(conn, account_id=1, status="queued", key_sfx=None):
content = {"vin": "WVWZZZ1KZAW000123", "nr_inmatriculare": "B1",
"data_prestatie": "2026-06-15", "odometru_final": "1",
"prestatii": [{"cod_prestatie": "OE-1"}]}
sfx = key_sfx or os.urandom(4).hex()
cur = conn.execute(
"INSERT INTO submissions (idempotency_key, account_id, status, payload_json) VALUES (?, ?, ?, ?)",
(f"k-{sfx}", account_id, status, json.dumps(content)),
)
return int(cur.lastrowid)
# --- (a) purge_after populat la 'sent' ---
def test_mark_sent_seteaza_purge_after(conn):
"""(a) mark(sent) seteaza purge_after = now + 90 zile."""
import app.worker.__main__ as w
sid = _insert_submission(conn)
w.mark(conn, sid, "sent", rar_status_code=200, id_prezentare=12345)
row = conn.execute("SELECT status, purge_after FROM submissions WHERE id=?", (sid,)).fetchone()
assert row["status"] == "sent"
assert row["purge_after"] is not None, "purge_after trebuie setat la 'sent'"
# purge_after trebuie sa fie in viitor (>= now)
is_future = conn.execute(
"SELECT purge_after > datetime('now') AS ok FROM submissions WHERE id=?", (sid,)
).fetchone()["ok"]
assert is_future, "purge_after trebuie sa fie in viitor"
def test_mark_other_status_nu_seteaza_purge_after(conn):
"""Alte statusuri (error, needs_data) nu seteaza purge_after."""
import app.worker.__main__ as w
sid = _insert_submission(conn)
w.mark(conn, sid, "error", rar_error="test")
row = conn.execute("SELECT purge_after FROM submissions WHERE id=?", (sid,)).fetchone()
assert row["purge_after"] is None, "purge_after nu trebuie setat la 'error'"
# --- (b) purge_expired sterge randurile expirate ---
def test_purge_expired_sterge_sent_expirat(conn):
"""(b) purge_expired sterge submissions cu purge_after < now si status=sent."""
import app.worker.__main__ as w
sid = _insert_submission(conn)
# Simuleaza un rand deja expirat (purge_after in trecut)
conn.execute(
"UPDATE submissions SET status='sent', purge_after=datetime('now', '-1 day') WHERE id=?",
(sid,),
)
stats = w.purge_expired(conn)
assert stats["submissions_purged"] == 1
row = conn.execute("SELECT id FROM submissions WHERE id=?", (sid,)).fetchone()
assert row is None, "randul expirat trebuie sters"
def test_purge_expired_pastreaza_neexpirat(conn):
"""Randul cu purge_after in viitor nu e sters."""
import app.worker.__main__ as w
sid = _insert_submission(conn)
conn.execute(
"UPDATE submissions SET status='sent', purge_after=datetime('now', '+90 days') WHERE id=?",
(sid,),
)
stats = w.purge_expired(conn)
assert stats["submissions_purged"] == 0
row = conn.execute("SELECT id FROM submissions WHERE id=?", (sid,)).fetchone()
assert row is not None, "randul ne-expirat trebuie pastrat"
def test_purge_expired_nu_sterge_queued(conn):
"""submissions in stare queued cu purge_after NULL nu sunt sterse."""
import app.worker.__main__ as w
sid = _insert_submission(conn, status="queued")
stats = w.purge_expired(conn)
assert stats["submissions_purged"] == 0
row = conn.execute("SELECT id FROM submissions WHERE id=?", (sid,)).fetchone()
assert row is not None
# --- (c) import_rows purjate cu batch-ul (CASCADE) ---
def test_import_rows_cascade_cu_batch(conn):
"""(c) Stergerea import_batches sterge import_rows via CASCADE."""
import app.worker.__main__ as w
# Creeaza batch cu purge_after in trecut
cur_batch = conn.execute(
"INSERT INTO import_batches (account_id, filename, status, purge_after) "
"VALUES (1, 'test.xlsx', 'committed', datetime('now', '-1 day'))"
)
batch_id = cur_batch.lastrowid
# Adauga import_rows
conn.execute(
"INSERT INTO import_rows (batch_id, row_index, raw_json, resolved_status) "
"VALUES (?, 0, 'test', 'ok')",
(batch_id,),
)
conn.execute(
"INSERT INTO import_rows (batch_id, row_index, raw_json, resolved_status) "
"VALUES (?, 1, 'test2', 'ok')",
(batch_id,),
)
# Verifica ca avem randuri
n_rows = conn.execute("SELECT COUNT(*) AS n FROM import_rows WHERE batch_id=?", (batch_id,)).fetchone()["n"]
assert n_rows == 2
stats = w.purge_expired(conn)
assert stats["batches_purged"] == 1
# Batch sters
batch = conn.execute("SELECT id FROM import_batches WHERE id=?", (batch_id,)).fetchone()
assert batch is None
# import_rows sterse via CASCADE
n_rows_after = conn.execute("SELECT COUNT(*) AS n FROM import_rows WHERE batch_id=?", (batch_id,)).fetchone()["n"]
assert n_rows_after == 0, "import_rows trebuie sterse odata cu batch-ul (CASCADE)"
def test_import_batch_neexpirat_pastrat(conn):
"""import_batch cu purge_after in viitor nu e sters."""
import app.worker.__main__ as w
cur = conn.execute(
"INSERT INTO import_batches (account_id, filename, status, purge_after) "
"VALUES (1, 'future.xlsx', 'committed', datetime('now', '+90 days'))"
)
batch_id = cur.lastrowid
stats = w.purge_expired(conn)
assert stats["batches_purged"] == 0
batch = conn.execute("SELECT id FROM import_batches WHERE id=?", (batch_id,)).fetchone()
assert batch is not None
def test_import_batch_fara_purge_after_pastrat(conn):
"""import_batch fara purge_after nu e sters (NULL = nu expira)."""
import app.worker.__main__ as w
cur = conn.execute(
"INSERT INTO import_batches (account_id, filename, status) "
"VALUES (1, 'no_purge.xlsx', 'staging')"
)
batch_id = cur.lastrowid
stats = w.purge_expired(conn)
assert stats["batches_purged"] == 0
batch = conn.execute("SELECT id FROM import_batches WHERE id=?", (batch_id,)).fetchone()
assert batch is not None

View File

@@ -0,0 +1,256 @@
"""Teste T1: accounts.rar_creds_enc durabile + worker re-login fallback + gate purjare.
Verify:
(a) Serie web, worker restart (sesiune goala), token expirat -> re-login din accounts -> trimite.
(b) Coada MIXTA API(efemer)+web(durabil): dupa login web, submission-urile API tot se trimit
(purjarea nu le-a rupt prematur).
"""
from __future__ import annotations
import json
import os
import tempfile
import pytest
from cryptography.fernet import Fernet
@pytest.fixture()
def env(monkeypatch):
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "t1.db"))
monkeypatch.setenv("AUTOPASS_CREDS_KEY", Fernet.generate_key().decode())
from app.config import get_settings
from app import crypto
get_settings.cache_clear()
crypto.reset_cache()
from app.db import init_db
init_db()
yield monkeypatch
get_settings.cache_clear()
crypto.reset_cache()
class FakeRar:
"""Stub RarClient pentru teste."""
def __init__(self, settings=None):
self.login_calls = 0
self.closed = False
def login(self, email, password):
self.login_calls += 1
return f"TOK-{email}"
def get_nomenclator(self, token):
return []
def close(self):
self.closed = True
def _insert(conn, account_id=1, creds_enc=None, status="queued", key_suffix=""):
content = {
"vin": "WVWZZZ1KZAW000123", "nr_inmatriculare": "B1",
"data_prestatie": "2026-06-15", "odometru_final": "1",
"prestatii": [{"cod_prestatie": "OE-1"}],
}
suffix = key_suffix or os.urandom(4).hex()
cur = conn.execute(
"INSERT INTO submissions (idempotency_key, account_id, status, payload_json, rar_creds_enc) "
"VALUES (?, ?, ?, ?, ?)",
(f"k-{suffix}", account_id, status, json.dumps(content), creds_enc),
)
return int(cur.lastrowid)
# --- (a) re-login din accounts dupa restart ---
def test_creds_from_account_fallback(env, monkeypatch):
"""Worker re-citeste creds din accounts daca submission n-are creds_enc."""
import app.worker.__main__ as w
from app.crypto import encrypt_creds
from app.db import get_connection
monkeypatch.setattr(w, "RarClient", FakeRar)
conn = get_connection()
try:
enc = encrypt_creds({"email": "web@test.ro", "password": "webpass"})
conn.execute("UPDATE accounts SET rar_creds_enc=? WHERE id=1", (enc,))
# Submission web fara creds_enc (ex: dupa ce s-au purjat)
_insert(conn, account_id=1, creds_enc=None)
# _creds_from_account trebuie sa returneze creds
creds = w._creds_from_account(conn, 1)
assert creds == {"email": "web@test.ro", "password": "webpass"}
finally:
conn.close()
def test_creds_from_account_no_creds(env):
"""Cont fara rar_creds_enc -> None (canal API pur, neatins)."""
import app.worker.__main__ as w
from app.db import get_connection
conn = get_connection()
try:
assert w._creds_from_account(conn, 1) is None
finally:
conn.close()
def test_worker_relogin_dupa_restart(env, monkeypatch):
"""(a) Worker restart: sesiune goala, submission fara creds -> re-login din accounts."""
import app.worker.__main__ as w
from app.crypto import encrypt_creds
from app.db import get_connection
FakeRar.login_calls_total = 0
monkeypatch.setattr(w, "RarClient", FakeRar)
conn = get_connection()
try:
enc = encrypt_creds({"email": "web@test.ro", "password": "webpass"})
conn.execute("UPDATE accounts SET rar_creds_enc=? WHERE id=1", (enc,))
# Submission web fara creds (creds deja purjate de primul login)
_insert(conn, account_id=1, creds_enc=None)
# Sesiune noua (simuleaza restart) — cache gol
sessions = w.AccountSessions(w.get_settings())
assert sessions.get_token(conn, 1, None) is None # fara creds directe
# Creds din account -> login posibil
creds = w._creds_from_account(conn, 1)
assert creds is not None
token = sessions.get_token(conn, 1, creds)
assert token == "TOK-web@test.ro"
finally:
conn.close()
# --- (b) coada MIXTA API+web ---
def test_coada_mixta_api_web(env, monkeypatch):
"""(b) Coada mixta: dupa login web, submission-urile API (efemere) tot se trimit.
Scenariul:
1. S1 = submission API cu creds efemere in submission.rar_creds_enc
2. S2 = submission WEB fara creds (foloseste accounts.rar_creds_enc)
3. Login cu creds S1 -> purjare S1.rar_creds_enc -> OK (worker are token)
4. S2 tot se poate procesa (creds din accounts)
"""
import app.worker.__main__ as w
from app.crypto import encrypt_creds
from app.db import get_connection
monkeypatch.setattr(w, "RarClient", FakeRar)
conn = get_connection()
try:
# Creds durabile pentru contul web
enc_web = encrypt_creds({"email": "web@test.ro", "password": "webpass"})
conn.execute("UPDATE accounts SET rar_creds_enc=? WHERE id=1", (enc_web,))
# S1: canal API cu creds efemere
enc_api = encrypt_creds({"email": "api@test.ro", "password": "apipass"})
s1 = _insert(conn, account_id=1, creds_enc=enc_api, key_suffix="api1")
# S2: canal web fara creds in submission
s2 = _insert(conn, account_id=1, creds_enc=None, key_suffix="web1")
sessions = w.AccountSessions(w.get_settings())
# Procesare S1: login cu creds API -> purjare rar_creds_enc pe TOATE submission-urile contului
creds_s1 = w._creds_for({"creds_enc": enc_api}, w.get_settings())
assert creds_s1 is not None
sessions.get_token(conn, 1, creds_s1) # login + purjare
# Verifica purjarea: S1.rar_creds_enc = NULL acum
row_s1 = conn.execute("SELECT rar_creds_enc FROM submissions WHERE id=?", (s1,)).fetchone()
assert row_s1["rar_creds_enc"] is None, "creds efemere trebuie sterse dupa login"
# S2 nu mai are creds in submission (nici nu a avut); fallback la accounts
creds_s2 = w._creds_for({"creds_enc": None}, w.get_settings()) or w._creds_from_account(conn, 1)
assert creds_s2 == {"email": "web@test.ro", "password": "webpass"}, \
"S2 trebuie sa ia creds din accounts.rar_creds_enc"
# accounts.rar_creds_enc NU a fost sters de purjare
row_acc = conn.execute("SELECT rar_creds_enc FROM accounts WHERE id=1").fetchone()
assert row_acc["rar_creds_enc"] is not None, \
"accounts.rar_creds_enc trebuie sa ramana dupa purjare submissions"
finally:
conn.close()
# --- Endpoint API set/delete rar-creds ---
@pytest.fixture()
def client(env):
from app.main import app
from fastapi.testclient import TestClient
with TestClient(app) as c:
yield c
def test_endpoint_set_rar_creds(client, env):
"""POST /v1/conturi/rar-creds seteaza creds criptate in accounts."""
from app.crypto import decrypt_creds
from app.db import get_connection
r = client.post("/v1/conturi/rar-creds", json={"email": "u@test.ro", "password": "pass123"})
assert r.status_code == 200
assert r.json()["ok"] is True
conn = get_connection()
try:
row = conn.execute("SELECT rar_creds_enc FROM accounts WHERE id=1").fetchone()
assert row["rar_creds_enc"] is not None
creds = decrypt_creds(row["rar_creds_enc"])
assert creds == {"email": "u@test.ro", "password": "pass123"}
finally:
conn.close()
def test_endpoint_delete_rar_creds(client, env):
"""DELETE /v1/conturi/rar-creds sterge creds durabile."""
# Mai intai seteaza
client.post("/v1/conturi/rar-creds", json={"email": "u@test.ro", "password": "pass123"})
# Sterge
r = client.delete("/v1/conturi/rar-creds")
assert r.status_code == 200
assert r.json()["ok"] is True
from app.db import get_connection
conn = get_connection()
try:
row = conn.execute("SELECT rar_creds_enc FROM accounts WHERE id=1").fetchone()
assert row["rar_creds_enc"] is None
finally:
conn.close()
def test_gate_purjare_nu_sterge_accounts(env, monkeypatch):
"""Gate purjare T1: stergerea submissions.rar_creds_enc NU atinge accounts.rar_creds_enc."""
import app.worker.__main__ as w
from app.crypto import encrypt_creds
from app.db import get_connection
monkeypatch.setattr(w, "RarClient", FakeRar)
conn = get_connection()
try:
enc = encrypt_creds({"email": "u@test.ro", "password": "p"})
conn.execute("UPDATE accounts SET rar_creds_enc=? WHERE id=1", (enc,))
_insert(conn, account_id=1, creds_enc=enc)
sessions = w.AccountSessions(w.get_settings())
sessions.get_token(conn, 1, {"email": "u@test.ro", "password": "p"})
# accounts.rar_creds_enc trebuie sa fie intact
row = conn.execute("SELECT rar_creds_enc FROM accounts WHERE id=1").fetchone()
assert row["rar_creds_enc"] is not None, \
"gate purjare: accounts.rar_creds_enc trebuie sa ramana intact"
finally:
conn.close()

202
tests/test_t6_auto_send.py Normal file
View File

@@ -0,0 +1,202 @@
"""Teste T6: gate auto_send pe coduri nou-mapate (OV-1).
Verify:
(a) cod nou-mapat cu auto_send=0 -> nu auto-send, review manual.
(b) REGRESIE: mapare existenta cu auto_send=1 tot se requeue ca azi.
"""
from __future__ import annotations
import json
import os
import tempfile
import pytest
@pytest.fixture()
def env(monkeypatch):
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "t6.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.db import init_db
init_db()
yield monkeypatch
get_settings.cache_clear()
@pytest.fixture()
def conn(env):
from app.db import get_connection
c = get_connection()
yield c
c.close()
@pytest.fixture()
def client(env):
from app.main import app
from fastapi.testclient import TestClient
with TestClient(app) as c:
yield c
def _insert_needs_mapping(conn, account_id=1, cod_op="ITP-CHECK"):
content = {
"vin": "WVWZZZ1KZAW000123", "nr_inmatriculare": "B1",
"data_prestatie": "2026-06-15", "odometru_final": "123456",
"prestatii": [{"cod_op_service": cod_op, "denumire": "Inspectie tehnica"}],
}
cur = conn.execute(
"INSERT INTO submissions (idempotency_key, account_id, status, payload_json) VALUES (?, ?, ?, ?)",
(f"k-{os.urandom(4).hex()}", account_id, "needs_mapping", json.dumps(content)),
)
return int(cur.lastrowid)
def _add_mapping(conn, account_id=1, cod_op="ITP-CHECK", cod_prestatie="OE-1", auto_send=True):
conn.execute(
"INSERT OR REPLACE INTO nomenclator_rar (cod_prestatie, nume_prestatie) VALUES (?, ?)",
(cod_prestatie, "Operatie test"),
)
conn.execute(
"INSERT OR REPLACE INTO operations_mapping (account_id, cod_op_service, cod_prestatie, auto_send) "
"VALUES (?, ?, ?, ?)",
(account_id, cod_op, cod_prestatie, 1 if auto_send else 0),
)
# --- load_mapping_meta ---
def test_load_mapping_meta_returns_auto_send(conn):
from app.mapping import load_mapping_meta
_add_mapping(conn, cod_op="ITP-1", cod_prestatie="OE-1", auto_send=True)
_add_mapping(conn, cod_op="ITP-2", cod_prestatie="OE-2", auto_send=False)
conn.execute("INSERT OR IGNORE INTO nomenclator_rar (cod_prestatie, nume_prestatie) VALUES ('OE-2', 'Test2')")
meta = load_mapping_meta(conn, 1)
assert meta["ITP-1"]["auto_send"] is True
assert meta["ITP-2"]["auto_send"] is False
# --- has_no_auto_send ---
def test_has_no_auto_send_detecteaza_false(conn):
from app.mapping import has_no_auto_send
mapping_meta = {
"ITP-1": {"cod_prestatie": "OE-1", "auto_send": False},
}
resolved = [{"cod_op_service": "ITP-1", "cod_prestatie": "OE-1"}]
assert has_no_auto_send(resolved, mapping_meta) is True
def test_has_no_auto_send_trece_cu_true(conn):
from app.mapping import has_no_auto_send
mapping_meta = {
"ITP-1": {"cod_prestatie": "OE-1", "auto_send": True},
}
resolved = [{"cod_op_service": "ITP-1", "cod_prestatie": "OE-1"}]
assert has_no_auto_send(resolved, mapping_meta) is False
def test_has_no_auto_send_direct_cod_prestatie(conn):
"""Item cu cod_prestatie direct (fara cod_op_service) nu e afectat de auto_send."""
from app.mapping import has_no_auto_send
mapping_meta = {}
resolved = [{"cod_prestatie": "OE-1"}]
assert has_no_auto_send(resolved, mapping_meta) is False
# --- reresolve_account cu auto_send=0 ---
def test_reresolve_auto_send_zero_nu_requeue(conn):
"""(a) cod nou-mapat cu auto_send=0 -> ramane needs_mapping (nu trece pe queued)."""
from app.mapping import reresolve_account
sid = _insert_needs_mapping(conn, cod_op="ITP-CHECK")
_add_mapping(conn, cod_op="ITP-CHECK", cod_prestatie="OE-1", auto_send=False)
stats = reresolve_account(conn, 1)
assert stats["review_manual"] == 1
assert stats["requeued"] == 0
row = conn.execute("SELECT status, rar_error FROM submissions WHERE id=?", (sid,)).fetchone()
assert row["status"] == "needs_mapping"
err = json.loads(row["rar_error"])
assert "auto_send" in err
def test_reresolve_auto_send_unu_requeue(conn):
"""(b) REGRESIE: mapare cu auto_send=1 tot se requeue ca azi."""
from app.mapping import reresolve_account
sid = _insert_needs_mapping(conn, cod_op="ITP-CHECK")
_add_mapping(conn, cod_op="ITP-CHECK", cod_prestatie="OE-1", auto_send=True)
stats = reresolve_account(conn, 1)
assert stats["requeued"] == 1
assert stats["review_manual"] == 0
row = conn.execute("SELECT status FROM submissions WHERE id=?", (sid,)).fetchone()
assert row["status"] == "queued"
# --- POST /v1/prezentari cu auto_send=0 ---
def _body_with_op(cod_op="ITP-CHECK"):
return {
"rar_credentials": {"email": "x@y.ro", "password": "s"},
"prezentari": [{
"vin": "WVWZZZ1KZAW000123",
"nr_inmatriculare": "B999TST",
"data_prestatie": "2026-06-15",
"odometru_final": "123456",
"prestatii": [{"cod_op_service": cod_op, "denumire": "Test"}],
}],
}
def test_post_auto_send_zero_nu_queued(client, env):
"""(a) Via API: cod nou-mapat cu auto_send=0 -> nu 'queued', review manual."""
from app.db import get_connection
conn2 = get_connection()
try:
_add_mapping(conn2, cod_op="ITP-X", cod_prestatie="OE-1", auto_send=False)
finally:
conn2.close()
r = client.post("/v1/prezentari", json=_body_with_op("ITP-X"))
assert r.status_code == 200
status = r.json()["results"][0]["status"]
assert status != "queued", f"auto_send=0 nu trebuie sa fie queued, e: {status}"
assert status == "needs_mapping"
def test_post_auto_send_unu_queued(client, env):
"""(b) REGRESIE: mapare existenta cu auto_send=1 -> queued ca azi."""
from app.db import get_connection
conn2 = get_connection()
try:
_add_mapping(conn2, cod_op="ITP-Y", cod_prestatie="OE-1", auto_send=True)
finally:
conn2.close()
r = client.post("/v1/prezentari", json=_body_with_op("ITP-Y"))
assert r.status_code == 200
status = r.json()["results"][0]["status"]
assert status == "queued", f"auto_send=1 trebuie queued, e: {status}"
def test_post_cod_prestatie_direct_queued(client):
"""Cod RAR direct (fara cod_op_service) -> queued indiferent de mapping."""
body = {
"rar_credentials": {"email": "x@y.ro", "password": "s"},
"prezentari": [{
"vin": "WVWZZZ1KZAW000123",
"nr_inmatriculare": "B999TST",
"data_prestatie": "2026-06-15",
"odometru_final": "123456",
"prestatii": [{"cod_prestatie": "OE-1"}],
}],
}
r = client.post("/v1/prezentari", json=body)
assert r.status_code == 200
assert r.json()["results"][0]["status"] == "queued"

View File

@@ -0,0 +1,197 @@
"""Teste T7: batch_id/row_index scope reresolve_account (R1 INCHIS).
Verify:
(a) salvare mapare in batch A NU trimite randuri din batch B / feed API.
(b) canal API (batch_id NULL) tot se re-rezolva ca azi (regresie).
"""
from __future__ import annotations
import json
import os
import tempfile
import pytest
@pytest.fixture()
def env(monkeypatch):
tmp = tempfile.mkdtemp()
monkeypatch.setenv("AUTOPASS_DB_PATH", os.path.join(tmp, "t7.db"))
from app.config import get_settings
get_settings.cache_clear()
from app.db import init_db
init_db()
yield monkeypatch
get_settings.cache_clear()
@pytest.fixture()
def conn(env):
from app.db import get_connection
c = get_connection()
yield c
c.close()
def _insert_batch(conn, account_id=1):
"""Creeaza un import_batch si returneaza id-ul."""
cur = conn.execute(
"INSERT INTO import_batches (account_id, filename, status) VALUES (?, ?, 'staging')",
(account_id, "test.xlsx"),
)
return int(cur.lastrowid)
def _insert_submission(conn, account_id=1, batch_id=None, cod_op="ITP-1", key_sfx=None):
"""Insereaza un submission needs_mapping (cu sau fara batch)."""
content = {
"vin": "WVWZZZ1KZAW000123", "nr_inmatriculare": "B1",
"data_prestatie": "2026-06-15", "odometru_final": "123456",
"prestatii": [{"cod_op_service": cod_op, "denumire": "Test"}],
}
sfx = key_sfx or os.urandom(4).hex()
if batch_id is not None:
cur = conn.execute(
"INSERT INTO submissions (idempotency_key, account_id, status, payload_json, batch_id) "
"VALUES (?, ?, 'needs_mapping', ?, ?)",
(f"k-{sfx}", account_id, json.dumps(content), batch_id),
)
else:
cur = conn.execute(
"INSERT INTO submissions (idempotency_key, account_id, status, payload_json) "
"VALUES (?, ?, 'needs_mapping', ?)",
(f"k-{sfx}", account_id, json.dumps(content)),
)
return int(cur.lastrowid)
def _add_mapping(conn, account_id=1, cod_op="ITP-1", cod_prestatie="OE-1", auto_send=True):
conn.execute(
"INSERT OR IGNORE INTO nomenclator_rar (cod_prestatie, nume_prestatie) VALUES (?, ?)",
(cod_prestatie, "Test operatie"),
)
conn.execute(
"INSERT OR REPLACE INTO operations_mapping (account_id, cod_op_service, cod_prestatie, auto_send) "
"VALUES (?, ?, ?, ?)",
(account_id, cod_op, cod_prestatie, 1 if auto_send else 0),
)
# --- Scoping ---
def test_reresolve_batch_specific_nu_atinge_alt_batch(conn):
"""(a) reresolve_account cu batch_id=A nu atinge randuri din batch_id=B."""
from app.mapping import reresolve_account
batch_a = _insert_batch(conn)
batch_b = _insert_batch(conn)
sid_a = _insert_submission(conn, batch_id=batch_a, cod_op="ITP-1", key_sfx="a1")
sid_b = _insert_submission(conn, batch_id=batch_b, cod_op="ITP-1", key_sfx="b1")
_add_mapping(conn, cod_op="ITP-1", cod_prestatie="OE-1", auto_send=True)
# Re-rezolva NUMAI batch_a
stats = reresolve_account(conn, 1, batch_id=batch_a)
assert stats["requeued"] == 1
# Batch B nemodificat
row_a = conn.execute("SELECT status FROM submissions WHERE id=?", (sid_a,)).fetchone()
row_b = conn.execute("SELECT status FROM submissions WHERE id=?", (sid_b,)).fetchone()
assert row_a["status"] == "queued", "batch A trebuie requeued"
assert row_b["status"] == "needs_mapping", "batch B trebuie sa ramana needs_mapping"
def test_reresolve_fara_batch_nu_atinge_batches(conn):
"""(a) reresolve fara batch (POST /v1/mapari) NU atinge batch submissions."""
from app.mapping import reresolve_account
batch_a = _insert_batch(conn)
sid_batch = _insert_submission(conn, batch_id=batch_a, cod_op="ITP-1", key_sfx="ba")
sid_api = _insert_submission(conn, batch_id=None, cod_op="ITP-1", key_sfx="ap")
_add_mapping(conn, cod_op="ITP-1", cod_prestatie="OE-1", auto_send=True)
# Fara batch (cum apeleaza POST /v1/mapari)
stats = reresolve_account(conn, 1)
assert stats["requeued"] == 1 # numai API canal
row_batch = conn.execute("SELECT status FROM submissions WHERE id=?", (sid_batch,)).fetchone()
row_api = conn.execute("SELECT status FROM submissions WHERE id=?", (sid_api,)).fetchone()
assert row_batch["status"] == "needs_mapping", "batch submission NU trebuie atins de reresolve global"
assert row_api["status"] == "queued", "API canal trebuie requeued"
def test_reresolve_canal_api_regresie(conn):
"""(b) Canal API (batch_id NULL) tot se re-rezolva ca azi."""
from app.mapping import reresolve_account
# Doua submission-uri API fara batch
sid1 = _insert_submission(conn, batch_id=None, cod_op="ITP-2", key_sfx="r1")
sid2 = _insert_submission(conn, batch_id=None, cod_op="ITP-2", key_sfx="r2")
_add_mapping(conn, cod_op="ITP-2", cod_prestatie="OE-1", auto_send=True)
stats = reresolve_account(conn, 1) # fara batch — re-rezolva tot API
assert stats["requeued"] == 2
for sid in (sid1, sid2):
row = conn.execute("SELECT status FROM submissions WHERE id=?", (sid,)).fetchone()
assert row["status"] == "queued"
def test_reresolve_batch_explicit_nu_atinge_api(conn):
"""Batch explicit: nu atinge feed API (batch_id IS NULL)."""
from app.mapping import reresolve_account
batch_a = _insert_batch(conn)
sid_batch = _insert_submission(conn, batch_id=batch_a, cod_op="ITP-3", key_sfx="ba3")
sid_api = _insert_submission(conn, batch_id=None, cod_op="ITP-3", key_sfx="ap3")
_add_mapping(conn, cod_op="ITP-3", cod_prestatie="OE-1", auto_send=True)
stats = reresolve_account(conn, 1, batch_id=batch_a)
assert stats["requeued"] == 1
row_batch = conn.execute("SELECT status FROM submissions WHERE id=?", (sid_batch,)).fetchone()
row_api = conn.execute("SELECT status FROM submissions WHERE id=?", (sid_api,)).fetchone()
assert row_batch["status"] == "queued"
assert row_api["status"] == "needs_mapping", "API canal nu trebuie atins de reresolve batch-specific"
def test_submissions_au_batch_id_si_row_index(conn):
"""Schema: submissions.batch_id si .row_index exista si se pot seta."""
batch_id = _insert_batch(conn)
conn.execute(
"INSERT INTO submissions (idempotency_key, account_id, status, payload_json, batch_id, row_index) "
"VALUES (?, ?, 'queued', '{}', ?, ?)",
("k-test-bi", 1, batch_id, 5),
)
row = conn.execute("SELECT batch_id, row_index FROM submissions WHERE idempotency_key='k-test-bi'").fetchone()
assert row["batch_id"] == batch_id
assert row["row_index"] == 5
def test_reresolve_multiple_batches_izolate(conn):
"""R1 INCHIS: 3 batches, fiecare re-rezolvat independent."""
from app.mapping import reresolve_account
batches = [_insert_batch(conn) for _ in range(3)]
sids = {
b: _insert_submission(conn, batch_id=b, cod_op="ITP-4", key_sfx=f"mb{i}")
for i, b in enumerate(batches)
}
_add_mapping(conn, cod_op="ITP-4", cod_prestatie="OE-1", auto_send=True)
# Re-rezolva batch 0, verifica ca 1 si 2 nu sunt atinse
reresolve_account(conn, 1, batch_id=batches[0])
statuses = {
b: conn.execute("SELECT status FROM submissions WHERE id=?", (sid,)).fetchone()["status"]
for b, sid in sids.items()
}
assert statuses[batches[0]] == "queued"
assert statuses[batches[1]] == "needs_mapping"
assert statuses[batches[2]] == "needs_mapping"

View File

@@ -30,9 +30,9 @@ from app.config import ROOT
from app.db import get_connection, init_db
from app.mapping import DEFAULT_ACCOUNT_ID
# DBF-urile vin din arhiva ROAAUTO din radacina repo-ului.
MAPARE_DBF = ROOT / "mapare_prestatii.DBF"
PREST_DBF = ROOT / "prestatii_rar.DBF"
# DBF-urile vin din arhiva ROAAUTO (legacy VFP), mutata in legacy-vfp/.
MAPARE_DBF = ROOT / "legacy-vfp" / "mapare_prestatii.DBF"
PREST_DBF = ROOT / "legacy-vfp" / "prestatii_rar.DBF"
# Language driver al DBF-urilor = 0x03 (Windows ANSI / cp1252). Diacriticele
# scrise ca literal '?' sunt in sursa, nu un artefact de encoding.

83
tools/rar_finalizate.py Normal file
View File

@@ -0,0 +1,83 @@
#!/usr/bin/env python3
"""Listeaza prezentarile finalizate la RAR (verificare end-to-end: au ajuns?).
Face login cu credentialele din settings.xml (blocul <test> sau <production>,
in functie de AUTOPASS_RAR_ENV) si afiseaza ce e inregistrat la RAR. Folosit ca
sa confirmi ca prezentarile trimise de worker au ajuns efectiv: compari
`id_prezentare` din coada locala (status='sent') cu `id`-urile de aici.
Utilizare:
AUTOPASS_RAR_ENV=test python3 -m tools.rar_finalizate
./start.sh test finalizate
"""
from __future__ import annotations
import sys
import xml.etree.ElementTree as ET
from app.config import ROOT, get_settings
from app.rar_client import RarClient, RarError
def _creds_for_env(env: str) -> dict | None:
"""Citeste credentialele pentru mediu (<test> / <production>) din settings.xml."""
path = ROOT / "settings.xml"
if not path.exists():
return None
block = "production" if env == "prod" else "test"
try:
root = ET.parse(path).getroot()
node = root.find(f"./{block}/credentials")
if node is None:
return None
email = (node.findtext("email") or "").strip()
password = (node.findtext("password") or "").strip()
if not email or not password or email.startswith("EMAIL_"):
return None
return {"email": email, "password": password}
except ET.ParseError:
return None
def main() -> int:
settings = get_settings()
env = settings.rar_env
creds = _creds_for_env(env)
if not creds:
print(
f"Lipsesc credentialele <{'production' if env == 'prod' else 'test'}> in settings.xml.\n"
"Copiaza settings.xml.example -> settings.xml si completeaza-le.",
file=sys.stderr,
)
return 2
print(f"[finalizate] login RAR ({env}) ca {creds['email']} ...")
rar = RarClient(settings)
try:
token = rar.login(creds["email"], creds["password"])
items = rar.get_finalizate(token)
except RarError as exc:
print(f"[finalizate] eroare RAR: {exc}", file=sys.stderr)
return 1
finally:
rar.close()
if not items:
print("[finalizate] RAR nu a intors nicio prezentare finalizata.")
return 0
print(f"[finalizate] {len(items)} prezentari inregistrate la RAR:\n")
print(f"{'id':>8} {'VIN':<18} {'data':<12} {'odometru':>10}")
print("-" * 54)
for it in items:
idp = it.get("id", "")
vin = (it.get("vin") or it.get("serieSasiu") or "")[:18]
data = it.get("dataPrestatie") or ""
odo = it.get("odometruFinal") or it.get("odometru") or ""
print(f"{str(idp):>8} {vin:<18} {str(data):<12} {str(odo):>10}")
return 0
if __name__ == "__main__":
sys.exit(main())