romfast/rar-autopass
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: curatare agresiva comentarii — scoatere referinte US/PRD din cod si template-uri

Browse Source 
Eliminat zgomotul de trasabilitate (US-xxx, PRD x.x, Rn, OV-x, Tn, decizii/naratiune
istorica) din 41 fisiere app/ + template-uri. Pastrate comentariile care documenteaza
invarianti si logica ne-evidenta (idempotenta/hash, reconciliere anti-duplicat, RAR 500
esec definitiv, creds per cont, WAF User-Agent, 422 fara echo de parola, scope NULL->1),
curatate doar de tokeni.

Verificare: pentru cele 27 module .py curatate, structura de cod (tokeni non-comentariu/
non-string) e IDENTICA fata de HEAD -> doar comentarii/docstring-uri schimbate. Singura
schimbare de cod e in tests/test_web_responsive.py (scos 3 assert pe markeri US-006/007/008,
inlocuite de asertiunile structurale alaturate). 0 tokeni US/PRD reziduali in app/.
Regresie: 896 passed, 1 deselected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Agent
2026-06-25 21:44:24 +00:00
parent f05fe5b221
commit 4a2afc68bf
43 changed files with 547 additions and 649 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
app/api/v1/import_router.py
View File

@@ -8,14 +8,13 @@ Endpointuri:
POST /v1/import/{id}/commit — gate HARD + enqueue randuri ok + log atestare
GET /v1/import/{id}/export-failed — CSV cu randuri esuate (needs_data/needs_mapping/needs_review)
Reguli cheie (plan §3.1-3.4, §12):
- Issue 6: scrieri bulk in tranzactie explicita BEGIN IMMEDIATE...COMMIT + executemany.
- Eng#5: already_sent lookup BATCH (IN chunk ~900), nu N+1.
- OV-3: duplicate_in_file EXCLUSIV la preview/commit. NU atinge reconcile.py/worker.
- Issue 1 (TOCTOU): commit per-rand cu ON CONFLICT(idempotency_key) DO NOTHING.
- Issue 5a: import_rows.raw_json CRIPTAT Fernet.
- Issue 5b: fuzzy coloane refoloseste mapping.normalize_for_match (DRY).
- T4/D3: drift semnatura coloane -> NU aplica orb, cere re-confirmare.
Reguli cheie:
- Scrieri bulk in tranzactie explicita BEGIN IMMEDIATE...COMMIT + executemany.
- already_sent lookup BATCH (IN chunk ~900), nu N+1.
- duplicate_in_file EXCLUSIV la preview/commit. NU atinge reconcile.py/worker.
- TOCTOU: commit per-rand cu ON CONFLICT(idempotency_key) DO NOTHING.
- import_rows.raw_json CRIPTAT Fernet.
- Drift semnatura coloane -> NU aplica orb, cere re-confirmare.
"""
from __future__ import annotations
@@ -60,7 +59,7 @@ router = APIRouter(prefix="/v1/import", tags=["import"])
# Marimea maxima a unui chunk pentru IN(...) SQLite (limite SQLite ~999)
_IN_CHUNK = 900
# Campuri canonice si sinonimele lor pentru sugestie fuzzy coloane (Issue 5b/Eng#4)
# Campuri canonice si sinonimele lor pentru sugestie fuzzy coloane
_CANONICAL_SYNONYMS: dict[str, list[str]] = {
"vin": ["VIN", "Serie sasiu", "Sasiu", "Serie", "Numar sasiu", "Nr sasiu", "Chassis"],
"nr_inmatriculare": ["Nr inmatriculare", "Numar inmatriculare", "Numar auto", "Nr auto", "Numar", "Nr"],
@@ -93,7 +92,7 @@ def _fuzzy_suggest_column(
) -> list[dict]:
"""Sugereaza campuri canonice pentru o coloana din fisier.
Refoloseste normalize_for_match + rapidfuzz.fuzz.token_sort_ratio (Issue 5b/Eng#4).
Refoloseste normalize_for_match + rapidfuzz.fuzz.token_sort_ratio.
Intoarce [{camp_canonic, score}] sortat descrescator.
"""
from rapidfuzz import fuzz, process
@@ -140,10 +139,10 @@ def _resolve_row_for_preview(
errors: lista erori validare
flags: motive needs_review
`override` (3.6, Approach B): patch CANONIC editat in preview, aplicat ULTIMUL
peste valorile mapate (dupa `json_mapare` si canonicalizare). Permite corectarea
unei valori sau completarea unui camp a carui coloana LIPSESTE din fisier, fara
sa atinga `raw_json`/idempotency.
`override`: patch CANONIC editat in preview, aplicat ULTIMUL peste valorile
mapate (dupa `json_mapare` si canonicalizare). Permite corectarea unei valori
sau completarea unui camp a carui coloana LIPSESTE din fisier, fara sa atinga
`raw_json`/idempotency.
"""
# Aplica maparea de coloane
mapped: dict[str, Any] = {}
@@ -151,7 +150,7 @@ def _resolve_row_for_preview(
if col_fisier in raw_row and camp_canonic:
mapped[camp_canonic] = raw_row[col_fisier]
# Detectie coloane cu formule (Issue 3) — nu blocheaza, dar adauga flag
# Detectie coloane cu formule — nu blocheaza, dar adauga flag
formula_flag: list[str] = []
for col_fisier, camp_canonic in json_mapare.items():
if col_fisier in formula_columns:
@@ -186,7 +185,7 @@ def _resolve_row_for_preview(
denumire = str(denumire_val).strip() if denumire_val not in (None, "") else str(operatie_val)
mapped["prestatii"] = [{"cod_op_service": str(operatie_val), "denumire": denumire}]
# Canonicalizare (T9): normalizeaza VIN/nr/odometru
# Canonicalizare: normalizeaza VIN/nr/odometru
canon = canonicalize_row(mapped)
mapped.update({
"vin": canon["vin"],
@@ -194,7 +193,7 @@ def _resolve_row_for_preview(
"odometru_final": canon["odometru_final"],
})
# Override editat in preview (3.6) — aplicat ULTIMUL, peste valorile mapate +
# Override editat in preview — aplicat ULTIMUL, peste valorile mapate +
# canonicalizate. Valorile din override sunt deja canonice (vezi _merge_override).
if override:
mapped.update(override)
@@ -230,7 +229,7 @@ def _resolve_row_for_preview(
"flags": all_flags,
}
# auto_send gate (T6/OV-1)
# auto_send gate
if has_no_auto_send(resolved, mapping_meta):
return {
"resolved_status": "needs_mapping",
@@ -261,8 +260,8 @@ def _build_idempotency_key(account_id: int | None, resolved: dict[str, Any]) ->
return build_key(account_id, canon)
# Campuri de continut editabile in preview (3.6). Operatia/codul RAR NU se editeaza
# aici (raman in panoul de mapare) — vezi Non-Goals din PRD 3.6.
# Campuri de continut editabile in preview. Operatia/codul RAR NU se editeaza
# aici (raman in panoul de mapare).
EDIT_FIELDS = ("vin", "nr_inmatriculare", "data_prestatie", "odometru_initial", "odometru_final")
@@ -350,7 +349,7 @@ def apply_row_override(
def _already_sent_lookup(conn, account_id: int, keys: list[str]) -> dict[str, dict]:
"""Cauta cheile de idempotenta in submissions (batch, nu N+1 — Eng#5).
"""Cauta cheile de idempotenta in submissions (batch, nu N+1).
Intoarce {idempotency_key: {id, id_prezentare, created_at}} pentru cheile gasite.
"""
@@ -371,7 +370,7 @@ def _already_sent_lookup(conn, account_id: int, keys: list[str]) -> dict[str, di
"id_prezentare": r["id_prezentare"],
"created_at": r["created_at"],
}
# Dual-lookup pentru chei legacy (OV-2: chei vechi cu account_id=None)
# Dual-lookup pentru chei legacy (chei vechi cu account_id=None)
legacy_keys_needed = [k for k in chunk if k not in found]
if legacy_keys_needed:
lph = ",".join("?" * len(legacy_keys_needed))
@@ -403,8 +402,7 @@ async def upload_import(
"""Upload fisier xlsx/csv -> staging in import_batches/import_rows.
Nu trimite nimic la RAR. Intoarce {import_id, columns, sample_rows, sheets?}.
PII (raw_json) criptat Fernet la rest (Issue 5a).
Scrieri bulk in tranzactie explicita (Issue 6).
PII (raw_json) criptat Fernet la rest. Scrieri bulk in tranzactie explicita.
"""
acct = account_or_default(account_id)
data = await file.read()
@@ -468,7 +466,7 @@ async def upload_import(
try:
sig = _signature(parsed.columns)
# Issue 6: tranzactie explicita BEGIN IMMEDIATE + executemany
# Tranzactie explicita BEGIN IMMEDIATE + executemany
conn.execute("BEGIN IMMEDIATE")
try:
# Insert import_batches
@@ -482,7 +480,7 @@ async def upload_import(
# Insert import_rows bulk (executemany) cu PII criptat
row_params = []
for i, row_dict in enumerate(parsed.rows):
raw_json_enc = encrypt_creds(row_dict) # Criptat Fernet (Issue 5a)
raw_json_enc = encrypt_creds(row_dict) # Criptat Fernet
row_params.append((batch_id, i, raw_json_enc, "pending", None))
conn.executemany(
@@ -506,11 +504,8 @@ async def upload_import(
# Sample rows (primele 3, fara PII)
sample = parsed.rows[:3]
# Persistam metadata parsedata (coercion_flags, date_col_format, formula_columns)
# in import_batches pentru refolosire la preview (stocam ca JSON in 'status' nu e OK,
# ci ca metadate suplimentare — le stocam intr-un rand separat sau returnam direct)
# Solutie: le returnam in raspuns; preview-ul le va recalcula din raw_json deja stocat
# SAU le stocam ca un camp extra. Cel mai simplu: stocam coloanele in batch.
# Metadata parsata (coercion_flags etc.) se intoarce in raspuns; preview-ul
# o recalculeaza din raw_json deja stocat.
conn.execute(
"UPDATE import_batches SET ok=?, needs_review=? WHERE id=?",
(0, len(parsed.coercion_flags), batch_id),
@@ -532,7 +527,7 @@ async def upload_import(
result["column_mapping"] = json.loads(existing_mapping["json_mapare"])
result["format_data"] = existing_mapping["format_data"]
else:
# Sugestii fuzzy per coloana (Issue 5b: refoloseste normalize_for_match)
# Sugestii fuzzy per coloana
suggestions: dict[str, list[dict]] = {}
for col in parsed.columns:
sugg = _fuzzy_suggest_column(col, limit=3)
@@ -676,7 +671,7 @@ def save_column_mapping(
# --------------------------------------------------------------------------- #
# GET /v1/import/{id}/preview — 6 stari per rand (T2 + T11) #
# GET /v1/import/{id}/preview — 6 stari per rand #
# --------------------------------------------------------------------------- #
@router.get("/{import_id}/preview")
@@ -686,8 +681,8 @@ def preview_import(
) -> dict:
"""Preview 6 stari per rand: ok/needs_mapping/needs_data/needs_review/already_sent/duplicate_in_file.
Nu enqueue-aza nimic. Already_sent = lookup batch (Eng#5). Duplicate_in_file = intra-batch
collision (OV-3: EXCLUSIV aici, NU in reconcile.py/worker).
Nu enqueue-aza nimic. Already_sent = lookup batch. Duplicate_in_file = intra-batch
collision (EXCLUSIV aici, NU in reconcile.py/worker).
"""
acct = account_or_default(account_id)
conn = get_connection()
@@ -708,7 +703,7 @@ def preview_import(
if not raw_rows_db:
return {"rows": [], "summary": {}}
# Decripteaza si reconstruieste randurile + override-urile editate (3.6)
# Decripteaza si reconstruieste randurile + override-urile editate
rows: list[dict] = []
overrides: list[dict] = []
for r in raw_rows_db:
@@ -747,22 +742,18 @@ def preview_import(
json_mapare: dict[str, str] = json.loads(mapping_row["json_mapare"])
format_data = mapping_row["format_data"]
# Incarca maparea de operatii o singura data (Eng#5: load_mapping o singura data)
# Incarca maparea de operatii o singura data
mapping_meta = load_mapping_meta(conn, acct)
mapping = {op: meta["cod_prestatie"] for op, meta in mapping_meta.items()}
# T2: validare nomenclator + reguli text incarcate O DATA, inainte de bucla pe randuri.
# Validare nomenclator + reguli text incarcate O DATA, inainte de bucla pe randuri.
valid_codes = load_nomenclator_codes(conn) or None
text_rules = load_text_rules(conn, acct)
# Reconstruieste parsed info (coercion_flags si date_col_format) din datele stocate
# Nota: import_rows stocheaza raw_json dupa coercion (din parse_file)
# Recalculam flags din valorile stocate (coercion_flags nu e stocat separat)
# Vom folosi o detectie simpla: VIN-uri care par numerice si odometru float
# Recalculam coercion_flags din valorile stocate (nu sunt persistate separat):
# detectie simpla de VIN numeric.
coercion_flags_map: dict[int, list[str]] = {}
# Detectam din valorile stocate
for i, row_dict in enumerate(rows):
flags = []
# Detectam VIN numeric: daca valoarea a fost stocata si arata ca numar
for col_f, camp_c in json_mapare.items():
if camp_c == "vin":
vin_val = row_dict.get(col_f)
@@ -830,11 +821,11 @@ def preview_import(
"idempotency_key": key,
})
# Already_sent: batch lookup (Eng#5 — nu N+1)
# Already_sent: batch lookup (nu N+1)
unique_keys = list(set(keys_for_lookup))
already_sent_map = _already_sent_lookup(conn, account_id, unique_keys)
# Duplicate_in_file (OV-3): detectie coliziuni intra-batch
# Duplicate_in_file: detectie coliziuni intra-batch.
# Grupam pe cheie de idempotenta: >1 rand cu aceeasi cheie = duplicate
key_to_indices: dict[str, list[int]] = {}
for row in preview_rows:
@@ -857,7 +848,7 @@ def preview_import(
row["already_sent_info"] = sent_info
continue
# Duplicate_in_file (OV-3): >1 rand cu aceeasi cheie in ACELASI fisier
# Duplicate_in_file: >1 rand cu aceeasi cheie in ACELASI fisier
indices_with_same_key = key_to_indices.get(k, [])
if len(indices_with_same_key) > 1 and row["resolved_status"] in ("ok", "needs_review", "needs_data"):
others = [idx for idx in indices_with_same_key if idx != row["row_index"]]
@@ -911,7 +902,7 @@ def preview_import(
# --------------------------------------------------------------------------- #
# POST /v1/import/{id}/commit — gate HARD + enqueue + log atestare (T5+T12) #
# POST /v1/import/{id}/commit — gate HARD + enqueue + log atestare #
# --------------------------------------------------------------------------- #
class CommitIn(BaseModel):
@@ -929,9 +920,9 @@ def commit_import(
req: CommitIn,
account_id: int = Depends(resolve_account_id),
) -> dict:
"""Gate HARD confirmare + enqueue randuri ok + log atestare (T5+T12).
"""Gate HARD confirmare + enqueue randuri ok + log atestare.
TOCTOU (Issue 1): INSERT per-rand cu ON CONFLICT(idempotency_key) DO NOTHING.
TOCTOU: INSERT per-rand cu ON CONFLICT(idempotency_key) DO NOTHING.
Randuri colidante = reclasificate already_sent in rezultatul commit-ului.
rows_hash + n_confirmed acopera DOAR randurile efectiv puse in coada.
"""
@@ -981,7 +972,7 @@ def commit_import(
elif r["resolved_status"] == "needs_review":
review_indices.add(r["row_index"])
# needs_review bifate explicit (Voce#1 — atestare pe valori)
# needs_review bifate explicit (atestare pe valori)
confirmed_review = [idx for idx in req.reviewed_rows if idx in review_indices]
for idx in confirmed_review:
# Gaseste randul needs_review si il adauga la ok_rows
@@ -1040,7 +1031,7 @@ def commit_import(
# Incarca maparea de operatii
mapping_meta = load_mapping_meta(conn, acct)
mapping = {op: meta["cod_prestatie"] for op, meta in mapping_meta.items()}
# T2: validare nomenclator + reguli text incarcate O DATA, inainte de bucla pe randuri.
# Validare nomenclator + reguli text incarcate O DATA, inainte de bucla pe randuri.
valid_codes = load_nomenclator_codes(conn) or None
text_rules = load_text_rules(conn, acct)
@@ -1049,10 +1040,9 @@ def commit_import(
toctou_collisions: list[int] = []
rows_for_hash: list[str] = []
# Enqueue in tranzactie explicita (Issue 6)
# Enqueue in tranzactie explicita
conn.execute("BEGIN IMMEDIATE")
try:
# purge_after pentru submissions noi (T16)
purge_after_sql = "datetime('now', '+90 days')"
for ok_row in ok_rows:
@@ -1100,7 +1090,7 @@ def commit_import(
"odometru_final": canon["odometru_final"],
})
# Override editat in preview (3.6) — aplicat ULTIMUL, ca in resolver.
# Override editat in preview — aplicat ULTIMUL, ca in resolver.
override = ok_row.get("override") or {}
if override:
mapped.update(override)
@@ -1127,7 +1117,7 @@ def commit_import(
payload_json = json.dumps(mapped, ensure_ascii=False)
# INSERT ON CONFLICT DO NOTHING (TOCTOU — Issue 1)
# INSERT ON CONFLICT DO NOTHING (TOCTOU)
cur = conn.execute(
"INSERT OR IGNORE INTO submissions "
"(idempotency_key, account_id, status, payload_json, batch_id, row_index, purge_after) "
@@ -1140,7 +1130,6 @@ def commit_import(
toctou_collisions.append(row_index)
else:
sub_id = cur.lastrowid
# US-010: telemetrie pentru itemii rezolvati prin regula text.
_emite_text_rule_hits(conn, acct, int(sub_id), resolved)
enqueued.append({
"submission_id": sub_id,
@@ -1155,7 +1144,7 @@ def commit_import(
n_enqueued = len(enqueued)
# Log atestare (Voce#9): rows_hash + n_confirmed acopera DOAR randurile puse in coada
# Log atestare: rows_hash + n_confirmed acopera DOAR randurile puse in coada
rows_hash = hashlib.sha256(
json.dumps(sorted(rows_for_hash), ensure_ascii=False).encode("utf-8")
).hexdigest() if rows_for_hash else ""
@@ -1185,7 +1174,7 @@ def commit_import(
# --------------------------------------------------------------------------- #
# POST /v1/import/{id}/rand/{row_index}/editeaza — editare celule preview (3.6) #
# POST /v1/import/{id}/rand/{row_index}/editeaza — editare celule preview #
# --------------------------------------------------------------------------- #
class RandEditIn(BaseModel):
@@ -1205,7 +1194,7 @@ def editeaza_rand(
req: RandEditIn,
account_id: int = Depends(resolve_account_id),
) -> dict:
"""Persista editarea unui rand de preview (mutatie pura — Approach B, 3.6).
"""Persista editarea unui rand de preview (mutatie pura).
NU recalculeaza statusul si NU atinge `submissions`; preview-ul rederiva statusul
prin `_resolve_row_for_preview` cu override aplicat ultimul.
@@ -1225,7 +1214,7 @@ def editeaza_rand(
# --------------------------------------------------------------------------- #
# GET /v1/import/{id}/export-failed — CSV randuri esuate (T8) #
# GET /v1/import/{id}/export-failed — CSV randuri esuate #
# --------------------------------------------------------------------------- #
_EXPORT_FAILED_COLUMNS = [