feat(embeddings): cache persistent de vectori in SQLite + warmup in fundal
Vectorii corpusului k-NN persista in tabela embedding_cache (PK model+text_hash,
blob float32 LE); la warmup se vectorizeaza doar textele lipsa din cache, deci
restartul cu corpus neschimbat nu mai plateste ~1-2 min de embed (embed=0).
- app/embedding_cache.py: serializare array('f'), load/save/purge chunk 500 cu
BEGIN/COMMIT explicit (conexiuni autocommit), validare dimensiune la scriere
si citire, orchestrare sync_corpus_vectors cu embed_fn injectat
- index_corpus(vectors=): vectori precalculati cu validare aliniere; mismatch
-> fallback embed complet
- ensure_embeddings_corpus: warmup in thread la startup (block=True), calea de
request ne-blocanta (acquire non-blocking pe lock; warmup in curs -> return
imediat); purjare orfane + modele vechi doar dupa indexare reusita
- log warmup: cache=N embed=M in Xs
- 31 teste noi (cold/warm/incremental, model schimbat, concurenta, ranking
exact, echivalenta float32); suita completa 1596 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -281,6 +281,17 @@ CREATE TABLE IF NOT EXISTS shared_mappings (
|
||||
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
);
|
||||
|
||||
-- Cache persistent vectori embeddings corpus (mapping_suggestions). Cheia (model,
|
||||
-- text_hash) leaga cache-ul de numele modelului: schimbare model = randuri vechi inerte,
|
||||
-- fara sa fie folosite (curatate ulterior de purge_stale).
|
||||
CREATE TABLE IF NOT EXISTS embedding_cache (
|
||||
text_hash TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
vector BLOB NOT NULL,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
PRIMARY KEY (model, text_hash)
|
||||
);
|
||||
|
||||
-- Heartbeat worker (un singur rand, id=1). /healthz citeste de aici.
|
||||
CREATE TABLE IF NOT EXISTS worker_heartbeat (
|
||||
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||
|
||||
Reference in New Issue
Block a user