Extraction finished (575/588 chunks; 6 content-filter-blocked, 7 await
re-extraction). DB rebuilt and frozen at 9418 activities — content_keys
are now stable for the enrichment overlay.
Part A (plumbing + UI):
- database.py: name_ro/description_ro/rules_ro/variations_ro, indoor_outdoor,
space_needed, estimated_fields, source_id/source_ids/chunk_key columns;
FTS5 indexes the 4 *_ro columns across CREATE + all 3 triggers; new equality
filters + category counts for both axes.
- activity.py: new fields + bilingual display helpers (get_display_*,
is_estimated, axis displays).
- config_taxonomy.py: INDOOR_OUTDOOR/SPACE_NEEDED enums + normalizers
(None on unrecognised, no fabrication).
- search.py / routes.py / config.py / templates / css: new dropdowns,
RO-primary rendering with "(estimat)" markers and collapsible original
text, and a /source/<id> download route shipped DARK behind
SOURCE_DOWNLOAD_ENABLED (copyright opt-in).
- build_database.py: source_id/chunk_key in dict_to_activity; merge_cluster
unions source_ids without touching enrichment fields.
Part B (enrichment pipeline, built not yet run):
- build_database.py: load_enrichment + apply_enrichment (post-dedup, keyed on
content_key) + --enrichment CLI + stated-vs-estimated QA.
- run_enrichment.py (resumable, --source/--limit pilot scoping, --collect),
ENRICHMENT_PROMPT.md.
Repair: scripts/repair_extractions.py fixes the subagents' systematic
unescaped-ASCII-quote bug with a faithful char-scanner (escapes, never
truncates) + schema validation + a strictly-more-text guard. json_repair was
tried first, truncated silently, and is NOT used. build_database has no repair
dependency.
Tests: tests/test_enrichment.py added; 99 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements the approved plan to replace the broken regex/index-master
extraction with an LLM-subagent pipeline. Four parallel lanes:
Lane A — scripts/extract_common.py (PDF/docx/doc/pptx/html/zip, no
max_pages truncation), normalize_sources.py, chunk_sources.py
(~20pg chunks + overlap, manifest registry), activity_schema.json.
Lane B — app/config_taxonomy.py (16 fixed category slugs), schema
rebuilt from scratch in app/models/ with content_type, language,
source_files, source_excerpt, normalized_name, extraction_confidence,
needs_review; FTS5 + 3 triggers extended with materials_list and
skills_developed.
Lane C — build_database.py (--rebuild, atomic swap, schema + fuzzy
source_excerpt validation, dedup with needs_review band),
validate_extractions.py, review_queue.py, new run_extraction.py
orchestrator, SUBAGENT_PROMPT.md.
Lane D — search.py content_type/language filters (default search
excludes non-game content), E7 schema-compat audit; fixed a NULL
keywords AttributeError in _boost_search_relevance.
Removes 8 orphaned/dead scripts and app/services/parser.py +
indexer.py. Adds tests/ (70 passing, 1 skipped — libreoffice absent).
Note: Lane D made one additive edit to app/models/database.py
(_update_category_counts) to surface content_type/language in
get_filter_options, outside its nominal lane boundary but after
Lane B completed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
