Compare commits

...

2 Commits

Author SHA1 Message Date
d1bc77e87d feat(voice): polish voice loop UX — filler kill, barge-in, DTX flush, time/RO TTS
End-to-end voice UX iteration after DAVE E2E shipped. Each change addresses a
real symptom Marius hit in live testing today:

- Kill the 3s filler ("mă gândesc"): Claude p50 is 4-7s so the filler always
  fired BEFORE the response and collided with it. Removed all filler infra
  from pipeline.py + tts_stream.py (FILLER_DELAY_S, _filler_task, push_filler,
  load_thinking_wav, thinking.wav cache).

- Barge-in: ttsq.clear() at the top of on_segment_done drops stale frames so
  a new utterance cuts off Echo's previous response cleanly.

- DTX silence flush: Discord stops sending RTP packets when the user goes
  silent (DTX), so the inline silence-check in sink.write() never fired for
  the trailing audio of an utterance — STT was missed entirely. Added a
  background poller thread that checks the silence-flush condition every
  200ms independent of incoming packets.

- Discord audio cadence fix: EchoStreamingAudioSource.read() blocked 100ms
  per call when pcm_queue was empty, wrecking Discord's 20ms frame pacing →
  client interpreted the stream as stutter and discarded leading frames
  (Marius heard "4 de minute în București" instead of the full sentence).
  Switched to get_frame_nowait() — instant return, silence frame on empty.

- RO time expansion: "23:09" was being read as "douăzeci și trei:nouă"
  with literal colon. Added expand_time() with feminine-correct minute
  formatting (un minut / două minute / douăzeci de minute / una de minute).

- Supertonic Unicode sanitize centralized in tools/tts.py: Romanian curly
  quotes (`„`, `"`, `"`, `—`, `…`) crash Supertonic with HTTP 500. Map them
  to ASCII at the synthesize() entry so BOTH voice mode and /audio command
  are covered without duplication. normalize.py re-exports for compat.

- Whisper offline: WhisperModel(..., local_files_only=True) — no more
  huggingface.co metadata GET on every startup. Model is already cached.

- Diagnostic logging across the chain: sink first-packet, VAD first-speech,
  voice stream block (Claude → callback), push_text (text → clauses queued),
  TTS pushed (clauses → frames). Lets future "spoke but Echo silent" bugs
  pinpoint exactly where the chain breaks.

- Captured Supertonic curly-quote lesson in tasks/lessons.md.

All 76 voice tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 20:33:24 +00:00
e4f3177fc1 feat(voice): DAVE E2E receive-side decrypt — unblocks Pas 12
Vendored fork: discord-ext-voice_recv 0.5.3a+echo.dave1

Patches the receive pipeline to handle Discord's mandatory DAVE E2E
encryption on voice gateway v=8. Without this, opus_decode raised
"corrupted stream" on every received packet in a DAVE-active room and
voice-to-voice never connected.

DAVE patch (vendor/discord-ext-voice-recv/reader.py):
- `_maybe_dave_decrypt(rtp_packet)`: gate mirrors discord.py 2.7.1
  `voice_state.can_encrypt`. Uses davey's `can_passthrough(user_id)` to
  branch — peers in passthrough send transport-only packets that pass
  through verbatim; peers in DAVE epoch go through `davey.decrypt`.
- Hooked in `callback()` between transport decrypt and feed_rtp;
  drops on decrypt failure without killing the reader thread.
- Bumps __version__ to '0.5.3a+echo.dave1' (PEP 440 local segment) so a
  contract test can fail fast on accidental upstream-sync overwrite.

Pipeline fixes uncovered while testing DAVE end-to-end:
- src/voice/pipeline.py: silero-vad v6+ requires exactly 512 samples per
  call at 16kHz; our 100ms window (1600 samples) was silently raising
  ValueError → VAD always returned False → STT never fired. Slice the
  window into 512-sample chunks. Bump whisper beam_size 1→5 and add a
  Romanian `initial_prompt` — transcriptions go from "Eco salt." gibberish
  to "Echo, salutare, te rog spune-mi cât este ora."
- src/voice/tts_stream.py: EchoStreamingAudioSource.read() returns a 20ms
  silence frame instead of b'' on empty queue. Empty return is treated
  by Discord as end-of-stream and kills the player, so any TTS pushed
  later would be silently discarded.
- src/adapters/discord_voice.py: actually attach EchoStreamingAudioSource
  to the voice client after the wakeup beep (chained via `after=`),
  which was missing entirely — TTS frames had no consumer.

Tests:
- tests/test_voice_recv_dave.py: 11 unit + callback integration tests
  covering bypass paths, can_passthrough gate, decrypt error handling.
- tests/test_voice_adapter_contract.py: +test_voice_recv_fork_version
  and +test_voice_connection_state_has_dave_attrs guards against
  upstream drift on either side.

Config:
- config.json: voice.allowed_user_ids whitelist for Marius's user id.

Status: voice-to-voice loop closes end-to-end (DAVE → VAD → Whisper →
Claude → Supertonic → audio out). Latency is ~8-13s per turn, which is
out of scope for this commit — see TODOS.md for the real-time UX
follow-up plan.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-27 19:48:36 +00:00
12 changed files with 712 additions and 126 deletions

View File

@@ -104,6 +104,12 @@
"ollama": { "ollama": {
"url": "http://10.0.20.161:11434" "url": "http://10.0.20.161:11434"
}, },
"voice": {
"allowed_user_ids": ["949388626146517022"],
"user_name": "Marius",
"default_voice": "M2",
"auto_leave_minutes": 5
},
"paths": { "paths": {
"personality": "personality/", "personality": "personality/",
"tools": "tools/", "tools": "tools/",

View File

@@ -21,6 +21,14 @@ from typing import Optional
import discord import discord
from discord import app_commands from discord import app_commands
# Optional DAVE dep (mandatory at runtime when discord.py 2.7.1 is paired with
# Discord voice gateway v=8; tolerated missing in tests / dev environments).
try:
import davey
_HAS_DAVE = True
except ImportError:
_HAS_DAVE = False
from src.config import Config from src.config import Config
from src.voice.pipeline import ( from src.voice.pipeline import (
VoiceSession, VoiceSession,
@@ -28,7 +36,7 @@ from src.voice.pipeline import (
_get_whisper_model, _get_whisper_model,
_get_silero_vad, _get_silero_vad,
) )
from src.voice.tts_stream import TTSQueue from src.voice.tts_stream import TTSQueue, EchoStreamingAudioSource
from src.voice._discord_voice_adapter import connect_voice from src.voice._discord_voice_adapter import connect_voice
log = logging.getLogger("echo-core.discord.voice") log = logging.getLogger("echo-core.discord.voice")
@@ -53,6 +61,11 @@ async def warmup_models() -> None:
""" """
global _voice_load_error global _voice_load_error
try: try:
if not discord.opus.is_loaded():
discord.opus.load_opus("libopus.so.0")
if _HAS_DAVE:
log.info("DAVE protocol v%d available (davey %s)",
davey.DAVE_PROTOCOL_VERSION, davey.__version__)
await asyncio.to_thread(_get_whisper_model) await asyncio.to_thread(_get_whisper_model)
await asyncio.to_thread(_get_silero_vad) await asyncio.to_thread(_get_silero_vad)
log.info("Voice models warm") log.info("Voice models warm")
@@ -167,11 +180,24 @@ def register(tree: app_commands.CommandTree, bot: discord.Client) -> app_command
) )
return return
_voice_sessions[guild_id] = session _voice_sessions[guild_id] = session
# Wake-up beep # Start TTS streaming source for the entire session. Chain the
# wake-up beep via `after=` so streaming takes over when beep ends.
def _start_stream(error: Optional[Exception] = None) -> None:
if error is not None:
log.warning("Beep playback ended with error: %s", error)
try: try:
vc.play(discord.FFmpegPCMAudio("assets/voice/beep_200ms.wav")) vc.play(EchoStreamingAudioSource(ttsq))
log.info("TTS streaming source attached")
except Exception: except Exception:
log.warning("Beep playback skipped", exc_info=True) log.exception("EchoStreamingAudioSource attach failed")
try:
vc.play(
discord.FFmpegPCMAudio("assets/voice/beep_200ms.wav"),
after=_start_stream,
)
except Exception:
log.warning("Beep playback skipped, starting stream directly", exc_info=True)
_start_stream()
# Attach sink # Attach sink
try: try:
bot_user_id = int(bot.user.id) if bot.user is not None else 0 bot_user_id = int(bot.user.id) if bot.user is not None else 0

View File

@@ -94,6 +94,47 @@ def expand_numbers_ro(text: str) -> str:
return _NUM_TOKEN.sub(_sub, text) return _NUM_TOKEN.sub(_sub, text)
# ---------- Time ----------
_TIME_PATTERN = re.compile(r'(?<!\d)([01]?\d|2[0-3]):([0-5]?\d)(?!\d)')
def _format_minutes_ro(n: int) -> str:
"""Romanian-correct feminine forms for minute counts (0-59)."""
if n == 1:
return "un minut"
if n == 2:
return "două minute"
if n < 20:
return f"{_int_to_ro(n)} minute"
last = n % 10
rest = n - last
if last == 0:
return f"{_int_to_ro(n)} de minute"
if last == 1:
return f"{_int_to_ro(rest)} și una de minute"
if last == 2:
return f"{_int_to_ro(rest)} și două de minute"
return f"{_int_to_ro(rest)} și {_int_to_ro(last)} de minute"
def expand_time(text: str) -> str:
"""Expand ``HH:MM`` clock times into colloquial Romanian.
23:09 -> "douăzeci și trei și nouă minute"
23:00 -> "douăzeci și trei fix"
"""
def _sub(match: re.Match) -> str:
h = int(match.group(1))
m = int(match.group(2))
hour_str = _int_to_ro(h)
if m == 0:
return f"{hour_str} fix"
return f"{hour_str} și {_format_minutes_ro(m)}"
return _TIME_PATTERN.sub(_sub, text)
# ---------- Currency ---------- # ---------- Currency ----------
_CURRENCY_MAIN = { _CURRENCY_MAIN = {
@@ -177,6 +218,9 @@ def expand_symbols(text: str) -> str:
return text return text
from tools.tts import sanitize_for_supertonic as sanitize_punctuation
# ---------- Abbreviations ---------- # ---------- Abbreviations ----------
# Longer patterns first so 'ș.a.m.d.' wins over 'ș.a.' # Longer patterns first so 'ș.a.m.d.' wins over 'ș.a.'
@@ -211,7 +255,9 @@ def normalize_for_tts(text: str) -> str:
response continues in the text channel mirror. response continues in the text channel mirror.
""" """
text = strip_markdown(text) text = strip_markdown(text)
text = sanitize_punctuation(text)
text = expand_abbreviations(text) text = expand_abbreviations(text)
text = expand_time(text)
text = expand_currency(text) text = expand_currency(text)
text = expand_numbers_ro(text) text = expand_numbers_ro(text)
text = expand_symbols(text) text = expand_symbols(text)

View File

@@ -19,7 +19,7 @@ the bot transcribe itself.
See plan: ``src/voice/pipeline.py`` (Pas 5), Engineering decisions #4 See plan: ``src/voice/pipeline.py`` (Pas 5), Engineering decisions #4
(VAD 100ms batched), #5 (cleanup centralizat), #7 (bot.user.id explicit (VAD 100ms batched), #5 (cleanup centralizat), #7 (bot.user.id explicit
guard), #8 (filler audio ``thinking.wav`` at 3s pre-first-block). guard).
""" """
from __future__ import annotations from __future__ import annotations
@@ -48,7 +48,6 @@ VAD_WINDOW_BYTES = PACKET_BYTES * (VAD_WINDOW_MS // PACKET_MS)
VAD_THRESHOLD = 0.5 VAD_THRESHOLD = 0.5
SILENCE_FLUSH_MS = 800 SILENCE_FLUSH_MS = 800
NO_SPEECH_DROP_THRESHOLD = 0.6 NO_SPEECH_DROP_THRESHOLD = 0.6
FILLER_DELAY_S = 3.0
PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
LOGS_DIR = PROJECT_ROOT / "logs" LOGS_DIR = PROJECT_ROOT / "logs"
@@ -76,6 +75,7 @@ def _get_whisper_model() -> Any:
from faster_whisper import WhisperModel from faster_whisper import WhisperModel
_whisper_model = WhisperModel( _whisper_model = WhisperModel(
"small", device="cpu", compute_type="int8", cpu_threads=4, "small", device="cpu", compute_type="int8", cpu_threads=4,
local_files_only=True,
) )
return _whisper_model return _whisper_model
@@ -164,8 +164,6 @@ class VoiceSession:
self._lock = threading.Lock() self._lock = threading.Lock()
self._cleaned_up = False self._cleaned_up = False
self._lock_owner_thread: Optional[int] = None self._lock_owner_thread: Optional[int] = None
self._filler_task: Optional[asyncio.Task] = None
self._first_block_seen = False
# ----- context manager ----- # ----- context manager -----
@@ -244,14 +242,7 @@ class VoiceSession:
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
log.warning("ttsq.stop failed: %s", e) log.warning("ttsq.stop failed: %s", e)
# 5. Cancel pending filler task. # 5. Release the session lock (held since __enter__).
if self._filler_task is not None and not self._filler_task.done():
try:
self._filler_task.cancel()
except Exception: # noqa: BLE001
pass
# 6. Release the session lock (held since __enter__).
try: try:
if self._lock.locked(): if self._lock.locked():
self._lock.release() self._lock.release()
@@ -276,6 +267,13 @@ class VoiceSession:
self.last_activity_ts = time.monotonic() self.last_activity_ts = time.monotonic()
speaker_name = self._resolve_speaker_name(speaker_id) speaker_name = self._resolve_speaker_name(speaker_id)
# Drop any TTS frames from the previous turn so a new utterance cuts off
# stale Echo speech (barge-in) and never mixes with the new response.
try:
self.ttsq.clear()
except Exception as e: # noqa: BLE001
log.warning("ttsq.clear failed: %s", e)
# 1. Mirror to text channel (one Unicode 🎤 — exception per plan). # 1. Mirror to text channel (one Unicode 🎤 — exception per plan).
if self.mirror_enabled and self.text_channel is not None: if self.mirror_enabled and self.text_channel is not None:
try: try:
@@ -302,33 +300,19 @@ class VoiceSession:
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
log.warning("voice transcript write failed: %s", e) log.warning("voice transcript write failed: %s", e)
# 3. Arm the 3s filler timer — fires only if no Claude block arrives. block_count = [0]
self._first_block_seen = False
if self._filler_task is not None and not self._filler_task.done():
self._filler_task.cancel()
try:
self._filler_task = asyncio.create_task(self._filler_after_delay())
except RuntimeError:
# No running loop (test path). Skip the timer.
self._filler_task = None
def voice_stream_callback(block: str) -> None: def voice_stream_callback(block: str) -> None:
"""Called once per Claude streamed text block — pushes to TTS """Called once per Claude streamed text block — pushes to TTS."""
and cancels the filler on first arrival.""" block_count[0] += 1
if not self._first_block_seen: log.info("voice stream block #%d (%d chars): %r",
self._first_block_seen = True block_count[0], len(block or ""), (block or "")[:80])
ft = self._filler_task
if ft is not None and not ft.done():
try:
ft.cancel()
except Exception: # noqa: BLE001
pass
try: try:
self.ttsq.push_text(block) self.ttsq.push_text(block)
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
log.warning("ttsq.push_text failed: %s", e) log.warning("ttsq.push_text failed: %s", e)
# 4. Dispatch to Claude. send_message is sync subprocess, run on # Dispatch to Claude. send_message is sync subprocess, run on
# a worker thread so the loop stays responsive for mirror/TTS. # a worker thread so the loop stays responsive for mirror/TTS.
try: try:
await asyncio.to_thread( await asyncio.to_thread(
@@ -343,20 +327,6 @@ class VoiceSession:
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
log.error("route_message voice path failed: %s", e) log.error("route_message voice path failed: %s", e)
async def _filler_after_delay(self) -> None:
"""Push ``assets/voice/thinking.wav`` after FILLER_DELAY_S if Claude
hasn't produced a first block yet."""
try:
await asyncio.sleep(FILLER_DELAY_S)
except asyncio.CancelledError:
return
if self._first_block_seen or self._cleaned_up:
return
try:
self.ttsq.push_filler()
except Exception as e: # noqa: BLE001
log.warning("ttsq.push_filler failed: %s", e)
# ----- helpers ----- # ----- helpers -----
def _resolve_speaker_name(self, speaker_id: int) -> str: def _resolve_speaker_name(self, speaker_id: int) -> str:
@@ -406,11 +376,28 @@ class EchoVoiceSink(AudioSink):
self._last_speech_ts: dict[int, float] = {} self._last_speech_ts: dict[int, float] = {}
self._has_speech: dict[int, bool] = {} self._has_speech: dict[int, bool] = {}
self._sink_lock = threading.Lock() self._sink_lock = threading.Lock()
# Diagnostics: log once-per-user when packets first arrive and when
# VAD first detects speech. Cheap, but tells us exactly where the
# chain breaks when "I spoke but Echo heard nothing" happens.
self._first_packet_logged: set[int] = set()
self._first_speech_logged: set[int] = set()
# Background poller that triggers the silence flush even when Discord
# DTX stops delivering RTP packets after the user stops speaking. Without
# this, sink.write would stop firing and STT would never run on the
# final utterance.
self._poller_stop = threading.Event()
self._poller_thread = threading.Thread(
target=self._silence_flush_poller,
name="echo-voice-flush-poller",
daemon=True,
)
self._poller_thread.start()
def wants_opus(self) -> bool: def wants_opus(self) -> bool:
return False return False
def cleanup(self) -> None: def cleanup(self) -> None:
self._poller_stop.set()
with self._sink_lock: with self._sink_lock:
self._user_buffers.clear() self._user_buffers.clear()
self._packet_accum.clear() self._packet_accum.clear()
@@ -435,6 +422,10 @@ class EchoVoiceSink(AudioSink):
if not pcm: if not pcm:
return return
if uid not in self._first_packet_logged:
self._first_packet_logged.add(uid)
log.info("voice sink: first PCM packet from user %s (%d bytes)", uid, len(pcm))
window_pcm: Optional[bytes] = None window_pcm: Optional[bytes] = None
pcm_for_stt: Optional[bytes] = None pcm_for_stt: Optional[bytes] = None
@@ -450,30 +441,57 @@ class EchoVoiceSink(AudioSink):
if window_pcm is not None: if window_pcm is not None:
if self._vad_detects_speech(window_pcm): if self._vad_detects_speech(window_pcm):
if uid not in self._first_speech_logged:
self._first_speech_logged.add(uid)
log.info("voice sink: VAD detected speech from user %s", uid)
with self._sink_lock: with self._sink_lock:
self._last_speech_ts[uid] = time.monotonic() self._last_speech_ts[uid] = time.monotonic()
self._has_speech[uid] = True self._has_speech[uid] = True
with self._sink_lock: pcm_for_stt = self._take_flushable_pcm(uid)
if self._has_speech.get(uid):
last = self._last_speech_ts.get(uid, 0.0)
silence_ms = (time.monotonic() - last) * 1000.0
if silence_ms >= SILENCE_FLUSH_MS:
pcm_for_stt = bytes(self._user_buffers.get(uid, b""))
self._user_buffers[uid] = bytearray()
self._packet_accum[uid] = bytearray()
self._has_speech[uid] = False
if pcm_for_stt: if pcm_for_stt:
self._flush_to_stt(uid, pcm_for_stt) self._flush_to_stt(uid, pcm_for_stt)
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
log.warning("EchoVoiceSink.write failed: %s", e) log.warning("EchoVoiceSink.write failed: %s", e)
def _take_flushable_pcm(self, uid: int) -> Optional[bytes]:
"""If user `uid` has buffered speech that's been silent ≥SILENCE_FLUSH_MS,
consume the buffer and return it. Otherwise return None."""
with self._sink_lock:
if not self._has_speech.get(uid):
return None
last = self._last_speech_ts.get(uid, 0.0)
silence_ms = (time.monotonic() - last) * 1000.0
if silence_ms < SILENCE_FLUSH_MS:
return None
pcm = bytes(self._user_buffers.get(uid, b""))
self._user_buffers[uid] = bytearray()
self._packet_accum[uid] = bytearray()
self._has_speech[uid] = False
return pcm if pcm else None
def _silence_flush_poller(self) -> None:
"""Background tick: Discord DTX stops sending RTP packets when the user
goes silent, so the inline flush check in `write()` never fires for the
last utterance. Poll every 200ms so the trailing audio actually reaches
Whisper."""
while not self._poller_stop.wait(0.2):
try:
with self._sink_lock:
pending = [uid for uid, has in self._has_speech.items() if has]
for uid in pending:
pcm = self._take_flushable_pcm(uid)
if pcm:
self._flush_to_stt(uid, pcm)
except Exception as e: # noqa: BLE001
log.warning("silence flush poller iter failed: %s", e)
# ----- VAD ----- # ----- VAD -----
def _vad_detects_speech(self, pcm48_stereo: bytes) -> bool: def _vad_detects_speech(self, pcm48_stereo: bytes) -> bool:
"""Run silero-vad on a 100ms window. Falls back to an RMS energy """Run silero-vad on a 100ms window. silero-vad v5+ requires exactly
threshold if torch / silero are unavailable.""" 512 samples per call at 16kHz, so we slice the window into 512-sample
chunks and return True if any chunk crosses the threshold."""
try: try:
mono16 = _pcm48_stereo_to_16_mono(pcm48_stereo) mono16 = _pcm48_stereo_to_16_mono(pcm48_stereo)
if mono16.size == 0: if mono16.size == 0:
@@ -484,10 +502,17 @@ class EchoVoiceSink(AudioSink):
rms = float(np.sqrt(np.mean(mono16.astype(np.float64) ** 2))) rms = float(np.sqrt(np.mean(mono16.astype(np.float64) ** 2)))
return rms > 0.02 return rms > 0.02
model, _ = _get_silero_vad() model, _ = _get_silero_vad()
chunk = 512 # silero-vad v5+ hard requirement at 16kHz
max_prob = 0.0
with torch.no_grad(): with torch.no_grad():
prob = float(model(torch.from_numpy(mono16), for start in range(0, mono16.size - chunk + 1, chunk):
SAMPLE_RATE_WHISPER).item()) seg = mono16[start:start + chunk]
return prob >= VAD_THRESHOLD p = float(model(torch.from_numpy(seg), SAMPLE_RATE_WHISPER).item())
if p > max_prob:
max_prob = p
if p >= VAD_THRESHOLD:
return True
return False
except Exception as e: # noqa: BLE001 except Exception as e: # noqa: BLE001
log.debug("VAD inference failed: %s", e) log.debug("VAD inference failed: %s", e)
return False return False
@@ -502,7 +527,12 @@ class EchoVoiceSink(AudioSink):
return return
model = _get_whisper_model() model = _get_whisper_model()
segments, _info = model.transcribe( segments, _info = model.transcribe(
mono16, language="ro", beam_size=1, mono16, language="ro", beam_size=5,
initial_prompt=(
"Echo Core, asistent personal AI românesc al lui Marius. "
"Conversație colocvială în română."
),
condition_on_previous_text=False,
) )
text_parts: list[str] = [] text_parts: list[str] = []
worst_no_speech = 0.0 worst_no_speech = 0.0
@@ -543,7 +573,6 @@ class EchoVoiceSink(AudioSink):
__all__ = [ __all__ = [
"VoiceSession", "VoiceSession",
"EchoVoiceSink", "EchoVoiceSink",
"FILLER_DELAY_S",
"SILENCE_FLUSH_MS", "SILENCE_FLUSH_MS",
"VAD_THRESHOLD", "VAD_THRESHOLD",
"VAD_WINDOW_MS", "VAD_WINDOW_MS",

View File

@@ -10,6 +10,7 @@ Engineering decisions #6, #8, #15.
from __future__ import annotations from __future__ import annotations
import io import io
import logging
import queue import queue
import re import re
import subprocess import subprocess
@@ -24,6 +25,9 @@ from src.voice.normalize import normalize_for_tts
from tools.tts import synthesize from tools.tts import synthesize
log = logging.getLogger(__name__)
# Discord wants 20ms of 16-bit 48kHz stereo PCM per frame. # Discord wants 20ms of 16-bit 48kHz stereo PCM per frame.
# 48000 Hz * 0.020 s * 2 channels * 2 bytes = 3840 bytes. # 48000 Hz * 0.020 s * 2 channels * 2 bytes = 3840 bytes.
FRAME_BYTES = 3840 FRAME_BYTES = 3840
@@ -31,13 +35,6 @@ TARGET_SAMPLE_RATE = 48000
TARGET_CHANNELS = 2 TARGET_CHANNELS = 2
TARGET_SAMPLE_WIDTH = 2 TARGET_SAMPLE_WIDTH = 2
_PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent
_THINKING_WAV = _PROJECT_ROOT / "assets" / "voice" / "thinking.wav"
# Cached filler frames (load + resample once, reuse forever).
_thinking_frames_cache: Optional[List[bytes]] = None
_thinking_cache_lock = threading.Lock()
# Sentinel pushed onto the text queue to ask the worker to exit cleanly. # Sentinel pushed onto the text queue to ask the worker to exit cleanly.
_POISON = object() _POISON = object()
@@ -149,27 +146,6 @@ def wav_to_pcm_20ms_frames(wav_bytes: bytes) -> List[bytes]:
return frames return frames
def load_thinking_wav() -> List[bytes]:
"""Load ``assets/voice/thinking.wav`` and cache it as 20ms PCM frames.
Safe to call from multiple threads; the load happens at most once.
Returns an empty list if the asset is missing or fails to decode
(callers treat that as a no-op filler).
"""
global _thinking_frames_cache
if _thinking_frames_cache is not None:
return _thinking_frames_cache
with _thinking_cache_lock:
if _thinking_frames_cache is not None:
return _thinking_frames_cache
try:
wav_bytes = _THINKING_WAV.read_bytes()
_thinking_frames_cache = wav_to_pcm_20ms_frames(wav_bytes)
except (FileNotFoundError, OSError, RuntimeError):
_thinking_frames_cache = []
return _thinking_frames_cache
# ---------- TTS worker queue ---------- # ---------- TTS worker queue ----------
class TTSQueue: class TTSQueue:
@@ -227,19 +203,13 @@ class TTSQueue:
if not text: if not text:
return return
cleaned = normalize_for_tts(text) cleaned = normalize_for_tts(text)
n = 0
for clause in clause_segments(cleaned): for clause in clause_segments(cleaned):
clause = clause.strip() clause = clause.strip()
if clause: if clause:
self._text_queue.put(clause) self._text_queue.put(clause)
n += 1
def push_filler(self) -> None: log.info("ttsq.push_text: input %d chars → %d clauses queued", len(text), n)
"""Enqueue pre-rendered filler frames (``thinking.wav``) directly.
Bypasses synthesis -- the filler plays even if Supertonic is down
or slow. No-op if the asset failed to load.
"""
for frame in load_thinking_wav():
self._pcm_queue.put(frame)
def clear(self) -> None: def clear(self) -> None:
"""Drop everything pending (used for barge-in).""" """Drop everything pending (used for barge-in)."""
@@ -251,10 +221,14 @@ class TTSQueue:
# --- consumer side (called by EchoStreamingAudioSource) --- # --- consumer side (called by EchoStreamingAudioSource) ---
def get_frame(self, timeout: float = 0.1) -> Optional[bytes]: def get_frame_nowait(self) -> Optional[bytes]:
"""Block up to ``timeout`` seconds for the next 20ms PCM frame.""" """Return the next PCM frame if available, else None — no blocking.
Blocking inside the player's read() loop wrecks Discord's 20ms cadence
and the client interprets the stream as stuttering / out-of-order.
"""
try: try:
return self._pcm_queue.get(timeout=timeout) return self._pcm_queue.get_nowait()
except queue.Empty: except queue.Empty:
return None return None
@@ -278,24 +252,25 @@ class TTSQueue:
break break
if not isinstance(item, str): if not isinstance(item, str):
continue continue
preview = item[:60]
try: try:
result = synthesize(item, voice=self.voice_id, lang=self.lang) result = synthesize(item, voice=self.voice_id, lang=self.lang)
except Exception: except Exception as e:
# Synth crashes shouldn't kill the worker -- log path is the log.warning("TTS synth raised for %r: %s", preview, e)
# caller's job (we have no logger here on purpose).
continue continue
if not result.get('ok'): if not result.get('ok'):
log.warning("TTS synth not ok for %r: %s", preview, result.get('error'))
continue continue
path = result.get('path') path = result.get('path')
if not path: if not path:
log.warning("TTS synth ok but no path for %r", preview)
continue continue
wav_bytes = b'' wav_bytes = b''
try: try:
wav_bytes = Path(path).read_bytes() wav_bytes = Path(path).read_bytes()
except OSError: except OSError as e:
pass log.warning("TTS WAV read failed for %r: %s", preview, e)
finally: finally:
# Best-effort cleanup of the synth tempfile.
try: try:
Path(path).unlink(missing_ok=True) Path(path).unlink(missing_ok=True)
except OSError: except OSError:
@@ -304,12 +279,18 @@ class TTSQueue:
continue continue
try: try:
frames = wav_to_pcm_20ms_frames(wav_bytes) frames = wav_to_pcm_20ms_frames(wav_bytes)
except RuntimeError: except RuntimeError as e:
log.warning("TTS WAV-to-PCM failed for %r: %s", preview, e)
continue
if not frames:
log.warning("TTS WAV-to-PCM produced 0 frames for %r", preview)
continue continue
for frame in frames: for frame in frames:
if self._stop_event.is_set(): if self._stop_event.is_set():
return return
self._pcm_queue.put(frame) self._pcm_queue.put(frame)
log.info("TTS pushed %d frames (%.1fs) for %r",
len(frames), len(frames) * 0.02, preview)
# ---------- Discord audio source ---------- # ---------- Discord audio source ----------
@@ -318,13 +299,17 @@ class EchoStreamingAudioSource(discord.AudioSource):
"""Pull PCM frames from a ``TTSQueue`` into Discord's audio thread. """Pull PCM frames from a ``TTSQueue`` into Discord's audio thread.
A single ``voice_client.play(EchoStreamingAudioSource(ttsq))`` call A single ``voice_client.play(EchoStreamingAudioSource(ttsq))`` call
spans the whole turn. The audio thread blocks on the PCM queue for spans the whole session. When the TTS queue is empty, ``read()``
up to 100ms per ``read()``; if it stays empty past that, ``read()`` returns a 20ms silence frame to keep the player alive — otherwise
returns ``b''`` which Discord interprets as end-of-stream and stops Discord would interpret an empty return as end-of-stream and stop
the player (which is exactly what we want at end-of-turn or after the player, so real TTS frames pushed later would be silently
``ttsq.clear()`` on barge-in). discarded. The player is explicitly terminated only via
``cleanup()`` (called on voice session teardown).
""" """
# 20ms of s16le stereo at 48kHz silence (960 samples × 2 channels × 2 bytes).
_SILENCE_FRAME = b'\x00' * (960 * 2 * 2)
def __init__(self, ttsq: TTSQueue): def __init__(self, ttsq: TTSQueue):
self._ttsq = ttsq self._ttsq = ttsq
self._closed = False self._closed = False
@@ -332,9 +317,9 @@ class EchoStreamingAudioSource(discord.AudioSource):
def read(self) -> bytes: def read(self) -> bytes:
if self._closed: if self._closed:
return b'' return b''
frame = self._ttsq.get_frame(timeout=0.1) frame = self._ttsq.get_frame_nowait()
if frame is None: if frame is None:
return b'' return self._SILENCE_FRAME
return frame return frame
def is_opus(self) -> bool: def is_opus(self) -> bool:

View File

@@ -17,6 +17,13 @@ Lecții capturate din corectările lui Marius. Citește acest fișier la începu
<!-- Lecțiile se adaugă mai jos, cele mai noi sus. --> <!-- Lecțiile se adaugă mai jos, cele mai noi sus. -->
## Supertonic rejectează ghilimelele curly (Unicode) cu HTTP 500
**Data:** 2026-05-27
**Context:** Marius a dat o comandă audio pe Discord cu un URL, iar răspunsul lui Claude conținea `„foo"` (ghilimele românești curly). Supertonic a returnat `HTTP 500: synthesis failed: Found 1 unsupported character(s): ['„']` și răspunsul nu s-a mai auzit. Fără retry logic vizibil în UX — pur și simplu tace.
**Greșeala:** Am presupus că `normalize_for_tts` produce text deja "TTS-safe" pentru Supertonic. În realitate `strip_markdown` păstrează ghilimelele Unicode (`„` U+201E, `"` U+201D, `—` U+2014, `…` U+2026, etc.) pe care Supertonic le refuză.
**Regula:** Înainte de orice apel HTTP la Supertonic, **sanitizează punctuația Unicode** la echivalentele ASCII (`„` `"` `"``"`, `'` `'` ```'`, `` `—``-`, `…``...`, `«` `»``"`). Funcția `sanitize_punctuation` în `src/voice/normalize.py` face asta și e apelată chiar după `strip_markdown` în pipeline. Dacă apar caractere noi care crapă Supertonic (ex: simboluri matematice, săgeți), adaugă-le în `_TTS_PUNCT_MAP`.
**Când se aplică:** Orice cod care trimite text la Supertonic (`tools/tts.py`, `src/voice/tts_stream.py`). Inclusiv testare manuală cu `curl` — folosește text românesc realistic (include `„foo"`, em-dash `—`, ellipsis `…`).
## Mai multe threads ≠ mai rapid — fitează `cpu_threads` pe physical cores, nu logical ## Mai multe threads ≠ mai rapid — fitează `cpu_threads` pe physical cores, nu logical
**Data:** 2026-05-27 **Data:** 2026-05-27
**Context:** Benchmark `tools/voice_bench.py` pentru faster-whisper `small` int8 pe i7-6700T (4 physical / 8 logical cores). Marius a urcat VM-ul de la 2 → 4 → 6 cores online, așteptând că mai multe = mai rapid. **Context:** Benchmark `tools/voice_bench.py` pentru faster-whisper `small` int8 pe i7-6700T (4 physical / 8 logical cores). Marius a urcat VM-ul de la 2 → 4 → 6 cores online, așteptând că mai multe = mai rapid.

View File

@@ -169,3 +169,54 @@ def test_voice_data_has_opus_property():
opus_attr = inspect.getattr_static(VoiceData, "opus", None) opus_attr = inspect.getattr_static(VoiceData, "opus", None)
assert isinstance(opus_attr, property), "VoiceData.opus must be a property" assert isinstance(opus_attr, property), "VoiceData.opus must be a property"
# --- Echo-core DAVE-decrypt fork guards -------------------------------------
#
# Two contract tests pinned by the DAVE receive-side decrypt patch.
# See plan: /home/moltbot/.claude/plans/wiggly-exploring-glade.md
#
# These fail fast on either:
# 1. An upstream voice-recv re-install wiping the fork's version marker
# (i.e. our patch is gone), OR
# 2. A discord.py upgrade renaming the connection-level DAVE attrs the
# patch reads (`dave_session`, `dave_protocol_version`).
def test_voice_recv_fork_version():
"""Echo-core fork tag for the DAVE-decrypt patch.
Lane A bumps `voice_recv.__version__` to `'0.5.3a+echo.dave1'` (PEP 440
local segment). If this assertion fails after a vendor reinstall, the
fork patch has been lost — re-apply `_maybe_dave_decrypt` + the
`callback()` hook before deploying, or live voice will regress to the
`opus_decode: corrupted stream` error chain.
"""
from discord.ext import voice_recv
assert voice_recv.__version__ == "0.5.3a+echo.dave1", (
f"voice_recv.__version__ is {voice_recv.__version__!r}; expected "
"'0.5.3a+echo.dave1'. The DAVE-decrypt fork patch has been "
"overwritten — re-apply before reinstalling the vendored package."
)
def test_voice_connection_state_has_dave_attrs():
"""`_maybe_dave_decrypt` reads `dave_session` and `dave_protocol_version`
off the discord.py `VoiceConnectionState`. If a future discord.py upgrade
renames either attr, fail loudly here rather than in a live voice call
(where the symptom is silent packet drops).
"""
from discord import voice_state
src = inspect.getsource(voice_state.VoiceConnectionState)
assert "dave_session" in src, (
"discord.voice_state.VoiceConnectionState source no longer mentions "
"'dave_session' — discord.py may have renamed the attr. Update "
"vendor/discord-ext-voice-recv/.../reader.py::_maybe_dave_decrypt."
)
assert "dave_protocol_version" in src, (
"discord.voice_state.VoiceConnectionState source no longer mentions "
"'dave_protocol_version' — discord.py may have renamed the attr. "
"Update _maybe_dave_decrypt accordingly."
)

View File

@@ -0,0 +1,302 @@
"""DAVE receive-side decrypt tests for the vendored voice-recv fork.
Exercises Lane A's patch on
`vendor/discord-ext-voice-recv/discord/ext/voice_recv/reader.py`:
* `_maybe_dave_decrypt(rtp_packet)` — DAVE E2E layer sandwiched between the
transport-layer decrypt and the routing into the opus decoder. No-op when
the room is non-DAVE, when davey isn't installed, or when the SSRC map
hasn't caught up to a new speaker yet.
* `callback()` hook — feeds the DAVE-unwrapped plaintext into
`packet_router.feed_rtp()` on success, drops the packet on failure WITHOUT
killing the reader thread.
The test fixtures mirror `tests/test_voice_session_cleanup.py:33-54`:
* Construct `AudioReader` via `AudioReader.__new__(AudioReader)` + manual
attr set so the reader thread is never started.
* `MagicMock` everything below the unit under test.
`_HAS_DAVE` / `_MEDIA_TYPE_AUDIO` on the reader module are monkey-patched per
test so the suite passes whether or not `davey` is importable in the venv.
The assertions only become meaningful once Lane A's patch has landed and the
package has been re-installed (`pip install -e vendor/discord-ext-voice-recv
--force-reinstall`); the FILE itself is valid Python regardless.
See plan: /home/moltbot/.claude/plans/wiggly-exploring-glade.md
"""
from __future__ import annotations
from unittest.mock import MagicMock
import pytest
from discord.ext.voice_recv.reader import AudioReader
# Sentinel for `_MEDIA_TYPE_AUDIO`. Using a plain object() keeps the tests
# independent of whether davey is importable — we just assert the value
# flows through to `dave_session.decrypt()` unchanged.
_FAKE_MEDIA_TYPE_AUDIO = object()
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture
def fake_dave_session():
sess = MagicMock(name="dave_session")
sess.ready = True
# Default: this user is NOT in passthrough — DAVE decrypt must run.
# Individual tests can override to True to exercise the passthrough path.
sess.can_passthrough = MagicMock(return_value=False)
sess.decrypt = MagicMock(return_value=b"plaintext_opus")
return sess
@pytest.fixture
def fake_connection(fake_dave_session):
conn = MagicMock(name="_connection")
conn.dave_protocol_version = 1
conn.dave_session = fake_dave_session
return conn
@pytest.fixture
def fake_voice_client(fake_connection):
vc = MagicMock(name="voice_client")
vc._connection = fake_connection
vc._ssrc_to_id = {12345: 999_000}
return vc
@pytest.fixture
def fake_rtp_packet():
pkt = MagicMock(name="rtp_packet")
pkt.ssrc = 12345
pkt.decrypted_data = b"ciphertext_after_transport_decrypt"
pkt.is_silence = MagicMock(return_value=False)
return pkt
@pytest.fixture
def reader(fake_voice_client):
"""`AudioReader` instance with no reader thread spawned.
Same pattern used by `tests/test_voice_session_cleanup.py` for
`VoiceSession` — bypass `__init__` so we can drive the public surface
against pure mocks.
"""
r = AudioReader.__new__(AudioReader)
r.voice_client = fake_voice_client
r.error = None
return r
@pytest.fixture
def dave_enabled(monkeypatch):
"""Force the reader module's DAVE-availability flags ON.
Pins `_MEDIA_TYPE_AUDIO` to a known sentinel so the happy-path test can
assert exactly what gets passed to `dave_session.decrypt`. `raising=False`
keeps the monkeypatch valid even if Lane A's patch hasn't landed yet —
the tests will still fail (no `_maybe_dave_decrypt` attr), just for the
right reason.
"""
import discord.ext.voice_recv.reader as reader_mod
monkeypatch.setattr(reader_mod, "_HAS_DAVE", True, raising=False)
monkeypatch.setattr(
reader_mod, "_MEDIA_TYPE_AUDIO", _FAKE_MEDIA_TYPE_AUDIO, raising=False
)
return reader_mod
# ---------------------------------------------------------------------------
# Unit tests: `_maybe_dave_decrypt`
# ---------------------------------------------------------------------------
class TestMaybeDaveDecrypt:
"""Seven unit tests on the DAVE-decrypt gate.
The gate mirrors `voice_client.can_encrypt` in discord.py 2.7.1 exactly
(`voice_state.py:272-273`). Bypass semantics on every "DAVE inactive"
branch let non-DAVE rooms and davey-less environments keep working.
"""
def test_protocol_version_zero_bypasses_decrypt(
self, dave_enabled, reader, fake_connection, fake_dave_session, fake_rtp_packet,
):
"""`dave_protocol_version == 0` → return the transport-decrypted
payload unchanged; never touch `dave_session.decrypt`."""
fake_connection.dave_protocol_version = 0
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result is fake_rtp_packet.decrypted_data
fake_dave_session.decrypt.assert_not_called()
def test_dave_session_none_bypasses_decrypt(
self, dave_enabled, reader, fake_connection, fake_rtp_packet,
):
"""`dave_session is None` → bypass. Pre-MLS-handshake state."""
fake_connection.dave_session = None
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result is fake_rtp_packet.decrypted_data
def test_dave_session_not_ready_bypasses_decrypt(
self, dave_enabled, reader, fake_dave_session, fake_rtp_packet,
):
"""`dave_session.ready is False` → bypass. Pre-MLS-epoch-1 packets
are transport-only on the wire."""
fake_dave_session.ready = False
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result is fake_rtp_packet.decrypted_data
fake_dave_session.decrypt.assert_not_called()
def test_unknown_ssrc_returns_none(
self, dave_enabled, reader, fake_voice_client, fake_dave_session, fake_rtp_packet,
):
"""SSRC not in `_ssrc_to_id` → drop (return None).
Accepted regression: davey requires per-user keys; when SPEAKING
events race behind the first audio packet, 1-5 packets per new
speaker per session are dropped. See plan §Edge cases.
"""
fake_voice_client._ssrc_to_id.clear()
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result is None
fake_dave_session.decrypt.assert_not_called()
def test_happy_path_invokes_decrypt_and_returns_plaintext(
self, dave_enabled, reader, fake_dave_session, fake_rtp_packet,
):
"""Full DAVE-active path: `decrypt(user_id, MediaType.audio, ciphertext)`
called exactly once with the expected args; method returns the
davey plaintext bytes verbatim."""
ciphertext = fake_rtp_packet.decrypted_data
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result == b"plaintext_opus"
fake_dave_session.decrypt.assert_called_once_with(
999_000, _FAKE_MEDIA_TYPE_AUDIO, ciphertext,
)
def test_decrypt_raises_returns_none_no_crash(
self, dave_enabled, reader, fake_dave_session, fake_rtp_packet,
):
"""davey.decrypt raising → drop the packet, don't propagate, and
leave `reader.error` untouched so the reader thread stays alive.
MLS epoch transitions can produce transient decrypt failures —
bumping `reader.error` would call `self.stop()` and kill the whole
receive pipeline."""
fake_dave_session.decrypt.side_effect = RuntimeError(
"simulated MLS epoch transition fail"
)
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result is None
assert reader.error is None
def test_has_dave_false_bypasses_even_with_session_present(
self, monkeypatch, reader, fake_dave_session, fake_rtp_packet,
):
"""`_HAS_DAVE = False` → bypass everything, even if a real session
somehow showed up on the connection. Defensive shim that keeps the
tests (and any davey-less deploys) green."""
import discord.ext.voice_recv.reader as reader_mod
monkeypatch.setattr(reader_mod, "_HAS_DAVE", False, raising=False)
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result is fake_rtp_packet.decrypted_data
fake_dave_session.decrypt.assert_not_called()
def test_can_passthrough_true_returns_payload_without_decrypt(
self, dave_enabled, reader, fake_dave_session, fake_rtp_packet,
):
"""`can_passthrough(user_id) == True` → return the transport-decrypted
payload as-is; never call `decrypt`. Mirrors Discord's protocol where
a passthrough-mode peer sends non-DAVE-wrapped packets that the
receiver must accept verbatim."""
fake_dave_session.can_passthrough = MagicMock(return_value=True)
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result is fake_rtp_packet.decrypted_data
fake_dave_session.can_passthrough.assert_called_once_with(999_000)
fake_dave_session.decrypt.assert_not_called()
def test_can_passthrough_raises_falls_through_to_decrypt(
self, dave_enabled, reader, fake_dave_session, fake_rtp_packet,
):
"""`can_passthrough` raising → swallow the error and try `decrypt`.
Defensive: an older davey build or transient internal state shouldn't
break the receive pipeline."""
fake_dave_session.can_passthrough = MagicMock(
side_effect=RuntimeError("simulated davey internal error")
)
result = reader._maybe_dave_decrypt(fake_rtp_packet)
assert result == b"plaintext_opus"
fake_dave_session.decrypt.assert_called_once()
# ---------------------------------------------------------------------------
# Integration tests: `callback()` exercises the DAVE hook
# ---------------------------------------------------------------------------
class TestCallbackIntegration:
"""Two integration tests for the hook Lane A inserts between transport
decrypt (reader.py:141) and the post-decrypt routing (reader.py:159).
Strategy: stub the transport-decrypt and RTP parsing path so `callback()`
reaches the hook, then mock `_maybe_dave_decrypt` directly on the reader
instance. The assertion focuses on `feed_rtp` being called (test 8) vs.
not called (test 9). The transport path correctness is covered by
voice-recv's own upstream tests.
"""
@staticmethod
def _wire_callback(reader, monkeypatch, fake_rtp_packet):
import discord.ext.voice_recv.reader as reader_mod
# Redirect rtp parsing — we want an RTP path (not RTCP) so the hook fires.
monkeypatch.setattr(reader_mod.rtp, "is_rtcp", lambda data: False)
monkeypatch.setattr(reader_mod.rtp, "decode_rtp", lambda data: fake_rtp_packet)
# Stub the instance attrs `callback()` touches besides the hook.
reader.decryptor = MagicMock(name="decryptor")
reader.decryptor.decrypt_rtp = MagicMock(return_value=b"ciphertext")
reader.packet_router = MagicMock(name="packet_router")
reader.packet_router.feed_rtp = MagicMock()
reader.speaking_timer = MagicMock(name="speaking_timer")
reader.sink = MagicMock(name="sink")
def test_callback_feeds_when_dave_returns_bytes(
self, monkeypatch, reader, fake_rtp_packet,
):
"""Hook returns plaintext → `feed_rtp` called once with the
rtp_packet whose `decrypted_data` is now the post-DAVE plaintext."""
self._wire_callback(reader, monkeypatch, fake_rtp_packet)
plaintext = b"dave_unwrapped_opus_payload"
reader._maybe_dave_decrypt = MagicMock(return_value=plaintext)
reader.callback(b"raw_packet_bytes")
reader._maybe_dave_decrypt.assert_called_once_with(fake_rtp_packet)
assert reader.packet_router.feed_rtp.call_count == 1
called_with = reader.packet_router.feed_rtp.call_args[0][0]
assert called_with is fake_rtp_packet
assert fake_rtp_packet.decrypted_data == plaintext
assert reader.error is None
def test_callback_drops_when_dave_returns_none(
self, monkeypatch, reader, fake_rtp_packet,
):
"""Hook returns None → `feed_rtp` NOT called, no exception propagated,
`reader.error` stays None (reader thread survives the drop)."""
self._wire_callback(reader, monkeypatch, fake_rtp_packet)
reader._maybe_dave_decrypt = MagicMock(return_value=None)
reader.callback(b"raw_packet_bytes")
reader._maybe_dave_decrypt.assert_called_once_with(fake_rtp_packet)
reader.packet_router.feed_rtp.assert_not_called()
assert reader.error is None

View File

@@ -23,6 +23,24 @@ VOICES = {"M1", "M2", "M3", "M4", "M5", "F1", "F2", "F3", "F4", "F5"}
DEFAULT_VOICE = "M2" DEFAULT_VOICE = "M2"
DEFAULT_LANG = "ro" DEFAULT_LANG = "ro"
# Punctuation Supertonic synthesis rejects with HTTP 500 (Romanian curly quotes,
# smart dashes, ellipsis, angle quotes). Mapped to ASCII so a stray „foo" in
# any caller's text doesn't kill the whole request.
_TTS_PUNCT_MAP = {
'': '"', '': '"', '': '"',
'': "'", '': "'", '': "'",
'«': '"', '»': '"',
'': '-', '': '-',
'': '...',
}
def sanitize_for_supertonic(text: str) -> str:
"""Replace Unicode punctuation Supertonic rejects with ASCII equivalents."""
for src, dst in _TTS_PUNCT_MAP.items():
text = text.replace(src, dst)
return text
def synthesize(text: str, voice: str = DEFAULT_VOICE, lang: str = DEFAULT_LANG) -> dict: def synthesize(text: str, voice: str = DEFAULT_VOICE, lang: str = DEFAULT_LANG) -> dict:
"""Call Supertonic server and save audio to a temp WAV file. """Call Supertonic server and save audio to a temp WAV file.
@@ -34,6 +52,8 @@ def synthesize(text: str, voice: str = DEFAULT_VOICE, lang: str = DEFAULT_LANG)
if not text or not text.strip(): if not text or not text.strip():
return {"ok": False, "error": "Text gol."} return {"ok": False, "error": "Text gol."}
text = sanitize_for_supertonic(text)
voice = voice.upper() voice = voice.upper()
if voice not in VOICES: if voice not in VOICES:
voice = DEFAULT_VOICE voice = DEFAULT_VOICE

View File

@@ -1,22 +1,76 @@
# Vendored: discord-ext-voice-recv # Vendored: discord-ext-voice-recv
**Upstream:** https://github.com/imayhaveborkedit/discord-ext-voice-recv **Upstream:** https://github.com/imayhaveborkedit/discord-ext-voice-recv
**Pinned commit:** `ac04ea7b0941112e83767cf1c1469b408fa06748` (bump version 0.5.3a) **Pinned commit:** `ac04ea7b0941112e83767cf1c1469b408fa06748` (bump version 0.5.3a, master HEAD Jun 2025)
**Vendored at:** 2026-05-27 **Vendored at:** 2026-05-27
**Echo Core fork version:** `0.5.3a+echo.dave1` (PEP 440 local segment)
**Reason:** Discord voice protocol is fragile, upstream is hobby fork. Adapter **Reason:** Discord voice protocol is fragile, upstream is hobby fork. Adapter
layer in `src/voice/_discord_voice_adapter.py` isolates upstream churn — if this layer in `src/voice/_discord_voice_adapter.py` isolates upstream churn — if this
package breaks, swap to py-cord by rewriting only that file. package breaks, swap to py-cord by rewriting only that file.
## Update procedure ## Echo Core patch: `+echo.dave1` (DAVE E2E receive-side decrypt)
### Why
Discord enforces DAVE (E2E media encryption) on voice gateway `v=8` whenever the
bot advertises `max_dave_protocol_version > 0` in IDENTIFY. discord.py 2.7.1 (the
version Echo Core pins) does so unconditionally — Discord then closes the WS
with code **4017** if the bot opts out by sending `max_dave_protocol_version=0`.
DAVE is **mandatory**.
Audio received from a DAVE-active room is **dual-wrapped**: transport layer
(`aead_xchacha20_poly1305_rtpsize`) + DAVE E2E. Upstream voice-recv decrypts
only the transport layer, then hands DAVE ciphertext to libopus, which raises
`OpusError: corrupted stream` on every packet.
### Patch shape
~30 lines, all in `discord/ext/voice_recv/reader.py`:
1. Module-level optional `davey` import (no-op when missing).
2. `AudioReader._maybe_dave_decrypt(rtp_packet) -> Optional[bytes]` — gate logic
mirrors discord.py 2.7.1 send-side `can_encrypt` exactly. Returns the
DAVE-unwrapped payload, the original payload (DAVE inactive), or `None` to
drop the packet (unknown SSRC, decrypt failure).
3. 4-line hook in `callback()` between transport-decrypt and `feed_rtp`:
overwrites `rtp_packet.decrypted_data` in place, or returns early to drop.
The post-decrypt `is_silence()` check (formerly at reader.py:172) still works
because we overwrite `decrypted_data` in place — silence frames produced by
davey reach the existing check unchanged.
### Dependency
`davey==0.1.5` — matches discord.py 2.7.1 expectation. Pin in
`echo-core/requirements.txt`. The import is optional at module level so tests
and non-DAVE environments still run; the gate degrades to a bypass.
### Re-sync strategy
When upstream voice-recv adds DAVE support natively:
1. Drop the three patch hunks in `reader.py` (davey import block,
`_maybe_dave_decrypt` method, hook in `callback()`).
2. Revert `__version__` to upstream value in `__init__.py`.
3. Update `Pinned commit` below.
4. Run `pytest tests/test_voice_recv_dave.py tests/test_voice_adapter_contract.py`.
The contract test `test_voice_recv_fork_version` asserts `__version__ ==
'0.5.3a+echo.dave1'` and will fail fast on any accidental wipe during a careless
upstream sync — forcing a conscious decision to either re-port or drop the
patch.
## Update procedure (vanilla upstream sync)
```bash ```bash
cd vendor/discord-ext-voice-recv cd vendor/discord-ext-voice-recv
git fetch origin master git fetch origin master
git log HEAD..origin/master --oneline # review what changed git log HEAD..origin/master --oneline # review what changed
git checkout <new-commit> git checkout <new-commit>
# RE-APPLY the +echo.dave1 patch if upstream still lacks DAVE
cd ../.. cd ../..
source .venv/bin/activate && pip install -e vendor/discord-ext-voice-recv --force-reinstall source .venv/bin/activate && pip install -e vendor/discord-ext-voice-recv --force-reinstall
pytest tests/test_voice_adapter_contract.py -v # MUST PASS — contract guard pytest tests/test_voice_adapter_contract.py tests/test_voice_recv_dave.py -v # MUST PASS — contract + DAVE guards
``` ```
Update this file's `Pinned commit` after a successful upgrade. Update this file's `Pinned commit` after a successful upgrade.

View File

@@ -17,4 +17,4 @@ __title__ = 'discord.ext.voice_recv'
__author__ = 'Imayhaveborkedit' __author__ = 'Imayhaveborkedit'
__license__ = 'MIT' __license__ = 'MIT'
__copyright__ = 'Copyright 2021-present Imayhaveborkedit' __copyright__ = 'Copyright 2021-present Imayhaveborkedit'
__version__ = '0.5.3a' __version__ = '0.5.3a+echo.dave1'

View File

@@ -19,6 +19,15 @@ try:
except ImportError as e: except ImportError as e:
raise RuntimeError("pynacl is required") from e raise RuntimeError("pynacl is required") from e
# Echo Core +echo.dave1 patch: DAVE E2E receive-side decrypt. See VENDOR_INFO.md.
try:
import davey
_MEDIA_TYPE_AUDIO = davey.MediaType.audio
_HAS_DAVE = True
except ImportError:
_MEDIA_TYPE_AUDIO = None
_HAS_DAVE = False
if TYPE_CHECKING: if TYPE_CHECKING:
from typing import Optional, Callable, Any, Dict, Literal, Union from typing import Optional, Callable, Any, Dict, Literal, Union
@@ -133,12 +142,63 @@ class AudioReader:
def _is_ip_discovery_packet(self, data: bytes) -> bool: def _is_ip_discovery_packet(self, data: bytes) -> bool:
return len(data) == 74 and data[1] == 0x02 return len(data) == 74 and data[1] == 0x02
def _maybe_dave_decrypt(self, rtp_packet) -> Optional[bytes]:
"""DAVE E2E layer applied after transport decrypt.
Returns the (possibly DAVE-unwrapped) opus payload, or None to drop the
packet. No-op when DAVE is inactive — non-DAVE rooms and environments
without `davey` installed pass through unchanged.
NOTE: `is_silence()` is NOT checked here. In a DAVE-active room the
transport-decrypted payload is ciphertext, so `is_silence()` (which
compares to plaintext OPUS_SILENCE ``b'\\xf8\\xff\\xfe'``) never matches.
Silence frames are handled either by davey.decrypt returning plaintext
silence (then caught at the existing post-decrypt silence check on
``decrypted_data``), or dropped via the decrypt-raises path. The
existing post-decrypt silence check continues to work because we
overwrite ``decrypted_data`` in place.
"""
if not _HAS_DAVE:
return rtp_packet.decrypted_data
conn = self.voice_client._connection
if getattr(conn, 'dave_protocol_version', 0) == 0:
return rtp_packet.decrypted_data
dave = getattr(conn, 'dave_session', None)
if dave is None or not dave.ready:
return rtp_packet.decrypted_data
user_id = self.voice_client._ssrc_to_id.get(rtp_packet.ssrc)
if user_id is None:
# ACCEPTED REGRESSION: davey requires per-user key. When SPEAKING
# event races behind the first audio packet, we drop 1-5 packets
# (~40-200ms) per new speaker per session.
return None
# can_passthrough(user_id) mirrors Discord's protocol: when this user's
# decryptor is in passthrough mode, packets are not DAVE-wrapped and
# must be returned as-is. Otherwise davey.decrypt unwraps DAVE E2E.
try:
if dave.can_passthrough(user_id):
return rtp_packet.decrypted_data
except Exception as e:
log.debug("can_passthrough check failed for ssrc=%s user=%s: %s: %s",
rtp_packet.ssrc, user_id, type(e).__name__, e)
try:
return dave.decrypt(user_id, _MEDIA_TYPE_AUDIO, rtp_packet.decrypted_data)
except Exception as e:
log.debug("DAVE decrypt failed for ssrc=%s user=%s: %s: %s",
rtp_packet.ssrc, user_id, type(e).__name__, e)
return None
def callback(self, packet_data: bytes) -> None: def callback(self, packet_data: bytes) -> None:
packet = rtp_packet = rtcp_packet = None packet = rtp_packet = rtcp_packet = None
try: try:
if not rtp.is_rtcp(packet_data): if not rtp.is_rtcp(packet_data):
packet = rtp_packet = rtp.decode_rtp(packet_data) packet = rtp_packet = rtp.decode_rtp(packet_data)
packet.decrypted_data = self.decryptor.decrypt_rtp(packet) packet.decrypted_data = self.decryptor.decrypt_rtp(packet)
# Echo Core +echo.dave1: DAVE E2E layer (no-op when inactive).
dave_payload = self._maybe_dave_decrypt(rtp_packet)
if dave_payload is None:
return # drop packet, do not feed_rtp; reader thread stays alive
rtp_packet.decrypted_data = dave_payload
else: else:
packet = rtcp_packet = rtp.decode_rtcp(self.decryptor.decrypt_rtcp(packet_data)) packet = rtcp_packet = rtp.decode_rtcp(self.decryptor.decrypt_rtcp(packet_data))