Multilingual Voice Lab

Testing low-latency neural speech synthesis and streaming voice translation.

TECHNOLOGY:WebSocketsWeb Audio APIElevenLabsTypeScript

01 // WHY I BUILT IT

Real-time speech applications frequently suffer from high latency and unnatural prosody when transitioning between languages like Indonesian and English.

02 // THE HYPOTHESIS & IDEA

Buffer incoming text tokens into semantic chunk boundaries (phrases/clauses) before dispatching to neural text-to-speech engines to reduce time-to-first-byte while preserving natural intonation.

03 // WHAT I TRIED

Built a bidirectional WebSocket streaming server that accepts text chunks, streams synthesized PCM audio back to the browser, and plays it via Web Audio API audio buffers.

04 // WHAT WORKED

Chunking at punctuation and conjunction boundaries reduced perceived latency by ~45% while eliminating audio clipping.

05 // WHAT FAILED

Direct word-by-word streaming resulted in robotic cadence and flat pitch transitions.

06 // KEY LEARNINGS

Natural speech synthesis requires small semantic units rather than raw token streaming.

07 // EXPERIMENTAL RESULT

A working prototype demonstrating under 350ms time-to-first-audio playback with smooth phrase boundaries.

08 // NEXT ITERATION

Experiment with local ONNX speech models running directly in WebAssembly to remove cloud dependency.

## Laboratory Notes This prototype tested techniques for bridging streaming LLM text responses into real-time audio playback without awkward stuttering. ### Architecture ``` LLM Stream (Tokens) ↓ Semantic Boundary Parser (Punctuation/Clauses) Text Chunks ↓ WebSocket Stream Neural TTS Model ↓ Chunked PCM Audio Web Audio API AudioBufferSourceNode ```