Indian classrooms are multilingual, but lectures are delivered in one language — usually English or Hindi. Students who aren't fluent in the medium of instruction fall behind, not for lack of ability but for lack of language. The tools that exist are batch or offline, English-centric, or break on the way multilingual speakers actually talk: switching fluidly between English and a regional language mid-sentence, sometimes mid-word.

The goal was narrow and hard: let a teacher speak naturally, and have every listener receive an accurate, low-latency caption — and optional spoken audio — in their language, on their own phone, with zero setup beyond a QR code. Real-time. Many listeners, each on a different language, at once. The difficulty isn't any single model call — it's holding a streaming speech pipeline stable in a noisy room, for the length of a real lecture, without dropping a word or a device.

A one-teacher → many-listener live translation pipeline, built and deployed solo:

• The teacher opens a web page and speaks into their mic.
• Audio streams to the server, is transcribed in real time, split into sentences, and translated concurrently into every language currently being listened to.
• Each listener's phone shows live captions in their chosen language (~1–3s behind), with an optional spoken voice reading it aloud.
• Supports 22 Indian languages, multiple simultaneous devices each on a different language, and per-device voice selection.

Producer–consumer with async fan-out. One translation is computed per active language and shared across all of that language's listeners (dedup), so work scales with languages in the room, not devices. WebSocket streaming end-to-end.

teacher mic (browser)
       │  16 kHz PCM over WebSocket
       ▼
┌─────────────────────────────┐
│  adaptive loudness gate     │  ← rejects room noise, never the teacher
└─────────────────────────────┘
       │
       ▼
┌─────────────────────────────┐
│  streaming STT (code-mix)   │
└─────────────────────────────┘
       │  finalized speech segments
       ▼
┌─────────────────────────────┐
│  sentence aggregation (VAD) │  ← emit one coherent sentence
└─────────────────────────────┘
       │
       ▼   fan-out: translate ONCE per active language (dedup)
   ┌────────────┬────────────┬────────────┐
   ▼            ▼            ▼            ▼
 Hindi        Tamil        Telugu       …  (LLM translation
   │            │            │               + automatic fallback)
   ▼            ▼            ▼            ▼
 broadcast captions to each listener in their language (WebSocket)
                     │  optional
                     ▼
             neural TTS → spoken audio per device

Each of these was a real production bug, root-caused from logs and fixed — the part that separates a demo from a system that survives a full lecture.

1. Background noise → transcription hallucination

Faint background chatter was being transcribed as if it were the teacher. A neural voice-activity detector can't reject it — chatter is speech; only loudness distinguishes it. I built an adaptive, peak-relative loudness gate that learns the room's noise floor and stays mathematically bounded belowthe teacher's own voice, so it rejects background without ever choking real speech.

2. “Stops working after 3–4 minutes”

Root-caused from production logs: the noise-floor threshold was slowly climbing above the teacher's voice and latching the gate shut permanently. Fixed with a peak-relative gate that can never exceed a fraction of recent peak, plus a browser mic watchdog that rebuilds a dead audio graph without dropping the connection.

3. Transcription flushing corrupted words

A fixed-interval force-flush was cutting words mid-token — “determinant” → “Dominant.” Replaced with silence-aware (VAD) flushing: flush only on a real pause, at a genuine word boundary.

4. Incoherent captions from sentence fragments

Speech pauses are not sentence boundaries. I added a sentence-aggregation layer that buffers fragments and emits one coherent unit only when the speaker genuinely finishes a thought — so translation runs on whole sentences, not stutters.

5. Multi-device, multi-language

A silent cap was treating language as a single global slot and blocking the 5th language. Re-architected to per-device language + voice, fixed the caps, and verified five phones in five different languages— captions and voice — end-to-end.

6. Final-sentence race

On teacher disconnect, an in-flight translation was being cancelled and the last sentence dropped. Fixed by awaiting the in-flight emit on teardown instead of cancelling it — the lecture's final line always lands.

The production pipeline is the proving ground. Beyond it, I'm building and refining our own voice AI models— speech models tuned for Indian languages and the code-mixed, accented, real-classroom speech that general models handle worst. The direction is deliberate: less dependence on generic engines, and better handling of the exact cases — names, code-switching, regional accents, and rare languages — that off-the-shelf systems miss. The live system tells me precisely where those models need to be better, and each iteration goes straight back into production.

Backend

Python 3.11, FastAPI, WebSockets, asyncio

Speech-to-text

Streaming, code-mix capable

Translation

LLM-based, per-language, with automatic fallback

Text-to-speech

Neural, per-device voice

Frontend

Vanilla JS, WebSocket client, responsive HTML/CSS

Audio / DSP

16 kHz PCM, RMS loudness gating, VAD

Infra

Cloud VM (Ubuntu), nginx (WS reverse proxy), systemd, TLS

• ~1–3s end-to-end caption latency, streaming.
• 22 Indian languages supported.
• Verified 5 simultaneous devices, 5 different languages (captions + voice).
• Sub-6% word error rate on code-mixed speech — the hardest case for general models.
• Deployed with nginx + systemd + TLS; full end-to-end test suite (surfaces, multi-language, noise, multi-device).

Being honest about the edges.

• Single speaker.Built for one teacher at a time — no multi-speaker diarization. Two people talking over each other is out of scope.
• Online only. Needs connectivity end-to-end; no offline / on-device mode yet. Low-bandwidth rooms degrade to captions-only by choice.
• Hardest on the rarest languages.Coverage is strongest for high-use languages; the rarest of the 22 are where quality is still climbing — which is exactly what the in-house models are aimed at, and I flag it rather than hide it.

Real-time systems (WebSockets, streaming, asyncio, producer–consumer) · applied AI pipelines (speech-to-text, machine translation, text-to-speech, multimodal) · building and refining custom voice models · systematic production debugging and root-cause analysis · audio / DSP (loudness gating, VAD) · measurement-driven evaluation and quality benchmarking · DevOps (Linux, nginx, systemd, TLS, cloud). Built and shipped solo, running in production.