Mark's Reports

research · 2026-04-13

Voice Cloning + Speaker ID — Landscape & Plan

Date: 2026-04-13

For: Mark Barnett (TP3 / OMI corpus / Legacy Soil narration)

1. TTS / Voice-Cloning Landscape (April 2026)

Open source — consolidated

Coqui AI shut down December 2025. The codebase lives on as Idiap's coqui-tts fork (last release Jan 2026), still shipping XTTS v2 — 6-second zero-shot cloning, 17 languages. CPML license forbids commercial use without a separate agreement, so OK for personal narration but not Legacy Soil customer videos.

The new center of gravity is Resemble AI's Chatterbox (2025, MIT, 500M params, trained on 500K hours). Zero-shot cloning from seconds of reference, emotion-exaggeration controls (a first for open source), Turbo variant runs sub-200ms latency. Every output is stamped with Resemble's "Perth" perceptual watermark. Blind evaluations have it beating ElevenLabs on quality. First reach on Apex.

Other serious open options: F5-TTS (MIT, 7x realtime, commercial-friendly), OpenVoice v2 (MIT, cross-lingual, granular emotion/accent control), Fish Speech V1.5, CosyVoice2, IndexTTS-2 (strong, with license caveats). Tortoise is legacy. No Anthropic / Google / Meta open-weight TTS worth using — Google is API-only (Chirp 3 HD), Meta has Voicebox papers but no public weights, Anthropic has nothing here.

Commercial — ElevenLabs still rules

ElevenLabs is still the quality leader and the easiest path. Pricing April 2026:

PVC is what matters for Legacy Soil narration — IVC drifts over multi-minute reads. ElevenLabs watermarks all output and requires a consent attestation.

Play.ht, Resemble.ai, Microsoft Custom Neural Voice are competent alternatives. Microsoft CNV is enterprise-gated (weeks of approval). Resemble.ai's hosted product is pricier than ElevenLabs. Play.ht is a distant third.

Sample length needed: ElevenLabs IVC ~1 minute clean; PVC ~30 minutes scripted; XTTS / Chatterbox / F5-TTS work from 6-30 seconds zero-shot. Mark's hundreds of hours of OMI audio is overkill. The bottleneck is quality of reference (no music/overlap/noise), not quantity.

2. Speaker Identification / Diarization

Two related tasks: diarization (who-talks-when segments) and identification (match a segment against Mark's known voiceprint).

pyannote.audio 3.1 is the open-source standard. DER ~10% on clean benchmarks, 15-25% on messy real-world audio. MIT-style license, runs on Apex's RTX 3090 fine. Pyannote also ships Precision-2, a paid premium model 28% more accurate than OSS 3.1 — only worth it for forensic-grade work (Mark doesn't need it).

WhisperX wraps Whisper transcription + pyannote diarization + word-level alignment. On a 3090 it's roughly 7x faster than pyannote alone (75s vs 520s on the same clip in published benchmarks) because it pins GPU utilization properly. For OMI processing, WhisperX is the right tool — transcript + diarization in one pass.

For identification (not just "speaker A vs B" but "is speaker A actually Mark"), the standard is an ECAPA-TDNN speaker embedding from SpeechBrain. Compute a 192-dim embedding of Mark's voice from a clean 30-second sample, store as a vector, then at inference cosine-compare against every diarized segment. >0.7 = Mark with high confidence. Forensic-grade, MIT-licensed, milliseconds per segment on the 3090.

3. Recommended Path for Mark

Use case 1 — auto-tag OMI audio "is this Mark?"

WhisperX (uses pyannote underneath) + a stored ECAPA-TDNN embedding in TP3's pgvector. Pipeline: clip arrives → WhisperX produces transcript with speaker labels → ECAPA embedding per label → cosine-compare against Mark's stored embedding → tag segments mark / other_1 / other_2. All open source, all on Apex, no per-call cost. Embedding is ~1KB per speaker.

Use case 2 — narration cloning for Legacy Soil and document read-alouds

Two-track:

On the OMI corpus as training data: not actually useful for cloning. Modern models need 30 sec to 30 min of clean audio, not 300 hours of bone-conduction wearable audio with HVAC hum and kitchen ambience. The corpus IS gold for (a) building a speaker-ID dataset by mining the cleanest Mark-only chunks, and (b) eventually fine-tuning a personal LLM on Mark's transcribed speech patterns — separate project.

4. Ethics & Safety

5. Concrete Next Steps

1. Record a clean 30-min reference take. Quiet room, USB condenser mic (not OMI, not laptop), scripted text in the Legacy Soil tone. This single file unlocks every option below. (~1 hr)

2. Sign up for ElevenLabs Creator ($22). Train a PVC from the 30-min file. Generate one test: first 500 words of the Legacy Soil proposal as audio. Decide if quality clears the customer-facing bar. (~1 hr)

3. Install WhisperX + pyannote + SpeechBrain on Apex. Process one week of OMI audio. Compute Mark's ECAPA embedding from the cleanest 60-sec chunk. Store in TP3 pgvector with a voice_embeddings table. (Cursor/Jules mission brief.)

4. Install Chatterbox on Apex as the local fallback. A/B against ElevenLabs on the same Legacy Soil paragraph. Document which Mark prefers.

5. Write the consent memo. One paragraph in AI_Library, dated, naming the authorized models. Future-proofs against any 2027 regulation that retroactively asks "when did you consent."

Sources