Moonshine + sherpa-onnx + Unsloth — Pre-Pivot Deep Research

Author: Claude Opus 4.7 (research agent) Date: 2026-05-10 (ET) Context: Tensor G3 NPU confirmed unavailable (LiteRT-LM 0.11 returns NOT_FOUND). Bidet AI v0.3 architecture has to land on routed STT → LLM. This document verifies, with citations, what we are integrating before we commit.

1. Moonshine architecture — verified

Key architectural takeaway: Moonshine is an encoder-decoder transformer like Whisper but with two structural wins — (a) RoPE replaces absolute position embeds so it handles variable-length audio without zero-padding, and (b) the encoder uses a learned conv stem on raw 16 kHz waveform, sidestepping mel-spectrogram preproc entirely. The non-padding property is where the 5× compute reduction comes from at inference: no wasted computation on padded silence.

Streaming reality check: the original Moonshine-Tiny is non-streaming (give it the full clip, it returns the full transcript). For incremental output as the user is still talking, the "Streaming" variants (Tiny-Streaming 34M, etc.) are needed. They use KV-cache and an "ergodic" encoder that processes audio in chunks. This is a separate model file, not a runtime mode flip on the same weights.

2. sherpa-onnx — verified

Integration shape on Android: drop the AAR into app/libs/, add implementation files('libs/sherpa-onnx-1.13.1.aar') to Gradle, place the Moonshine ONNX files (encoder + decoder + tokenizer/tokens.txt) under assets/, then call OfflineRecognizer.fromConfig(...) with paths. ~50 LOC swap from a whisper.cpp setup.

Pre-converted Moonshine ONNX bundles ship with tokens.txt, so we don't have to convert anything ourselves — sherpa-onnx hosts them as release downloads.

3. Integration into bidet-phone — sized

3a. What changes in the codebase

Total Kotlin LOC delta: ~100–150 LOC net (replace whisper JNI wrapper, simplify Gradle).

3b. APK size delta

Current bidet-phone APK: ~186 MB (per memory file project_bidet_phone_v0.1_working_2026-05-09.md).

Caveat: the 30 MB / 104 MB encoder/decoder split from the GitHub README is for the .ort flatbuffer format (memory-mappable, not the smallest). The published sherpa-onnx INT8 quantized bundle (sherpa-onnx-moonshine-tiny-en-int8) is in the 30–50 MB total range — that's what we'd ship.

Safe APK target: ~196–210 MB. Worth re-measuring the day we land it.

3c. Runtime memory peak (Moonshine + Gemma 4 E2B simultaneously)

Pixel 8 Pro has 12 GB RAM. Comfortable headroom. E4B (3.66 GB on disk → ~4.5 GB peak) would push closer to the OOM line — confirms the rule we already wrote: ship E2B for the contest demo.

3d. Streaming partial-transcript UX

Yes — sherpa-onnx supports streaming Moonshine via the OnlineRecognizer API + the streaming Moonshine checkpoints. Microphone + VAD examples are in the Android demo. We can surface partial text to the RAW tab as the user is still talking. This is a major UX win over current whisper.cpp setup which is batch-only.

Sequence: OnlineStream accepts 100 ms PCM chunks → recognizer.decode(stream) returns partial text → emit to RecordingViewModel.partialText StateFlow → bind to TextView. ~30 LOC addition.

3e. Blockers / risks

1. NDK ABI cohabitation with LiteRT-LM. Both LiteRT-LM and sherpa-onnx ship libonnxruntime.so (LiteRT-LM uses TFLite + a built-in ORT; sherpa-onnx ships its own ORT). If both AARs ship libonnxruntime.so at different versions, the second one loaded silently overwrites the first, and you get version-mismatch crashes. Mitigation: extract sherpa-onnx's libs and use Gradle packagingOptions { pickFirst "**/libonnxruntime.so" } deliberately — verify which version wins works for both. This is the #1 risk and needs a 30-min spike on day 1 of the migration.
2. ProGuard/R8. sherpa-onnx Kotlin bindings use reflection to load native methods. Add -keep class com.k2fsa.sherpa.onnx.** { *; } to proguard-rules.pro. Trivial.
3. Maven Central non-availability. Have to vendor the AAR (commit it to app/libs/ or pull via Gradle from URL on build). Not a blocker — just operationally unfamiliar.
4. Streaming Moonshine model file is a different download. Standard sherpa-onnx-moonshine-tiny-en-int8 is non-streaming. To get partial output we'd need the streaming variant — verify it's published as INT8 sherpa-onnx bundle (it is for moonshine-tiny streaming per [4], but double-check before committing in PR).

4. Where Unsloth fine-tune fits

This is the load-bearing question for the Unsloth $10K side prize.

Option A — Fine-tune Gemma 4 E2B on the 79 paired (raw_text → clean_text) triples

Option B — Fine-tune Moonshine-Tiny on Mark's voice (acoustic adaptation)

Option C — Both, sequenced (Moonshine for voice, Gemma for vocabulary)

Sequencing: Option A first (it's the prize-eligible one and ships in 1 day), Option B as bonus week-2 polish if there's time.

Recommendation: Option A only for contest deadline. Option B is a v0.4 stretch goal.

Reasoning: 1. Option A is the only path that wins the Unsloth $10K (Moonshine fine-tune doesn't qualify — wrong model family). 2. Option A is 1 day of work vs Option B's 2–3 days minimum. 3. Option A's improvement is directly visible in the demo (cleanup tab generates Mark-correct output instead of hallucinating "Hasspin"). Option B's improvement (1–3 WER points on raw STT) is invisible to a 3-min video judge. 4. Option A doesn't risk APK / sherpa-onnx packaging surprises.

Carry Option B as v0.4 backlog. If we win, finish Moonshine fine-tune in June.

5. Cactus prize narrative fit

Verbatim prize description (from reference_kaggle_gemma4_prize_tree_2026-05-09.md, sourced via Mark's logged-in Kaggle Chrome MCP capture 2026-05-09)

Cactus Special Technology Prize — $10,000: "For the best local-first mobile or wearable application that intelligently routes tasks between models."

Does Moonshine → Gemma 4 routing fit?

Yes — almost word-for-word.

• "Local-first mobile application": Bidet-phone is an Android app, runs Moonshine + Gemma entirely on-device, no cloud required.
• "Intelligently routes tasks between models": small-fast STT model (Moonshine, 27M) handles the audio-to-text task; large-context LLM (Gemma 4 E2B, 2.59 GB) handles the cleanup-and-summarize task. Different models, different strengths, routed by task type — exactly the description.

The one real concern: Cactus's own SDK

Cactus runs a separate branded hackathon (the "Cactus x DeepMind Hackathon" via AI Tinkerers) where their cactus-compute/functiongemma-hackathon repo requires building against the Cactus SDK [20]. That is NOT the Kaggle Cactus Special Tech Prize.

The Kaggle Cactus prize description (verified via Mark's Kaggle session) does not require their SDK. It rewards the architectural pattern (local-first, routed). Bidet's Moonshine + Gemma routing is eligible. Confirmed by the prize's own wording — it doesn't say "using Cactus" anywhere.

Hedge: if we want to double-down, we could write a tiny CactusFallback.kt that detects when Gemma 4 E2B refuses or hits a complexity ceiling and falls back to Gemini cloud (Cactus's signature pattern). Adds ~30 LOC, narrative gold. This is what Cactus's own blog champions [21].

Past Cactus-funded projects / what Cactus values

From docs.cactuscompute.com blog [21]: - "0.3 sec from end-of-audio to first token" on M4 Mac with Gemma 4 — they care about latency. - They champion single integrated multimodal models philosophically (Gemma 4 reasons over raw audio in one pass). - BUT their hybrid feature (Cactus = "low-latency engine for mobile devices & wearables") explicitly supports cloud handoff — small on-device model handles 80%, frontier cloud handles 20%.

The "single-model" philosophy is their preference for the audio→text→reasoning pipeline (Gemma does it all). The "routing" they reward in the Kaggle prize is task routing (small for STT, large for LLM, OR local for fast / cloud for complex).

These aren't contradictory — they're two layers of "routing": 1. Task-level routing (STT vs cleanup) — what Bidet does today. 2. Confidence-level routing (local vs cloud) — what Cactus champions.

We can do both in v0.3 by adding a 30-LOC "if Gemma's response was empty/low-confidence, optionally fall back to Gemini" path. That hits BOTH definitions of routing and locks the Cactus narrative.

Is there any reason the prize would prefer single-model?

Reading the Cactus blog posts [21] suggests yes-ish — they philosophically prefer single-model because it eliminates latency between stages. But the prize text doesn't say that — it says "intelligently routes tasks between models," which is explicitly multi-model.

My read: Cactus wrote the prize text as a separate definition from their own product preference. They're funding the broader ecosystem of routed local-first apps. Moonshine→Gemma fits.

6. Compatibility / risk matrix

7. Recommendation

v0.3 architecture diagram

┌─────────────────────────────────────────────────────┐
│  bidet-phone v0.3 (Android, Pixel 8 Pro target)    │
│                                                     │
│   16 kHz mic ──► AudioCaptureService                │
│                       │                             │
│                       │ Float[] PCM (100 ms chunks)  │
│                       ▼                             │
│                ┌──────────────────┐                 │
│                │  sherpa-onnx     │ ← AAR 53.9 MB    │
│                │  OnlineRecognizer│                 │
│                │  + Moonshine-Tiny│ ← ONNX ~40 MB    │
│                │  Streaming       │                 │
│                └──────┬───────────┘                 │
│                       │ partial text events         │
│                       ▼                             │
│              ┌─────────────────────┐                │
│              │ RAW tab (live)      │                │
│              └─────────┬───────────┘                │
│                        │ on stop: full transcript    │
│                        ▼                             │
│                ┌──────────────────┐                 │
│                │ LiteRT-LM        │                 │
│                │ + Gemma 4 E2B    │ ← .litertlm 2.6GB│
│                │   (UNSLOTH-      │   ← fine-tuned  │
│                │    fine-tuned    │     on 79 Mark   │
│                │    on Mark)      │     triples      │
│                │  Backend.CPU     │                  │
│                └──────┬───────────┘                 │
│                       │ cleaned text                │
│                       ▼                             │
│                ┌──────────────────┐                 │
│                │ CLEAN tab        │                 │
│                │ (3 modes:        │                 │
│                │  for-others,     │                 │
│                │  for-self,       │                 │
│                │  bullet-list)    │                 │
│                └──────────────────┘                 │
│                                                     │
│   [Optional Cactus-narrative fallback: if Gemma     │
│    output empty/low-conf, route to Gemini cloud]    │
└─────────────────────────────────────────────────────┘

Right Unsloth path

Option A: fine-tune Gemma 4 E2B on the 79 paired (raw_text → clean_text) triples. - 1 evening data prep + 1 hour training on Kaggle T4 + 2 hours convert/test = ~1 day end-to-end - Wins the Unsloth $10K side prize (Moonshine fine-tune doesn't qualify) - Improvement is visible in the demo video (cleanup tab generates Mark-correct vocab) - LoRA r=16, alpha=32, merged before MediaPipe convert

Contest framing

Kaggle Gemma 4 Hackathon Writeup angle:

"Bidet AI is a local-first mobile brain-dump cleanup tool that intelligently routes between two models: Moonshine-Tiny (27M params) handles real-time speech-to-text on-device, and a Gemma 4 E2B fine-tuned with Unsloth on a personalized vocabulary handles the cleanup. Total RAM footprint 3.8 GB — fits a Pixel 8 Pro with no NPU, no cloud, no compromise."

This single sentence hits three prize buckets: 1. Cactus Special Tech ($10K) — "intelligently routes tasks between models" verbatim 2. Unsloth Special Tech ($10K) — "fine-tuned Gemma 4 with Unsloth, optimized for a specific impactful task" 3. Future of Education Impact ($10K) OR Digital Equity Impact ($10K) — depends on framing (lecture-recorder for nephews vs accessibility tool)

Plus eligible for Main Track ($10K–$50K). Stacking ceiling per Kaggle rules: $70K (Main + Impact + ONE Special Tech). Realistic-best target: $30K (4th Main + Future of Ed + Cactus) or $30K (4th Main + Impact + Unsloth).

DEV.to writeup angle (separate prize ecosystem): different cover letter — emphasize the Apache-2.0 licensing story ("Gemma 4 is now Apache 2.0, this app ships free forever"), the on-device privacy story (raw audio never leaves the phone), and the educator-built-it angle.

Right "what to ship in next 7 days"

Budget if Day 1 (Mon) shows sherpa-onnx + LiteRT-LM ABI conflict: drop to Plan B — keep whisper.cpp + Whisper-tiny in v0.2 build, ship Unsloth-finetuned Gemma flavor only. Still hits Unsloth $10K + Cactus framing (Whisper→Gemma is also routed). Lose nothing but the "Moonshine 51× faster" sizzle. Re-attempt Moonshine in v0.4 post-contest.

Sources

[1] arXiv 2410.15608 — Moonshine: Speech Recognition for Live Transcription and Voice Commands. https://arxiv.org/abs/2410.15608 [2] HF transformers docs — Moonshine model. https://huggingface.co/docs/transformers/en/model_doc/moonshine [3] github.com/usefulsensors/moonshine — README and config docs (16 kHz, raw waveform, ≤30 s). [4] HF — UsefulSensors/moonshine-streaming-medium / -small / -tiny. https://huggingface.co/UsefulSensors/moonshine-streaming-medium [5] arXiv 2602.12241 — Moonshine v2: Ergodic Streaming Encoder ASR. https://arxiv.org/html/2602.12241v1 [6] HF model card — UsefulSensors/moonshine-tiny. https://huggingface.co/UsefulSensors/moonshine-tiny — LibriSpeech clean 4.55, other 11.68; MIT; 27M params. [7] HF model card — UsefulSensors/moonshine. https://huggingface.co/UsefulSensors/moonshine — MIT. [8] github.com/moonshine-ai/moonshine (current upstream). [9] github.com/k2-fsa/sherpa-onnx — README, Apache-2.0, supported model families, Android/iOS support. [10] sherpa-onnx LICENSE. https://github.com/k2-fsa/sherpa-onnx/blob/master/LICENSE [11] sherpa-onnx releases — v1.13.1 May 8 2026. https://github.com/k2-fsa/sherpa-onnx/releases [12] mvnrepository — com.bihe0832.android:lib-sherpa-onnx (third-party Maven mirror). [13] sherpa-onnx jitpack.yml. https://github.com/k2-fsa/sherpa-onnx/blob/master/jitpack.yml [14] sherpa-onnx official docs. https://k2-fsa.github.io/sherpa/onnx/index.html [15] sherpa-onnx Moonshine docs. https://k2-fsa.github.io/sherpa/onnx/moonshine/index.html — pre-converted INT8 bundles, microphone+VAD streaming examples. [16] sherpa-onnx Android docs. https://k2-fsa.github.io/sherpa/onnx/android/index.html [17] sherpa-onnx Android build doc. https://k2-fsa.github.io/sherpa/onnx/android/build-sherpa-onnx.html — NDK 22.1.7171670, ABIs, .so sizes (15 MB ORT + 3.7 MB JNI). [18] sherpa-onnx CHANGELOG. https://github.com/k2-fsa/sherpa-onnx/blob/master/CHANGELOG.md — Moonshine Kotlin/Java APIs added. [19] github.com/pierre-cheneau/finetune-moonshine-asr — community Moonshine fine-tune toolkit (full fine-tune, no LoRA, ONNX export). [20] github.com/cactus-compute/functiongemma-hackathon — separate hackathon, requires Cactus SDK. NOT the Kaggle Cactus Special Tech Prize. [21] Cactus blog: Gemma 4 on Cactus. https://docs.cactuscompute.com/latest/blog/gemma4/ — single-model philosophy + cloud-handoff hybrid. [22] Gemma 4 Apache-2.0 announcement. https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/ ; https://ai.google.dev/gemma/apache_2 — fully commercial-fine-tuneable. [23] Useful Sensors Moonshine README — license (MIT). https://github.com/usefulsensors/moonshine [24] Memory file — reference_kaggle_gemma4_prize_tree_2026-05-09.md — Cactus prize verbatim text, captured via Mark's Kaggle Chrome MCP session 2026-05-09.

Generated 2026-05-10 by Claude Opus 4.7 deep-research agent. All claims sourced. Items marked unverified are: actual streaming Moonshine ONNX file size (need to download and weigh), and final APK size after AAR + assets land — both will be verified Day 1 of the v0.3 branch spike.