Whisper Model for Edge AI on Mobile

OpenAI's Whisper is a transformer-based automatic speech recognition (ASR) system that has become a cornerstone for on-device, edge AI applications. By running Whisper locally on mobile devices, developers can ensure user privacy, eliminate latency from cloud round-trips, and provide offline functionality. This article explores the key considerations for deploying Whisper on mobile, including model size trade-offs, language variants, handling long audio, and performance optimization.

Model Sizes and Trade-offs

Whisper is available in several sizes, each balancing accuracy, speed, and resource consumption. The primary difference is the number of parameters, which dictates how well the model handles complex audio, accents, and background noise. The tiny model (39M parameters) is the fastest and lightest, ideal for low-end devices where latency is critical, though it is more error-prone. The base model (74M parameters) offers a better balance for general-purpose transcription and is often the largest that runs comfortably on very constrained hardware. The small model (244M parameters) is often considered the sweet spot for mobile, providing a significant accuracy jump while still fitting within modern smartphone memory. Medium (769M) and Large (1.55B) models offer the highest accuracy but require substantial RAM and processing power, making them infeasible for real-time mobile use without dedicated hardware acceleration.

English-Only vs Multilingual Models

Most Whisper sizes (tiny, base, small, medium) come in two versions: multilingual and English-only (.en). Multilingual models are trained on a diverse dataset of 99 languages, making them versatile but slightly larger and more complex. English-only variants are trained exclusively on English data, generally offering better accuracy for English tasks and more efficient resource usage. For mobile apps targeting English-speaking users, the .en models are often preferred to reduce model size and improve performance.

Handling Long Audio: The 30-Second Window

Whisper has a fixed receptive field of 30 seconds. To transcribe audio longer than this, developers use a sliding window or chunking approach. The audio is split into 30-second segments, processed individually, and then stitched together. To prevent loss of context at boundaries, implementations often use overlapping segments (e.g., 3–10 seconds of overlap) to ensure that words or phrases cut off by a segment boundary are captured correctly in the next window. This technique is essential for processing files longer than 30 seconds, such as podcasts or recorded meetings.

Deployment on Mobile: ExecuTorch and CoreML

Running Whisper on mobile requires optimization to manage thermal and memory limits. Developers can use frameworks like ExecuTorch for cross-platform PyTorch deployment or convert models to CoreML for iOS. CoreML allows the model to leverage the Apple Neural Engine (ANE), significantly improving performance and reducing power consumption compared to CPU-only execution. Quantization is also common: models are quantized (e.g., to GGML or GGUF formats) to fit into mobile RAM, sometimes reducing the footprint by several gigabytes. On modern mobile hardware, smaller models like tiny or base can run significantly faster than real-time. The small model is often the practical limit for real-time, interactive dictation, while larger models are reserved for offline file processing where speed is less critical than accuracy.

In summary, choosing the right Whisper model for mobile involves balancing accuracy, speed, and resource constraints. For most edge AI applications, the small model with English-only variant offers the best compromise, while deployment frameworks like ExecuTorch and CoreML ensure efficient execution on device hardware.