On-Device AI Models: Speech & Decision Apps ⚡ (2026)
Learn how on-device AI models power local speech recognition and Swift machine learning using AnySpeechRecognizer & AnyDecisionModel—setup tips, limits, fast results.
Speech and Decision AI Come to Apple Devices
From the outside, a voice support flow sounds straightforward: transcribe what the customer said, figure out what they need, then route the request. But once you build it in an app with a general-purpose language model, those steps can turn into a bunch of extra work - plus fragility.
You might be sending sensitive audio or text to a service, paying for generated tokens, and then parsing the model's output just to get a simple category or yes/no decision.
A more practical setup is to use a model that's designed for each job. AnySpeechRecognizer and AnyDecisionModel are Swift packages for Apple platforms that follow this idea. AnySpeechRecognizer gives you a common speech-to-text interface with a choice of recognition engines, plus speaker diarization. AnyDecisionModel returns typed decisions and probabilities, so you're not stuck asking the model to "write the answer" and then trying to extract it from prose.
The question isn't whether these packages make everything local, reliable, or magically solved. They don't. The real question is where specialist models can improve privacy, latency, cost, and control - and where device limits, model quality, and evaluation still matter. This guide explains how the packages work, what their reported benchmarks show, and how to decide whether they fit a production system.
Why use specialist models for speech and decisions?
General-purpose language models are flexible, but flexibility has a cost. If the task is to transcribe audio or pick one option from a known set, generation can add unnecessary steps. The model has to produce text, your app has to interpret it, and you still need to handle cases where the output doesn't match what your routing logic expects.
Specialist models narrow the job. Speech recognition turns audio into text. A decision model compares a defined question against an allowed set of answers. That also tends to make testing easier, because the inputs and outputs are clearer.
Separate the pipeline into explicit stages
Think about a support call. A solid system usually ends up with a few clear stages, even if you hide them behind one feature in the UI:
- Transcribe the audio. Pick a speech engine based on language coverage, accuracy, and whether the input is a file or live audio.
- Identify speakers. If more than one person is talking, align words with speaker turns.
- Classify the request. Choose from known teams, like billing, technical support, or sales.
- Apply business rules. Route, escalate, or send for human review based on the result and its confidence.
That separation isn't just "clean design." It lets you swap the transcription model without rewriting your whole routing system. It also makes it easier to compare a model's classification against human-labeled tickets. And when the model is unsure, you can route to a person instead of pretending every answer is equally trustworthy.
Local inference changes the trade-offs
Both packages can download open models from the Hugging Face Hub and run them on the device. That can reduce network dependence and keep audio or text from leaving the user's machine. It can also make inference costs more predictable as usage grows, since you're not paying a hosted provider per request.
But local inference isn't automatically cheaper or better. Your app still has to manage model downloads, storage, memory, device compatibility, and updates. A model that runs smoothly on an Apple silicon Mac might not fit on an older iPhone. Some models are large - hundreds of megabytes or multiple gigabytes - and compilation or initial loading can affect first-run experience.
So the useful decision isn't "cloud versus local" as an ideology. It's whether local inference, task by task, meets your quality bar while staying within device constraints. A hosted model can still be the better fit for broader language coverage, centralized updates, or devices with limited compute. A local model makes more sense when privacy, offline use, or high recurring request volume is the dominant constraint.
What does AnySpeechRecognizer offer?
AnySpeechRecognizer gives Swift developers a speech API modeled on Apple's SpeechAnalyzer interface, while letting you choose the underlying recognition engine. Apple's Speech framework is the default. If you have a compatible app, you can switch its import to AnySpeechRecognizer and keep much of your analyzer setup, asset installation, and result-reading flow.
That shared interface matters because model choice will likely change as you iterate. You might start with Apple's engine and later test a local model for a language or workload where it performs better. Keeping the app code stable reduces the cost of those comparisons.
Choosing between Parakeet and Qwen3-ASR
The package supports several Core ML speech models, each with its own strengths:
- Parakeet TDT v3 recognizes 25 European languages and can detect the language automatically.
- Parakeet Ultra is a post-trained Parakeet v3 variant that performed better in most of the reported measurements.
- Parakeet Unified is an English-only model.
- Qwen3-ASR comes in 0.6B and 1.7B sizes, supports 30 languages and 22 Chinese dialects, and can automatically detect a language or use one you specify.
It's not just a "which model is best" situation. Parakeet supports both file and live input, produces volatile interim results, and provides word timing and word confidence. Qwen3-ASR produces a final result for each audio segment (up to 40 seconds) and does not provide interim results, word-level timing, or confidence. It runs on the GPU and is slower, so the package positions it as a better fit for file transcription than live interaction.
If you're building live captions, interim output and responsiveness probably matter more than a small bump in an offline benchmark. For batch processing multilingual recordings, language coverage may matter more. In both cases, test with the audio your customers actually produce - noisy calls, different accents, interruptions, and microphone quality can change everything.
Speaker diarization adds another layer
Transcription answers what was said. Speaker diarization estimates who said each part, and when. AnySpeechRecognizer adds a SpeakerDiarizer module using NVIDIA's Streaming Sortformer model. It can track up to four speakers, including overlapping speech.
You can run the diarizer on the same timeline as an Apple or Parakeet transcriber. Then combine results into a speaker timeline that attributes timed words to turns. That's especially useful for meeting notes, interviews, and support calls, where a transcript without speaker identity is hard to use.
Just don't overread what diarization is doing. It labels distinct voices; it doesn't prove that a speaker is a specific person, customer, or employee. Treat the labels as an analysis of the recording, not verified identity data.
Understand the benchmark context
On an M4 Max Mac Studio running macOS 27.0.1, reported Word Error Rate on LibriSpeech files was between 1.43% and 2.78% for Parakeet file transcription, depending on the model and test set. Apple's transcriber measured 2.12% on test-clean and 3.97% on test-other. Live Parakeet results varied more; for instance, Parakeet v3 measured 4.73% on test-clean.
Those results are useful, but they aren't a guarantee for your app. LibriSpeech is read speech. Results from one Mac don't automatically translate to iPhones or to real production calls. The source also reports that all tested engines had Word Error Rates above 23% on AMI meeting recordings, with overlapping speech.
On the tested Mac, Parakeet processed an hour of audio from a file in about 20-40 seconds (roughly 1.5-2.5 minutes in live mode). After Core ML compilation, reported memory use was around 100 MB. Qwen3-ASR was substantially slower and required several gigabytes of memory. Those differences should influence deployment decisions as much as headline accuracy.
How does AnyDecisionModel return decisions?
A lot of apps prompt a language model with something like "Which team should handle this ticket?" and then parse whatever it says back. It can work, but it gives you a blob of text instead of a constrained result. The model might format the department name differently, add explanations, or pick something outside your intended categories.
AnyDecisionModel structures this by defining questions around a state (text or JSON) and returning typed results. The API supports yes-or-no questions, choices from a set, and rating scales. Each choice can include the selected option, probabilities for each option, and a confidence value. A score can include the expected level and the probability distribution.
Probabilities are useful - but not proof
For general-purpose language models, one backend derives answers from probabilities assigned to allowed next-token labels. Instead of generating an open-ended response, it compares the possible answers directly. A yes-or-no question returns a numeric probability; a multiple-choice question returns a typed choice plus probabilities across the options.
That gives you more structure than parsing prose, but it doesn't automatically mean the model's judgment is calibrated. Language models can be overconfident, and the position of an option in a list can affect its score. AnyDecisionModel includes calibration support for local backends and rotation debiasing for certain choice questions. Rotation debiasing tests different option orders and averages the results, but it costs additional model evaluations.
Confidence still needs careful interpretation. Here, confidence for a local choice or score is based on how concentrated the probabilities are. Even a concentrated distribution can be wrong. Measure error rates on your own labeled data before using any automation threshold. For high-impact decisions, keep a human review path and record enough information to audit outcomes later.
Backends serve different deployment models
AnyDecisionModel offers multiple ways to run decisions:
- MLX runs a language model locally on Apple silicon.
- MLX decision heads use models trained with a dedicated decision head, such as Cloudflare's Clef.
- Core AI provides a local backend for supported Apple operating systems and models.
- llama.cpp runs a local GGUF model through llama.swift.
- Jev calls a hosted decision service through the System One API.
- Ollama connects to a compatible local server or Ollama's cloud service.
Decision-head support matters because general-purpose models aren't necessarily trained to make structured decisions. A model such as Clef-Flash has a trained decision head that returns probabilities for typed questions. The documented 4-bit MLX version is a 6.2 GB download, and its model card recommends a Mac with at least 16 GB of memory. That's not just an implementation detail - it's a real product constraint.
For next-token backends, the package uses labels for the possible answers. That limits a choice to 26 options, and a rating scale to 10 levels. Decision heads don't have the same label limit. Pick the backend based on the task, hardware, and output contract - not just because a model name is newer.
How should you evaluate these packages for an app?
These packages require Swift 6.2 or later and macOS 26 or iOS 26 or later. Parakeet, Sortformer, and the MLX backend require Apple silicon. Some build paths need extra tooling: MLX build tasks use Xcode's Swift 6.4 or later and the Metal toolchain, while Core AI requires Xcode 27 and an OS 27 runtime.
Because of that, do a small feasibility test early. The packages share a familiar Swift interface, but hardware support, dependency compatibility, download size, and model behavior still vary by backend.
Run a test against representative data
A useful evaluation isn't a single demo transcript or one hand-picked classification example. Build a small test set that matches your real operating conditions:
- Collect representative inputs. Include the languages, accents, background noise, audio lengths, and edge cases you expect in production. For decisions, add ambiguous cases and out-of-scope examples.
- Set task-specific success measures. Measure transcription error and latency for speech. For decisions, measure per-category precision and recall, calibration quality, and how often you should escalate.
- Test on target hardware. Measure first-run model installation, memory use, inference time, and battery or thermal impact where relevant. Don't assume Mac performance will match iPhone performance.
- Compare local and hosted paths. Include network failures, privacy requirements, and ongoing operating costs. A fallback path only helps if you can define its behavior clearly.
- Define a safe failure mode. Decide what the app does when audio is unclear, when the model returns low confidence, or when the request doesn't match your supported categories.
AnySpeechRecognizer includes a command-line tool called hear for transcribing local files, labeling speakers, and evaluating results. It requires a backend selection, and hear prepare installs assets before analysis. A command-line workflow makes repeatable experiments easier before you start shipping model downloads and inference inside a user-facing app.
Check model licensing and package compatibility
The published Core ML conversions list licenses that vary by model. Parakeet v3 and Parakeet Ultra conversions are listed under CC BY 4.0; Parakeet Unified and one Sortformer version use the NVIDIA Open Model License; the Qwen3-ASR conversions are listed under Apache 2.0. Check the terms for the exact model and version you plan to ship. Just because something is "open" doesn't mean every model weight you can load is covered the way you assume.
Downloads add up too. The source lists Parakeet models at about 600 MB each, plus an additional live encoder at about 590 MB. Qwen3-ASR models are roughly 0.9-2.4 GB, and Sortformer is about 250 MB. Assets download only when the app requests them, using the standard Hugging Face cache. That reduces app size, but your product still needs to explain downloads, handle interrupted installation, and manage storage expectations.
Finally, verify dependency compatibility before combining packages. The source notes a current constraint between AnySpeechRecognizer and AnyDecisionModel's Core AI trait due to differing swift-huggingface version requirements. Treat this like a real integration check: resolve dependencies in a small project before planning a release around a specific combination.
Conclusion
AnySpeechRecognizer and AnyDecisionModel point toward a practical Apple app architecture: use a focused model for transcription, another for structured decisions, and keep the product logic around those results explicit.
The shared Swift patterns make it easier to compare engines, and local inference can help with privacy, offline behavior, and recurring service costs.
The trade-off is real. Models take space, use device resources, vary in what they can handle, and can fail in the situations that matter most to your users. Treat the published benchmarks as a starting point, not the final answer. Test with representative inputs on your target devices, verify licensing and dependencies, and define what your app does when confidence is low. That's the difference between "adding AI" and building something you can safely rely on.