Imagine leaving a meeting with a recording on your phone and boarding a plane before you've written the notes. A summary would help now. An assistant waiting for a connection won't. Google shipped a concrete answer to this problem in December 2023: Gemini Nano could summarize recordings on the Pixel 8 Pro without a network connection. That's a useful, bounded example of on-device AI. Google's launch announcement also names Smart Reply in Gboard as an on-device feature powered by Tensor G3.

The interesting question is how much of that assistant can stay in your pocket. Summarizing one transcript, checking a claim against current information, and analyzing a folder of documents place very different demands on a device. Building a useful local assistant means understanding those differences before deciding when to ask a server for help.

Follow the recording

For a meeting assistant, speech recognition turns audio into text. A language model then turns that transcript into a summary. These are separate jobs: a fluent summary cannot repair a name that the transcription step heard incorrectly. Google's announcement attributes Recorder's summarization to Gemini Nano; it doesn't establish that Nano performs every step of the recording pipeline.

Consider an illustrative meeting where someone says, "We can ship Friday if the security review passes." A useful summary preserves the condition. "We're shipping Friday" is shorter and wrong. This is the kind of test I'd use to decide whether a local model is good enough: does it retain decisions, conditions, owners, and dates without inventing commitments?

Local processing gives the app an immediate advantage when the connection disappears. Once the necessary models and software are installed, inference can run without a remote service. It also removes the network trip from each request and avoids sending the transcript to a model provider. Those benefits apply to the local processing path; backups and other features need separate attention.

They don't establish how fast the summary arrives or how much battery it consumes. Google's launch announcement documents the offline feature, but supplies no task-level latency, battery measurement, or summary-accuracy rate. Those remain device-and-workload tests. For a product, I would measure the complete recording-to-summary path, including a cold start and repeated use, alongside whether the summary preserves the meeting's meaning.

A model has to fit beside everything else

A phone is already running an operating system, a display, and the apps its owner cares about. An AI feature gets a share of the memory and power budget. The model's download size is only the beginning of that calculation.

The arithmetic explains why compression matters. A dense 70-billion-parameter model stored at 16 bits per weight needs about 140 GB for weights alone. For a 7B model, that's about 14 GB. Storing each weight in 8 bits reduces the raw weight storage to about 7 GB; 4 bits reduces it to about 3.5 GB. These are decimal storage calculations, not measured runtime footprints. Quantization metadata, working buffers, and the memory used to track the conversation add more.

Quantization is a little like rounding numbers on a spreadsheet. Keeping fewer digits saves space, but whether the rounding matters depends on the calculation. With models, the effect also depends on the quantization method and task. A summary that drops a condition is a meaningful failure even if most sentences still read well.

Apple's 2025 model report provides a concrete example of that tradeoff. It describes 2-bit quantization-aware training for the on-device decoder weights, 4-bit embeddings, and an 8-bit key-value cache, the working memory used during text generation. Additional adapter training helped recover quality. Apple reports a roughly 4.6% regression on the MGSM math benchmark and a 1.5% improvement on MMLU after compression. That is a mix of outcomes on named benchmarks, not a promise that every task keeps its original quality. Apple's model report

Two other techniques help make smaller models practical. Pruning removes selected weights or larger structures such as channels; unstructured pruning zeroes individual weights. The runtime must be able to exploit that sparsity to turn it into useful savings. PyTorch's pruning tutorial shows the distinction. Distillation trains a student to reproduce useful behavior from a teacher, so the deployed model can be smaller from the start. The original distillation paper demonstrates transferring knowledge from an ensemble into a model that's easier to deploy.

Small-model training also depends on data quality. Microsoft's Phi-3-mini is a 3.8B model trained on synthetic and filtered public data, followed by supervised fine-tuning and preference optimization. Calling every compact model a distilled version of a larger one would hide those differences. Phi-3-mini model card

Fast enough for which part of the job?

A small classifier can return one label. A meeting summary has to read a transcript and generate many words. Treating both as "an AI call" makes latency comparisons nearly meaningless.

Apple reported about 0.6 milliseconds per prompt token for time to first token and a generation rate of 30 tokens per second on an iPhone 15 Pro for its 2024 on-device model. Those are vendor measurements for a particular model and device. At the reported generation rate, producing 150 tokens would take about five seconds, before adding input processing and other application work. That's a useful scale for a short summary, quite different from claiming that a complete local response takes tens of milliseconds. Apple's 2024 inference results

Dedicated accelerators help. An NPU, or neural processing unit, handles operations used heavily in neural networks with hardware designed for that work. But a chip's advertised TOPS, trillions of operations per second, doesn't tell you how long your summary will take. The runtime must support the model's operations, and moving weights through memory also takes time. A model may execute across CPU, GPU, and NPU rather than using one accelerator throughout.

For the recording app, sustained behavior matters. Does the third summary take longer after the phone warms up? Does a long transcript cause memory pressure? How much battery does the feature use across a day? Moving computation onto a device removes per-request API charges for that path. It gives the computation to the user's hardware, which still consumes energy and has limited capacity.

Where the local job ends

Suppose the meeting summary is accurate, and the next request is, "Check whether the regulation we discussed changed this morning." The missing ingredient is current information. A larger local model won't create access to a source it hasn't retrieved. The app needs a permitted connection or an honest explanation that it can't verify the claim offline.

A request to reconcile many long documents creates a different problem. More text means more input processing and working memory. Model context limits aren't a universal 4K or 8K: Phi-3-mini, for example, has both 4K and 128K variants. Supporting that context length in the model doesn't establish that a particular phone can process it comfortably. Splitting documents into sections or retrieving relevant passages can reduce the workload, provided the approach preserves the relationships the question depends on. Microsoft's model specifications

Model size helps estimate resource demands, but it isn't a quality certificate. Examples span roughly 1B to 8B: Llama 3.2 1B, Phi-3-mini at 3.8B, Mistral 7B, Qwen2.5-7B, and Llama 3.1 8B. They are candidates for different local budgets, not interchangeable assistants. Evaluate the quantized model on the actual task. A difficult reasoning request can exceed a local model's demonstrated ability even when the input fits in memory.

Apple's 2024 architecture illustrates a hybrid design: an approximately 3B on-device foundation model alongside a larger server model for Private Cloud Compute. That was one documented generation, not a fixed description of every Apple Intelligence model. It shows how a product can use local processing for supported work while retaining a server path for more demanding requests. Apple's 2024 foundation model report

Multimodal support also depends on the model and device. Apple's 2025 report describes image and text inputs in its updated model family; "local" no longer tells you which input types are supported. Training is a separate resource question. Deploying compressed weights for inference doesn't give a phone the memory budget needed to train that same model. Apple's 2025 architecture

Privacy depends on the whole path

Local inference removes a specific exposure: the model provider doesn't need to receive that request. For a therapist's notes or a firm's internal documents, that's a substantial architectural advantage. It doesn't secure every copy of the information.

Recordings may still enter cloud backups, diagnostics, or a later sharing workflow. Google, for example, documents an optional Recorder backup and sync path to a Google Account. A locally generated summary and a cloud-backed recording can exist in the same app. Recorder backup documentation

A design review should therefore follow the data: where the audio is stored, who can read the transcript, what logs contain, and what happens when the app asks for help. Local execution alone doesn't establish compliance or eliminate access-control and retention responsibilities. If the product promises to keep a confidential transcript on the device, silently uploading it when the model struggles would break that promise. Let the user review the transcript, narrow the request, or explicitly choose an allowed remote service.

Hardware and tools behind the feature

For implementation, these are useful reference points rather than a chip ranking:

  • Apple's M4 announcement specifies a 16-core Neural Engine with up to 38 TOPS. Qualcomm's Snapdragon 8 Gen 3 includes a Hexagon NPU. Intel introduced an integrated NPU with Meteor Lake Core Ultra, while MediaTek's Dimensity 9300 is a flagship chip with an APU 790. Check the exact device and supported runtime before treating an accelerator as available to your model.
  • llama.cpp provides local GGUF model serving. MLC LLM compiles and deploys models for supported platforms. ONNX Runtime execution providers connect inference to supported hardware backends.
  • Apple's Core ML is its model-deployment framework. Google's MediaPipe supplies on-device task APIs, including vision and audio pipelines. Framework support and accelerator support are separate checks.
  • For transcription, whisper.cpp includes iOS and Android examples. MacWhisper offers local transcription on the Mac. Whether a particular Whisper model keeps up with live audio requires a test on the target hardware.

The following llama.cpp command starts a local prototype: serve a downloaded Mistral 7B GGUF model with a 4,096-token context. Run it from a directory containing the llama-server binary, with the model at the indicated path and a build that supports your GPU:

# Running a 7B model locally with llama.cpp ./llama-server \ --model models/mistral-7b-instruct-v0.3.Q4_K_M.gguf \ --ctx-size 4096 \ --n-gpu-layers 35 \ --port 8080

The GPU-layer setting requests offloading; it doesn't invoke an NPU or prove the model fits. This is a starting point for testing summaries on a local machine, not a benchmark or a reproduction of Pixel Recorder. The server documentation explains these options.

Give the device a job it can finish

For the meeting assistant, I would keep supported summaries local when the model preserves the transcript's meaning within the device's measured time, memory, and battery budget. Requests needing external information need retrieval. Work that exceeds the tested model or device needs an explicit choice: reduce the scope, defer it, or use an authorized remote path. A model's own confident answer isn't enough to decide that it passed.

That gives the offline feature a clear promise. The reader on the plane gets useful notes from a supported recording, with the original transcript available to check important details. They also know which questions must wait. On-device AI earns its place when that promise holds through an ordinary day of use, including the moments when the phone has no connection and the user has no patience for an excuse.