Edge AI Deployment

Edge AI is genuinely useful — in specific scenarios. The hype around it is mostly noise, but the signal is real: if your use case involves sensitive data that can’t leave the device, requires inference with no network dependency, or needs sub-20ms latency that a round-trip to a cloud GPU can never deliver, then running the model closer to the user is the right architectural decision. For everything else, a centralized API is cheaper, simpler, and more capable.

Here’s when edge inference actually makes sense, how the major deployment surfaces work today, and how to decide which one belongs in your stack.

What “Edge AI” Actually Means

“Edge” is overloaded. Depending on who’s talking, it means four different things:

Each has a completely different set of tradeoffs, tooling, and viable model sizes. Treating them as one category is where most architectural mistakes start.

Browser / WASM Inference: What’s Actually Usable Today

Running a model in the browser is not a novelty anymore, but it’s still gated by device capability. A MacBook with an M-series chip and a mid-range Android phone are an order of magnitude apart in inference throughput.

Transformers.js

Transformers.js from Hugging Face is the most production-ready option for browser inference. It runs ONNX-exported models via the @huggingface/transformers package (v4, released February 2026) and supports a wide range of encoder-based tasks: sentiment analysis, named entity recognition, zero-shot classification, embeddings, and speech recognition via Whisper. The v4 release added a WebGPU runtime with 10-15x speedup over WASM, making larger model architectures usable in the browser.

What works well:

• Sentence embeddings with all-MiniLM-L6-v2 — fast, small (23MB), genuinely useful for semantic search that runs without a backend
• Whisper tiny and base for in-browser transcription — acceptable quality, ~40-150ms per utterance on decent hardware
• Zero-shot classification with bart-large-mnli — heavier (891MB), only reasonable if you cache aggressively

What doesn’t work well: anything requiring a decoder-only LLM at quality levels users expect from GPT-4-class models. The gap between what fits in browser memory and what produces usable generative output is still large.

WebLLM

WebLLM runs quantized LLMs directly in the browser using MLC-LLM and WebGPU. Models like Llama 3.1 8B (4-bit) and Phi-3 Mini are genuinely runnable — but “runnable” means:

• First load: 2-6GB download, cached via IndexedDB afterward
• Inference speed on M3 MacBook: 15-40 tok/s. On a mid-range Windows laptop: 3-8 tok/s
• WebGPU is required — Safari support is partial, Firefox is behind a flag

Use WebLLM when you need a generative LLM that processes data that must never leave the client — legal documents, medical notes, internal comms. The download penalty is a one-time cost that some user segments will tolerate.

WebGPU Directly

WebGPU is the API that makes browser inference viable for larger models. It gives JavaScript access to the device GPU with near-native performance — but browser support is still uneven. Chrome and Edge ship it enabled by default. Safari 26+ supports it on iOS, with partial support on macOS (some rendering bugs remain). Firefox still requires a flag in stable releases as of mid-2026.

If you’re building a WebGPU-dependent feature today, gate it properly:

if (!navigator.gpu) {
  // fall back to server-side inference
}
const adapter = await navigator.gpu.requestAdapter();
if (!adapter) {
  // WebGPU not available even with the API present
}

Don’t assume WebGPU availability. You need a server fallback in production.

Cloudflare Workers AI: Real-World Characteristics

Cloudflare Workers AI runs inference at Cloudflare’s edge PoPs using GPUs colocated in their network. The pitch is compelling: sub-50ms TTFB from most of the world, serverless billing, no cold starts.

The reality is more constrained.

What models are available: Cloudflare’s model catalog is curated and limited. As of mid-2026, the catalog was refreshed — Llama 3.1 8B and Mistral 7B were deprecated in May 2026. Current text generation models include Llama 3.2 3B, Llama 4 Scout, Gemma 4 26B, Mistral Small 3.1 24B, and a handful of embedding, classification, and image generation models. You cannot bring your own model — you’re restricted to whatever Cloudflare has decided to host.

Latency: Genuinely low for first-token latency — typically 100-300ms depending on PoP proximity and model size. For short completions, the total response time can beat a centralized GPU API. For long generations, throughput becomes the bottleneck and the advantage narrows.

Pricing: $0.011 per 1000 neurons (Cloudflare’s unit) for text generation, which translates to roughly $0.30-4.00 per million output tokens depending on model. Cheaper than OpenAI GPT-4, more expensive than self-hosted open models, similar to many inference API providers.

Real limitations:

• No fine-tuned models — you get base models only
• Context windows are shorter than what the underlying models support (many models are artificially limited at the Workers layer)
• AI Gateway streaming has improved but configuration-dependent limitations remain
• Rate limits per account are aggressive on the free tier

Workers AI is a good fit for: classification, summarization, and short-form generation where you control the prompt and expected output length, and where you’re already running logic in Workers. It’s a poor fit for: complex RAG pipelines, long-context reasoning, or anything requiring models not in their catalog.

Vercel Edge Functions for AI: What You’re Actually Getting

Vercel’s original Edge Functions (V8 isolates) were deprecated in June 2025 and replaced by Vercel Functions with an Edge Runtime option. The architecture changed, but the constraints for inference workloads remain similar. When people talk about “AI on Vercel Edge,” they mean one of two things:

1. Streaming responses from a centralized API (OpenAI, Anthropic) through a function to reduce connection latency
2. Very lightweight inference using WASM-compiled models

The real constraints:

• 250MB bundle size limit (Node.js runtime) makes ONNX models deployable, but cold starts remain a concern
• No GPU access — CPU-only WASM on the Edge Runtime
• 300s maximum duration — rules out sustained inference, but short streaming sessions are fine
• No native modules on the Edge Runtime — anything that needs Node.js crypto, fs, or native addons won’t work on the Edge Runtime path

What Vercel Edge Functions are actually good for in an AI stack: streaming proxy, token counting, prompt injection detection with a small WASM classifier, rate limiting, and routing logic. They’re not an inference runtime.

The Vercel AI SDK’s edge support is specifically about streaming — it wraps fetch calls to upstream AI APIs and pipes the response stream to the client with low overhead. That’s genuinely useful. The inference is still happening at OpenAI or Anthropic’s data centers.

Mobile On-Device Inference: The Mature Option

Mobile is where on-device inference has the most production history. The tooling is stable, the hardware acceleration is real, and user expectations are calibrated — Siri, on-device translation, and keyboard autocomplete have been running models locally for years.

Apple Core AI and Core ML

Apple’s on-device AI framework landscape shifted at WWDC 2026. Core AI is now the primary framework for neural network and generative model inference on Apple Silicon, handling tasks formerly served by Core ML. Core ML continues to serve classical, non-neural ML. The conversion path from PyTorch is through coreai-torch (for Core AI) or coremltools (for legacy Core ML workflows):

import coremltools as ct
model = ct.convert(traced_model, inputs=[ct.TensorType(shape=input_shape)])
model.save("model.mlpackage")

Both Core AI and Core ML route to the ANE (Apple Neural Engine), GPU, or CPU depending on model architecture and chip. The ANE is extremely fast for MobileNet/EfficientNet-class models but doesn’t support all ops — if your model uses unsupported layers it silently falls back to CPU, which can be much slower. Profile with Instruments.

Core AI supports quantized models (4-bit and 8-bit) and is optimized for LLM inference, supporting models up to 70B parameters on iPhone. Apple’s MLX framework (now at v0.31.3) remains the best choice for research, fine-tuning, and custom training on Apple Silicon — it exploits unified memory and shows strong throughput for quantized models.

Android NNAPI / LiteRT

Android’s Neural Networks API provides hardware-accelerated inference across the heterogeneous mess of Android devices. In practice, you’re using LiteRT (Google’s successor to TensorFlow Lite), which handles NNAPI delegation automatically:

val options = Interpreter.Options().apply {
    addDelegate(NnApiDelegate())
}
val interpreter = Interpreter(modelBuffer, options)

The challenge on Android is device fragmentation. A Samsung Galaxy S24 has a Snapdragon 8 Gen 3 with a strong NPU. A budget Android device from 18 months ago may fall back to CPU entirely. Test on a representative set of target devices — don’t profile only on your own phone.

MediaPipe

MediaPipe (now under Google AI Edge) provides task-level APIs for common on-device AI workflows: face detection, pose estimation, image segmentation, text classification, and embeddings. It handles the model, the inference pipeline, and platform adaptation. If your use case maps to one of its tasks, it’s the fastest path to production.

On-Prem / Private Cloud with Ollama and LM Studio

For enterprise workloads where data residency is non-negotiable — healthcare, finance, legal, defense — running open-weight models on private infrastructure is the only viable path.

Ollama

Ollama is the fastest way to get a production-grade inference server running locally or on a private VM. It handles model downloading, quantization selection, CUDA/Metal acceleration, and exposes an OpenAI-compatible API:

ollama serve &
ollama pull llama3.1:8b-instruct-q4_K_M
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b-instruct-q4_K_M", "messages": [{"role": "user", "content": "Hello"}]}'

The OpenAI-compatible endpoint means you can drop Ollama behind your existing OpenAI SDK calls by changing the base URL. For multi-GPU setups, it handles tensor parallelism transparently for supported models.

For production on-prem deployment, pair Ollama with a reverse proxy (Nginx or Caddy), add authentication at the proxy layer, and use a load balancer if you need to serve multiple instances.

LM Studio

LM Studio (v0.4.16 as of mid-2026) is optimized for developer ergonomics — local model browsing, quantization comparison, and a GUI. Its newer llmster headless daemon extends it to server deployments, though Ollama remains the simpler default for production on-prem use.

The Real Tradeoffs

Stop treating edge vs. centralized as a binary. The decision matrix is:

The privacy argument is the most compelling. If you’re building a legal document analyzer, a medical coding tool, or any application where the inputs are confidential and regulated, moving inference on-prem or on-device isn’t an optimization — it’s a requirement.

The latency argument is weaker than it looks. A well-tuned centralized API with streaming can deliver perceived responsiveness that’s indistinguishable from local inference for most text generation tasks. Users perceive the first token, not the total generation time.

Decision Framework: When to Go Edge

Use this as a starting point, not a formula:

Go edge (browser/mobile) if:

• User data is too sensitive to leave the device under any circumstances
• You need offline functionality that must work without any network
• Your use case is a well-defined narrow task (classification, embeddings, transcription) that small models handle well
• You’re building a mobile-first product on iOS where Apple’s Neural Engine gives you genuinely fast inference

Go CDN edge (Cloudflare Workers AI) if:

• You’re already building on the Cloudflare stack
• You need low-latency short-form completions and the model catalog covers your use case
• You want serverless billing without managing GPU infrastructure

Go on-prem if:

• Data residency is a hard regulatory requirement
• You have existing hardware or a private cloud contract
• Query volume justifies the infrastructure investment over API costs

Stay centralized if:

• You need GPT-4 / Claude / Gemini-level output quality
• Your context windows exceed 32K tokens regularly
• Model capability matters more than latency or privacy
• You’re early-stage and operational complexity is expensive

The honest default is still centralized inference through a managed API. Most applications don’t have privacy requirements that force the move to edge, and most don’t have latency budgets tight enough to justify the model quality tradeoff. Edge inference is a deliberate architectural choice with real costs — don’t make it because it sounds modern.

The tooling has matured enough that edge AI is no longer an experiment. Transformers.js runs in production. Cloudflare Workers AI handles real traffic. Core AI inference on Apple Silicon is fast. But “mature enough to use” is different from “the right choice for your system.” Know which constraint is actually driving you there, and make the tradeoff deliberately.