Vision-Language Models at the Edge 2026 — GPU & Jetson Sizing for Multimodal AI

A vision-language model (VLM) fuses an image encoder with a large language model: one network that can read a gauge, describe a weld defect in a sentence, answer "why did this part fail?" from a photo, and even drive robots through vision-language-action policies. In 2026 the open-model ecosystem — Qwen2.5-VL, NVIDIA VILA, LLaVA, Gemma 3, MiniCPM-V — has matured enough that OEMs are replacing multi-model pipelines (detector + OCR + classifier + rules) with a single multimodal model. But VLMs change the hardware math: they are memory-hungry, token-hungry and quantization-sensitive. This guide sizes Jetson modules, workstation GPUs and data-center accelerators for VLM workloads, anchored on NVIDIA's own measured Orin-vs-Thor numbers.

Why multimodal models change the edge vision conversation

Classic edge vision is a chain: a fast detector (YOLO-class) finds regions, a separate OCR or classifier labels them, and business logic decides. Each stage is small, cheap and runs at 30+ FPS on a 20–100 TOPS device. VLMs collapse that chain into one open-vocabulary model — prompt "describe any anomaly you see and read the serial number" without training a custom class. That suits low-volume, high-variety inspection (random defects, worn tooling), document and display reading, safety monitoring, and natural-language search over surveillance video (NVIDIA's Metropolis VSS pattern).

The trade-off is real: VLMs cost 10–100× more compute and memory per image than a detector, and they do not replace it in every role. A 60 FPS, 16-camera PCB line still belongs to a small YOLO-class model; the VLM earns its place as the reasoning layer that catches rare, unmodeled failures and writes the human-readable report. The architecture that wins in practice is hybrid: detector at the edge, VLM gatekeeper on anomalies, both on the same Jetson or GPU box.

The memory math: weights, vision tokens and context

VLM memory = model weights + KV cache for every token (text and image tokens) + runtime headroom. Weights dominate, and the rule of thumb is unchanged: ~2 GB per billion parameters at FP16, ~1 GB at INT8, ~0.6 GB at INT4. What surprises first-time buyers is the image side: every image is expanded into hundreds or thousands of tokens before it reaches the language layers, and each of those tokens occupies KV-cache memory for the whole generation.

Vision-token counts vary by architecture: Gemma 3 encodes each image into 256 tokens, PaliGemma's SigLIP encoder produces 1,024 tokens at 448×448, and Qwen2.5-VL's dynamic resolution can pass 1,500 tokens for a high-resolution frame. Video scales linearly: a 30-second clip sampled at 2 FPS is 60 frames — at 500–1,000 tokens per frame that is 30,000–60,000 tokens of context. Budget 1–3 GB of headroom above weights at the 7–8B scale, and 4–16 GB for 32B-class models with long contexts or video. "The model fits in 24 GB" is not the same as "the deployment fits in 24 GB."

Platform fit in 2026: measured on Orin, Thor and workstation GPUs

NVIDIA publishes output-tokens-per-second numbers for the same VLM on Jetson AGX Orin and AGX Thor — the cleanest apples-to-apples edge data available. On Qwen2.5-VL-3B (W4A16 on Orin, FP4 on Thor), Orin sustains ~216 output tokens/s and Thor ~357 tokens/s (1.65×). Qwen2.5-VL-7B on Thor with FP4 plus Eagle-style speculative decoding is up to 3.5× faster than Orin at W4A16, and Thor holds 16 concurrent VLM + LLM requests with time-to-first-token under 200 ms and time-per-output-token under 50 ms.

Three conclusions follow. First, Orin is a 7–13B platform: with W4A16 it runs the models most inspection and document tasks need, but Ampere has no FP4 tensor cores, so squeezing 32B-class accuracy onto Orin means INT4 or INT8 with a real quality check. Second, Thor (Blackwell) is the first edge module where the 27–32B tier is comfortable and 70B-class quantized models are plausible — NVFP4/FP4, 128 GB unified memory and 273 GB/s are exactly what video and agentic VLM workloads starve on without. Third, workstation and server GPUs still win on concurrency: vLLM-style batching on an L40S-class 48 GB card can serve several inspection lines at once, which no single Jetson can match.

Deployment reality: quantization, runtimes and a pre-purchase checklist

Quantization is the biggest deployment lever, and it is silicon-dependent. W4A16 (INT4 weights, FP16 activations) is the default for Orin and RTX GPUs; modern 4-bit schemes keep most VLM QA accuracy within a point or two of FP16 — but verify on your own defect set, because OCR and fine-grained attribute questions degrade faster than captioning. On Blackwell parts (Thor, newer data-center GPUs), NVFP4 delivers near-FP8 accuracy at half the memory traffic of INT4, and FP8 spans both generations through NVIDIA's NGC containers. Run with the runtime you will ship: vLLM and TensorRT-LLM containers on Jetson, llama.cpp for light single-model cases. One field lesson from the Thor launch: VLM prompts are prefix-heavy (system prompt + image tokens repeat on every frame), and enabling vLLM's automatic prefix caching produced the largest single speedup seen — bigger than model or hardware changes alone.

• Define the task tier first — closed-set detection at 30+ FPS stays on a YOLO-class model; open-vocabulary description, reasoning and reporting belong to the VLM. Most 2026 designs are hybrid.
• Count tokens, not just parameters — weight GB is table stakes; add vision tokens × KV cache, then multiply by your FPS and concurrency target.
• Match precision to silicon — W4A16 on Orin/Ampere, NVFP4/FP4 on Thor/Blackwell, FP8 on both; budget an accuracy eval with your own images before the PO.
• Benchmark with your production runtime — numbers from eager PyTorch mislead; measure with the vLLM/TensorRT-LLM container and the camera feed you will actually process.
• Size the camera and pre-processing path — aligned, undistorted frames matter to VLMs; confirm the carrier's CSI/GMSL2 or GigE input keeps up before blaming the model.
• Plan concurrency and thermal headroom — one AGX Orin gates about one VLM stream comfortably; Thor runs 16 concurrent requests but at up to 130 W — a real enclosure-thermal number, not a paper one.

VLMs have made edge AI genuinely multimodal in 2026, but they reward buyers who do the memory math before the hardware order. Match the VLM class to the silicon's precision capabilities, keep classic detectors for the high-FPS work, and verify throughput on the actual inference runtime — that combination separates deployments that demo well from deployments that ship.