Running LLMs on Edge AI Dev Kits 2026 — Jetson Orin vs Rockchip RK3588 vs Hailo-8 for On-Device Language Models

Edge AI development kits have always been benchmarked on computer vision — YOLO FPS, ResNet latency, segmentation throughput. But the landscape shifted in 2025–2026: small language models (SLMs) like Llama 3.2 3B, Gemma 2 2B, and Phi-3.5-mini now run on devices that cost under $600. The question every 开发套件 buyer needs to answer: which dev kit actually runs LLMs at usable speeds?

We tested three popular platforms — NVIDIA Jetson Orin NX 16GB, Rockchip RK3588 (16GB), and Hailo-8L + Raspberry Pi 5 — across three small LLMs. Here's what we found.

The Contenders: Hardware at a Glance

Key insight: The Hailo-8L's 13 TOPS are irrelevant for LLMs — it's a CNN/vision accelerator with no transformer support. LLM inference on Hailo falls back to the Raspberry Pi's CPU. The RK3588 NPU fares slightly better with RKLLM, but the ecosystem is immature. Only Jetson Orin has a production-grade LLM runtime (TensorRT-LLM).

Real-World LLM Benchmarks

All models quantized to INT4 (Q4_K_M) for fair comparison. Measured tokens/sec on single-stream interactive generation (not batched).

Finding: Jetson Orin NX runs 3B-class LLMs at 22–35 tok/s — faster than human reading speed. The RK3588 and Hailo-8L setups deliver 3–6 tok/s, which is functional for batch processing but frustrating for interactive chat. For 8B models, only the Orin NX (16GB) has enough memory bandwidth and GPU compute to deliver usable performance.

Runtime Stack Comparison

llama.cpp on the RK3588's CPU cluster is surprisingly capable for its price — 5 tok/s on a $180 board. But it's a CPU-only path; the NPU remains underutilized. RKLLM promises NPU acceleration but currently supports only two model architectures and lacks batching.

Who Should Buy Which Kit for LLM Workloads

Jetson Orin NX ($599): You need interactive-speed on-device LLMs, RAG pipelines, or multi-model serving. TensorRT-LLM is the only edge LLM runtime with production features (continuous batching, speculative decoding, KV cache quantization). Ideal for voice assistants, industrial copilots, and on-premise chatbots.

RK3588 ($180): Budget-sensitive projects where 5 tok/s is acceptable — batch document classification, offline text summarization, or non-interactive inference pipelines. The price-per-token is unbeatable if interactivity isn't required. Pair with a lightweight API server for HTTP-based LLM access.

Hailo-8L + RPi 5 ($139): Stick to computer vision. The Hailo accelerator is excellent for YOLO/ResNet but provides zero benefit for transformer models. For LLMs, this setup is a $139 CPU-only inference node — functional but not recommended for language workloads.