A $8 Microcontroller Now Runs a 28.9M-Parameter Language Model Entirely Offline at 9 Tokens Per Second

An $8 Chip Just Ran a Model 100x Bigger Than Anyone Managed Before

An ESP32-S3 microcontroller costs about $8 and carries 512 kilobytes of fast memory. A developer known as slvDev_ has now used one to run a 28.9-million-parameter language model entirely on-chip, at roughly 9 tokens per second, with no internet connection 1. The previous state of the art on this class of chip was 260,000 parameters 1. That is a two-orders-of-magnitude jump, and the technique that made it possible tells you where the floor of on-device AI is heading.

The obstacle on a chip this small has always been memory, not compute. The ESP32-S3, a Wi-Fi and Bluetooth microcontroller from Espressif Systems, provides 512KB of SRAM for fast access, alongside 8MB of PSRAM and 16MB of flash storage 12. A language model normally needs its parameters in fast memory because the chip touches most of them on every token. At 512KB, you are capped near 260,000 parameters, and spending more on RAM does not help.

The developer applied an architecture idea from Google's Gemma models called Per-Layer Embeddings, which appears in Gemma 3n and Gemma 4 13. The principle: most of a language model's parameters are not computed on every token. They sit in an embedding table that the model samples from selectively. So the developer parked 25 million of the model's 28.9 million parameters in flash, the chip's slowest but largest tier, and reads only about six rows per token into fast memory. That amounts to roughly 450 bytes per token 13. The dense computational core that stays in SRAM is only about 560,000 parameters 3. The 25 million parameters sitting in flash cost almost nothing at runtime because the chip never loads most of them.

Google designed Per-Layer Embeddings for phone-tier AI, where the goal is running useful models on devices with gigabytes of memory. Applying it to a chip with 512KB of SRAM is a step down in silicon class that the developer writes nobody had publicly demonstrated before 1.

The model's output is modest. It was trained on TinyStories, a dataset of short synthetic stories from Microsoft Research designed to test how small a model can be while still writing coherent English 1. It produces simple stories and keeps them mostly on track. It cannot answer questions, follow instructions, write code, or reason over live data 1.

For anyone designing connected hardware, this reframes a cost structure that has been static since cloud AI became the default. Every IoT device that depends on cloud inference carries a recurring toll: connectivity, an API call, a per-query bill, and data leaving the device 2. A factory sensor, an agricultural monitor, or a consumer wearable that has to function when Wi-Fi drops cannot rely on a round trip to a data center. When inference happens on the chip, those constraints disappear. No latency, no API cost, no data leaving the device, no connectivity requirement.

The project comes with honest caveats. Not every ESP32-S3 board can run it: the setup requires the specific variant with 16MB of flash and 8MB of OPI PSRAM, sold as the N16R8V in Espressif's DevKitC-1 line 2. The model file is 14.9MB at 4-bit quantization 1. The developer also disclosed that an early version of the project contained a parameter accounting bug that inflated an initial figure, and left the full correction visible in the repository's commit history 1. The 28.9 million figure is the corrected count.

Two market projections add context. Fact.MR estimates the IoT microcontroller market at $8.4 billion in 2026, projected to reach $31.7 billion by 2036 2. Grand View Research projects the broader edge AI market will reach $118.69 billion by 2033 2. Those are single-source projections, not measurements. They do explain why techniques that push local inference onto cheaper silicon attract capital.

The comparison that matters is not to ChatGPT. It is to the 260,000-parameter ceiling that existed on this chip class a month ago. An ESP32-S3 will not run a frontier model. It may now be able to run enough local intelligence to classify an event, parse a constrained command, or decide whether a sensor reading warrants waking up a larger system. A crop monitor does not need to draft a memo. It needs to decide, locally and instantly, whether a reading matters. For the first time, the chip making that decision can hold a real language model while doing it.