AI with the ESP32: TinyML on the Board and Integrating an LLM

"Can you put artificial intelligence on an ESP32?" The short answer is yes, but not the way many people imagine. A language model like the ones that chat does not fit on a board costing a few dollars: the smallest of them take hundreds of megabytes, and the DevKit V1's ESP32 has 4 MB of flash and, in MicroPython, under 200 KB of free memory for the program. What does fit — and works very well — are small models, trained for a single question: "is this vibration normal?", "is this sound a clap?", "is this reading from a stuffy room?".

For open-ended questions, in free text, the path is different: the ESP32 talks over the network to a large model, which runs on a school computer, on a server or in the cloud. This article shows both paths and a third one that combines them. The examples with the Simulate button run in the browser; the network ones are to take to the real board.

Three paths

1. AI on the board (what is called TinyML, tiny machine learning). Data is collected with the sensors, the model is trained on the computer and then deployed to the ESP32, which starts deciding on its own. It works without internet, answers in milliseconds and sends the data nowhere. In return, it only answers the question it was trained for.
2. AI on the server. The ESP32 reads the sensors and sends the data to a server, which asks an LLM (large language model) and returns an answer. The LLM understands free text and knows many things, but needs a network, takes seconds to answer, may cost money per request and can be wrong with full confidence.
3. The two together. The small model, on the board, solves the common cases right away; only when it is unsure does the board ask the LLM. Most decisions never leave the board, and the network is used little.

AI on the board: collect, train, deploy

Every TinyML project follows the same three-step cycle.

Collect. The model learns from labeled examples: sensor readings accompanied by the right answer. To recognize a stuffy room, for example, you measure temperature and humidity over several days and note, on each reading, how the room was. The data can go into a CSV file on the board itself, as in the article storing data on the ESP32, and then to the computer. This is the most laborious step and the one that most decides whether the model will work: few examples, or examples from a single day, give a model that is right in the lab and wrong in real life.

Train. On the computer, a program looks for a way to go from the readings to the label. For the small models in the examples below, "training" is just storing the examples or computing averages. For larger models, a machine-learning library is used on the computer, and training can take minutes.

Deploy. The trained model becomes a file that goes to the board — sometimes a simple list of numbers inside main.py. From then on, the ESP32 reads the sensor, passes the reading through the model and acts on the answer.

First model: the k nearest neighbors

The simplest model to understand is k-nearest neighbors (kNN). It stores the labeled examples and, for a new reading, looks for the k most similar examples and answers with the label that appears most among them. "Most similar" is the distance between readings, the same ruler as in geometry.

The example uses ten temperature and humidity readings, like those of a DHT22, already labeled. The numbers are made-up examples for the class, not measurements: in a project, they come from the collection step.

import math

# (temperature in °C, humidity in %, label) — example data for the class
exemplos = [
    (18.0, 55, 'cold'),       (17.5, 60, 'cold'),      (19.0, 50, 'cold'),
    (23.0, 50, 'pleasant'),  (24.5, 45, 'pleasant'), (22.0, 55, 'pleasant'),
    (29.0, 80, 'stuffy'),    (31.0, 75, 'stuffy'),   (28.0, 85, 'stuffy'),
    (33.0, 40, 'hot and dry'),
]

def distancia(a, b):
    # humidity varies in tens and temperature in units: dividing by the range
    # gives both the same weight (this is called normalizing)
    dt = (a[0] - b[0]) / 20
    du = (a[1] - b[1]) / 50
    return math.sqrt(dt * dt + du * du)

def classificar(leitura, k=3):
    perto = sorted(exemplos, key=lambda e: distancia(leitura, e))[:k]
    votos = {}
    for e in perto:
        votos[e[2]] = votos.get(e[2], 0) + 1
    rotulo = max(votos, key=votos.get)
    return rotulo, votos[rotulo], k

for leitura in [(30.0, 78), (21.0, 52), (18.5, 58), (26.0, 65)]:
    rotulo, a_favor, k = classificar(leitura)
    print('%.1f °C, %d %% -> %s (%d of %d neighbors)' % (leitura[0], leitura[1], rotulo, a_favor, k))

Notice the reading of 26 °C and 65 %: it falls between the groups, and the neighbors do not all agree. The vote count is a confidence measure the model gives for free — and it is what makes the third path possible, further on.

kNN has a cost: on each reading, it compares with all the examples. With ten, it is instant; with thousands, the board starts to spend time and memory. That is why, in bigger projects, models that summarize the examples come in.

Training and deploying with the potentiometer

A model that summarizes the examples is the nearest centroid: training is computing the average of each group, and classifying is seeing which average the reading is closest to. The deployed model is three numbers.

In the circuit, the potentiometer plays the role of any sensor (a lever, a light sensor, a soil-moisture sensor). The program already brings the "collected" examples at three positions, trains, and then reads the real pin and classifies. When the reading is far from all the centroids, the model says it is unsure — and, in the third path, that is the moment to ask the server.

from machine import ADC, Pin
import time

# 1. collect: readings in mV with the shaft at three positions (examples for the class)
coletados = {
    'low': [210, 260, 190, 240, 230],
    'medium': [1580, 1660, 1700, 1620, 1640],
    'high':  [2950, 3010, 2900, 3050, 2980],
}

# 2. train: the model is the average of each group
modelo = {}
for rotulo, valores in coletados.items():
    modelo[rotulo] = sum(valores) / len(valores)
print('trained model:', {r: round(m) for r, m in modelo.items()})

# 3. deploy: read the sensor and choose the nearest centroid
LIMITE_INSEGURO = 400   # mV: farther than this from all of them, the model does not trust itself
adc = ADC(Pin(34), atten=ADC.ATTN_11DB)

def prever(mv):
    rotulo = min(modelo, key=lambda r: abs(mv - modelo[r]))
    return rotulo, abs(mv - modelo[rotulo])

for i in range(3):
    mv = adc.read_uv() // 1000
    rotulo, dist = prever(mv)
    if dist > LIMITE_INSEGURO:
        print('%d mV -> unsure (closest to %s, %d mV away): would ask the server' % (mv, rotulo, dist))
    else:
        print('%d mV -> %s' % (mv, rotulo))
    time.sleep(1)

Turn the potentiometer in the simulator and run again: near the ends, the model answers "low" or "high"; between the groups, it declares itself unsure. LIMITE_INSEGURO is a project choice: larger, the model takes more risks; smaller, it asks more often.

Libraries for bigger models

When the problem asks for more than averages and neighbors — recognizing a movement on an accelerometer, a sound, a gesture —, the heavy lifting of training stays on the computer, with machine-learning libraries, and the board receives just the ready model. For MicroPython, an open project that does this is emlearn-micropython. According to the project's page, it offers classification with decision trees and random forests, kNN (including learning on the board itself), small convolutional neural networks, linear regression and clustering with k-means, plus filters and FFT to prepare the signals. The models run as native modules (.mpy) that install without reflashing the firmware; the project says it is mainly tested on the xtensawin architecture, that of the ESP32 and ESP32-S3. For the ESP32-C3, which is RISC-V, check the project's documentation before planning the class.

TinyML's golden rule holds for any library: the model is only as good as the data it was trained on. Always set aside part of the examples for testing, without using them in training, and count how many the model gets right. A model that gets 100 % of the training examples right and gets the test ones wrong memorized the list instead of learning.

AI on the server: the ESP32 talking to an LLM

The second path swaps the small model for a language model, which understands and writes text. The ESP32 comes in as eyes and hands: it measures, sends, receives an answer and acts. The conversation is over HTTP, the same requests as in the article Wi-Fi, HTTP and MQTT.

There are two common ways to have an LLM at hand.

A model on the local network, with Ollama. Ollama is an open program that runs language models on an ordinary computer. It serves HTTP requests on port 11434; according to its documentation, it listens only on the computer itself (127.0.0.1) until the OLLAMA_HOST environment variable tells it to listen on the network, for example with 0.0.0.0:11434. The request goes to /api/generate with the model name, the text (prompt) and "stream": false, to receive the whole answer at once in the response field. The data does not leave the school, and there is no key or per-request cost; in return, the computer must stay on and have memory for the model.

A cloud API. Companies like Anthropic, which makes Claude, offer models through a paid API, accessed with a secret key. The models are bigger and more capable, and the school computer does not need to do anything; in return, each request costs money, the data leaves for the company and the key needs to be very well kept.

Why not call the LLM straight from the board

It seems natural to put the API key in main.py and call the cloud straight from the ESP32. Do not do this. The key is like a password that spends money: anyone with access to the board — just plug it into a computer and list the files — copies the key and starts making requests on your account. Anthropic itself recommends, in its key best practices, never putting them in distributed code and keeping them in environment variables or a secrets vault.

The safe architecture has an intermediate server of yours — a school computer, a hosting service — between the board and the LLM, as in the image. It:

• keeps the API key in an environment variable, outside the code;
• recognizes the board by a token of its own, which, if it leaks, can be swapped without touching the API account;
• limits how many requests each board makes per hour, so a faulty loop does not drain the balance in one afternoon;
• builds the prompt with the instructions and returns to the board only a short JSON, already checked.

The server also relieves the board: API responses come as large JSON over HTTPS, and every byte less is more memory for the ESP32's program.

Ask for JSON and check the answer

An LLM answers with text, and text can come in any format — including a wrong one. The board must never run what the model wrote as if it were a trusted command. The pattern that works has three parts:

1. the request tells the model exactly which format to return: a JSON with an acao from a list;
2. the answer is read with json.loads inside a try, because it may not be JSON;
3. the acao is checked against an allowlist of commands the board knows, and the values are limited to safe ranges. Whatever is not on the list is ignored.

The JSON keys and values of this protocol (acao, pino, motivo, ventilar, aquecer, nada) stay in Portuguese on purpose, as in the original article: they are the agreement between the board and the server, and the same words appear in the request text and in the examples below.

The example builds the request and tests the check with five possible answers, among them a made-up one and one that tries to ask for a pin that does not exist in the project. It runs entirely in the simulator, with no network.

import json

ACOES = {'ventilar', 'aquecer', 'nada'}      # what this board knows how to do
PINOS_PERMITIDOS = {2, 4}                     # and where

def montar_pedido(temp, umid):
    return json.dumps({
        'leitura': {'temperatura': temp, 'umidade': umid},
        'instrucao': 'Answer only a JSON {"acao": "ventilar"|"aquecer"|"nada", "pino": 2|4, "motivo": texto curto}',
    })

def conferir(texto):
    try:
        r = json.loads(texto)
    except ValueError:
        return None, 'answer is not JSON'
    if not isinstance(r, dict):
        return None, 'answer is not an object'
    acao = r.get('acao')
    if acao not in ACOES:
        return None, 'unknown action: %r' % (acao,)
    pino = r.get('pino', 2)
    if pino not in PINOS_PERMITIDOS:
        return None, 'pin not on the list: %r' % (pino,)
    return (acao, pino), str(r.get('motivo', ''))[:40]

print('request:', montar_pedido(31.2, 78))
print()

respostas = [
    '{"acao": "ventilar", "pino": 2, "motivo": "calor e umidade altos"}',
    '{"acao": "nada", "motivo": "ambiente ok"}',
    'Sure! I think you should turn on the fan.',
    '{"acao": "abrir_porta", "pino": 2}',
    '{"acao": "aquecer", "pino": 13}',
]
for texto in respostas:
    comando, info = conferir(texto)
    if comando and comando[0] == 'nada':
        print('accepted: nothing to do (%s)' % info)
    elif comando:
        print('RUNS %s on GPIO%d (%s)' % (comando[0], comando[1], info))
    else:
        print('ignores:', info)

Only two of the five answers are accepted, and only one turns something on. The other three — loose text, an action the board does not have, a pin not on the list — are discarded without risk. This check is what separates a demo project from one that can be trusted.

The board's code

With an Ollama on the local network, the ESP32 talks to it directly, without a key. The format: "json" field asks Ollama for JSON output, and stream: False makes the answer arrive at once. The address and the model name are examples: use those of the school's computer.

import json
import requests

OLLAMA = 'http://192.168.0.50:11434/api/generate'   # computer with Ollama

def perguntar(prompt):
    r = requests.post(OLLAMA, json={
        'model': 'model-name',
        'prompt': prompt,
        'format': 'json',
        'stream': False,
    }, timeout=30)
    try:
        if r.status_code != 200:
            return None
        return r.json()['response']   # the model's text, to pass through conferir()
    finally:
        r.close()

With the intermediate server, the board sends only the readings and its token; the one talking to the LLM is the server.

import requests

SERVIDOR = 'https://my-server.example/api/decide'
TOKEN_DA_PLACA = 'this-boards-token'    # not the API key: it is a token only for your server

r = requests.post(SERVIDOR, json={'temperatura': 31.2, 'umidade': 78},
                  headers={'Authorization': 'Bearer ' + TOKEN_DA_PLACA}, timeout=20)
try:
    if r.status_code == 200:
        comando = r.json()     # already checked by the server; the board checks again
finally:
    r.close()

On the server side, an example in Node.js with Anthropic's official SDK (@anthropic-ai/sdk). The key comes from the ANTHROPIC_API_KEY environment variable; the claude-haiku-5-5 model is from the Haiku line, the smaller and more economical models, and is enough for short answers like this one.

import http from 'node:http';
import Anthropic from '@anthropic-ai/sdk';

const cliente = new Anthropic();            // reads ANTHROPIC_API_KEY from the environment
const TOKENS = new Set([process.env.TOKEN_PLACA_1]);
const ACOES = new Set(['ventilar', 'aquecer', 'nada']);

http.createServer(async (req, res) => {
  const token = (req.headers.authorization ?? '').replace('Bearer ', '');
  if (req.method !== 'POST' || !TOKENS.has(token)) { res.writeHead(401).end(); return; }
  let corpo = '';
  for await (const parte of req) corpo += parte;
  const { temperatura, umidade } = JSON.parse(corpo);

  const msg = await cliente.messages.create({
    model: 'claude-haiku-5-5',
    max_tokens: 100,
    system: 'You control a classroom. Answer only a JSON {"acao": "ventilar"|"aquecer"|"nada", "pino": 2|4, "motivo": texto curto}.',
    messages: [{ role: 'user', content: `Temperature ${Number(temperatura)} °C, humidity ${Number(umidade)} %.` }],
  });
  const texto = msg.content.filter((b) => b.type === 'text').map((b) => b.text).join('');

  let resposta = { acao: 'nada', pino: 2, motivo: 'invalid answer' };
  try {
    const r = JSON.parse(texto);
    if (ACOES.has(r.acao) && [2, 4].includes(r.pino)) resposta = r;
  } catch { /* the safe answer stays */ }
  res.writeHead(200, { 'Content-Type': 'application/json' }).end(JSON.stringify(resposta));
}).listen(process.env.PORT ?? 3000);

For a real project, the per-hour request limit and HTTPS would be missing, but the essentials are there: the key only on the server, the board's token checked, the request built by the server and the answer checked twice, on the server and on the board.

The two together

The third path joins the potentiometer example with the LLM one: the board uses the small model and only makes the network request when the reading falls in "unsure". On a sensor that measures every minute, this can mean one question to the server per day instead of 1440 — less cost, less waiting and much less battery. The article batteries on the ESP32 shows how much the radio weighs in the math: each HTTP request keeps Wi-Fi on for seconds.

Precautions that hold for all three paths

• The model is wrong sometimes. A small model errs when the situation differs from the training data; an LLM errs with convincing sentences. Nothing that could hurt someone or break something — a motor, a heater, a lock — should depend only on a model's answer. Physical limits (a limit switch, a thermostat) and limits in the program always stay.
• Text that comes from outside is data, not orders. If the board sends the LLM text that came from another person (a message, a typed name), that text may contain instructions for the model. The allowlist on the board is the last barrier.
• Privacy. Readings from a room, sounds and images say a lot about who is there. Prefer processing on the board or on the local network, and only send to the cloud what is necessary.
• The network fails. Use timeout on every request and always have a safe default decision for when the server does not answer.

Where to go next

The Analog track teaches how to read the potentiometer and the moving average lesson shows how to clean a reading before passing it to the model. The Sensors track brings the DHT22, which supplies the data for the kNN example, and the Wi-Fi and HTTP track covers the network side. To store the collected examples, see the article storing data on the ESP32; to serve a page with the model's result, the article web server on the ESP32. The datasheets for the ESP32 DevKit V1, the potentiometer and the DHT22 complete the build.

Sources

• emlearn-micropython: available models, native .mpy modules and tested architectures.
• Ollama documentation: FAQ: port 11434, default address and OLLAMA_HOST.
• Ollama documentation: /api/generate: the model, prompt, format, stream and response fields.
• Anthropic's TypeScript/JavaScript SDK: messages.create and the key in ANTHROPIC_API_KEY.
• API key best practices (Anthropic): do not put keys in code and keep them in environment variables.
• MicroPython documentation: requests: the ESP32's HTTP module.