{ "video": { "id": "akk6KRlcwW4", "title": "OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack", "duration": 1476, "upload_date": null, "channel": "AI Engineer", "source": "AI Engineer", "event_format": "Online" }, "analysis": { "video_id": "akk6KRlcwW4", "title": "OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack", "one_liner": "A Callstack engineer with a PhD in physics shows the handheld, dual-display (OLED + e-paper) ESP32 terminal he built over ~3 months to drive OpenClaw and a local 120B open-source model on an NVIDIA DGX Spark — a deliberately quiet, distraction-free \"Game Boy for LLMs\" that also runs generative text RPGs.", "summary": "Lech Kalinowski set out to build a remote control for his OpenClaw instance running on a DGX Spark, and ended up building what he calls an AI-native operating system for a small microcontroller. The device pairs a fast one-colour OLED 'live surface' for typing with a bistable e-paper panel that renders the committed page, driven by an ESP32 dual-core MCU with fixed pre-allocated one-bit image buffers — no markdown engine and no malloc on the MCU. A backend handles all the agentic work and serves an open-source 120-billion-parameter GPT model through TensorRT behind an OpenAI-style LLM proxy. Beyond controlling OpenClaw, the surprise hit was an RPG mode with four generated worlds, which he argues proves the market niche: a calm text device with no colours, pop-ups or distractions.", "key_points": [ "Dual-display design: a small one-colour OLED is the 'live surface' where typed text appears dynamically; hitting enter triggers a full render onto the bistable e-paper panel — e-paper alone is perfect for reading but too slow for dynamic text.", "Rendering uses fixed static buffers of one-bit images held in pre-allocated memory: 'the pages live in a pre-allocated memory. There's no markdown engine and no malloc on the MCU side.'", "Hardware: ESP32 dual-core MCU, OLED, keyboard, rotary encoder, e-paper, and a substantial custom power-management/regulator unit — needed because he blew up two displays during prototyping and replacement parts took a couple of weeks to arrive.", "Four system modes/classes: an internal shell for terminal settings, Wi-Fi and basic configuration; an assistant control mode; and the RPG mode. Numbers cited: 130 commits, ~3 months of work, 2 displays, 4 modes, 4 worlds, 16 classes, one lithium-polymer cell.", "Backend architecture: firmware on the terminal plus a backend that handles the agentic OpenClaw work and the LLM. The demo runs an open-source 120-billion-parameter GPT model served with TensorRT, exposed as an OpenAI-style API through an LLM proxy — because not all open-source models match the OpenAI API style.", "Field notes / failures: software I2C needed the right approach to work without extra physical pull-ups; GPIO 13 produced a silent failure and the pin had to be moved; the regulator killed LED and fragile display parts; a cheap encoder gave rotational noise that required pull-ups and extra capacitors.", "Graceful degradation is designed in: if the OLED dies the e-paper still works, if the keyboard dies there's the encoder, if Wi-Fi drops you still have the local shell.", "RPG mode generates characters, worlds, maps, skills, NPCs and memory, with images converted to one-bit matrices for the displays; the four worlds are 'Older Guns', 'Neon Abyss', 'The Hollows' and 'The Void Witch'. Demo shows OpenClaw checking disk space on the DGX Spark and writing a Java example to the local machine.", "He filed a provisional patent, arguing everyone else is building around audio interfaces and video capture, leaving a niche for a quiet, distraction-free reading-and-writing device." ], "takeaways": [ "Match the display to the job — there are plenty on the market; pairing a fast small OLED for live input with slow bistable e-paper for the rendered page gets you both responsiveness and energy efficiency.", "Keep the model off the metal: today's LLMs are too heavy for super-tiny MCUs, so put the terminal on the microcontroller and the model on a backend behind an OpenAI-style proxy.", "Build the proxy layer up front — not all open-source models match the OpenAI API style, and he 'hit a lot of walls' before normalising on one.", "On constrained MCUs, pre-allocate fixed one-bit page buffers instead of running a markdown engine or malloc at runtime.", "Budget seriously for power supply and part quality: a bad regulator destroyed two displays and a cheap encoder cost extra pull-ups and capacitors — weeks of lost time from a few cheap components.", "For the LLM content itself, 'try narrative' — context matters more than numbers." ], "topics": [ "hardware", "embedded", "local-llms", "agents", "e-paper", "generative-games", "edge-ai" ], "tools": [ "OpenClaw", "NVIDIA DGX Spark", "ESP32", "TensorRT", "OpenAI API", "LLM proxy", "open-source 120B-parameter GPT model", "Callstack" ], "quotes": [ { "text": "I just wanted to build a device which is physical and AI native, like the device which comes from the future.", "at": "01:44", "url": "https://www.youtube.com/watch?v=akk6KRlcwW4&t=104s" }, { "text": "The pages live in a pre-allocated memory. That means there's no markdown engine and no malloc on the MCU side.", "at": "07:05", "url": "https://www.youtube.com/watch?v=akk6KRlcwW4&t=425s" }, { "text": "Try narrative. The context matters, and in general not numbers.", "at": "17:42", "url": "https://www.youtube.com/watch?v=akk6KRlcwW4&t=1062s" }, { "text": "In general, it was about the Game Boy. The device to play with LLMs.", "at": "24:03", "url": "https://www.youtube.com/watch?v=akk6KRlcwW4&t=1443s" } ], "words": 2890 }, "summary_url": "/#akk6KRlcwW4", "transcript": { "html": "/transcripts/akk6KRlcwW4.html", "txt": "/transcripts/akk6KRlcwW4.txt", "vtt": "/transcripts/akk6KRlcwW4.vtt" } }