OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

AI Engineer · 24 min · 166 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:02Probably all of you once planned the clean up activities.
2. 00:07And you can imagine I also planned one weekend to clean up of my basement.
3. 00:13And I just uh went there uh and went to clean up all of the stuff there.
4. 00:20You know, throw up uh old washing machine.
5. 00:24And I just look at the corner there and realized there's something weird there lying on the floor.
6. 00:34So, I just grab it, look at the the strange device looks a bit familiar, but a bit old like from '80s with keyboard but with a weird displays.
7. 00:48Of course, covered with dust.
8. 00:51I just take it to my hand.
9. 00:53Look at it.
10. 00:57And I was completely surprised that um device have a shell.
11. 01:04And it's on.
12. 01:07So, um I used my superpowers skills in a terminal and I asked I provided there a simple question.
13. 01:20Who am I?
14. 01:22And the device responds in the weirdest way I couldn't imagine.
15. 01:28In the natural way language.
16. 01:32So, the story I just described to you it can be a great story for a marketing.
17. 01:40But honestly, um I just wanted to build a device which is physical and AI native like the device which comes from the future.
18. 01:57Hello.
19. 01:58My name is Lech Kalinowski.
20. 02:01I hold a PhD in physics and today I want to present you a device and uh backend and a system I built over last couple of months.
21. 02:14And the whole story begins because I just simply wanted to build a remote controller to my open claw instance on my DJX park.
22. 02:31And um the story is simple.
23. 02:36Because when you are just first think about the LLMs, then you're not thinking about the audio, about the generative uh generative uh videos or pictures.
24. 02:50First, you're thinking about the text.
25. 02:53And then I just get the bright idea that maybe I can use some kind of better display to read and type with my LLM.
26. 03:06Of course, on the market, there is a electronic paper, but um I realized electronic paper can be a bit slow.
27. 03:14Of course, it's perfect for reading, but if you just want to provide dynamically text to it, you proba- probably need uh faster display.
28. 03:26And then I just uh used and uh in my design uh a simple connection of the OLED display, the small one, one color, with the electronic paper display.
29. 03:46And then with this simple, let's say, dual display approach, I realized I can build quite a powerful and energy-efficient terminal to play with my claw.
30. 04:08And um of course, behind it uh there is uh some kind of lot of complications.
31. 04:16Because from one side, I just wanted to build a remote control, but from other side there is such a niche on the market and this is the AI-native operational systems.
32. 04:30And when I just started to think more clearly about it I just realized I can build such a system for my purposes.
33. 04:40But not with a super powerful powerful computers, but with a small microcontrollers.
34. 04:48And I just designed a system a built in let's say with a with a say uh four classes.
35. 04:57One is the internal shell to control the terminal itself, you know, system settings, basic configurations, Wi-Fi connection.
36. 05:09And within my idea to build an AI-native device I build an assist control and uh one thing which really surprised me and what I will explain you later, RPG
37. 05:25mode.
38. 05:28And uh here it is.
39. 05:32I started to prototype DGX Park at the um at the background here on the picture, two displays.
40. 05:43One it's a live surface, that means the dynamic part.
41. 05:47You can type the whole text provided is written there.
42. 05:54And when you adjust trigger action, hit enter, then the whole render also happens on the second bistable electronical paper.
43. 06:09But there is a lot of complications and lot of tech and lot of steps you need to do to get an clear user experience and clear interface.
44. 06:23So, I just found out a lot of the different approaches how to render properly on the two displays properly render the the the the interface.
45. 06:35I just interact to control my LLMs and to get in responses and to do whatever I want with my DGX park through the through the open claw.
46. 06:45And this is the one of the approaches that you can just have fixed static buffers and render one-bit images stored in the in the memory.
47. 07:00That means the pages live in a pre-allocated memory.
48. 07:06There's a no markdown engine and no malloc on the MCU side.
49. 07:16I just wanted to really to show you how the system is really complicated here.
50. 07:22And this is not because I just wanted to use that much components.
51. 07:30It's because there is a huge need to build a to build a power management system because I just blow up two these place over my prototype build.
52. 07:45And um I just needed to be sure that the current and the voltage is stable as possible.
53. 07:56So, here inside you have an MCU, which is ESP32 dual-core microcontroller.
54. 08:02Here you have an OLED display, keyboard, and an encoder.
55. 08:08And of course, the mention um power supply.
56. 08:16But, the A native end-to-end device is not uh let's say that simple because of the, you know, high power consumption and the and the hunger for the compute capabilities of your device.
57. 08:32So, here is a full backend created.
58. 08:36That means I just created a full backend and here's the presented architecture of it.
59. 08:40That first we have a vault firmware deployed on the on the terminal, and there's a huge backend, which um handle all of the all of the agentic work with the open claw
60. 08:55and handle the LLM.
61. 08:57I just want to point out here um within my today's presentation, I just want to demonstrate you um the device connected to an open-source GPT 120 billion parameters model.
62. 09:15Um and I just serve that model with a let's say most possible optimal way using and I I I use their TensorRT serving system.
63. 09:28And I just exposed OpenAI style to uh also like LLM proxy because I just hit a lot of walls with uh other open source models um because that's not all models
64. 09:43match the style of the Open AI API.
65. 09:50Um as I said uh the first idea was to build and mhm play with the Open Claw through that terminal and this is the stuff I just do with this.
66. 10:04Here you can look at the picture and see an example.
67. 10:07I was I was possible to hit a command like write a Java example uh and store it on my local machine and Open Claw with the with the LLM support just did
68. 10:22work and make it happened.
69. 10:28Okay, and here's my field notes, the things that doesn't go as expected.
70. 10:35Uh lessons learned.
71. 10:37Uh first of all the uh software I2C and because I just wired up things there and it doesn't work correctly.
72. 10:48Um and there's a way to um to do proper control over over I2C without any additional physical pull-ups mounted.
73. 11:01From other side GPIO 13 um there's a silent failure.
74. 11:09Uh there was a need to move to the other port.
75. 11:12Um to um to be sure that everything works correctly.
76. 11:17From other side I need to build up the regulator the power supply um the whole unit of the power supply um because the the regulator uh kills the LED and and fragile parts of the display
77. 11:35and cost me a lot of time and and getting the replacement parts from the market took so long.
78. 11:44A couple of weeks.
79. 11:47And the last part, quality of your parts, that means encoder cheap and low quality give me a lot of rotational noise and there was a need to build up the the pull-ups
80. 12:01and to and to wire it up additional capacitors there.
81. 12:08But this is the funnest stuff.
82. 12:12My favorite one.
83. 12:14Because the this part really surprised me.
84. 12:17You already know there's a huge part called RPG and behind it is a bit sad maybe story.
85. 12:26Because uh I just never played RPGs game like on the paper.
86. 12:32That means with a with a lot of friends.
87. 12:35You can go to a to apartment, you know, open a book and make an someone game master and play in a real text-based RPG.
88. 12:48And after a lot of those years, I just never tried it.
89. 12:52But I built um a RPG game and a console um which give me a pure experience of the text-based um RPG games, role-playing games.
90. 13:09And actually I it's a bit funny because those device it's really like perfect fits for that kind of gaming.
91. 13:22I build out an NPC and memory around that.
92. 13:26I just created the mood of the world, the contents, omnious, and I just used all of the, you know, the LNM um advantages to build uh the best uh role-playing game experience
93. 13:41ever.
94. 13:42Um I just created four different worlds.
95. 13:45From one side, I just wanted to build something cyberpunk related.
96. 13:48From other side, I just wanted something uh The Witcher part related, that means some kind of fantasy world with the dragons.
97. 13:56But also my favorite one is the is the void in a deep space somewhere um in the cosmos.
98. 14:06And this is really uh really good example how we can just use uh generative AI to build computer games.
99. 14:17From one side, I just generate the characters, their worlds, maps, the skills, uh and all of those converted with the mechanism I just described at the beginning of that presentation
100. 14:30with the one-bit memory allocation, that means all of the pictures transform to the to the matrices, to the uh to the to the maps.
101. 14:42And um one um important uh important thing, actually uh the device is really bulletproof.
102. 14:52Um if the LED doesn't work, you the e-paper works.
103. 14:56If the cardboard doesn't work, that means the keyboard, uh then you'll you have an encoder.
104. 15:04Wi-Fi falls, you always can use your uh local uh shell.
105. 15:11Um And um funny stuff, the area of the intellectual property there.
106. 15:21I just uh do some kind of research in the field and everyone just wants to build the devices around the audio interfaces, the video capture, you know, but for the quiet places,
107. 15:34when you just want to, you know, sit down, keep calm, and play your RPG game, you don't need any of the, you know, colorful, powerful displays, you know, all of these,
108. 15:45you know, popping up with the colors and all of the distractions, commercials, web pages, and other things.
109. 15:53Sometimes you just want to sit down and you know, do do the right job with the with the with the open claw or or or you know, just chat with your LLM,
110. 16:04read and write in the quiet environment without any of the distractions.
111. 16:10And there's a niche on the market.
112. 16:12So, I also decided to build a the provisional patent here in that field and maybe it will bring some commercial device in the future to a to a market.
113. 16:27Some numbers.
114. 16:30In my project I just pushed 130 comments around 3 months of work.
115. 16:41Two deep displays.
116. 16:44Each of the display do other job.
117. 16:47Four modes.
118. 16:49For worlds, for gaming.
119. 16:51In general, 16 classes.
120. 16:54That means you can generate RPG worlds inside of it.
121. 16:58Firmware is is really tiny, really small, and it works extremely fast.
122. 17:03And everything is powered by one single polymer cell.
123. 17:11Lithium polymer cell.
124. 17:18But maybe some remarks and takeaways.
125. 17:24Much of this place to the job.
126. 17:26Uh there's a plenty of them on the market.
127. 17:29Keep the model of the metal because it's really heavy.
128. 17:33That means for now we don't have such a models which runs really on the super tiny MCUs.
129. 17:39In in means LLMs.
130. 17:42And try narrative narrative.
131. 17:44The context matters and in general not numbers.
132. 17:54So here I also record a the demo, the usage, how the device work and some kind of uh some kind of um last um last session.
133. 18:09Um please enjoy.
134. 18:12And I will place a couple of comments there.
135. 18:41Device is booting.
136. 18:42Connecting to a local Wi-Fi network.
137. 18:46Here's the welcome screen and the main screen.
138. 18:55Small display.
139. 19:04Keyboard first simple help command.
140. 19:13Here you will have all of the things you can do with the device.
141. 19:22There's also a tutorial for gaming and tutorial for the device itself.
142. 19:28And when you just switch it on for the first time, then you need to or can go you don't have to, but you can go through the tutorial itself to learn how to use it.
143. 19:49Usage of the Open Claw agent, that means right now I'm just checking my DGX Spark.
144. 19:55And want to look at the disk space.
145. 20:12Here's that.
146. 20:40Let's maybe now look on the my favorite part with gaming.
147. 21:17For worlds Older guns, neon abyss The hollows and the void witch.
148. 21:43You can select your mm your character.
149. 22:04And L and just generates the stuff.
150. 22:08Everything is generated there.
151. 22:10Um pictures, uh uh personalities of the of the people.
152. 22:15And here, there's a mood.
153. 22:18There's a breach.
154. 22:19You're in the deep space and can't deep dive into the other world.
155. 22:50Yeah.
156. 23:10So, that's it.
157. 23:15That was the the video.
158. 23:17Funny part is that the game like this without the three-dimensional graphics use the most advanced Nvidia uh processor mounted in the DGX Park that means the the the this is the bug and currently I just
159. 23:34prepare for the demonstration.
160. 23:36And so, the quiet game with the you know, text interface requires the the most powerful computer computer graphics card in the in the world.
161. 23:50And maybe another thing is that the presentation was about the handle device to control uh agent and open close with the local LLMs, but in general, it was about the
162. 24:05Game Boy.
163. 24:06The device to to play with LLMs.
164. 24:12Thank you for the attention thank you very much for the for the Kostak support because I'm just working in at Kostak in the technological incubator uh and the whole work was supported by
165. 24:28by Kostak company.
166. 24:32Thank you very much.