WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=akk6KRlcwW4 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/akk6KRlcwW4.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:02.320 --> 00:00:07.240 Probably all of you once planned the clean up activities. s2 00:00:07.240 --> 00:00:13.800 And you can imagine I also planned one weekend to clean up of my basement. s3 00:00:13.800 --> 00:00:20.040 And I just uh went there uh and went to clean up all of the stuff there. s4 00:00:20.040 --> 00:00:24.040 You know, throw up uh old washing machine. s5 00:00:24.040 --> 00:00:34.160 And I just look at the corner there and realized there's something weird there lying on the floor. s6 00:00:34.160 --> 00:00:48.760 So, I just grab it, look at the the strange device looks a bit familiar, but a bit old like from '80s with keyboard but with a weird displays. s7 00:00:48.760 --> 00:00:51.320 Of course, covered with dust. s8 00:00:51.320 --> 00:00:53.280 I just take it to my hand. s9 00:00:53.280 --> 00:00:55.640 Look at it. s10 00:00:57.360 --> 00:01:04.559 And I was completely surprised that um device have a shell. s11 00:01:04.559 --> 00:01:07.000 And it's on. s12 00:01:07.440 --> 00:01:20.200 So, um I used my superpowers skills in a terminal and I asked I provided there a simple question. s13 00:01:20.200 --> 00:01:22.240 Who am I? s14 00:01:22.240 --> 00:01:28.520 And the device responds in the weirdest way I couldn't imagine. s15 00:01:28.720 --> 00:01:31.720 In the natural way language. s16 00:01:32.040 --> 00:01:39.520 So, the story I just described to you it can be a great story for a marketing. s17 00:01:40.240 --> 00:01:57.120 But honestly, um I just wanted to build a device which is physical and AI native like the device which comes from the future. s18 00:01:57.120 --> 00:01:58.880 Hello. s19 00:01:58.880 --> 00:02:01.560 My name is Lech Kalinowski. s20 00:02:01.600 --> 00:02:14.600 I hold a PhD in physics and today I want to present you a device and uh backend and a system I built over last couple of months. s21 00:02:14.600 --> 00:02:26.760 And the whole story begins because I just simply wanted to build a remote controller to my open claw instance on my DJX park. s22 00:02:31.800 --> 00:02:36.480 And um the story is simple. s23 00:02:36.480 --> 00:02:50.440 Because when you are just first think about the LLMs, then you're not thinking about the audio, about the generative uh generative uh videos or pictures. s24 00:02:50.440 --> 00:02:53.160 First, you're thinking about the text. s25 00:02:53.160 --> 00:03:05.360 And then I just get the bright idea that maybe I can use some kind of better display to read and type with my LLM. s26 00:03:06.120 --> 00:03:14.640 Of course, on the market, there is a electronic paper, but um I realized electronic paper can be a bit slow. s27 00:03:14.640 --> 00:03:26.680 Of course, it's perfect for reading, but if you just want to provide dynamically text to it, you proba- probably need uh faster display. s28 00:03:26.680 --> 00:03:46.800 And then I just uh used and uh in my design uh a simple connection of the OLED display, the small one, one color, with the electronic paper display. s29 00:03:46.800 --> 00:04:05.600 And then with this simple, let's say, dual display approach, I realized I can build quite a powerful and energy-efficient terminal to play with my claw. s30 00:04:08.080 --> 00:04:16.959 And um of course, behind it uh there is uh some kind of lot of complications. s31 00:04:16.959 --> 00:04:30.960 Because from one side, I just wanted to build a remote control, but from other side there is such a niche on the market and this is the AI-native operational systems. s32 00:04:30.960 --> 00:04:40.880 And when I just started to think more clearly about it I just realized I can build such a system for my purposes. s33 00:04:40.880 --> 00:04:48.520 But not with a super powerful powerful computers, but with a small microcontrollers. s34 00:04:48.520 --> 00:04:57.200 And I just designed a system a built in let's say with a with a say uh four classes. s35 00:04:57.200 --> 00:05:09.000 One is the internal shell to control the terminal itself, you know, system settings, basic configurations, Wi-Fi connection. s36 00:05:09.000 --> 00:05:25.320 And within my idea to build an AI-native device I build an assist control and uh one thing which really surprised me and what I will explain you later, RPG s37 00:05:25.320 --> 00:05:27.320 mode. s38 00:05:28.840 --> 00:05:32.840 And uh here it is. s39 00:05:32.840 --> 00:05:43.280 I started to prototype DGX Park at the um at the background here on the picture, two displays. s40 00:05:43.280 --> 00:05:47.880 One it's a live surface, that means the dynamic part. s41 00:05:47.880 --> 00:05:54.000 You can type the whole text provided is written there. s42 00:05:54.000 --> 00:06:07.640 And when you adjust trigger action, hit enter, then the whole render also happens on the second bistable electronical paper. s43 00:06:09.000 --> 00:06:23.080 But there is a lot of complications and lot of tech and lot of steps you need to do to get an clear user experience and clear interface. s44 00:06:23.080 --> 00:06:35.560 So, I just found out a lot of the different approaches how to render properly on the two displays properly render the the the the interface. s45 00:06:35.560 --> 00:06:45.400 I just interact to control my LLMs and to get in responses and to do whatever I want with my DGX park through the through the open claw. s46 00:06:45.400 --> 00:07:00.480 And this is the one of the approaches that you can just have fixed static buffers and render one-bit images stored in the in the memory. s47 00:07:00.480 --> 00:07:06.760 That means the pages live in a pre-allocated memory. s48 00:07:06.760 --> 00:07:16.000 There's a no markdown engine and no malloc on the MCU side. s49 00:07:16.840 --> 00:07:22.480 I just wanted to really to show you how the system is really complicated here. s50 00:07:22.480 --> 00:07:30.320 And this is not because I just wanted to use that much components. s51 00:07:30.320 --> 00:07:45.680 It's because there is a huge need to build a to build a power management system because I just blow up two these place over my prototype build. s52 00:07:45.680 --> 00:07:56.640 And um I just needed to be sure that the current and the voltage is stable as possible. s53 00:07:56.640 --> 00:08:02.960 So, here inside you have an MCU, which is ESP32 dual-core microcontroller. s54 00:08:02.960 --> 00:08:08.160 Here you have an OLED display, keyboard, and an encoder. s55 00:08:08.160 --> 00:08:15.160 And of course, the mention um power supply. s56 00:08:16.080 --> 00:08:32.400 But, the A native end-to-end device is not uh let's say that simple because of the, you know, high power consumption and the and the hunger for the compute capabilities of your device. s57 00:08:32.400 --> 00:08:36.360 So, here is a full backend created. s58 00:08:36.360 --> 00:08:40.919 That means I just created a full backend and here's the presented architecture of it. s59 00:08:40.919 --> 00:08:55.400 That first we have a vault firmware deployed on the on the terminal, and there's a huge backend, which um handle all of the all of the agentic work with the open claw s60 00:08:55.400 --> 00:08:57.720 and handle the LLM. s61 00:08:57.720 --> 00:09:15.600 I just want to point out here um within my today's presentation, I just want to demonstrate you um the device connected to an open-source GPT 120 billion parameters model. s62 00:09:15.600 --> 00:09:28.040 Um and I just serve that model with a let's say most possible optimal way using and I I I use their TensorRT serving system. s63 00:09:28.040 --> 00:09:43.360 And I just exposed OpenAI style to uh also like LLM proxy because I just hit a lot of walls with uh other open source models um because that's not all models s64 00:09:43.360 --> 00:09:47.760 match the style of the Open AI API. s65 00:09:50.120 --> 00:10:04.360 Um as I said uh the first idea was to build and mhm play with the Open Claw through that terminal and this is the stuff I just do with this. s66 00:10:04.360 --> 00:10:07.960 Here you can look at the picture and see an example. s67 00:10:07.960 --> 00:10:22.160 I was I was possible to hit a command like write a Java example uh and store it on my local machine and Open Claw with the with the LLM support just did s68 00:10:22.160 --> 00:10:26.080 work and make it happened. s69 00:10:28.760 --> 00:10:35.520 Okay, and here's my field notes, the things that doesn't go as expected. s70 00:10:35.520 --> 00:10:37.960 Uh lessons learned. s71 00:10:37.960 --> 00:10:48.080 Uh first of all the uh software I2C and because I just wired up things there and it doesn't work correctly. s72 00:10:48.080 --> 00:11:01.880 Um and there's a way to um to do proper control over over I2C without any additional physical pull-ups mounted. s73 00:11:01.880 --> 00:11:09.400 From other side GPIO 13 um there's a silent failure. s74 00:11:09.400 --> 00:11:12.640 Uh there was a need to move to the other port. s75 00:11:12.640 --> 00:11:17.680 Um to um to be sure that everything works correctly. s76 00:11:17.680 --> 00:11:35.760 From other side I need to build up the regulator the power supply um the whole unit of the power supply um because the the regulator uh kills the LED and and fragile parts of the display s77 00:11:35.760 --> 00:11:44.960 and cost me a lot of time and and getting the replacement parts from the market took so long. s78 00:11:44.960 --> 00:11:47.720 A couple of weeks. s79 00:11:47.720 --> 00:12:01.160 And the last part, quality of your parts, that means encoder cheap and low quality give me a lot of rotational noise and there was a need to build up the the pull-ups s80 00:12:01.160 --> 00:12:08.720 and to and to wire it up additional capacitors there. s81 00:12:08.720 --> 00:12:12.040 But this is the funnest stuff. s82 00:12:12.040 --> 00:12:14.120 My favorite one. s83 00:12:14.120 --> 00:12:17.920 Because the this part really surprised me. s84 00:12:17.920 --> 00:12:26.360 You already know there's a huge part called RPG and behind it is a bit sad maybe story. s85 00:12:26.360 --> 00:12:32.840 Because uh I just never played RPGs game like on the paper. s86 00:12:32.840 --> 00:12:35.600 That means with a with a lot of friends. s87 00:12:35.600 --> 00:12:48.760 You can go to a to apartment, you know, open a book and make an someone game master and play in a real text-based RPG. s88 00:12:48.760 --> 00:12:52.800 And after a lot of those years, I just never tried it. s89 00:12:52.800 --> 00:13:09.800 But I built um a RPG game and a console um which give me a pure experience of the text-based um RPG games, role-playing games. s90 00:13:09.800 --> 00:13:22.560 And actually I it's a bit funny because those device it's really like perfect fits for that kind of gaming. s91 00:13:22.560 --> 00:13:26.240 I build out an NPC and memory around that. s92 00:13:26.240 --> 00:13:41.040 I just created the mood of the world, the contents, omnious, and I just used all of the, you know, the LNM um advantages to build uh the best uh role-playing game experience s93 00:13:41.040 --> 00:13:42.680 ever. s94 00:13:42.680 --> 00:13:45.120 Um I just created four different worlds. s95 00:13:45.120 --> 00:13:48.720 From one side, I just wanted to build something cyberpunk related. s96 00:13:48.720 --> 00:13:56.000 From other side, I just wanted something uh The Witcher part related, that means some kind of fantasy world with the dragons. s97 00:13:56.000 --> 00:14:05.720 But also my favorite one is the is the void in a deep space somewhere um in the cosmos. s98 00:14:06.280 --> 00:14:17.080 And this is really uh really good example how we can just use uh generative AI to build computer games. s99 00:14:17.080 --> 00:14:30.360 From one side, I just generate the characters, their worlds, maps, the skills, uh and all of those converted with the mechanism I just described at the beginning of that presentation s100 00:14:30.360 --> 00:14:41.480 with the one-bit memory allocation, that means all of the pictures transform to the to the matrices, to the uh to the to the maps. s101 00:14:42.200 --> 00:14:52.360 And um one um important uh important thing, actually uh the device is really bulletproof. s102 00:14:52.360 --> 00:14:56.360 Um if the LED doesn't work, you the e-paper works. s103 00:14:56.360 --> 00:15:04.480 If the cardboard doesn't work, that means the keyboard, uh then you'll you have an encoder. s104 00:15:04.480 --> 00:15:11.800 Wi-Fi falls, you always can use your uh local uh shell. s105 00:15:11.800 --> 00:15:21.120 Um And um funny stuff, the area of the intellectual property there. s106 00:15:21.120 --> 00:15:34.120 I just uh do some kind of research in the field and everyone just wants to build the devices around the audio interfaces, the video capture, you know, but for the quiet places, s107 00:15:34.120 --> 00:15:45.320 when you just want to, you know, sit down, keep calm, and play your RPG game, you don't need any of the, you know, colorful, powerful displays, you know, all of these, s108 00:15:45.320 --> 00:15:53.920 you know, popping up with the colors and all of the distractions, commercials, web pages, and other things. s109 00:15:53.920 --> 00:16:04.120 Sometimes you just want to sit down and you know, do do the right job with the with the with the open claw or or or you know, just chat with your LLM, s110 00:16:04.120 --> 00:16:10.240 read and write in the quiet environment without any of the distractions. s111 00:16:10.240 --> 00:16:12.560 And there's a niche on the market. s112 00:16:12.560 --> 00:16:27.160 So, I also decided to build a the provisional patent here in that field and maybe it will bring some commercial device in the future to a to a market. s113 00:16:27.160 --> 00:16:30.000 Some numbers. s114 00:16:30.600 --> 00:16:40.839 In my project I just pushed 130 comments around 3 months of work. s115 00:16:41.200 --> 00:16:44.040 Two deep displays. s116 00:16:44.040 --> 00:16:47.760 Each of the display do other job. s117 00:16:47.760 --> 00:16:49.440 Four modes. s118 00:16:49.440 --> 00:16:51.760 For worlds, for gaming. s119 00:16:51.760 --> 00:16:54.839 In general, 16 classes. s120 00:16:54.839 --> 00:16:58.839 That means you can generate RPG worlds inside of it. s121 00:16:58.839 --> 00:17:03.920 Firmware is is really tiny, really small, and it works extremely fast. s122 00:17:03.920 --> 00:17:11.640 And everything is powered by one single polymer cell. s123 00:17:11.640 --> 00:17:14.640 Lithium polymer cell. s124 00:17:18.319 --> 00:17:24.040 But maybe some remarks and takeaways. s125 00:17:24.040 --> 00:17:26.560 Much of this place to the job. s126 00:17:26.560 --> 00:17:29.280 Uh there's a plenty of them on the market. s127 00:17:29.280 --> 00:17:33.520 Keep the model of the metal because it's really heavy. s128 00:17:33.520 --> 00:17:39.640 That means for now we don't have such a models which runs really on the super tiny MCUs. s129 00:17:39.640 --> 00:17:42.200 In in means LLMs. s130 00:17:42.200 --> 00:17:44.200 And try narrative narrative. s131 00:17:44.200 --> 00:17:53.640 The context matters and in general not numbers. s132 00:17:54.120 --> 00:18:09.640 So here I also record a the demo, the usage, how the device work and some kind of uh some kind of um last um last session. s133 00:18:09.640 --> 00:18:12.040 Um please enjoy. s134 00:18:12.040 --> 00:18:17.040 And I will place a couple of comments there. s135 00:18:41.200 --> 00:18:42.760 Device is booting. s136 00:18:42.760 --> 00:18:46.840 Connecting to a local Wi-Fi network. s137 00:18:46.840 --> 00:18:52.520 Here's the welcome screen and the main screen. s138 00:18:55.000 --> 00:18:57.280 Small display. s139 00:19:04.960 --> 00:19:12.520 Keyboard first simple help command. s140 00:19:13.800 --> 00:19:18.200 Here you will have all of the things you can do with the device. s141 00:19:22.280 --> 00:19:28.720 There's also a tutorial for gaming and tutorial for the device itself. s142 00:19:28.720 --> 00:19:40.560 And when you just switch it on for the first time, then you need to or can go you don't have to, but you can go through the tutorial itself to learn how to use it. s143 00:19:49.400 --> 00:19:55.840 Usage of the Open Claw agent, that means right now I'm just checking my DGX Spark. s144 00:19:55.880 --> 00:20:00.200 And want to look at the disk space. s145 00:20:12.160 --> 00:20:14.480 Here's that. s146 00:20:40.720 --> 00:20:49.920 Let's maybe now look on the my favorite part with gaming. s147 00:21:17.760 --> 00:21:27.560 For worlds Older guns, neon abyss The hollows and the void witch. s148 00:21:43.880 --> 00:21:52.720 You can select your mm your character. s149 00:22:04.160 --> 00:22:08.280 And L and just generates the stuff. s150 00:22:08.280 --> 00:22:10.520 Everything is generated there. s151 00:22:10.520 --> 00:22:15.880 Um pictures, uh uh personalities of the of the people. s152 00:22:15.880 --> 00:22:18.360 And here, there's a mood. s153 00:22:18.360 --> 00:22:19.760 There's a breach. s154 00:22:19.760 --> 00:22:28.560 You're in the deep space and can't deep dive into the other world. s155 00:22:50.200 --> 00:22:52.200 Yeah. s156 00:23:10.760 --> 00:23:13.920 So, that's it. s157 00:23:15.760 --> 00:23:17.560 That was the the video. s158 00:23:17.560 --> 00:23:34.000 Funny part is that the game like this without the three-dimensional graphics use the most advanced Nvidia uh processor mounted in the DGX Park that means the the the this is the bug and currently I just s159 00:23:34.000 --> 00:23:36.880 prepare for the demonstration. s160 00:23:36.880 --> 00:23:50.560 And so, the quiet game with the you know, text interface requires the the most powerful computer computer graphics card in the in the world. s161 00:23:50.560 --> 00:24:05.280 And maybe another thing is that the presentation was about the handle device to control uh agent and open close with the local LLMs, but in general, it was about the s162 00:24:05.280 --> 00:24:06.440 Game Boy. s163 00:24:06.440 --> 00:24:12.480 The device to to play with LLMs. s164 00:24:12.480 --> 00:24:28.680 Thank you for the attention thank you very much for the for the Kostak support because I'm just working in at Kostak in the technological incubator uh and the whole work was supported by s165 00:24:28.680 --> 00:24:32.360 by Kostak company. s166 00:24:32.360 --> 00:24:34.720 Thank you very much.