I Built a Personal AI Agent on a Raspberry Pi — Jeremy Adams, Neo4j

AI Engineer · 20 min · 340 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:12How's everybody doing?
  2. 00:15Day what is it?
  3. 00:164 400 something.
  4. 00:20Thank you for bearing with us as we had a little technical difficulty getting started.
  5. 00:23But I am Jeremy Adams.
  6. 00:25I'm from Neo4j.
  7. 00:27You might have seen our booth.
  8. 00:28It's kind of right over that way.
  9. 00:31And we're all about graphs and graph databases.
  10. 00:34Today I'm going to expand that a little bit to a side project I've been playing with.
  11. 00:40Uh who here has done something with uh open claw or Hermes agent or any of your kind of claw of choice.
  12. 00:51Okay.
  13. 00:51So a lot of some folks but most folks are kind of new to the space which is cool.
  14. 00:56How about uh things that are more like edge devices, hardware?
  15. 01:01Um thinking about like you know offline mode kind of stuff.
  16. 01:04Yeah.
  17. 01:04Okay.
  18. 01:05Some interest there.
  19. 01:06Um how about who's experienced with graph in general?
  20. 01:10A little bit of graph some graph people.
  21. 01:13Cool.
  22. 01:13And how who's been doing like building agents with memory and things like that?
  23. 01:20Okay.
  24. 01:20Whoa.
  25. 01:20Cool.
  26. 01:20That's the the biggest group so far.
  27. 01:23And I want to keep it going.
  28. 01:24We're going to try and do more things that have to do with uh statistics of some sort uh as we get to some findings that we're going to do in a little bit.
  29. 01:33So, um what I'm going to do in this talk today is talk about this agent memory project that I did that involves a bit of hardware that you can see on my chest.
  30. 01:43And um as we go through it, we're going to talk about how I also built up some software for agent memory and how that all went.
  31. 01:53So, this is like the livest of the live demos I've done with this so far because I'm like literally assembling the pieces in front of you.
  32. 02:00So, let's see.
  33. 02:01We're I have a microphone, little extra microphone.
  34. 02:05Yes, I'm going to be using this to talk to my claw in a second.
  35. 02:09It's not It's not yet powered on.
  36. 02:11So, this is really crazy.
  37. 02:13And I've got this uh I've got like a capture a video capture from my HDMI, which is over here.
  38. 02:21I'm going to plug that in.
  39. 02:24Whoa.
  40. 02:24Somewhere.
  41. 02:25Plug that in.
  42. 02:26Okay.
  43. 02:27It's not on yet.
  44. 02:28I'm also going to This is the receiver for the USB for the mic.
  45. 02:33So, we hope this works too today.
  46. 02:37If I can see where it goes.
  47. 02:38There we go.
  48. 02:39Plug that in.
  49. 02:43Guess that's why they made USBC.
  50. 02:44You have to keep flipping the thing over.
  51. 02:46Okay.
  52. 02:47So, there that goes in.
  53. 02:48And now gonna try and sync these two mics.
  54. 02:53Okay, they're green.
  55. 02:54That's okay.
  56. 02:55So, I got a mic there.
  57. 02:56Now for some power.
  58. 02:58Good old good old battery power.
  59. 03:01All right.
  60. 03:01So, what I'm going to do really quick, just for fun before we get into the main presentation.
  61. 03:07I'm going to show you this trick.
  62. 03:09So, with this weird video capture situation I've got going on, I'm going to go over here to my QuickTime player, and I'm going to say that I want a
  63. 03:23I want something that it will not let me do right now.
  64. 03:26Let me quit QuickTime Player and let's reopen it.
  65. 03:31All right, here we go.
  66. 03:33And let's try and do a f a new movie recording.
  67. 03:37All right, we can see there we go.
  68. 03:38You see the claw?
  69. 03:40All right, we don't want to see that.
  70. 03:41We want to see what's on the claw.
  71. 03:43So, what I'm going to do instead is I'm going to use the source will be from my from the output of this video here.
  72. 03:52All right.
  73. 03:52So, let's power it up and I'll try my best.
  74. 03:57There we go.
  75. 03:57To catch it during the boot sequence.
  76. 04:00All right.
  77. 04:01So, I'm going to go here to my I'm going to change to this USB3 capture device and I'm going to move this around a little bit so we can see it.
  78. 04:13And hopefully, I mean, I can't give you any guarantees, but hopefully what's going to happen is as this guy goes, we're going to see something actually happen.
  79. 04:23So, I'm going to give a little reset to that and back to that.
  80. 04:28Oh, and let me push power on this.
  81. 04:31Okay, there we go.
  82. 04:32All right.
  83. 04:32So, again, hopefully I see power is now flowing through into here.
  84. 04:38Okay.
  85. 04:38Oh, something's happening.
  86. 04:40Okay.
  87. 04:40And um this gives me high confidence because I've done this only like one million times as you can imagine, hacking on this silly thing.
  88. 04:50Okay.
  89. 04:50Raspberry Pi desktop.
  90. 04:52Oh, yeah.
  91. 04:52So, this is a Raspberry Pi 4B, not the latest model.
  92. 04:57Okay.
  93. 04:57So, um, but very serviceable, and I was running a 32-bit operating system for a long time until I realized it was 64-bit ARM capable.
  94. 05:07It's pretty cool.
  95. 05:08Um, you can't run everything on this thing, but you can run quite a bit.
  96. 05:12And, um, we're going to get into exactly what I did put on this to end up getting this presentation together.
  97. 05:19Okay, cool.
  98. 05:20All right.
  99. 05:20So, it's alive and I've got a little extra Bluetooth keyboard and that's how I can drive things on here.
  100. 05:26And we will do that um in a second.
  101. 05:29But I mostly just kind of wanted to show you the setup so you could, you know, believe that it actually is a real thing.
  102. 05:36And uh there we go.
  103. 05:39All right.
  104. 05:39So, now I'm going to get into the presentation.
  105. 05:41We'll come here.
  106. 05:41There we go.
  107. 05:42I'll come back to this Pi here in just a second after I open up.
  108. 05:47There we go.
  109. 05:48And just to give you a a flavor, uh, this one has nano claw, but it also has Docker on here.
  110. 05:55And I'm running Neo Forj on on here.
  111. 05:58I've got a little Docker based Neo Forj database.
  112. 06:00Okay, now let's continue with the talk and we'll get back to the claw in a minute.
  113. 06:07So small claw.
  114. 06:09What did I I I said small claws are beautiful.
  115. 06:11What did I mean by that?
  116. 06:13Well, there was a lot of hype happening and still is about these personal agents, but I did not want one that was on my laptop.
  117. 06:19I'm sorry.
  118. 06:20I'm a little conservative.
  119. 06:21I come from the old kind of cisadmin devops space.
  120. 06:24I'm like, I want to wait for this stuff to be really solid before I do something like that.
  121. 06:28But I I also wanted to be cheap, open, hackable, right?
  122. 06:33And for me, I wanted to understand what was happening more than it being featurerich.
  123. 06:38That was more important to me.
  124. 06:39And so I went and I saw like lo I live in Portland, Oregon.
  125. 06:42I looked on Craigslist.
  126. 06:43Somebody's selling a Mac Mini with OpenClaw pre-installed.
  127. 06:47I'm already like pre-installed.
  128. 06:48That's a red flag.
  129. 06:50And you know, Mac Mini could be cool, but um I know now it's not going to be 600 anymore with the recent price increases, right?
  130. 06:57So that's not a small claw as far as I'm concerned.
  131. 07:00So then I reached back and this is uh I did I think it's pandemic or at the end of the pandemic I did something where I was hacking with this very Raspberry Pi
  132. 07:09doing something like measuring the weight of the dishes in your sink to tell you you should wash the dishes.
  133. 07:15Anyway, you could check out the video there if you like that kind of nonsense.
  134. 07:18But that's this exact guy.
  135. 07:19And I saw I said I have this in the closet.
  136. 07:22Will it claw?
  137. 07:23I don't know.
  138. 07:24I should try it and see.
  139. 07:25And uh yeah, it claws real good.
  140. 07:28It was and um that that that little that it's not a sticker.
  141. 07:32It's like more like me, you know, photoshopping a a claw on there just like, you know, oh, maybe someday I could actually, you know, uh do something really cool with this.
  142. 07:41And uh but you see my old setup.
  143. 07:43I have this external monitor.
  144. 07:45I've got my external keyboard.
  145. 07:47There's a mouse off screen and everything.
  146. 07:49This guy, this is a game changer right here.
  147. 07:51I'm like so it's so sleek now.
  148. 07:54Uh, it's got onboard Wi-Fi, which is great, or you can plug into wired.
  149. 07:57So, hopefully I'm actually on the network.
  150. 07:59We'll have to check that in a second.
  151. 08:01Um, okay.
  152. 08:03And then addition, uh, I used nano claw.
  153. 08:07So, I didn't use open claw.
  154. 08:09I found this other project that more suited my tastes because it was like 15 source files and it was quite compact and small number of lines of code.
  155. 08:18It just used Docker containers to run the agent processes.
  156. 08:21So they couldn't run a mock on the system.
  157. 08:24I was like, okay, this is my kind of thing.
  158. 08:26So I said, I'm going for it.
  159. 08:27Uh, and so I like the other boxes that check for me.
  160. 08:31I could run Linux on here.
  161. 08:32It's got a Docker.
  162. 08:33I understand.
  163. 08:34Uh, it was based on cloud agent SDK, nano clause, and I could and they actually encourage you modify it.
  164. 08:42Use cloud code.
  165. 08:43We got skills in there.
  166. 08:44Hack on it.
  167. 08:45Make it work for you.
  168. 08:46And it use messaging like, you know, Telegram if you want.
  169. 08:49But I was like, well, I had WhatsApp on my phone, so I'll just use WhatsApp.
  170. 08:53Perfect.
  171. 08:54So, the initial architecture was something like this to go from my iPhone uh or my MacBook WhatsApp, you know, and then go through and talk to NanoClaw on this Raspberry Pi.
  172. 09:04That's it.
  173. 09:05And you just had to get like a credential.
  174. 09:07You go on the NanoClon, you get it to register.
  175. 09:09So, now you know WhatsApp's in the cloud.
  176. 09:11And this thing can pull messages down from it.
  177. 09:14And of course, you know, on board, I said we're using Claude stuff in the background.
  178. 09:18So you got all the power.
  179. 09:19There's no inference, no LLM inference happening on this thing.
  180. 09:23I am using the CL the cloud for all that.
  181. 09:25So I've got a big brain though it's over over a wire to the cloud, right?
  182. 09:31But then uh I said well what can I do with this?
  183. 09:33You know uh I I've got also in the claw in the cloud I also have Neo forj.
  184. 09:39I've got Neo Forj database up there.
  185. 09:41And for the folks here that have never done graph before this is your graph 101.
  186. 09:45There is a small graph and this is a graph.
  187. 09:47Can you see which ones are the nodes, right?
  188. 09:50The round ones.
  189. 09:51And you see the relationships, the edges between them.
  190. 09:54So, there's two person nodes and a movie node.
  191. 09:57And Tom Hanks, the person acted in Horus Gump and Robert Zmechus directed it.
  192. 10:03Pretty easy.
  193. 10:04That's it.
  194. 10:04Now, you know, graphs.
  195. 10:05So, I asked in a WhatsApp message once I had this database full of movies and I had my claw connected.
  196. 10:13I'm like, "Hey, tell me what movies Tom Hanks acted in."
  197. 10:16and it went through there with an MCP server connecting into my Neoforj instance with the movie database in there, right?
  198. 10:24And then it responded back to me.
  199. 10:26It's like, "Oh, well, first your stuff's not set up right.
  200. 10:29It's broken."
  201. 10:29I'm like, "Oh, okay.
  202. 10:31Let me fix it."
  203. 10:32So, I got on the claw, made sure O was working, and then I got a bunch of results back on the right there.
  204. 10:37And those all came right out of pulling data from the database, feeding it back up through the cloud agent SDK, forming a response, and sending it back through the WhatsApp channel.
  205. 10:46Got it?
  206. 10:47So then I was like, well, what else could I do with this?
  207. 10:49And I'm on the airplane, and I found that even without paying for Wi-Fi, I could use the messaging stuff and WhatsApp was working.
  208. 10:57I was like, oh my god, I could talk to Claw right now.
  209. 11:00So I was like, hey, could I do memory?
  210. 11:03And I said, well, I'm not exactly sure what I want, but I was reading blog posts about this thing called poll plus O. So, this is a way of doing memory where you just think of like person, object,
  211. 11:14location, event, uh, uh, an organization, right?
  212. 11:18And it's this actually came from the European policing and you know those folks have been drawing graphs for a long time and every every one of your favorite cop shows that does a string diagram, right?
  213. 11:28And so then I just was on the plane.
  214. 11:29I'm like, hey, could we build a memory system using pole plus o?
  215. 11:33And then it's like, "Yeah, sure.
  216. 11:34Yeah, we can do it.
  217. 11:35I'm your right of skill for that right now.
  218. 11:36I've got this mount point.
  219. 11:38It'll persist even when you reboot me.
  220. 11:40No problem."
  221. 11:40I was like, "This is amazing."
  222. 11:42And so now I have this thing where I've got Neo4j in the cloud.
  223. 11:45I've got this new memory database with this super advanced schema memories and related things very early, you know, before Carpathy's, you know, wiks and be after Obsidian, but you know, you get the idea.
  224. 11:58It's it's that kind of thing.
  225. 12:00And this is what ends up being in Neo forj memory nodes that show like places I was going and as I went to do little Devril events I was having this thing record it for me and all that.
  226. 12:10So then we can like actually jump in and look at this database.
  227. 12:14So I'm going to go over here to um to the web browser which I'm already in but I'm going to go over to this other tab and over here I've got Neo forj Aura
  228. 12:24and I've got this demo instance.
  229. 12:26I'm going to connect to it in query.
  230. 12:27And this is where I've been storing all this information kind of over time.
  231. 12:31And so I'm going to jump in and show you the memory nodes.
  232. 12:36I got a bunch of stuff.
  233. 12:37It's kind of a dumping ground in here, I admit.
  234. 12:40And um here's all the here's the memories.
  235. 12:44So he these are those original memory types that I had in here.
  236. 12:47And I actually want to show you the memories.
  237. 12:50I want to show you all the memories uh as they're connected to other things.
  238. 12:55And I'm going to run that.
  239. 12:56So you can see more like the connected graph.
  240. 12:59And as I zoom in on this thing, you can see there's like me going to there's, you know, talking to a person going to the snowflake office in Menllo Park.
  241. 13:09Menlo Park is itself a location in that framework and then I was at a certain AI meetup event, right?
  242. 13:17So it follows and there's people in here like Rebecca, like Jeremy, there's different people, right?
  243. 13:22So this is one way of doing it.
  244. 13:23So then going back over here, we see that we can definitely store memories.
  245. 13:29Well, what else?
  246. 13:29I was thinking can I fit on this physical claw and somebody was doing like they had a microphone out their window and they were recording all the bird song in the backyard
  247. 13:39and they could there was a bird net model that that could tell you that.
  248. 13:43And I was like, "Oh, whoa.
  249. 13:44I want to do that."
  250. 13:45And then I was like, "Oh, wait.
  251. 13:46Or maybe I could do voice to text.
  252. 13:47Is that possible?"
  253. 13:48So then I was like, "Sure enough, let me hook up a USB microphone, right, and do something like that."
  254. 13:53And then do some voice to text and and do something like that.
  255. 13:57So, um, and just to get ahead of it, just for fun, I'm going to go ahead and try and use Oh, that's funny.
  256. 14:06Oh, there it is.
  257. 14:06The button fell off.
  258. 14:08So, I I made this button and it's connected into into some pins.
  259. 14:13This is me getting ahead a little bit, but just in the interest of time, I'm going to go ahead and plug this back in.
  260. 14:18And what this allows me to do is trigger that model and I'm going to do some quick recording.
  261. 14:25I'm saying I am actually on stage at AI Engineer World delivering the talk I've been preparing for, right?
  262. 14:33And so I you see the little lights went on.
  263. 14:35So it did some recording.
  264. 14:36Hopefully we'll be able to look and see if we actually if I'm actually online, if we got a message and so forth.
  265. 14:41But this is the dream, right?
  266. 14:42And I started doing it with all sorts of microphones, playing around with it.
  267. 14:46And then I was like, "Oh, I want to bring this to AI engineer and I want to like walk around, right?
  268. 14:52And with these recording little LEDs on it, and I want to like record something.
  269. 14:56I don't know what I want what I want to record."
  270. 14:59And I was like, I know what I'll do.
  271. 15:00I'll go booth to booth and I'll record what messaging people have up at their booths.
  272. 15:06And then I was like, let me build this thing.
  273. 15:08And then I actually started working on, you know, adding in that kind of capability for recording.
  274. 15:13And then I was like, okay, well, the booths, they got numbers and names.
  275. 15:16Okay, I had to map this out.
  276. 15:18So I was like, I'm just going to like create a very simple data model.
  277. 15:21It's just exhibitors and they're not even connected.
  278. 15:24They're just there in the database.
  279. 15:26So I tried to get as many as I could and there was some data cleaning to do.
  280. 15:29And now then then uh I was worried.
  281. 15:31I was like, "Oh god, conference Wi-Fi is terrible typically."
  282. 15:35So I need like an offline mode.
  283. 15:36So then I had this other thing.
  284. 15:38I put Neo forj running on here and then I was like, well I I could have this thing so if I'm offline it'll use some reaxes and stuff and parse out of the audio voice to text the booth number and maybe it can write a query to insert it in this local database.
  285. 15:54And you know again if this if this thing is uh if this thing is up which I at this point I'm not even I'm not even sure.
  286. 16:03Um, we're going to try cipher.
  287. 16:06Uh, my cipher shell browser.
  288. 16:09Oh, I should spell right.
  289. 16:10Let's go.
  290. 16:11Cipher.
  291. 16:13Shell browser.
  292. 16:14So, this is like a little program I wrote to talk to Neoforj databases without a very expensive guey and things like that.
  293. 16:23And I should be able to match all my exhibitors and return those.
  294. 16:28So, this is talking to the database that's running in here around my neck.
  295. 16:31And as I go in here and I look, I could say, "Oh, yeah, there's Microsoft, Minmax, CI, right?"
  296. 16:37And so this is me and I had on these like my raw notes.
  297. 16:40It's like, "Oh, this is like the raw notes that I took.
  298. 16:43I'm at this booth and blah blah blah."
  299. 16:46Right?
  300. 16:46So I and I took that data and then I uploaded it later up into Neo4j in the cloud.
  301. 16:54And in the cloud, um, it looks a little bit different because I've now enriched it.
  302. 17:00And so if we look at this, let's see if I can just move back into Neo forj for a second.
  303. 17:07And here I'm going to look for exhibitors, right?
  304. 17:11And here are my exhibitors.
  305. 17:12And the exhibitors, I can click on one.
  306. 17:15And you can see they're actually connected to something else.
  307. 17:18So I did load all their data in.
  308. 17:20So here's, you know, got the notes all cleaned up now on them.
  309. 17:23But you'll notice that these other things are connected to.
  310. 17:26And the things that they're connected to is I'm going to put in here they're connected to all sorts of other things.
  311. 17:33And I'm going to put a path through them and return the path.
  312. 17:38And we end up with something like this.
  313. 17:42This shows how they're all connected to some other theme nodes.
  314. 17:47So at some point I figured out, oh look, there's some themes that exist here.
  315. 17:52Evaluation and observability.
  316. 17:54Build kite has that theme.
  317. 17:55Lang chain has that theme, right?
  318. 17:57So I I was like, okay, I've got I've actually generated some knowledge that was maybe novel that maybe nobody has any, right?
  319. 18:06So are there themes?
  320. 18:07Yes, there are themes.
  321. 18:08So these this is the data dump that I collected by walking around the booth and figuring out what the themes were.
  322. 18:14So there's a whole bunch out there.
  323. 18:16And um I know I'm just about at my time.
  324. 18:20The thing uh I will point out for folks that are interested uh in talking more, I did subsequently upgrade the agent memory that I have using the agent memory service.
  325. 18:30Basically, I wanted to load all of the WhatsApp messages that I ever had in all and it's been a while I've been talking to it as you see.
  326. 18:39I wanted to upload all of those in there and see if I could get some kind of insights out of those.
  327. 18:44And sure enough, I use this Neo Forj agent memory service.
  328. 18:47And if you log into this sort of thing, at some point I just connected it up and that allowed me to get a whole bunch of memories from the conversations
  329. 18:57and then it distilled them out.
  330. 18:59And you can see here that uh I've got all these different things, these conversations that I've had and so forth.
  331. 19:06And this will allow me to browse those memories and um for example, all the people I've spoken to or the locations, the concepts, whatever.
  332. 19:17So this these are all the things that I've been doing talking about with my claw and they're all distilled out into memories that are now accessible via an MCPU server from this very claw.
  333. 19:29So anyway, that gives you some flavor of how this stuff works.
  334. 19:33Please come see us at booth P3 over there.
  335. 19:36I can tell you a lot more.
  336. 19:37If you're just getting started with graph, you can go to Graph Academy, which is this link right there, and we've got all these hands-on courses.
  337. 19:45It's all totally free to use.
  338. 19:47So, please either uh visit us there or even better at the booth.
  339. 19:51Thank you so much.
  340. 20:07[music]