Stop Renting Your AI's Memory — Dylan Couzon, Qdrant

AI Engineer · 15 min · 194 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Hi everyone, thank you for joining me today.
00:15I hope everyone is having a great conference so far.
00:18Um, so my name is Dylan Kuzan.
00:20I am a developer relations engineer at Quadrant.
00:24And the title of my talk today is the frontier is coming home.
00:29So what do I mean by that?
00:30And somewhere in the next few years, something smarter than all of us is going to exist.
00:36We've spent years asking when, but I think when is kind of the boring question.
00:41I think the real one is who it belongs to because there are two versions of this.
00:47In one, super intelligence lives in someone else's data center and you pay by the token.
00:54In the other, it's yours.
00:56It learns from your life.
00:57your context stays private and nobody can switch it off, throttle it or take it away from you.
01:04That second version is what I want to talk about today.
01:07That's the frontier coming home.
01:10Quick thought experiments.
01:12What would you build if your assistants actually remembered everything you ever taught it?
01:18Every preference, every correction, every dead end for years.
01:22That's not a chatbot any anymore.
01:24That's a second mind.
01:26But the the assistant you're using right now forgets almost all of it the second the conversation ends.
01:34Every session starts from zero.
01:36We call these things intelligence.
01:38And they are they're just geniuses with no long-term memory.
01:44This talk is about fixing that.
01:46Not with a bigger prompt with actual memory.
01:51That genius used to only exist behind someone's API in a data center you never see.
01:56Now it runs on a on hardware that most of us own.
02:00Not the absolute frontier, I'm going to be honest, but shockingly close.
02:04A machine under $2,500 today can run what was basically last year's frontier.
02:12And open weights models are closing on fast.
02:15The gap is shrinking every quarter.
02:19You'd expect this to feel like a landmark uh the frontier on your desk for the price of a laptop and yet it doesn't quite land because a model on its own is just a brilliant stranger.
02:32It's powerful.
02:34It's but it it isn't yours.
02:36What make what makes it yours is everything it knows about you and that part hasn't come home yet.
02:44So you can run a frontier class model on your own hardware today and yet look at the AI stack that most of us are using today.
02:52The compute is rented, the models is rented, the harness is rented and the part that's supposed to become you, the thing that should compound over the years also rented.
03:04If we don't fix that, the most powerful technology any technology any any of us will ever touch ends up owned by whoever holds the lease, not the people it was built for.
03:17So let's look at what renting actually costs today.
03:21Starts where we starts with where the model leads because when you rent it, you don't really control it.
03:29This shows up in three different ways.
03:32Last month, a government order pulled access to two flagship models, Fable and Mythos 5 for every single customer.
03:40Now, compare that to weights you already that are already sitting on your desk.
03:45Nobody can reach into your machine and flip those off.
03:49Even when the model the model stays up, it doesn't stay put.
03:53Providers retire versions and they throttle what's left.
03:56When the economics get gets tight, the model you rely on can quietly get worse or vanish.
04:04Weights your own never change unless you change them.
04:08People also don't don't write prompts anymore.
04:10They run agents or loops for hours.
04:14And one task can burn thousands of times more tokens than a chat message.
04:18Some developers burn through over $5,000 a month of of compute on a $200 plan.
04:25That's a subsidy.
04:26And those subsidies will ultimately disappear.
04:30If you own the compute and the weights, the control comes back to you.
04:34But that's still only half the problem.
04:37Here's the deeper issue.
04:39Owning inference gives you autonomy.
04:42Nobody can take the model away.
04:44But only memory gives you continuity.
04:46And continuity is exactly what every major lab is trying to package up and sell back to you.
04:52Look at last year.
04:53Every Frontier Lab shipped a memory feature, not because the models got smarter, but because they don't carry anything between sessions.
05:01That gap is the actual product.
05:04A system that starts simple and keeps learning you will always feel smarter than one that starts brilliant, but forget everything about you.
05:14That compounding is where intelligence starts to feel personal.
05:19So what actually is memory?
05:22It's not a bigger prompt.
05:23It's a systems with three verbs.
05:26Write, retrieve, and forget.
05:29Same three things your brain actually does.
05:33And you know, there's an obvious push back.
05:35Why not just throw everything in a big folder of markdown files.
05:40And because retrieving the right memory beats dumping everything and praying the model finds it.
05:46Retrieval gives you something a prompt cannot.
05:49It is control.
05:50You can filter by topic.
05:52You can decay by recency and frequency.
05:55You can let relevance shift over time.
05:58The way human memory actually works.
06:02And the infrastructure has been ready for years.
06:05HNSW in 2016.
06:08Um, embeddings have been running locally since 2019, long before any any model could really use them at scale.
06:16The memory side of super intelligence was never the hard part.
06:20We were just pointing it out at ourselves.
06:25So today, the memory that actually knows you lands in one of two places.
06:29Either it's stuck in an unindexed file you can't really query or it's in someone else's cloud.
06:36One terms one terms of service change away from being being reshaped or moved.
06:42Corp Carpathy framed it in a way that really stuck with me.
06:46It's the model is the CPU, the memory uh the context window is the RAM and the memory is the disk.
06:52RAM is fast but forgets forgets the moment the session ends.
06:57The disk is what remembers you across every conversation.
07:01And right right now for almost every AI assistant on on Earth, the that disk is in someone else's data center.
07:09So, you know, today I'm not going to argue which memory architecture wins or is the best one.
07:15That's definitely a different talk.
07:18But the disk, the parts that holds the actual record of you should be yours fully and forever.
07:25So, this is the piece that I work on and it's just one way to do it.
07:30Um, a ve I work on a vector search engine that embeds directly into your app.
07:34It opens look a local store inside your process.
07:38You write embeddings with payloads.
07:40Then query offline sub millisecond time.
07:44Same rust score as quadrant in the cloud just running where you are.
07:48With quantization, a million memories can fit in less than a gigabyte.
07:53Small enough to live on a phone or even a Raspberry Pi.
07:57Close the app, come back a year later, the memory is intact.
08:01Every app gets its own isolated store.
08:04And that folder is the thing that actually follows you.
08:08So, I'm going to do a quick live demo here.
08:12Um, so right now, um, I have a video recording of a drone like going over a house.
08:20So, basically that drone, it's it first time starting.
08:23It's it's first time out in the world.
08:25It doesn't know anything.
08:27And so, this this is actually running live right now fully offline on my laptop.
08:34And I'm using yellow which is an object detection model.
08:38So it detects the object and creates a um label for every items it sees here.
08:45Then we take take that picture, we take that label and we turn that into embeddings to create a live realtime memory for that drone.
08:56And and that way we can re the drone can remember everything it has ever seen before.
09:02And so here you can see a 2D representation of that vector space with all the memories inside.
09:09And memories that are the most similar will will be the closest to each other.
09:16So here you can see that we've already recognized 92 different objects and we have over three 300 vectors.
09:23So that's 300 different representations of those 90 objects.
09:28And you can see that the entire memory and quadrants engine uh footprints is only 15 megabytes.
09:38And then what we can do uh on those memory?
09:41Well, we enable um semantic search over those memories.
09:46So, you know, if I click here on the um coffee table, you can see that in less than one millisecond, I was able to pull up every single coffee table that this drone has
09:59seen before.
10:00And so, you know, we have the image here.
10:03We have the first sin time stamp.
10:05We have the last s time stamp.
10:07We have a definition of that coffee table.
10:10and and also we can see how many times that u table has been seen.
10:16So this is all being run and processed locally.
10:20There's no network attachment at all.
10:22Um but we also have a capability of uh cloud sync.
10:28So for example, you know, if you have like a swarm of drones or robots navigating the world, they can have like a hive mind in the in the cloud where
10:37I you know, one system uh learns something and they can all learn about it.
10:46And you know, this is what this was like an example for a drone, but you can apply that to to everything.
10:53just a chatbot, an agent that you call to a code assistant.
10:58And you know, if you create those memories for just a few weeks, the stranger just disappears.
11:05It remembers your corrections, your preferences, the dead ends you had to only hit once.
11:11And a model that's merely good, but never forgets you and keeps compounding over the years starts to feel super human in practice.
11:21Not because the model changed, but because it never stopped learning you.
11:26And it's portable.
11:27Swap the resulting model, change the hardware, none of it matters.
11:31It all comes with you if you keep the same embedding model and that folder becomes permanent.
11:37A lifetime of context fully yours moving from device to device.
11:42But we can go even further than that.
11:44You know, this can go way past a laptop or a drone.
11:49points that same memory at your whole day.
11:51Where did I leave my badge?
11:53What was her daughter's name again?
11:56What was on that whiteboard?
11:58And what what was the song that was playing when we met?
12:02Every one of those is just a a retrieval query.
12:06And you know, you just watched a drone remember objects.
12:10No, extend that to everything you see and everything you hear.
12:15And you know, that's not science fiction.
12:17Those are the smart glasses that are already shipping today and that we have seen today in this room.
12:25It's not a smarter you.
12:27It's a you that doesn't forget.
12:30And right now, none of those memories share uh none of those devices share a memory that you actually own.
12:38They could on your own terms because Frontier models are already building an index of you today.
12:44They have your relationships.
12:45They have your habits.
12:47They they have your whole inner life.
12:50That index is common whether you opt in or not.
12:54The only question is whether you own it or if you're renting access to yourself.
13:01And one last idea because owning memory doesn't doesn't mean keeping it to yourself.
13:07Edge can sync parts of your store to the quad quadrant cloud out of the box when you choose to.
13:13Picture a family, each wearing glasses that record their own day.
13:18Separate memories, but they can pull into a hive mind the whole family owns.
13:23Mom's day, dad's day, your day, searchable together.
13:28Now, scale that up.
13:29Instead of one company's super intelligence serving billions of identical people, you get thousands of small private ones, each shaped by a life, a family, a team.
13:42But that same mechanism cuts both ways.
13:45A memory can be shared without a memory share can be shared with consent or it can be extracted without it.
13:52That's exactly why it has to start local and why sharing should always be optin never the default.
14:00You should really decide who gets your continuity and your memories.
14:06And this is where I'm going to leave you.
14:08You know, every everyone here is going to watch super intelligence show up over the next few years.
14:14That's that part is basically settled.
14:16What's not settled is who it belongs to.
14:19And that's the fight worth having right now.
14:22While the AR while the architecture is still up for grabs.
14:26So tonight, don't go home and just try a tool.
14:29Take the stranger home and give it memory.
14:32Then picture that memory 5 years from now.
14:36not a chatbot, a private mind that never forgets you, that nobody can switch off, and that knows knows you better than any system ever has because you were the only one who ever trained it.
14:48That's not a that's not a smaller super intelligence.
14:52That's the only one worth wanting.
14:56All right, that's all for me.
14:58Thank you very much.
15:14[music]