A Genius With Amnesia - Victor Savkin, Nx

AI Engineer · 20 min · 277 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:04Imagine you find a magic lamp in an antique store.
00:07You rub it, a genie appears, and asks how it can help.
00:12You burn it in the outline, so you say, "I need the best engineer to help with an impossible project at work."
00:18And the genie grants your wish.
00:20For me, the best engineer is probably John Carmack from his eight days, so you get Carmack.
00:25But the genie had a sense of humor and imposes restrictions, maybe for safety.
00:31Carmack can only see one small part of your code base, maybe 1/1000 of it.
00:36And he remembers nothing he did before.
00:38Every conversation starts fresh.
00:41So, that would be maddening, right?
00:42You would know there is a standard way to do stuff, and Carmack wouldn't.
00:46You would have to explain the same thing over and over and over again.
00:50You would have a genius on one side and something deeply deficient on the other.
00:55And that's what agents are.
00:57Let me walk you through an example of how many times we re-explain things in a simple interaction.
01:03We have four repos, UI, module one, module two, and platform.
01:09I want to change the UI and propagate the change through the system.
01:12Okay?
01:13First, we change the UI library.
01:14Say we, I don't know, change a button or whatever.
01:17That's the first explanation, unavoidable.
01:19We have to express the intent, okay?
01:22Then we publish it.
01:22We go to module one, and we have to re-explain what just has happened in the UI library, so it can consume the package here.
01:30Note that that's often a different person, right?
01:32Every box in this diagram can be done by a different person.
01:37And then we discover that the published UI library doesn't work with module one.
01:43So, we go back to UI, and we have to re-explain the original change and the issue.
01:51Right?
01:51Because it's a new agent, it doesn't know the original change.
01:54It obviously doesn't know about the issue.
01:57Let's say we fix it, right?
01:59And uh publish it again.
02:01We go and again we explain the new change in the context of module one, same module.
02:07I mean, do the same for module two again.
02:10And then we go to the platform repo and we explain everything fits together and we implement the change there.
02:17Let's imagine a week after release, a bug appears in the UI component and we have to fix it.
02:23So we start an agent in the UI repo.
02:26And we have to explain again the original change from a week ago and this production issue we're seeing.
02:32So we have seven explanations for what essentially is one change.
02:37And also it may not be one person making all these seven explanations, but they still occurred, right?
02:43So that's very very typical with agents.
02:46So how do we solve it?
02:50Well, there are many problems in here that contribute to this experience, but they roughly fall into two categories.
02:58The first one is that an agent essentially is repo bound.
03:03Agent sees and changes generally one repo at a time.
03:08It never sees the whole system, which can be hundreds or thousands of repos.
03:14So that's kind of the space component of the problem.
03:18Second is amnesia.
03:19Agent forget the work.
03:21Every session start with a blank slate.
03:24The human becomes the memory in this case.
03:26That's the time component of the problem.
03:28Look at the two closer.
03:29Take the repo boundary first.
03:32Without a model how repos fit together, the agent leans on the human to do the research.
03:38It can't align the code with the rest of the system.
03:40It couldn't align the UI change with module one.
03:43The human didn't explain it, so a bad version shipped.
03:48It can't reliably reference best practices and standards either because those often live in other repos.
03:55Worse than even worse, the agent writes to one repo at a time, it means it can't validate changes downstream.
04:01Modules one CI should have failed on the UI change, but it didn't.
04:05The agent can't update consumers at the same time even though, you know, while making the UI change, it has perfect information to do so.
04:12It knows exactly what it's doing.
04:14So, the user has to re-explain stuff imperfectly to each consumer.
04:20Changing something across 20 repos means re-explaining things 20 times.
04:24A lot of developer time spent, but also a lot of tokens burned.
04:28The second category is that the agent forgets.
04:32The agent has no episodic memory.
04:35Every session is a blank slate.
04:38And the human, in this case, becomes the memory.
04:42Here's what the graph of your work actually looks like.
04:46At the bottom, there is a repository graph.
04:49The artifacts your organization produces plus every open source repo you depend on.
04:55Maybe a thousand repos you own and tens of thousands of open source repos.
05:00At the top, there are all agentic sessions that create and modify that code.
05:05Sessions relate to each other, repos relate to each other.
05:09So, this graph is a faithful picture of the work in your organization.
05:14It describes what's there at the bottom and how it came to be at the top.
05:20That's what you want your agent to see.
05:24Here's what it actually sees.
05:26There is one session, one small fraction of the code base, no memory, okay?
05:33Because it sees so little, it leans on the one who understands the system, the developer.
05:41Every developer has a part of that graph, right?
05:44In their hand, at least in the domain they know.
05:47Agent generically speaking doesn't.
05:50If this doesn't sound crazy, right?
05:52Imagine an agent that could see one file at a time maximum and can only look five messages back.
05:59Sort of constraint again both in space, what can see, and time, how far in the past it could see.
06:05You would say that's impossible to work in.
06:08What we have now is similar to that crazy picture.
06:12And the more complex the organization is, the more apparent it becomes.
06:18I'll show you how we solved it.
06:21Other organizations I talked to have similar solutions, so uh look at the problem and the solution conceptually, not the specific tool, although the tool is pretty cool.
06:32We built an agent agnostic meta harness called Polygraph.
06:39Okay, let me show you what it does and how it fixes the issues we just discussed.
06:44The first idea that we uh arrived at is that if a GitHub user, any user, has access to thousands of repos, some of them they own, many of them are open source,
06:58we can analyze them and extract a lot of metadata out of them to build unified dependency graph.
07:07Uh no line of code changes in those repos, that all happens kind of on a side, right?
07:12And then we can get this metadata and feed it to the meta harness and create an illusion of one big code base the agent can read and write anywhere.
07:24This is my personal graph.
07:26I want to have about 300 repos I own, right?
07:30And thousands of open source repos my projects depend on.
07:33Polygraph computes what each one produces, each repo, each project in each repo, what each project in each repo consumes package-wise, what API they produce and consume, and lots of other stuff, right?
07:44And this stitches this together uh into this like one big body of code that your agent can work with.
07:53So, let's see what it does, right?
07:56The first thing it it does is uh it lets you start a session to bring the relevant repositories in, right?
08:05So, what it needs to do, it needs to uh set up the source code, install dependencies, set up an agent for each repo, wire them up so they can work together,
08:18and provide a clean, beautiful GUI to make non-trivial changes without getting lost.
08:25I will show you how it all works in a second, right?
08:28So, that's kind of pull information in.
08:31Pulling information in is only one part of the story, all right?
08:34Honestly, it's an easy part.
08:36Making changes is harder.
08:38If you have 10 repos in one session, it means you can have 10 pull requests, all right?
08:45You need to run CI.
08:47You need to coordinate all of it, right?
08:51You need to do all this stuff, all right?
08:53What if one of them fails, right?
08:55Polygraph treats all the CI as one vector.
09:00Like if you look at the earlier example, uh when we run CI for UI module one and module two, if module one fails within a polygraph session, it will figure out who fixes it, whether module one needed a patch
09:13or the UI component itself is broken and incompatible with module one, at which point everyone will need a patch, right?
09:20Polygraph lets you treat complex multi-repo change as if it was a single repo change.
09:27The same machinery, by the way, fixes episodic memory.
09:32Because we capture your work, no matter how many repos are involved, we We your intent, the repository is involved, PRs.
09:38We also capture all agent traces.
09:41Because we capture all of this stuff, we can relate it.
09:43So, now we can say your work in one repo connects to another work in another repo.
09:49Right?
09:50And all of that lets us restore any session, any piece of work on any machine, or reference it from anywhere.
09:58And I will show you again how it works in a second.
10:01What you get is an agent with eidetic or photographic memory of your entire organization.
10:09It understands how repos are written, how they relate, how they put together, and remembers every session from every repo by basically every developer.
10:19Right?
10:20And that creates a completely different development experience.
10:24Let me show you.
10:26First, let's look at how we create a session, something simple.
10:30You run a command, and you pick some repositories from a list.
10:36Here's a tiny GitHub work with only three repos because it's a demo.
10:40I pick back end and a front end.
10:43Let's say I need to make a change that, you know, changes the API and has to update both the API and how stuff is being displayed.
10:51I need to give my session a name.
10:54I need to pick an agent from the ones I have installed.
10:57I picked Claude.
10:59But any installed agent works same way.
11:01Remember, Polygraph isn't an agent.
11:04It's a meta harness around an agent that makes them uh more capable.
11:10And in a second, uh the agent boots.
11:17And here I could interact with it as if I was in a single repo, even though multiple repos are involved, right?
11:24I could give it instructions.
11:28It's going to uh plan out the change.
11:33There is some cool animations in the UI as well.
11:38Eventually,
11:40[snorts]
11:41it figures out how the two repos relate and what the change is.
11:45I can ask it to implement the change.
11:47My interaction with this uh exactly same as if it I was working in a single repo.
11:53The fact that there are multiple repos involved is not really important, right?
11:57Uh the only uh part where it becomes important that I have multiple pull requests, right?
12:02Uh but I also get a polygraph session where those pull requests are.
12:06All right?
12:07If I look at the session, I will see I have a description uh that uh description of the session describes the work conceptually uh kind of bypassing the repo boundary, saying we have to change stuff in this repo, change stuff in that repo.
12:19It gives me a good view of which repos are involved, pull requests involved, CI in those repos, everything I need to know.
12:25A lot of this stuff is basically what I would have in a single repo but many, right?
12:30And I also have all the agent logs captured as well, which is important for resuming, which I'm going to show you in a second.
12:36Now it gets interesting.
12:38I already saved one re-explanation.
12:41I didn't re-explain the back end change uh in the in the front end repo, right?
12:46I explained the change once and I got it implemented in both repos and it's all in agreement.
12:50Now let's resume a session.
12:52Say I want a coworker to finish the back end change.
12:54Perhaps they own the back end repo.
12:57I send them the session.
12:58They resume it on their machine, all right?
13:00So this I'm sending them the session.
13:02They could run the command.
13:04Uh different machine, different everything.
13:06They use different terminal, all right?
13:08Uh they would reconstruct it on their machine.
13:11They don't have the session, right?
13:13They have never worked on it.
13:15They they can pick an agent.
13:16Uh the agent they pick could be a different agent, right?
13:19I used Cloud in the original session, but let's say they're using a different one, Cortex.
13:23The same setup happens to their machine, same repos, same SHAs, everything set up correctly.
13:29Agent starts in a triple like in mine, all right?
13:32They're all connected again, so they work together.
13:34They're all primed with a trace captured from my machine, so the back end of the repo agent on their machine has the same shine and the same history.
13:42The front end repo situation is the same.
13:44It's It's checked out at the same at the correct shard, has the agent running with the correct history.
13:50So, my agent was Claude.
13:51They coded, but they share memory.
13:54And they could actually actually make changes in here as shown in a small video.
13:59Um but I think what the memory sharing part is key, all right?
14:03Uh I can work, they can work, and we can share our memories as though we used to different agents in different machine.
14:08The full state of my session can I get materialized on their machine.
14:13It kind of less memory and more about the state, right?
14:15The state of the world that's attached to the session, uh you know, is what enables them to continue my session even though they had didn't do anything with it originally.
14:25It's close to the transporter in Star Trek.
14:27Like a whole copy of my session is all of its state materialized on their machine so they can continue.
14:32And that's how I often work when there is a pull request for me to review and I have questions, I usually don't ask the person.
14:39I resume their session on my machine, I get the exact state, fully functional, zero setup, and then I just talk to my agent about the decisions we made, right?
14:48[snorts]
14:48Because all these decisions are in the traces captured.
14:52So, my agent knows exactly what the other person talked to their agent.
14:56Right?
14:57So, a side note, this is also useful when I want to switch from say Claude to Codex mid-session when something goes down, okay?
15:05Okay.
15:06Take the earlier case I talked about where a bug landed in production.
15:10Here, I'm going to reference the session and say it's basically broken.
15:16Uh and you know, can you figure out what's wrong and fix it?
15:21The agent will look it up, will download what it needs.
15:27If description is like high-level information is enough, that's great.
15:30If not, it's going to pull relevant repos, relevant shards, agent logs, right?
15:37It's going to get all this information from the original session to reconstruct that state such that it can do the necessary fixes as shown here.
15:44He actually provided the fix, right?
15:47I only had to say, "This happened.
15:50There is a bug."
15:51That's it.
15:52No extra information was required for me to provide.
15:58Okay.
15:59So far, we have manually selected repos and sessions, but we don't have to, all right?
16:04Instead of selecting repos by hand, I can also tell the agent what I want.
16:08Remember that graph has all this intelligence, right, about how repos relate.
16:13I could tell my agent, "Find every repo that depends on a particular version of a library and update it."
16:20Right?
16:21[snorts]
16:23I mean, it knows, right?
16:24I I didn't have to select them.
16:25It knows a lot of metadata about what's going on.
16:28I can also ask loose questions.
16:30Things like, you know, uh "What if I I want to write a blog post, right, or an article?"
16:37I could describe it and it will figure out which repo is the most relevant based on relationships between repos and what's in them.
16:45Another example.
16:46Let's say I want to add vector index into the PR collection.
16:51And I want to know if anyone at any point did something relevant in any repo that I can draw from.
16:57So, in this case, if I do it, I'll see that it will find several sessions that appear to be relevant.
17:03[snorts]
17:03And I can load one of them or both of them, right?
17:06Um it's useful for many reasons.
17:08Uh let's just one small example.
17:10It helps with best practices and consistency.
17:13Instead of doing stuff from scratch, where, you know, every single implementation is bespoke, I can make it replicate the approach used in the session by an engineer I respect.
17:23Now, our code cross repos is consistent.
17:26That's a big deal.
17:28There is a lot more to it, of course.
17:29If you are in a repo, I can ask, you know, for sessions, it will prioritize sessions that's relevant to the repo, and vice versa.
17:36If I'm asking for repos, it will look at my session and see what similar sessions tend to bring in, right?
17:41There's a lot of interesting intelligence that make it a lot more useful that appear at first glance, okay?
17:49Lastly, uh everything so far I I used uh everything I shown uh used the Polygraph CLI, the current meta harness CLI, to start it, and then you can start Cloat or Cortex or whatever from within it.
18:02But, you don't have to use it this way.
18:03So, in this case, I'm already in a Cloat session, but it works with anything.
18:07And I could just say, "Hey, you know, I actually think a separate repo would be useful.
18:11Like, maybe I'm working on a Vitest plugin in this NX repo, and I could say, 'Can you add the Vitest uh repository to this session so I know what's going on?'"
18:21In this case, we'll engage Polygraph, and it'll set it up, you know, configure everything, and we'll bring the Vitest library, which is a the Vitest repo, the open-source repo, to my session.
18:31So, now uh my agent can, you know, explore it.
18:36It could, you know, uh figure out how it works, and maybe resolve an issue I have in my repo.
18:41I much prefer this to say context seven, because if I have the real code, the agent can go really deep.
18:47So, the deep problems are discovered all this way.
18:52All right.
18:53So, agents are constrained in space and time.
18:56They only see a small fraction of the code base, as they don't know the past, okay?
19:02Uh and both limits could be lifted.
19:05Polygraph uh gives agents access to the entire code your organization can reach, the one you own and open source.
19:13So, it's no longer constrained in space.
19:15Any agent can bring all of it.
19:18Right?
19:19And it gives you agent a perfect memory of what happened.
19:23Every session, every decision made is within reach.
19:28Because it crosses developer boundaries, not per developer, the agent can have more context than any single developer.
19:34Like a thousand engineers have an organization, create all these sessions, they all accessible to to each of them.
19:41Almost like sort of the Borg.
19:43Every agent can run by every developer contributes to kind of one big this hive mind, right?
19:49So, uh if it's interesting, my name is Victor.
19:51You can follow me on Twitter.
19:53If you want to check it out, go to try.poligraph.com and see if it works for you.
19:58Thank you.