WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=x4e5O9zN0TE NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/x4e5O9zN0TE.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.639 --> 00:00:18.240 Hi everyone, I'm Arena, former engineer at Microsoft and Supercell. s2 00:00:18.240 --> 00:00:24.000 And today I want to talk about auto research in a multi- aent AI village. s3 00:00:24.000 --> 00:00:34.960 I will use a video game like AI Village as a running example here, but the broader question is one I think many AI engineers are starting to run into. s4 00:00:34.960 --> 00:00:42.640 How do we evaluate and improve agents that carry state over a long period of time? s5 00:00:43.680 --> 00:00:49.920 Before I get into the auto research layer, I want to talk a bit about project paradox. s6 00:00:49.920 --> 00:00:54.719 We developed project paradox at supercell's AI innovation lab. s7 00:00:54.719 --> 00:00:59.199 Me and my teammate are natural money. s8 00:00:59.199 --> 00:01:14.159 We built a modular AI framework that allows any developer to plug in intelligent autonomous agents within a video game that can interact, compete or cooperate with other players or agents as well s9 00:01:14.159 --> 00:01:19.600 and place them uh and make them into dynamic game companions. s10 00:01:20.479 --> 00:01:26.880 Now to give examples of what these agents can do, the agents can move with intent. s11 00:01:26.880 --> 00:01:35.680 They can go to any location or person and they're guided by their own memories, emotion, or curiosity. s12 00:01:35.680 --> 00:01:38.159 These agents can interact with the world. s13 00:01:38.159 --> 00:01:49.280 They can pick up objects, drop them anywhere, and they're also aware about the context in their own environment, such as the objects or other characters or agents as well. s14 00:01:49.280 --> 00:01:58.000 I would also like to note that game developers can also add new actions for these agents to accomplish within our framework as well. s15 00:01:58.000 --> 00:02:08.720 Instead of just dropping or uh placing objects, agents can also obviously react to what's happening around them. s16 00:02:08.720 --> 00:02:16.879 And these events that happen around them affect their own beliefs and emotions on the fly as well. s17 00:02:16.879 --> 00:02:21.680 And of course, it wouldn't be complete if agents can't start conversations, right? s18 00:02:21.680 --> 00:02:31.599 agents can in this scenario approach other agents or even the player as well and this makes the game feel more alive. s19 00:02:31.599 --> 00:02:43.599 And of course these conversations are stored within their memory and is according to their own um and affect their own emotions and beliefs or goals as well. s20 00:02:43.599 --> 00:02:49.360 And al together these agents make our multi- aentic framework. s21 00:02:51.440 --> 00:03:02.480 Um yeah yeah one second so the architecture was intentionally stateful behind this. s22 00:03:02.480 --> 00:03:07.040 The first important part was per agent memory. s23 00:03:07.040 --> 00:03:12.239 Each agent has its own memory namespace backed by rag. s24 00:03:12.239 --> 00:03:16.239 So memory did not bleed between agents. s25 00:03:16.239 --> 00:03:19.920 Second, we tracked emotion as a small vector. s26 00:03:19.920 --> 00:03:29.599 So after an event or conversation, the system could update values like joy, sadness, fear, anger, or disgust. s27 00:03:29.920 --> 00:03:36.080 Third, agents had belief scores towards other agents and the player. s28 00:03:36.080 --> 00:03:38.319 You can think of this as a trust matrix. s29 00:03:38.319 --> 00:03:48.959 Basically like after the interaction happens, the LM basically decides whether the trust score should go up, down, or whether it shouldn't change at all. s30 00:03:48.959 --> 00:03:53.840 And fourth, every memory receives an important score. s31 00:03:53.840 --> 00:04:01.599 Um to to explain this better, like let's say you had dinner a few days ago, you probably wouldn't remember what you had for dinner, right? s32 00:04:01.599 --> 00:04:08.080 But um if someone was murdered a few days ago, you definitely remember that. s33 00:04:08.080 --> 00:04:21.919 So the agent will evaluate or the LM will evaluate uh an important score of an event and if it crosses a threshold, it will store that specific memory uh in a separate cache s34 00:04:21.919 --> 00:04:28.160 so that important context can be retrieved better later on. s35 00:04:28.160 --> 00:04:32.080 And here's an example of it just working. s36 00:04:32.080 --> 00:04:36.880 Um, we going to ask one of the characters to go on a picnic with us. s37 00:04:36.880 --> 00:04:46.880 Here, uh, our character Blossom um, decides to pick up a pastry and go to the picnic area because we asked her to do so. s38 00:04:46.880 --> 00:04:54.160 Keep in mind during the conversation in the background, she plans all of these sequence of actions to accomplish. s39 00:04:54.160 --> 00:05:02.400 And one when we talk to her afterwards, she will also reply within context as well. s40 00:05:03.919 --> 00:05:05.919 Yeah. s41 00:05:06.080 --> 00:05:11.680 But this is where an interesting problem actually started. s42 00:05:11.680 --> 00:05:19.600 As you saw in the last example, like for shortterm game play, this our architecture worked pretty well. s43 00:05:19.600 --> 00:05:28.960 like a character could make a plan, move around, talk and remember the recent interaction and respond to us or other characters as well. s44 00:05:28.960 --> 00:05:36.160 But over longer horizons, this is where we notice the social consistency start to get weaker. s45 00:05:36.160 --> 00:05:49.759 So in this example, we have one agent spreading a rumor about a sale on mangoes to another agent and that agent receives that information and goes and tells another agent about it. s46 00:05:49.759 --> 00:06:06.240 Later on, after a number of events that occurred in between, when the player asks one of the agents about the mangoes, it doesn't exactly store that context that we were expecting or it doesn't give us the context that we kind of wanted to. s47 00:06:06.240 --> 00:06:11.199 And this is where things are starting to get messy naturally. s48 00:06:11.199 --> 00:06:16.880 Like the system may remember the rough topic but lose the source of the topic. s49 00:06:16.880 --> 00:06:32.560 A rumor may become certain instead of just a rumor like the agent might state it as a fact or um an agent might know a fact but fail to execute fail to remember it while creating a plan for its actions. s50 00:06:32.560 --> 00:06:43.759 So the question here became how do we improve a multi-agentic system over longunning social behavior and not just over one response. s51 00:06:44.000 --> 00:06:48.319 And this is where we wanted to bring in auto research. s52 00:06:48.319 --> 00:06:58.560 As you all know, uh few months ago, Karpathi posted out auto research and this this made us immediately very curious. s53 00:06:58.560 --> 00:07:07.520 Uh perhaps we can make the system run experiments uh on itself and can we use this for our system as well. s54 00:07:07.520 --> 00:07:21.840 So what we understood is instead of manually tuning a prompt or watching one nice demo, we could define sen a scenario suit, run the agents, collect traces, score the behavior s55 00:07:21.840 --> 00:07:28.319 and change a small policy surface and only keep the changes that actually improve the score. s56 00:07:28.319 --> 00:07:33.759 And this is where we're trying to bridge project paradox with auto research. s57 00:07:33.759 --> 00:07:45.199 So at this point basically our multi-agentic framework project paradox is more like a lab bench and auto research becomes the experimental loop around it. s58 00:07:45.199 --> 00:07:50.319 And importantly this is not only about improving rag retrieval. s59 00:07:50.319 --> 00:08:05.039 The broader framing is optimizing the agent protocol like how do agents write memories, retrieve them, communicate uncertainty, update trust attribute sources and replan around new facts. s60 00:08:05.039 --> 00:08:24.800 basically um yeah in this context uh oh yeah in this context art research is not another agent in the village like I said it's a meta system outside the village s61 00:08:24.800 --> 00:08:37.440 the villagers have local perspectives of course they only know what they saw heard remembered or inferred because there isn't a common memory database in between them. s62 00:08:37.440 --> 00:08:43.839 Information only travels once uh other agents communicate them properly. s63 00:08:43.839 --> 00:08:46.800 The auto research layer has a different job here. s64 00:08:46.800 --> 00:09:03.600 It reads the full traces of a run, compares what happens against the scenario ground truth, uh scores the behavior and proposes a constrained change to the agent protocol or cognitive policy. s65 00:09:03.600 --> 00:09:10.560 Then it reruns the scenario and asks society level behavior like did society level behavior get better. s66 00:09:10.560 --> 00:09:14.560 This is the key shift we were trying to look for. s67 00:09:14.560 --> 00:09:17.440 So we were no longer evaluating one answer. s68 00:09:17.440 --> 00:09:21.200 We were evaluating an entire run. s69 00:09:21.839 --> 00:09:25.760 And this is what one of the loops would look like. s70 00:09:25.760 --> 00:09:31.519 Like first we define a control scenario which I'll elaborate a bit more about later. s71 00:09:31.519 --> 00:09:37.600 For example, one agent learns a public fact or one agent hears a rumor. s72 00:09:37.600 --> 00:09:40.640 Uh that could be a controlled scenario. s73 00:09:40.640 --> 00:09:42.800 Then we run the simulation. s74 00:09:42.800 --> 00:09:55.680 During the run, we collect structured traces, observations, conversations, memory rights, retrievals, belief updates, whatever is relevant to us in that case, we collect. s75 00:09:55.680 --> 00:09:58.240 Then we score this behavior. s76 00:09:58.240 --> 00:10:01.760 Did the information spread as we expected it to? s77 00:10:01.760 --> 00:10:04.000 Did the source attribution survive? s78 00:10:04.000 --> 00:10:08.000 Such as, does the agent remember who started the rumor? s79 00:10:08.000 --> 00:10:10.480 Did uncertainty stay uncertain? s80 00:10:10.480 --> 00:10:14.640 Did agents act on what they actually knew? s81 00:10:14.640 --> 00:10:20.160 And then the auto research layer here proposes a small policy change. s82 00:10:20.160 --> 00:10:22.160 And this is important. s83 00:10:22.160 --> 00:10:26.160 It should not rewrite the whole application of course. s84 00:10:26.160 --> 00:10:31.680 it should only edit a controlled policy surface and then we rerun. s85 00:10:31.680 --> 00:10:41.440 If the score improves and the guardrails hold, we we keep the improvement and if not we simply just revert back. s86 00:10:43.200 --> 00:10:55.200 And talking about controlled scenarios, the reason why uh scenario design matters is that social behavior is otherwise a bit fuzzy. s87 00:10:55.200 --> 00:11:12.000 uh in general in the sense if you just let the agents in our environment wander around it might look cool and you might get nice interactions but it's actually very hard to evaluate on whether the system actually improved. s88 00:11:12.000 --> 00:11:16.720 So this is why we believe you need controlled scenarios. s89 00:11:16.720 --> 00:11:21.120 For example, one scenario could test a public fact diffusion. s90 00:11:21.120 --> 00:11:26.320 Let's say agent A learns uh the bakery will close tomorrow. s91 00:11:26.320 --> 00:11:28.480 Do the right agents learn it? s92 00:11:28.480 --> 00:11:31.040 Do they remember who said what? s93 00:11:31.040 --> 00:11:35.600 Do they rem do they change their plans based on this fact? s94 00:11:35.600 --> 00:11:39.680 Another scenario could test rumor uncertainty. s95 00:11:39.680 --> 00:11:45.519 Agent let's say agent A hears that agent C might leave the village. s96 00:11:45.519 --> 00:11:54.079 When this rumor spreads, does might leave suddenly become is leaving or does it stay as might leave? s97 00:11:54.079 --> 00:11:59.839 Like does it become a fact or does it still stay as a a rumor? s98 00:11:59.839 --> 00:12:03.279 Another scenario could test replanning. s99 00:12:03.279 --> 00:12:10.480 The group has a plan, but one agent learns, let's say, the route they wanted to take is blocked. s100 00:12:10.480 --> 00:12:20.079 Do agents update this and communicate this uh with each other to avoid uh a improper plan or scale actions? s101 00:12:20.079 --> 00:12:24.959 The point is not that these exact scenarios are universal here. s102 00:12:24.959 --> 00:12:32.079 The point we're trying to make is that long horizon agent behavior needs scenario suits. s103 00:12:32.720 --> 00:12:49.680 And talking about our Mango example again, after running one of our auto research loops, this time after uh a a long pro period of time, when the player finally asked one of the agents about the sale on mangoes, s104 00:12:49.680 --> 00:13:02.000 we did find that u the the agent was able to respond within context this time like compared to last time. s105 00:13:02.880 --> 00:13:13.920 Um yeah and for this talk the form the exact formula we believe is less important than the shape of the scorecard. s106 00:13:13.920 --> 00:13:20.160 Uh you do not want a single vague met metric like agent quality. s107 00:13:20.160 --> 00:13:22.959 This will hide all the interesting failures. s108 00:13:22.959 --> 00:13:26.399 Instead you want a balanced scorecard. s109 00:13:26.399 --> 00:13:33.600 For diffusion, you might measure reach like how many agents know the fact after end steps. s110 00:13:33.600 --> 00:13:40.000 For provenence, you measure source retention among agents who know it. s111 00:13:40.000 --> 00:13:48.480 How many remember it, where it came from, etc. For rumors, you can measure uncerny preservation and false surn rate. s112 00:13:48.480 --> 00:13:53.600 For planning, you can measure action consistency and time to replan. s113 00:13:53.600 --> 00:13:57.040 And for privacy, you can measure containment. s114 00:13:57.040 --> 00:14:02.880 This matters because optimizing only one metric can create bad behavior. s115 00:14:02.880 --> 00:14:10.079 Because let's say if you only optimize for diffusion, the agents may learn to overshare everything. s116 00:14:10.079 --> 00:14:17.920 And let's say if you only optimize for memory recall, you might create noisy or steal um like memories. s117 00:14:17.920 --> 00:14:31.199 So this scorecard is what keeps the system honest and prevents the auto research agent from gamifying the system to just increase one specific score. s118 00:14:32.399 --> 00:14:44.800 The other important engineering lesson that we learned over this project is that um it's important to keep the editable surface really small. s119 00:14:44.800 --> 00:14:51.440 The auto research layer should not have permission to randomly rewrite the whole codebase. s120 00:14:51.440 --> 00:14:58.399 Instead, it's really important to freeze the harness, the scenarios, and the metrics. s121 00:14:58.399 --> 00:15:03.600 So, we're only exposing the part of the system that we actually want to optimize. s122 00:15:03.600 --> 00:15:28.480 Here in project paradox for us that meant things like memory writing policy, retrieval policy, communication prompt, belief, trust rules, source attribution, replanning triggers, etc. This gives the search pro process room to improve behavior, but also prevents it from gaming the evaluation directly as we mentioned before. s123 00:15:28.480 --> 00:15:39.760 And this is the difference between the LM writing random patches versus the LM actually searching within a controlled policy space. s124 00:15:40.880 --> 00:15:48.320 And here are here are examples of the kind of changes I want this kind of loop to search over. s125 00:15:48.320 --> 00:15:59.040 If if source attribution disappears, the policy change might be preserve source in memory and uh write uh memory rights and summaries. s126 00:15:59.040 --> 00:16:10.399 If rumors harden into facts, the policy change might be stored confidence mark firstand versus secondhand and require hedging when retelling uncertain claims. s127 00:16:10.399 --> 00:16:22.399 If if facts if public facts stay local, the policy change might be classify useful public facts differently and make agents proactively share important source evidence. s128 00:16:22.399 --> 00:16:34.560 The key is that these are small changes to the agent protocol, but they can have larger effects on a society level behavior for multi-agentic systems. s129 00:16:34.560 --> 00:16:46.160 This is also where I kind of want to be careful about our claims here because with we believe without repeated current loop results like I wouldn't say the system s130 00:16:46.160 --> 00:16:49.199 just generally improved. s131 00:16:49.199 --> 00:17:04.800 We're trying to say this is the right kind of surface to expose to an auto research layer uh loop because it is small enough to control but it's still rich enough to change the social behavior to some extent at least. s132 00:17:05.199 --> 00:17:12.480 And the biggest lesson for me perhaps was that memory is not enough here. s133 00:17:12.480 --> 00:17:22.160 You can add a rag memory to an agent and still not get the current long-term uh horizon behavior that you were looking for. s134 00:17:22.160 --> 00:17:28.799 Um because agents need to sometimes know where that information came for uh came from. s135 00:17:28.799 --> 00:17:34.000 You need to preserve whether it was firsthand, secondhand, verified or uncertain. s136 00:17:34.000 --> 00:17:39.200 Sometimes you need to separate raw episodic memories from what the agent currently believes too. s137 00:17:39.200 --> 00:17:44.640 And you need to test behavior through scenarios not not just through vibes. s138 00:17:44.640 --> 00:17:50.320 So the other lesson is that uh roll back also is not optional. s139 00:17:50.320 --> 00:17:56.400 When you optimize social behavior, a change can improve one thing and damage another. s140 00:17:56.400 --> 00:18:02.640 So, a policy that spreads public facts uh faster might also leak private information. s141 00:18:02.640 --> 00:18:07.360 A policy that increases recall might increase stale memory usage. s142 00:18:07.360 --> 00:18:10.799 So, the loop should basically be like a ratchet. s143 00:18:10.799 --> 00:18:18.559 Try a change, score it, keep it only if the scorecard improves and guard rails whole. s144 00:18:19.840 --> 00:18:40.720 And we we definitely believe this is not only relevant for game agents cuz although I gave you an example using a game village um we believe like let's say for example support agents support agents need to know which policy update comes from where right and whether it supersedes an older answer. s145 00:18:40.720 --> 00:18:53.919 Personal assistants for example need to remember commitments that they previously made and h make corrections if uh if the user u wants to change those personal commitments. s146 00:18:53.919 --> 00:19:00.960 Research agents need pro uh provenence citations contradiction handling and hypothesis updates. s147 00:19:00.960 --> 00:19:07.919 Coding agents need longunning context across issues, files, teammates and changing requirements. s148 00:19:07.919 --> 00:19:13.280 Workflow agents need access controls, handoffs, and replplanning when the world changes. s149 00:19:13.280 --> 00:19:17.440 All of these systems have the same underlying problem. s150 00:19:17.440 --> 00:19:20.080 They maintain state over time. s151 00:19:20.080 --> 00:19:24.480 And that state affect affects future action. s152 00:19:24.480 --> 00:19:31.280 So they need control scenarios and behavioral scorecards is what we are proposing. s153 00:19:31.520 --> 00:19:36.559 So again in brief, a recipe for long horizon agents. s154 00:19:36.559 --> 00:19:51.440 If there's one practical recipe I want you to take away, freeze the harness, define scenarios, log traces, score behavior, and expose only a small policy surface. s155 00:19:51.440 --> 00:19:56.960 Search over these changes, keep only changes that survive your measurement. s156 00:19:56.960 --> 00:20:03.840 And this is an engineering pattern that we believe would uh make sense for longunning agents. s157 00:20:03.840 --> 00:20:11.840 The real question we believe is across controlled runs, does the system behave better? s158 00:20:13.039 --> 00:20:19.679 To close, project paradox start as an attempt to make game agents feel alive in a 3D world. s159 00:20:19.679 --> 00:20:24.240 But the deeper engineing problem was not animation or dialogue for us. s160 00:20:24.240 --> 00:20:36.559 It was the state such as which agent knows what, which agent told whom, what is true, uncertain or outdated, and do agents act on what they remember. s161 00:20:36.559 --> 00:20:37.679 Otter research. s162 00:20:37.679 --> 00:20:42.640 Otter research gave us a way to approach this a bit more systematically. s163 00:20:42.640 --> 00:20:53.760 Not by trusting one demo and not by endlessly handtuning prompts, but by running control experiments and keeping only the changes that survived our measurement. s164 00:20:53.760 --> 00:20:57.280 Long horizon agents need experiments and not just prompts. s165 00:20:57.280 --> 00:21:01.360 And I hope that's the takeaway that you get from this talk. s166 00:21:01.360 --> 00:21:03.840 And yes, please do connect with us. s167 00:21:03.840 --> 00:21:06.559 We'd love to talk if you have any questions. s168 00:21:06.559 --> 00:21:08.960 Thank you so much for listening. s169 00:21:08.960 --> 00:21:10.960 Yeah.