We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI

AI Engineer · 20 min · 251 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Uh what's up guys?
00:13Thank you guys for coming to this talk.
00:15I know we're getting to the end of the day.
00:17You had lunch, you're getting tired.
00:19So I really appreciate the patience.
00:20You've been hearing a lot from all of us at Our Eyes.
00:22You just heard Sophia's talk maybe if you were in the room earlier.
00:25So thank you again.
00:27My name's Fuad.
00:27I did a talk yesterday if you weren't here.
00:29I work on product at Our Eyes.
00:31So I'm what they call an AI PM which I'll get into a little bit about you kind of how that relates to shipping agents.
00:37Um today I want to talk about uh two years of shipping our in-product production agent and some of the lessons we've learned.
00:46Hopefully there are good takeaways for you guys as you are kind of building your own agents in the wild.
00:53Cool.
00:53So before I start, we're going to get a little bit meta here uh and hopefully you guys are all kind of following along.
00:59But you know this slide was made with an agent.
01:01Yes, it was.
01:02Um we have coding agents um that kind of operate with Alex, with Signal, with other parts of our product that are themselves agents.
01:11Uh we have Signal which is kind of the trace reading agent that reads our agent's traces, right?
01:17So we have an agent reading the agent traces.
01:19We have Alex which is the in-product agent.
01:21You can think of it like kind of a cursor for AI engineering.
01:24That's the agent I'm talking about today.
01:26Uh we actually use Alex to debug Alex.
01:30Um so it gets a little bit meta.
01:32Um and then hopefully you get you know kind of an idea of how Alex could help you evaluate your agent and some of the design patterns to be taken from Alex and how that could be translated
01:41to your agent.
01:42So anyways, turtles all the way down, agents all the way down.
01:45Uh and if you are confused about which agent we're talking about, I will make sure to make that clear.
01:51So um what we're going to cover is kind of like how Alex got here.
01:55I've got some nice throwback screenshots from 2023 which feels like ages ago, about uh of how Alex used to look.
02:03Uh where it used to be just a router over skills tool calling loop.
02:08Um and kind of how we've changed that.
02:10Uh some of the hardest decisions we made along the way.
02:13So, specifically who owns the plan, how to end a turn, how we dealt with memory across the agent, should we use one agent, should we use many sub agents?
02:24Uh what default model to choose and sort of how we test it as well.
02:28Um then, like I mentioned, it gets a bit meta, but we debug Alex with Alex.
02:33So, and now we have signals as well.
02:35So, how signals is finding Alex's retry loops and some of the real issues we found with our agent using signal.
02:41Uh and then, importantly, what what still needs a person as well.
02:45And so, where the human value kind of adds judgment and the AIPM world kind of comes in.
02:50Uh and some of the decisions we kept that have lasted throughout the architecture of Alex.
02:55Um Alex is the agent inside Arize X. Alex can read your traces, turn findings into tests, so building your evals for you, data sets, experiments.
03:06It can plan work.
03:07It can run multi-step tasks.
03:10It can create synthetic data for you based on your traces.
03:13Um and it can do a lot of different things.
03:16I would say, yeah, this is a quote from one of our staff engineers, on the first prototype back in late 2023 with like 3.5 error models, it was about 50% hallucination,
03:2650% And you saw that say we're getting a bit better as models have gotten better as well.
03:32Uh this is just a little before and after of the visuals just to give you an idea.
03:37So, you can give it, you know, prompts that you stored in Arize, you can ask it to optimize those prompts.
03:43And obviously, it had, you know, kind of tools to read prompts from our database and things like that.
03:49Now, we have, you know, much more in-depth flow with planning, with much more beautiful UI.
03:54Alex kind of explains what it's doing.
03:57Uh it'll reason through tasks much more strongly.
04:00So, here I have like a request to optimize Alex's own system prompt and actually makes a plan to check, you know, kind of resources around the application as well.
04:09So, yeah, this window shows up in a rise.
04:11You can use it to debug your traces.
04:13Um How do we get here?
04:16So, quick kind of like timeline.
04:18We started the prototype in late 2023.
04:22This is like with 3.5 error models.
04:24Um We then, you know, very quickly decided to move a lot of math out of the model and into the harness code because if you remember asking agents to do math back then, I mean nowadays they're solving
04:37Navier-Stokes level problems, but back then they were not very good at math.
04:41Um Then we kind of had the official launch.
04:45We called it the copilot back then when things were still called copilots.
04:49A router of a set of skills, a decision tree.
04:52We had some rails.
04:53We rewrote a lot of it in August 2025.
04:57Um Iterative tool calling and a to-do tool come in.
05:00So, the agent can actually kind of plan what it's doing.
05:03Alex 2.0 was our big relaunch in February 2026.
05:07Sonar 4.5 became the default model and we'll talk through some of how we test that model as well.
05:13One Alex was just released in June.
05:15So, instead of specialist agents, we merged it into kind of like one agent that just loads skills on demand.
05:23And finally, memory.
05:25We now have long-term memory, so as the user kind of interacts with Alex throughout the course of their session, they are kind of continuously the agent continuously knows sort of the history of that user and that's written in the background.
05:39[snorts]
05:40Um and obviously, we didn't train these models.
05:43Everything we rebuilt was harness code, which I think um actually gives me a bit of faith in the world of engineering that there's still very interesting problems to solve here and it's not all up to the model and you can't just run a frontier model and expect it to do everything for you.
05:56[snorts]
05:57Uh models got better on their own schedule for sure and we made some decisions around that, but pretty much everything we built was that harness code.
06:05We don't use an agent framework.
06:07Um The loop, the plan, the stopping rule, and the memory are all ours.
06:12What that means is we have a lot of control over it and you know we get to learn a lot along the way as well.
06:19[snorts]
06:19So first one planning and to-do list.
06:23The analogy I have a bunch of stuff on the slides but the analogy I kind of think about around you know planning is like Let's say you I don't I don't have kids maybe some of you guys have kids and you can kind of relate to this analogy but let's say you want you know your kid to
06:37do some chores.
06:38You can give it a list of chores and ask it to or ask them to check off the list of chores as they go.
06:46Um You can check in with them continuously and hold the list yourself and not tell them about the list except for the next task, right?
06:54So that's one paradigm.
06:56Um And there are trade-offs to kind of like each scenario.
07:01We initially just gave the model to do list and what we saw was it spent most of its time checking off the to do list.
07:09So a lot of LM calls for um pretty much nothing like no useful unit of work other than updating a to do list.
07:17It also we also realized it tended to lose focus after multiple tool calls and yeah 20 20 plus iterations on the to do tools themselves.
07:27So we added this as an extra tool in its arsenal.
07:29And we eventually moved the plan into the harness one piece at a time.
07:33So what I mean by that is first thing we did was we added an in progress so the the model wouldn't lose track of where it was.
07:41Um Then we decided to put the plan as a message so the harness renders the plan to the model call every time the model gets a new call and the rules are next to the data so it's a little bit easier for the model to interpret where it's at.
07:56We actually ended up dropping the in progress identifier because we realized we're just making two calls for effectively you know anything that isn't finished is in in a stack ranked order is the next thing to do, so we remove that.
08:09Um and we're actually testing now without a to-do tool at all, uh just keeping everything in line in the messages uh completely.
08:16So, the harness is keeping the state and the completion check, and the model just consumes that and goes and does the next thing.
08:22Um so, maybe some lessons for you guys as you're building to-dos, uh models can be very eager with uh lists.
08:32Um then, kind of like what ends the turn, uh finishing became a gate the harness enforces.
08:39Um so, what we at first tried was, you know, kind of text without tool calls, uh meant and done.
08:46That broke on models that announce a step in one turn and make the call in the next, which
08:50[snorts]
08:51um you know, as you change models, you'll you'll quickly find some of these breaking things.
08:55And one of the things we do in Aux is we actually allow users to choose their own provider.
09:00Uh so, we can't necessarily guarantee that we've tested every single model someone will try with Aux.
09:06So, a lot of um you know, some of the decisions we made here are to guard against those.
09:10And yes, we run regression tests and we check against a bunch of different models and we benchmark different models on Aux.
09:16We have an opinion on a default, but you know, obviously some of our customers are exclusive Anthropic customers or exclusive OpenAI customers.
09:24So, to actually empower them to use their own models, we kind of had to design against everything that was available, right?
09:31Um so, now we have an explicit finish call that bounces back uh when plan items are open up to three times, uh a warning set one step before the iteration limit, and then a force finish as well.
09:42Um memory, this is a good one.
09:45Um there are actually a number of problems here, and I'm I I like analogies, so I'm going to use another analogy here.
09:50It's like, you know, you you're working at your desk, um and a big book lands on your desk.
09:57And maybe one big book is fine.
09:58You can open the book, kind of search through the book.
10:01Uh if the book gets too long, then it's kind of useless to now cuz you're going to spend your entire time reading that book.
10:07That's one issue.
10:09Uh and that's sort of like, you know, one big tool result comes back and it's too big for you to read, right?
10:14Um but you can imagine there's actually multiple books on your desk, right?
10:18So, eventually not just are the books too long for you to read on your desk, but there's no space left on your desk, right?
10:24You just have too many books, right?
10:27So, there's a yeah, you know, in this analogy, um there's a lot of ways you can approach this, right?
10:32You can decide to store that book and just have a pointer to the book, right?
10:36So, that's kind of what we did with the tool results, where Okay, I now have a sticky note on my desk that says where the book is and what, you know, kind of high level the book contains.
10:47And so, if I need to go read that book, I'll just reference my sticky note and then go find the book on the shelf.
10:53My desk clears up a little bit, right?
10:54So, that's kind of what we do with tool calls uh and tool results.
10:58For example, like Alex's uh you know, can help you with trace analysis.
11:02Some of these traces are other agents' traces, and they can have a ton of stuff in the trace, right?
11:08So, um if it's a big JSON blob, for example, that just blows up our context window and then you can pretty much do no useful work.
11:14So, instead, what we do is we truncate within uh the JSON and we just let the agent code to go find it if it needs the full context with JQ.
11:22Um one thing that agents are very, very good at is writing code and using code-like tools to uh kind of infer things.
11:30And so, that's kind of what we try to leverage with that.
11:33Um the agent's own trajectory also going past uh a certain uh window.
11:39Um so, what we now have is the harness makes the model write a checkpoint uh and then drops what it replaced.
11:47Uh we have fixed token budget for history um and across sessions, so I kind of got to like the long-term memory piece.
11:54Uh basically, we just ask the a background job to edit one small file per user.
12:00I'll dive a little bit more into like why we decided to do this with with long-term memory uh later in the presentation, but essentially, that's how it works.
12:08Um kind of talk through this.
12:11So, treat a huge result like a file and just scrap it.
12:14Uh don't dump it all into context.
12:16Don't worry about doing anything fancy.
12:19Uh for long-term memory, a small file actually beat an all graph and we we tried things like vector retrieval, episodic and semantic stores, uh temporal knowledge graphs, etc. etc.
12:30What we eventually landed on was we just have one file per user.
12:34It's a markdown file.
12:35It's capped at 8,000 characters.
12:37Um and you know, I I think there there is a lesson here that, you know, as engineers, um we like to build things and it's really fun to build things and sometimes we get in our own way and we over-engineer.
12:49Um and this file is just loaded in on every full request.
12:53Uh it's written into in a background job after the session finishes.
12:58It's small.
12:58It's exact.
13:00It's easy to load in.
13:01It doesn't blow up context and it's very easy for a human to audit as well.
13:06Uh so, you can actually kind of have a human labeler go look at these memory files and and determine whether the right things are being stored and kind of edit instructions for the background job.
13:15So, um and then you also only have one job at the very end, which uh you can make very safe as well.
13:21Uh so, you're you're making sure you're not writing memory from other users into the file.
13:26Um so, yeah, very good lesson on not over-engineering uh your problems.
13:32Um briefly on specialists, um you know, we had one trace agent that initially just did trace analysis back in 2025.
13:40Then we had 10 specialists, so every page kind of had its own specialist.
13:44If you're on the data set page, there's, you know, special sub agent that can generate synthetic data for you and look look and introspect on all uh all your data sets.
13:53Uh we ended up kind of rolling that back as well and just using one agent where the agent can load skill instructions on demand and tools on demand and we just let the agent decide
14:04what tools to do, right?
14:05So, it is a bit of an evolution also in in model capability, but I think simplification as well.
14:10[snorts]
14:11Um we on model capability, we also kind of stand by that the newest model isn't always the best by default.
14:21Um there are times where we upgrade the model by default, right?
14:24And so, we have a perspective on that.
14:27Um I think we have um you know, what is the best model is a very subjective question.
14:33Uh what is the best model on a generalized frontier is very different from what may be best for you guys in what you're building or what we're building.
14:41Uh we generally just default to fast and good enough and spend a lot of time on the harness instead.
14:46Um we are now starting to explore auto routing, uh so choosing models per task and training our own models, so small post training SLMs, things like that on very narrow surfaces.
14:57Um and some of these surfaces actually don't even need the loop at all.
15:00You can just make a single model call.
15:02So, an example of that is I wonder if I can pull it up.
15:06We have this like, you know, filter with AI.
15:09Um we've come a long way from SQL generation in the in the past and it's actually very, very easy to generate filters with very, very small models now.
15:17So, um that's an example where maybe we don't need the entire loop and we're just over-engineering again.
15:23Um Cool.
15:26So, that's a good takeaway.
15:28Um production sessions becoming our test.
15:30So, this is where I'll talk a little bit more about arise as a platform.
15:34Uh tracing everything is really important, right?
15:36So, we trace Actually, if you look at this, these are the traces from this agent.
15:41Uh so, it's very meta here.
15:43Um but every time you ask Alex a ques- question here, the the traces these are the traces from Alex.
15:50We we still call it Copilot prod.
15:52Um so, those are our live traces coming in from the tool.
15:55Um this makes it very easy to add, you know, regression data sets and and keep track of different user queries that you want to go test against and iterate against.
16:05Um and so, those production sessions become our test cases and become our living fixtures over time.
16:10They grow, they change um as we see new error cases.
16:14Um and we also now have the ability to use session data sets as well.
16:18So, you can store an entire session as a data set and kind of play a specific turn through the agent if you need.
16:24Um finally, we use arise experiments throughout all this as well to make sure that we're you know, kind of AB testing uh harness changes as we go through this.
16:33Uh one takeaway I want you guys to to kind of keep in mind also is that in agents, bugs rarely show up as explicit errors.
16:41Um in the old world software, this was a lot I don't know if it was easier, but it was different.
16:46Um now it's more of like a qualitative okay, the status says okay, but if there are 43 identical tool calls in the same turn, that's probably something we want to fix.
16:58Um software metrics like latency, right?
17:00So, I think we can all agree 227 seconds is way too long to wait for a response, but what actually is too raw too long it really depends on your use case.
17:09Um so, we use signal a lot now to kind of automate this reveal.
17:15We we use less and less of the human reading the trace.
17:18Um our staff engineers are very grateful for that.
17:21Uh but we still do use kind of that human expertise, which I'll get to in a bit.
17:24[snorts]
17:25Um the fix can also be the next box.
17:28So, these regression data sets are are really important and I would encourage you guys to keep them in mind.
17:32Judges need that human baseline, so that's where we kind of impart the human task.
17:36We align our evaluators and we make sure that our judges actually accurately reflect what you know, our engineers think as well.
17:42Uh so, that's very important.
17:44Um I think we we still keep a lot of things with the human, those gold labels, those golden data sets uh for judges come from people.
17:55Um the speed versus quality is a human judgment call.
18:00I don't think there's any single answer across the stack on when you should be a little bit faster, but it's okay to degrade quality.
18:08It is still you know, kind of like a human judgment call even if you have the metrics around it like okay, changing models will increase latency by X and decrease
18:17uh or increase quality response by Y. If we just use Astra everywhere, would that be the right decision?
18:23Still a human judgment call, I think.
18:24We have the metrics to support that, but we still have to make those decisions.
18:28Um And these are kind of the the six things that survived all these rebuilds.
18:36Tracing to our own products that we of course still trace everything that Alex does.
18:40Math and data in code.
18:42So, code computes the aggregations, the model reason and explains.
18:46And then exposing those as tools to them to the model.
18:48So, we're not giving the model plain text math questions.
18:51We're actually asking it to use code to go do that.
18:55Golden data sets in CI, errors with an obvious fix, streaming and persistent, and skills.
19:00Um We've changed what we call skills.
19:03I think, you know, routers, tools, etc. But really it's give the model a capability around an external system.
19:12So, a lot of decisions you'll face.
19:14And here's some of what we chose.
19:16So, if you'll go ahead and take a picture of the slide.
19:18Who owns the plan?
19:19What ends the turn?
19:22If your tool output is too big for your context, what do you do?
19:25Who should write the long-term memory?
19:27Should you have one?
19:28Should you have multiple sub agents?
19:30What model is good enough and fast enough?
19:33And who checks the agent's own fixes?
19:37So, yeah, that's me.
19:39On the bottom right of the screen there my LinkedIn and X. The tagline is everyday we use Alex to build Alex.
19:46We have a bunch of blog posts on sort of like technical deep dives into how we built this.
19:52And yeah, I encourage you guys to all reach reach out.
19:54And big shout out to the Alex team and our engineers for building this.
19:57Thank you.
19:59[applause]