WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=YIhvrvHtonk

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/YIhvrvHtonk.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:13.440
Uh what's up guys?

s3
00:00:13.440 --> 00:00:15.400
Thank you guys for coming to this talk.

s4
00:00:15.400 --> 00:00:17.520
I know we're getting to the end of the day.

s5
00:00:17.520 --> 00:00:19.280
You had lunch, you're getting tired.

s6
00:00:19.280 --> 00:00:20.600
So I really appreciate the patience.

s7
00:00:20.600 --> 00:00:22.920
You've been hearing a lot from all of us at Our Eyes.

s8
00:00:22.920 --> 00:00:25.280
You just heard Sophia's talk maybe if you were in the room earlier.

s9
00:00:25.280 --> 00:00:27.080
So thank you again.

s10
00:00:27.080 --> 00:00:27.760
My name's Fuad.

s11
00:00:27.760 --> 00:00:29.920
I did a talk yesterday if you weren't here.

s12
00:00:29.920 --> 00:00:31.600
I work on product at Our Eyes.

s13
00:00:31.600 --> 00:00:37.760
So I'm what they call an AI PM which I'll get into a little bit about you kind of how that relates to shipping agents.

s14
00:00:37.760 --> 00:00:46.320
Um today I want to talk about uh two years of shipping our in-product production agent and some of the lessons we've learned.

s15
00:00:46.320 --> 00:00:53.000
Hopefully there are good takeaways for you guys as you are kind of building your own agents in the wild.

s16
00:00:53.320 --> 00:00:53.560
Cool.

s17
00:00:53.560 --> 00:00:59.000
So before I start, we're going to get a little bit meta here uh and hopefully you guys are all kind of following along.

s18
00:00:59.000 --> 00:01:01.400
But you know this slide was made with an agent.

s19
00:01:01.400 --> 00:01:02.240
Yes, it was.

s20
00:01:02.240 --> 00:01:11.680
Um we have coding agents um that kind of operate with Alex, with Signal, with other parts of our product that are themselves agents.

s21
00:01:11.680 --> 00:01:17.520
Uh we have Signal which is kind of the trace reading agent that reads our agent's traces, right?

s22
00:01:17.520 --> 00:01:19.520
So we have an agent reading the agent traces.

s23
00:01:19.520 --> 00:01:21.880
We have Alex which is the in-product agent.

s24
00:01:21.880 --> 00:01:24.760
You can think of it like kind of a cursor for AI engineering.

s25
00:01:24.760 --> 00:01:26.920
That's the agent I'm talking about today.

s26
00:01:26.920 --> 00:01:30.280
Uh we actually use Alex to debug Alex.

s27
00:01:30.280 --> 00:01:32.200
Um so it gets a little bit meta.

s28
00:01:32.200 --> 00:01:41.560
Um and then hopefully you get you know kind of an idea of how Alex could help you evaluate your agent and some of the design patterns to be taken from Alex and how that could be translated

s29
00:01:41.560 --> 00:01:42.040
to your agent.

s30
00:01:42.040 --> 00:01:45.840
So anyways, turtles all the way down, agents all the way down.

s31
00:01:45.840 --> 00:01:51.560
Uh and if you are confused about which agent we're talking about, I will make sure to make that clear.

s32
00:01:51.560 --> 00:01:55.600
So um what we're going to cover is kind of like how Alex got here.

s33
00:01:55.600 --> 00:02:03.172
I've got some nice throwback screenshots from 2023 which feels like ages ago, about uh of how Alex used to look.

s34
00:02:03.172 --> 00:02:08.252
Uh where it used to be just a router over skills tool calling loop.

s35
00:02:08.252 --> 00:02:10.532
Um and kind of how we've changed that.

s36
00:02:10.532 --> 00:02:13.252
Uh some of the hardest decisions we made along the way.

s37
00:02:13.252 --> 00:02:24.372
So, specifically who owns the plan, how to end a turn, how we dealt with memory across the agent, should we use one agent, should we use many sub agents?

s38
00:02:24.372 --> 00:02:28.612
Uh what default model to choose and sort of how we test it as well.

s39
00:02:28.612 --> 00:02:33.012
Um then, like I mentioned, it gets a bit meta, but we debug Alex with Alex.

s40
00:02:33.012 --> 00:02:35.372
So, and now we have signals as well.

s41
00:02:35.372 --> 00:02:41.452
So, how signals is finding Alex's retry loops and some of the real issues we found with our agent using signal.

s42
00:02:41.452 --> 00:02:45.172
Uh and then, importantly, what what still needs a person as well.

s43
00:02:45.172 --> 00:02:50.292
And so, where the human value kind of adds judgment and the AIPM world kind of comes in.

s44
00:02:50.292 --> 00:02:55.692
Uh and some of the decisions we kept that have lasted throughout the architecture of Alex.

s45
00:02:55.692 --> 00:03:06.212
Um Alex is the agent inside Arize X. Alex can read your traces, turn findings into tests, so building your evals for you, data sets, experiments.

s46
00:03:06.212 --> 00:03:07.612
It can plan work.

s47
00:03:07.612 --> 00:03:10.812
It can run multi-step tasks.

s48
00:03:10.812 --> 00:03:13.812
It can create synthetic data for you based on your traces.

s49
00:03:13.812 --> 00:03:16.732
Um and it can do a lot of different things.

s50
00:03:16.732 --> 00:03:26.812
I would say, yeah, this is a quote from one of our staff engineers, on the first prototype back in late 2023 with like 3.5 error models, it was about 50% hallucination,

s51
00:03:26.812 --> 00:03:32.612
50% And you saw that say we're getting a bit better as models have gotten better as well.

s52
00:03:32.612 --> 00:03:37.292
Uh this is just a little before and after of the visuals just to give you an idea.

s53
00:03:37.292 --> 00:03:43.292
So, you can give it, you know, prompts that you stored in Arize, you can ask it to optimize those prompts.

s54
00:03:43.292 --> 00:03:49.132
And obviously, it had, you know, kind of tools to read prompts from our database and things like that.

s55
00:03:49.132 --> 00:03:54.932
Now, we have, you know, much more in-depth flow with planning, with much more beautiful UI.

s56
00:03:54.932 --> 00:03:57.012
Alex kind of explains what it's doing.

s57
00:03:57.012 --> 00:04:00.292
Uh it'll reason through tasks much more strongly.

s58
00:04:00.292 --> 00:04:09.174
So, here I have like a request to optimize Alex's own system prompt and actually makes a plan to check, you know, kind of resources around the application as well.

s59
00:04:09.174 --> 00:04:11.294
So, yeah, this window shows up in a rise.

s60
00:04:11.294 --> 00:04:13.774
You can use it to debug your traces.

s61
00:04:13.774 --> 00:04:16.294
Um How do we get here?

s62
00:04:16.294 --> 00:04:18.534
So, quick kind of like timeline.

s63
00:04:18.534 --> 00:04:22.014
We started the prototype in late 2023.

s64
00:04:22.014 --> 00:04:24.534
This is like with 3.5 error models.

s65
00:04:24.534 --> 00:04:37.374
Um We then, you know, very quickly decided to move a lot of math out of the model and into the harness code because if you remember asking agents to do math back then, I mean nowadays they're solving

s66
00:04:37.374 --> 00:04:41.254
Navier-Stokes level problems, but back then they were not very good at math.

s67
00:04:41.254 --> 00:04:45.774
Um Then we kind of had the official launch.

s68
00:04:45.774 --> 00:04:49.814
We called it the copilot back then when things were still called copilots.

s69
00:04:49.814 --> 00:04:52.294
A router of a set of skills, a decision tree.

s70
00:04:52.294 --> 00:04:53.854
We had some rails.

s71
00:04:53.854 --> 00:04:57.014
We rewrote a lot of it in August 2025.

s72
00:04:57.014 --> 00:05:00.934
Um Iterative tool calling and a to-do tool come in.

s73
00:05:00.934 --> 00:05:03.894
So, the agent can actually kind of plan what it's doing.

s74
00:05:03.894 --> 00:05:07.934
Alex 2.0 was our big relaunch in February 2026.

s75
00:05:07.934 --> 00:05:13.454
Sonar 4.5 became the default model and we'll talk through some of how we test that model as well.

s76
00:05:13.454 --> 00:05:15.974
One Alex was just released in June.

s77
00:05:15.974 --> 00:05:23.694
So, instead of specialist agents, we merged it into kind of like one agent that just loads skills on demand.

s78
00:05:23.694 --> 00:05:25.734
And finally, memory.

s79
00:05:25.734 --> 00:05:39.293
We now have long-term memory, so as the user kind of interacts with Alex throughout the course of their session, they are kind of continuously the agent continuously knows sort of the history of that user and that's written in the background.

s80
00:05:39.293 --> 00:05:40.174
[snorts]

s81
00:05:40.174 --> 00:05:43.614
Um and obviously, we didn't train these models.

s82
00:05:43.614 --> 00:05:56.979
Everything we rebuilt was harness code, which I think um actually gives me a bit of faith in the world of engineering that there's still very interesting problems to solve here and it's not all up to the model and you can't just run a frontier model and expect it to do everything for you.

s83
00:05:56.979 --> 00:05:57.014
[snorts]

s84
00:05:57.014 --> 00:06:05.134
Uh models got better on their own schedule for sure and we made some decisions around that, but pretty much everything we built was that harness code.

s85
00:06:05.134 --> 00:06:07.734
We don't use an agent framework.

s86
00:06:07.734 --> 00:06:12.335
Um The loop, the plan, the stopping rule, and the memory are all ours.

s87
00:06:12.335 --> 00:06:19.175
What that means is we have a lot of control over it and you know we get to learn a lot along the way as well.

s88
00:06:19.249 --> 00:06:19.375
[snorts]

s89
00:06:19.375 --> 00:06:23.775
So first one planning and to-do list.

s90
00:06:23.775 --> 00:06:37.055
The analogy I have a bunch of stuff on the slides but the analogy I kind of think about around you know planning is like Let's say you I don't I don't have kids maybe some of you guys have kids and you can kind of relate to this analogy but let's say you want you know your kid to

s91
00:06:37.055 --> 00:06:38.495
do some chores.

s92
00:06:38.495 --> 00:06:46.575
You can give it a list of chores and ask it to or ask them to check off the list of chores as they go.

s93
00:06:46.575 --> 00:06:54.775
Um You can check in with them continuously and hold the list yourself and not tell them about the list except for the next task, right?

s94
00:06:54.775 --> 00:06:56.375
So that's one paradigm.

s95
00:06:56.375 --> 00:07:01.975
Um And there are trade-offs to kind of like each scenario.

s96
00:07:01.975 --> 00:07:09.415
We initially just gave the model to do list and what we saw was it spent most of its time checking off the to do list.

s97
00:07:09.415 --> 00:07:17.575
So a lot of LM calls for um pretty much nothing like no useful unit of work other than updating a to do list.

s98
00:07:17.575 --> 00:07:27.175
It also we also realized it tended to lose focus after multiple tool calls and yeah 20 20 plus iterations on the to do tools themselves.

s99
00:07:27.175 --> 00:07:29.815
So we added this as an extra tool in its arsenal.

s100
00:07:29.815 --> 00:07:33.935
And we eventually moved the plan into the harness one piece at a time.

s101
00:07:33.935 --> 00:07:41.775
So what I mean by that is first thing we did was we added an in progress so the the model wouldn't lose track of where it was.

s102
00:07:41.775 --> 00:07:56.655
Um Then we decided to put the plan as a message so the harness renders the plan to the model call every time the model gets a new call and the rules are next to the data so it's a little bit easier for the model to interpret where it's at.

s103
00:07:56.655 --> 00:08:09.457
We actually ended up dropping the in progress identifier because we realized we're just making two calls for effectively you know anything that isn't finished is in in a stack ranked order is the next thing to do, so we remove that.

s104
00:08:09.457 --> 00:08:16.777
Um and we're actually testing now without a to-do tool at all, uh just keeping everything in line in the messages uh completely.

s105
00:08:16.777 --> 00:08:22.377
So, the harness is keeping the state and the completion check, and the model just consumes that and goes and does the next thing.

s106
00:08:22.377 --> 00:08:32.297
Um so, maybe some lessons for you guys as you're building to-dos, uh models can be very eager with uh lists.

s107
00:08:32.297 --> 00:08:39.937
Um then, kind of like what ends the turn, uh finishing became a gate the harness enforces.

s108
00:08:39.937 --> 00:08:46.257
Um so, what we at first tried was, you know, kind of text without tool calls, uh meant and done.

s109
00:08:46.257 --> 00:08:50.557
That broke on models that announce a step in one turn and make the call in the next, which

s110
00:08:50.557 --> 00:08:51.217
[snorts]

s111
00:08:51.217 --> 00:08:55.777
um you know, as you change models, you'll you'll quickly find some of these breaking things.

s112
00:08:55.777 --> 00:09:00.537
And one of the things we do in Aux is we actually allow users to choose their own provider.

s113
00:09:00.537 --> 00:09:06.017
Uh so, we can't necessarily guarantee that we've tested every single model someone will try with Aux.

s114
00:09:06.017 --> 00:09:10.817
So, a lot of um you know, some of the decisions we made here are to guard against those.

s115
00:09:10.817 --> 00:09:16.857
And yes, we run regression tests and we check against a bunch of different models and we benchmark different models on Aux.

s116
00:09:16.857 --> 00:09:24.537
We have an opinion on a default, but you know, obviously some of our customers are exclusive Anthropic customers or exclusive OpenAI customers.

s117
00:09:24.537 --> 00:09:31.297
So, to actually empower them to use their own models, we kind of had to design against everything that was available, right?

s118
00:09:31.297 --> 00:09:42.937
Um so, now we have an explicit finish call that bounces back uh when plan items are open up to three times, uh a warning set one step before the iteration limit, and then a force finish as well.

s119
00:09:42.937 --> 00:09:45.417
Um memory, this is a good one.

s120
00:09:45.417 --> 00:09:50.137
Um there are actually a number of problems here, and I'm I I like analogies, so I'm going to use another analogy here.

s121
00:09:50.137 --> 00:09:57.417
It's like, you know, you you're working at your desk, um and a big book lands on your desk.

s122
00:09:57.417 --> 00:09:58.937
And maybe one big book is fine.

s123
00:09:58.937 --> 00:10:01.817
You can open the book, kind of search through the book.

s124
00:10:01.817 --> 00:10:07.817
Uh if the book gets too long, then it's kind of useless to now cuz you're going to spend your entire time reading that book.

s125
00:10:07.817 --> 00:10:09.337
That's one issue.

s126
00:10:09.337 --> 00:10:14.618
Uh and that's sort of like, you know, one big tool result comes back and it's too big for you to read, right?

s127
00:10:14.618 --> 00:10:18.098
Um but you can imagine there's actually multiple books on your desk, right?

s128
00:10:18.098 --> 00:10:24.818
So, eventually not just are the books too long for you to read on your desk, but there's no space left on your desk, right?

s129
00:10:24.818 --> 00:10:27.058
You just have too many books, right?

s130
00:10:27.058 --> 00:10:32.298
So, there's a yeah, you know, in this analogy, um there's a lot of ways you can approach this, right?

s131
00:10:32.298 --> 00:10:36.898
You can decide to store that book and just have a pointer to the book, right?

s132
00:10:36.898 --> 00:10:47.018
So, that's kind of what we did with the tool results, where Okay, I now have a sticky note on my desk that says where the book is and what, you know, kind of high level the book contains.

s133
00:10:47.018 --> 00:10:53.098
And so, if I need to go read that book, I'll just reference my sticky note and then go find the book on the shelf.

s134
00:10:53.098 --> 00:10:54.458
My desk clears up a little bit, right?

s135
00:10:54.458 --> 00:10:58.418
So, that's kind of what we do with tool calls uh and tool results.

s136
00:10:58.418 --> 00:11:02.538
For example, like Alex's uh you know, can help you with trace analysis.

s137
00:11:02.538 --> 00:11:08.178
Some of these traces are other agents' traces, and they can have a ton of stuff in the trace, right?

s138
00:11:08.178 --> 00:11:14.698
So, um if it's a big JSON blob, for example, that just blows up our context window and then you can pretty much do no useful work.

s139
00:11:14.698 --> 00:11:22.978
So, instead, what we do is we truncate within uh the JSON and we just let the agent code to go find it if it needs the full context with JQ.

s140
00:11:22.978 --> 00:11:30.498
Um one thing that agents are very, very good at is writing code and using code-like tools to uh kind of infer things.

s141
00:11:30.498 --> 00:11:33.858
And so, that's kind of what we try to leverage with that.

s142
00:11:33.858 --> 00:11:39.738
Um the agent's own trajectory also going past uh a certain uh window.

s143
00:11:39.738 --> 00:11:47.058
Um so, what we now have is the harness makes the model write a checkpoint uh and then drops what it replaced.

s144
00:11:47.058 --> 00:11:54.578
Uh we have fixed token budget for history um and across sessions, so I kind of got to like the long-term memory piece.

s145
00:11:54.578 --> 00:12:00.018
Uh basically, we just ask the a background job to edit one small file per user.

s146
00:12:00.018 --> 00:12:08.138
I'll dive a little bit more into like why we decided to do this with with long-term memory uh later in the presentation, but essentially, that's how it works.

s147
00:12:08.138 --> 00:12:11.258
Um kind of talk through this.

s148
00:12:11.258 --> 00:12:14.579
So, treat a huge result like a file and just scrap it.

s149
00:12:14.579 --> 00:12:16.099
Uh don't dump it all into context.

s150
00:12:16.099 --> 00:12:19.019
Don't worry about doing anything fancy.

s151
00:12:19.019 --> 00:12:30.259
Uh for long-term memory, a small file actually beat an all graph and we we tried things like vector retrieval, episodic and semantic stores, uh temporal knowledge graphs, etc. etc.

s152
00:12:30.259 --> 00:12:34.219
What we eventually landed on was we just have one file per user.

s153
00:12:34.219 --> 00:12:35.499
It's a markdown file.

s154
00:12:35.499 --> 00:12:37.739
It's capped at 8,000 characters.

s155
00:12:37.739 --> 00:12:49.659
Um and you know, I I think there there is a lesson here that, you know, as engineers, um we like to build things and it's really fun to build things and sometimes we get in our own way and we over-engineer.

s156
00:12:49.659 --> 00:12:53.619
Um and this file is just loaded in on every full request.

s157
00:12:53.619 --> 00:12:58.179
Uh it's written into in a background job after the session finishes.

s158
00:12:58.179 --> 00:12:58.819
It's small.

s159
00:12:58.819 --> 00:13:00.179
It's exact.

s160
00:13:00.179 --> 00:13:01.659
It's easy to load in.

s161
00:13:01.659 --> 00:13:06.139
It doesn't blow up context and it's very easy for a human to audit as well.

s162
00:13:06.139 --> 00:13:15.779
Uh so, you can actually kind of have a human labeler go look at these memory files and and determine whether the right things are being stored and kind of edit instructions for the background job.

s163
00:13:15.779 --> 00:13:21.819
So, um and then you also only have one job at the very end, which uh you can make very safe as well.

s164
00:13:21.819 --> 00:13:26.059
Uh so, you're you're making sure you're not writing memory from other users into the file.

s165
00:13:26.059 --> 00:13:32.379
Um so, yeah, very good lesson on not over-engineering uh your problems.

s166
00:13:32.499 --> 00:13:40.939
Um briefly on specialists, um you know, we had one trace agent that initially just did trace analysis back in 2025.

s167
00:13:40.939 --> 00:13:44.659
Then we had 10 specialists, so every page kind of had its own specialist.

s168
00:13:44.659 --> 00:13:53.899
If you're on the data set page, there's, you know, special sub agent that can generate synthetic data for you and look look and introspect on all uh all your data sets.

s169
00:13:53.899 --> 00:14:04.219
Uh we ended up kind of rolling that back as well and just using one agent where the agent can load skill instructions on demand and tools on demand and we just let the agent decide

s170
00:14:04.219 --> 00:14:05.179
what tools to do, right?

s171
00:14:05.179 --> 00:14:10.854
So, it is a bit of an evolution also in in model capability, but I think simplification as well.

s172
00:14:10.854 --> 00:14:11.739
[snorts]

s173
00:14:11.739 --> 00:14:21.261
Um we on model capability, we also kind of stand by that the newest model isn't always the best by default.

s174
00:14:21.261 --> 00:14:24.741
Um there are times where we upgrade the model by default, right?

s175
00:14:24.741 --> 00:14:27.061
And so, we have a perspective on that.

s176
00:14:27.061 --> 00:14:33.421
Um I think we have um you know, what is the best model is a very subjective question.

s177
00:14:33.421 --> 00:14:41.501
Uh what is the best model on a generalized frontier is very different from what may be best for you guys in what you're building or what we're building.

s178
00:14:41.501 --> 00:14:46.301
Uh we generally just default to fast and good enough and spend a lot of time on the harness instead.

s179
00:14:46.301 --> 00:14:57.781
Um we are now starting to explore auto routing, uh so choosing models per task and training our own models, so small post training SLMs, things like that on very narrow surfaces.

s180
00:14:57.781 --> 00:15:00.661
Um and some of these surfaces actually don't even need the loop at all.

s181
00:15:00.661 --> 00:15:02.341
You can just make a single model call.

s182
00:15:02.341 --> 00:15:06.901
So, an example of that is I wonder if I can pull it up.

s183
00:15:06.901 --> 00:15:09.261
We have this like, you know, filter with AI.

s184
00:15:09.261 --> 00:15:17.501
Um we've come a long way from SQL generation in the in the past and it's actually very, very easy to generate filters with very, very small models now.

s185
00:15:17.501 --> 00:15:23.581
So, um that's an example where maybe we don't need the entire loop and we're just over-engineering again.

s186
00:15:23.581 --> 00:15:26.781
Um Cool.

s187
00:15:26.781 --> 00:15:28.061
So, that's a good takeaway.

s188
00:15:28.061 --> 00:15:30.261
Um production sessions becoming our test.

s189
00:15:30.261 --> 00:15:34.421
So, this is where I'll talk a little bit more about arise as a platform.

s190
00:15:34.421 --> 00:15:36.301
Uh tracing everything is really important, right?

s191
00:15:36.301 --> 00:15:41.501
So, we trace Actually, if you look at this, these are the traces from this agent.

s192
00:15:41.501 --> 00:15:43.261
Uh so, it's very meta here.

s193
00:15:43.261 --> 00:15:50.341
Um but every time you ask Alex a ques- question here, the the traces these are the traces from Alex.

s194
00:15:50.341 --> 00:15:52.581
We we still call it Copilot prod.

s195
00:15:52.581 --> 00:15:55.421
Um so, those are our live traces coming in from the tool.

s196
00:15:55.421 --> 00:16:05.621
Um this makes it very easy to add, you know, regression data sets and and keep track of different user queries that you want to go test against and iterate against.

s197
00:16:05.621 --> 00:16:10.421
Um and so, those production sessions become our test cases and become our living fixtures over time.

s198
00:16:10.421 --> 00:16:14.181
They grow, they change um as we see new error cases.

s199
00:16:14.181 --> 00:16:18.422
Um and we also now have the ability to use session data sets as well.

s200
00:16:18.422 --> 00:16:24.542
So, you can store an entire session as a data set and kind of play a specific turn through the agent if you need.

s201
00:16:24.542 --> 00:16:33.702
Um finally, we use arise experiments throughout all this as well to make sure that we're you know, kind of AB testing uh harness changes as we go through this.

s202
00:16:33.702 --> 00:16:41.102
Uh one takeaway I want you guys to to kind of keep in mind also is that in agents, bugs rarely show up as explicit errors.

s203
00:16:41.102 --> 00:16:46.742
Um in the old world software, this was a lot I don't know if it was easier, but it was different.

s204
00:16:46.742 --> 00:16:58.262
Um now it's more of like a qualitative okay, the status says okay, but if there are 43 identical tool calls in the same turn, that's probably something we want to fix.

s205
00:16:58.262 --> 00:17:00.662
Um software metrics like latency, right?

s206
00:17:00.662 --> 00:17:09.862
So, I think we can all agree 227 seconds is way too long to wait for a response, but what actually is too raw too long it really depends on your use case.

s207
00:17:09.862 --> 00:17:15.302
Um so, we use signal a lot now to kind of automate this reveal.

s208
00:17:15.302 --> 00:17:18.702
We we use less and less of the human reading the trace.

s209
00:17:18.702 --> 00:17:21.462
Um our staff engineers are very grateful for that.

s210
00:17:21.462 --> 00:17:24.736
Uh but we still do use kind of that human expertise, which I'll get to in a bit.

s211
00:17:24.736 --> 00:17:25.502
[snorts]

s212
00:17:25.502 --> 00:17:28.382
Um the fix can also be the next box.

s213
00:17:28.382 --> 00:17:32.582
So, these regression data sets are are really important and I would encourage you guys to keep them in mind.

s214
00:17:32.582 --> 00:17:36.302
Judges need that human baseline, so that's where we kind of impart the human task.

s215
00:17:36.302 --> 00:17:42.782
We align our evaluators and we make sure that our judges actually accurately reflect what you know, our engineers think as well.

s216
00:17:42.782 --> 00:17:44.862
Uh so, that's very important.

s217
00:17:44.862 --> 00:17:55.782
Um I think we we still keep a lot of things with the human, those gold labels, those golden data sets uh for judges come from people.

s218
00:17:55.782 --> 00:18:00.182
Um the speed versus quality is a human judgment call.

s219
00:18:00.182 --> 00:18:08.502
I don't think there's any single answer across the stack on when you should be a little bit faster, but it's okay to degrade quality.

s220
00:18:08.502 --> 00:18:17.422
It is still you know, kind of like a human judgment call even if you have the metrics around it like okay, changing models will increase latency by X and decrease

s221
00:18:17.422 --> 00:18:23.143
uh or increase quality response by Y. If we just use Astra everywhere, would that be the right decision?

s222
00:18:23.143 --> 00:18:24.543
Still a human judgment call, I think.

s223
00:18:24.543 --> 00:18:28.743
We have the metrics to support that, but we still have to make those decisions.

s224
00:18:28.743 --> 00:18:36.343
Um And these are kind of the the six things that survived all these rebuilds.

s225
00:18:36.343 --> 00:18:40.823
Tracing to our own products that we of course still trace everything that Alex does.

s226
00:18:40.823 --> 00:18:42.423
Math and data in code.

s227
00:18:42.423 --> 00:18:46.023
So, code computes the aggregations, the model reason and explains.

s228
00:18:46.023 --> 00:18:48.983
And then exposing those as tools to them to the model.

s229
00:18:48.983 --> 00:18:51.823
So, we're not giving the model plain text math questions.

s230
00:18:51.823 --> 00:18:55.183
We're actually asking it to use code to go do that.

s231
00:18:55.183 --> 00:19:00.183
Golden data sets in CI, errors with an obvious fix, streaming and persistent, and skills.

s232
00:19:00.183 --> 00:19:03.743
Um We've changed what we call skills.

s233
00:19:03.743 --> 00:19:12.303
I think, you know, routers, tools, etc. But really it's give the model a capability around an external system.

s234
00:19:12.303 --> 00:19:14.783
So, a lot of decisions you'll face.

s235
00:19:14.783 --> 00:19:16.463
And here's some of what we chose.

s236
00:19:16.463 --> 00:19:18.383
So, if you'll go ahead and take a picture of the slide.

s237
00:19:18.383 --> 00:19:19.783
Who owns the plan?

s238
00:19:19.783 --> 00:19:22.263
What ends the turn?

s239
00:19:22.383 --> 00:19:25.343
If your tool output is too big for your context, what do you do?

s240
00:19:25.343 --> 00:19:27.343
Who should write the long-term memory?

s241
00:19:27.343 --> 00:19:28.343
Should you have one?

s242
00:19:28.343 --> 00:19:30.503
Should you have multiple sub agents?

s243
00:19:30.503 --> 00:19:33.943
What model is good enough and fast enough?

s244
00:19:33.943 --> 00:19:37.423
And who checks the agent's own fixes?

s245
00:19:37.863 --> 00:19:39.983
So, yeah, that's me.

s246
00:19:39.983 --> 00:19:46.223
On the bottom right of the screen there my LinkedIn and X. The tagline is everyday we use Alex to build Alex.

s247
00:19:46.223 --> 00:19:52.303
We have a bunch of blog posts on sort of like technical deep dives into how we built this.

s248
00:19:52.303 --> 00:19:54.623
And yeah, I encourage you guys to all reach reach out.

s249
00:19:54.623 --> 00:19:57.703
And big shout out to the Alex team and our engineers for building this.

s250
00:19:57.703 --> 00:19:59.943
Thank you.

s251
00:19:59.973 --> 00:20:01.973
[applause]