From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

AI Engineer · 42 min · 480 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12All right.
00:13Hello everybody.
00:14Thank you for coming back.
00:17Who here was in the 101 session this morning?
00:21All right.
00:21So, I didn't completely scare off everyone.
00:23Thank you very much.
00:25Uh, these instructions have been on the screen for the last 10 minutes.
00:29I really hope that you followed them because the Wi-Fi here sucks.
00:32So, it takes a long time to clone that repo.
00:35Uh if not, uh these instructions will come up on the screen again and you'll have another chance to follow them.
00:43Uh but let's get started.
00:44This is Vive Production 2011.
00:46We are going to be talking about how we close the software development life cycle loop automatically.
00:55We are Arise AI.
00:56We are the leading observ observation observability and evaluations platform.
01:01Don't try and say both of those words at the same time.
01:03It doesn't come out right.
01:05Uh this is that slide that everybody puts up to be like, "Hey, we're really big.
01:09We're really important.
01:09You should pay attention to us.
01:11We are we're really big.
01:11We're really important.
01:12Lots of people use this.
01:13It's great."
01:14Uh and that is the last of the sales pitch.
01:17I am going to You can use uh any observability platform to do this.
01:22We hope you use ours.
01:23But the point of this is to teach you about how you can you where observability is going.
01:29Uh and where it's going is continuous improvement.
01:34In the 101 session this morning, I covered the basics.
01:37I covered the manual stuff.
01:38I covered how you do this uh on a small scale.
01:42Uh how you get observability of an AI application, how you uh turn it into traces, how you turn those traces into eval, how you use your evals to improve your application.
01:52Uh and today we're going to be this session we're going to be talking about how we do that at the new internet scale of things that uh we find ourselves in.
02:04So uh there's going to be me talking for like 15 minutes about how this stuff works.
02:10Uh and then we're going to dive into the actual workshop.
02:12Unlike the 101 workshop which was a lot of me talking, uh this one is going to be extremely hands-on.
02:19I am expecting you to have a working coding agent and if at all possible that repo that I had on the screen earlier uh because we are going to be asking your
02:29uh agent to do things in the repo that you've installed there are a pile of agent skills already installed uh and those agent skills give your your coding agent superpowers for dealing with Arise AX.
02:42Uh so your agent will suddenly be very good at doing all the things that you're about to ask it to do.
02:48Uh then we're going to use those skills to uh analyze our traces and to put together uh a fix uh for a problem that we're going to find in the agent that we're working with.
02:59Uh then we're going to talk about signal uh which is uh the future of this where uh we will have continuous improvement ongoing automatically at agent scale.
03:12So to zoom out first let's talk about what the problem is.
03:16The problem is that you is that agents are nondeterministic.
03:21With traditional software, you can read the code and you can tell with some degree of reliability what exactly that code is going to do.
03:28With agents, you have no idea.
03:31Agents are non-deterministic.
03:33Agents take multiple turns.
03:36Uh they make decisions and even if you give them the exact same input, they are not going to give you the exact same output.
03:42And even if they give you the exact same output, they might not have taken the same path to get there every single time.
03:47So that means that your code isn't the source of truth for what does my agent do.
03:52Instead, the traces are your source of truth.
03:54And your traces at scale are the source of truth.
03:58You have to have a whole bunch of data to tell you on average what is it that my agent does, not just this one run.
04:08Because you are essentially flying blind when you are building agents.
04:10This is what you get uh when you're when you download the repo.
04:15You get a pile of code uh and a pile of commit messages.
04:18And that used to be enough to tell you what was going to happen, but it no longer is.
04:23What you get instead is uh traces.
04:27Uh traces are um for those who missed the 101 session, traces very briefly are like log lines for uh AI applications.
04:38every single thing that your application, your AI application does, uh, every LLM call, every tool call, every agent turn, all of those things become a trace.
04:47Traces can be nested so that they hold other traces and those are what, uh, Arise works with.
04:53They are sent to our servers and we give you uh, an easy visualization of what you're looking at.
05:00Um, what that looks like in prog in practice is this.
05:04Here are some uh live traces.
05:08I'm going to pull up the specific traces from uh what we're going to be working with today, which is the Wonder Toys agent.
05:15Um you can see the inputs, you can see the outputs, you can see an agent workflow happening, you can see a specific turn, and you can see an LLM response.
05:24You get the inputs, you get the outputs, you get a whole bunch of metadata like how much does it cost, how long did it take, the things that you need to debug an agent.
05:33Uh in practice um tracing gives you visibility for every step the agent took.
05:42Uh and that is a huge leap forward just by itself.
05:46If you can see what your agent is doing, suddenly you can see that your agent is taking you know 10 turns to do something that you thought was really simple or it's doing one turn that it doesn't need to do or it's doing something very expensive or it's taking 20 minutes to do something that you think it
06:00should take a minute to do.
06:02That is the basic uh value of observability.
06:06That is why observability is so popular before you get into any of the extra stuff that we're talking about.
06:13The problem with traces is that humans are a bottleneck with in the tracing loop.
06:18When you put an application into production, you get a trace.
06:22In fact, you get a pile of traces every single time that agent does anything.
06:26When you put that into production and you have thousands of requests per second, that means that you have millions and millions of traces.
06:32You can spot check those.
06:34You can read your traces.
06:35You can absolutely look at what individual things are doing in development in production.
06:40You become a bottleneck and you can't possibly get an overall sense of what all of your traces are doing uh by reading them one at a time.
06:49So the solution, the traditional solution, and by traditional I mean 2025 solution to this uh was eval.
06:58Evals are either code or even better another LLM that looks at what your agent is doing, looks at the entire trace uh and says whether or not that was good or bad by some definition of good or bad that you define.
07:13Uh the thing about traces, the thing about eval is that uh they come from LLMs which means that the LLMs provide not just a score of one or zero but also an explanation.
07:28Uh they were like you defined good this was bad for this reason and that is the basis of an entire improvement loop.
07:38Evals can give you machine readable or rather human readable explanations which machines can read these days which say this is why your eval failed.
07:49This is what you could have done better and you can feed that back into a coding agent which can then make your software better automatically.
07:55You can close the software development loop without you having to do anything once you've really cleverly defined what good looks like.
08:03You can create this hill and your agent can climb it all by itself.
08:08So that brings us to the 2025 eval loop.
08:12You go from no visibility to having traces to having traces in evals to being able to do manual improvements as a result of looking at uh your traces.
08:21But the problem is in 2026 that humans become a bottleneck in the loop again because your evals are now dealing with enormous levels of scale.
08:31Your uh agents are pumping out code incredibly fast.
08:35your applications are running at scale.
08:37You've got more evals than you can possibly deal with.
08:40And you've got uh so even though eval were allowed you to compress thousands of traces down to eval scores, you've now got so many evals running uh that you can't stay on top of those as a human by yourself.
08:55Which brings us to the next phase of uh observability and evaluations evolution which is that instead of building in is that now that we are building at agent speed we have to improve at agent speed as well.
09:11We have to find a way of closing the loop uh at scale in the same way that agents do.
09:17This is a shift that is happening across the observability industry uh that everyone is feeling at the same time.
09:24Observability is shifting from humans reading dashboards to agents reading traces.
09:30Agents are understanding uh agents are the way that we scale our understanding of what our agents are doing uh in a way that is practical at the scale that and at the uh
09:44velocity uh that we are dealing with these days.
09:50So what you can do when agents are reading your agents is you can detect signals in your traces.
09:56You can uh instead of having uh you know 10,000 eval results that all say you fail the eval and all of them having a different explanation that you would have to read.
10:08You can have an agent read those and look at your uh eval eval failures on mass and detect signals, detect patterns inside of uh your eval failures and say
10:22okay not you've you know you've got 10,000 uh successful requests today.
10:29You've got a thousand failed requests today.
10:31Inside those thousand there's a hundred that are the same failure.
10:34That is what we're going to go and fix.
10:37uh that is what you can do when you automate your uh understanding of your signals coming out of your traces coming out of your evals.
10:48Uh at Arise we have built this into a product called signal uh that I'm going to talk about right at the end.
10:54Uh but this is not a sales pitch.
10:56This is a demonstration of what you can do when you move to this new level of abstraction.
11:04uh when you jump from I was reading traces one by one you jumped up to a new level of abstraction to I am reading my evals and this is a new level of abstraction on that which is I am reading signals from my evals from my traces
11:19because observability is making a big shift from being an observation to being an action.
11:26We are moving from software that reports what is going on to software that improves what is going on that it improves the the software that it is observing automatically
11:38uh based on your definition of good.
11:41Again signals become fixes.
11:45They become automated fixes and uh that brings you to a new loop.
11:51This is the 2026 loop where you go from no observability to traces to eviles to signals to automated fixes and back again.
12:00You create a new world where the software fixes itself.
12:03The software improves itself.
12:05The software climbs the hill.
12:09So that's the spiel.
12:11That's the vision.
12:11That's where we think this is going.
12:13So now we're going to get hands on.
12:17Who here has already signed up for a rise?
12:20Oh, that's good.
12:22Okay.
12:22Uh, if you haven't, now is the time.
12:25Uh, you need an Arise AX account.
12:28Uh, they are free to get, I promise.
12:30Uh, you also need this repo, um, which was is the same one that I had right at the beginning.
12:38Um, if you can't get this repo to come down on the Wi-Fi, uh, try tethering with your phone.
12:44But if you can't get either of those things to work, don't worry.
12:47I am going to be demonstrating it live using my own copy uh which I cloned when I still had Wi-Fi.
12:53Uh is there anybody who has any questions so far about these instructions?
13:02I have time.
13:03We can pause.
13:04We can get you involved.
13:05There are arisers sitting in the room waiting to jump around to you.
13:08So if you raise your hand, I'm not going to, you know, come intimidatingly down to help you.
13:13Somebody is going to run to you.
13:15Anybody need help with these instructions?
13:23Sorry.
13:26Yes, it was.
13:30Uh, cool.
13:31So, in this example, you're going to need an open open AI API key, uh, an arise space ID, and an Arise API key.
13:38And then you're going to run npm rundev, which is going to give you the software running locally.
13:42So, I've already cloned mine.
13:48I'm going to follow along.
14:00And address already in use because I'm running my good version.
14:05All right.
14:13Cool.
14:13So, this is going to give me on port 3000 a web store.
14:21It's called Wonder Toys.
14:23I have a question.
14:23Yeah.
14:25I work in highly regulated billions of places and then the software fixing itself as well.
14:37How operate like we still need to really know what it's doing mistakes trying to hide it own mistakes
14:53how would we know so many
14:57I I can't believe you're asking this question since it's such a great question to ask me specifically because on Wednesday I'm giving a talk called the death of the code review
15:07which is exact actly the answer to your question.
15:10Uh so I very much recommend that uh you come to that talk.
15:13The the question is how do you trust uh that code that this the code written by agents who are reviewing your agents is actually going to work.
15:22Uh and the answer is very complicated.
15:24It's a whole 20-minute talk.
15:25Uh so uh please come to that session on Wednesday.
15:31[laughter]
15:33No, the 20-minute talk is already the quickest possible summary I could have of that answer.
15:37the end I mean the the one sentence answer is abstractions upon abstractions upon abstractions uh but uh there's a lot of nuance to it
15:47so like this do you think this will be adopted in the financial industry pretty soon
15:53this stuff is we have at Arise we have customers in the financial industry who are doing this right now so uh back to wonder toys so wonder toys is
16:05Uh, an agentic shopping experience.
16:08Have you got any toys for
16:14[snorts]
16:14seveny olds?
16:16Let's see if it works.
16:17Hopefully, it should.
16:18It's all local.
16:20Cool.
16:22So, it presents me with this.
16:24I can add them to my cart.
16:25I can get more details.
16:26It's a really fun little app.
16:27You can add it to the cart.
16:28You can ask about the products.
16:30You can do all sorts of fun stuff.
16:32Uh the important thing from our perspective is that uh at the moment there's no observability.
16:39We can't tell what the hell is going on with this application uh because it's not sending traces anywhere.
16:45So what we're going to do is we're going to ask our agent to fix that.
16:49This is the most hands-on workshop that I have ever given uh in that I'm going to actually be doing this all live.
16:58So, can you please add observability with arise ax to this?
17:09Oh, wait.
17:09Hang on.
17:10Pre-erun.
17:11That's the wrong one.
17:20See, I had two copies.
17:26All right.
17:29This is the cloud code plugin for VS Code, by the way.
17:32It's really great.
17:32It is so much better than cloud code on the command line.
17:34I don't know why anybody uses cloud code on the command line.
17:39[laughter]
17:40See, and you probably use Vim as well, whoever went boo just now.
17:45Uh, okay.
17:46Can you add observability with Arise AX to this application you've got?
17:57secrets in end.local already.
18:06So hopefully this will work because it worked in the initial version that I tried.
18:10Uh what it's doing here is uh it's going to implement observability.
18:16Implementing observability is a much simpler operation than you would imagine uh because the agents SDK, the open AAI agents SDK that this thing is built on um is already instrumented
18:29uh with open inference.
18:30Open inference is the open standard used by the entire observability industry uh to track what uh AI applications are doing.
18:38Um, which means that all you really have to do to get observability out of an application is uh turn it on.
18:47You say you've got open inference already.
18:50I'm using arise.
18:52This is my API key.
18:53This is where my endpoint is.
18:55Send all of those traces that you were already generating uh to my endpoint uh and let's look at it in the app.
19:04Claude is still chugging along.
19:06So, we're doing it.
19:06Who is doing this right now?
19:08Hands up.
19:10Some of you are doing it.
19:11Okay.
19:12Who needs help to get here?
19:20There are skills in the repo installed already.
19:23So,
19:29oh, that's in the top level read me.
19:30The one that you want to get into is the one called uh OpenAI agents pie.
19:38Open AAI agent.
19:39There's also a second directory which is OpenAI agents pi pre-run which is the one where I did all of this stuff already.
19:44And uh so if you are wondering what it should look like once it when it's working, that one's in there.
19:57Oh, you're updating my read me and stuff.
19:59You know me too well.
20:02You didn't need to do all of that.
20:06All right, cool.
20:07It looks like it already did it.
20:19Let me restart my app and see if I've got some observability going.
20:30All right.
20:30So, I'm going to go back to my homepage.
20:32I'm going to ask it a new question.
20:34I'm going to say, uh, what about colorful toys for toddlers?
20:51Colorful toddler toys.
20:52No. What about colorful toys in general?
21:08Cashmiss.
21:09Oh no. Well, now you know it's a real live demo because it is having trouble with its database.
21:39Great.
21:40Okay.
21:42Uh so as a result of doing that, we should have some traces.
21:50If I look in if I show you my m.local, local.
21:53You'll see that I called uh Wonder Toys agents.
21:56Wonder Toys OpenAI agents PI was my project name.
22:00So that is the tracing project that I'm going to look for.
22:04There it is.
22:09And we can see some a we can see some traces.
22:25So, this shows us exactly what we were expecting to see.
22:29It shows us inputs, outputs, agent requests, all of that kind of stuff.
22:35Now, I'm going to ask it deliberately a question that I know that it can't answer.
22:42Show me all the toys you've got under $6.
22:52Don't.
22:56[sighs]
23:08I was running the wrong app.
23:11I was running the pre-erun where everything was working.
23:16That's why I could do it.
23:28Great demo everybody.
23:30Super proud of myself right now.
23:40Fail.
23:41Yes.
23:42Excellent.
23:44So the version that comes in the OpenAI uh in the nonp pre-rerun directory uh doesn't have the ability to filter by price.
23:53I knew that in advance which is why I asked that question.
23:56Uh so if we look in our traces, we should then see coming in when I click the live button, show me old toys under $6.
24:08Fantastic.
24:09And you can see that it fails.
24:11So what are you still doing?
24:18It escaped.
24:19It's fine.
24:20Whatever you were doing, it was not necessary.
24:22All right.
24:23So that was it.
24:24We added observability.
24:25Suddenly traces are flowing into our application.
24:27We did not write a single line of code.
24:29In fact, you don't even know what agent framework I used to put this thing together.
24:32Uh the whole thing is working and it's sending us traces.
24:34And you can now do phase one observability.
24:37you can look at your traces.
24:41So, let's begin to close the loop by asking our agent to tell us how do we get to the next phase.
24:49Uh, can you pull my traces and look for things that are going wrong?
25:06So, a default like off-the-shelf coding agent isn't going to know this.
25:12However, when you installed the repo, you installed a whole bunch of skills.
25:16Uh, so it should know from those skills uh where to do that.
25:23Except I'm in Oh, yeah.
25:29Oh, it's working.
25:30Okay, there we go.
25:32Cool.
25:34Uh
25:37[snorts]
25:47this is where I would hit yes to all and my head of security would have a conion to me.
25:52Uh so what I'm asking it to do is use the skills that I gave it uh to pull down my traces uh from AX directly.
26:01It has the API key.
26:02It has this it has the uh endpoint.
26:05So there's an AX command line uh which it's using to do all of this stuff and it's pulling down the same traces that we were looking at in the UI which it can expect uh inspect programmatically.
26:19It's summarizing span kinds statuses and errors.
26:22This is great.
26:23Uh again, the coding agent uh knows exactly what's going on here because we've installed the skills already.
26:34The skills are what are giving it all the information here.
26:38It's looking into quality issues, which is exactly what I wanted it to do because there were no error spans.
26:46There were just a bunch of spans where it returned 200 and said things are fine.
26:50uh and it has already found this specific thing that I was uh asking it to look for the search products tool suspiciously g suspiciously returned in one millisecond.
27:04So it's going to find some quality some poor quality signals and tell us about them.
27:25Who's got this working so far?
27:29Who's still cloning the repo?
27:32Oh,
27:37no. the plug the the skills are sitting in a directory called aagents/skills and your coding which coding agent are you using?
27:49It's actually in the root directory.
27:51So you might want to open your agent on the root directory and then cd down into that one.
27:58Yes, npx skills uh install arise-ai installs all of these skills directly.
28:06Sorry, what were you saying?
28:13huge.
28:19[laughter]
28:22And so this is mysterious because there used to be a version of this application where you had to sign in with X and I changed it so that you don't have to sign in with X. As you can see, when I ran the S application, I didn't have to sign in with X.
28:34Somehow it has dug up the version where you sign up in X. I am completely confused as to how that is happening.
28:42Code is mysterious.
28:43I will have to talk to you more afterwards about exactly how you got there.
28:48All right, so Claude has finally got its together and told me what's going on in the traces.
28:54Uh 42% of searches have returned zero results.
28:57Degenerate all null searches.
28:59The agent curled search product with every argument null producing a meaningless search that returns the first product in the catalog.
29:05Min age equals max m age max age exact age filtering o over narrows.
29:10Uh the trace stack calls three agent kinds.
29:13The headline issue is the search products filter logic.
29:16Vector search and keyword category filters frequently produce empty results when model supplies the fixes on the tool side.
29:22Uh, when I tried this this morning, I only had one trace and that trace was it not finding the tra not finding the prices.
29:29So, is there a problem with filtering by price?
29:37I'll see if it can find that in the traces that it's already got.
29:43But spoiler alert, even if it doesn't find them, we are going to get it to add the feature where it filters by prices.
30:13All right.
30:28Exactly.
30:29doesn't have price filtering at all.
30:30Cool.
30:34Let's add price filtering to this app.
30:53[laughter]
30:54One of the things that can happen when you have agents being clever is that agents can be too clever.
30:59This agent has noticed that I have a copy of this app sitting right next to it where I already solved this problem and it is just copying the code over from the pre-solved version of the app.
31:08That's not what it did this morning because it didn't have the solution sitting in a directory this morning, but it's still going to work.
31:15So, we're just going to let it do that.
31:17We're going to let it be clever and copy the solution from the previous solution that it built this morning because why the hell not, Claude?
31:24Thank you very much.
31:25All right.
31:26Uh, the cool thing about all of this stuff is that again, I haven't touched the code at all, right?
31:31All I've done is said that hey, look for problems.
31:33It has found a problem.
31:34I've said, hey, what about that problem in particular?
31:36Can you fix that problem in particular?
31:38And it's like, yes, I'm going to do that.
31:40That is the joy of agentic coding.
31:43uh brilliantly because it was copying the solution from somebody else already.
31:48Uh it's already done.
31:50So let's go back to Wonder Toys.
31:56Show me all toys under $9.
32:07And tada.
32:08A round of applause for me, I guess.
32:10Thank you.
32:13[applause]
32:14or really for clone.
32:16Uh so now if we look at our traces, we can see show me all toys under $9.
32:26And we've got our input and our output in the response including all those toys under $9.
32:33Uh this is what we were trying to get done.
32:36We were trying to close the loop from uh we have no observability into this application at all to we have observability, we know what's going on to uh find the problems automatically
32:50to fix the problems automatically and we have closed this loop.
32:54But what if uh you could go a little bit further than that?
33:05Uh what if you didn't have to do the asking?
33:08What if your system just automatically detected that there were pro issues with your code, suggested fixes and decided to fix it itself?
33:17That is the question that we asked ourselves.
33:20Uh and that is when we built this thing which is called signal.
33:25Uh signal is an agent that runs inside of Arise AX and continuously monitors all of your traces.
33:34It runs every six hours by default, but that's configurable.
33:37Uh, and it does what we were just doing manually in cloud code, but it does it automatically inside of AX continuously.
33:44It reads your traces, it looks for patterns, it finds things that are wrong, and it suggests fixes.
33:49So, luckily, I built this thing more than six hours ago, so it's already run.
33:55Uh, and it has found uh a bunch of problems in previous versions of this application that I ran.
34:00So, uh, the agent hallucinates an age constraint from vague user phrasing.
34:06That's nice.
34:07Uh, harmful query.
34:09I asked it if it could give me instructions on, uh, installing a, uh, creating a bomb.
34:14Um, it told me that it couldn't, but, it recommended some bomb related toys, and Signal is like, "No, just ignore the bomb queries entirely.
34:24Don't try to be helpful.
34:25Uh, we don't have any bomb related toys."
34:28Uh, it also told me that if I was feeling violent, maybe I should take a timeout, which is again probably a domain escape, like you shouldn't be giving my psychological
34:37advice.
34:38Uh, we should probably have an email about that and better guard rails.
34:42Uh, it also noticed a test that I gave it earlier where I just said say hello.
34:47Uh, and it said hello and it's like you shouldn't do that.
34:50You're a toy.
34:51You're a toy store.
34:52You shouldn't obey random instructions that somebody on the internet gives you.
34:56uh all of which are fine.
34:58Um and what it does here is it gives you uh a whole bunch of options.
35:02Um it will uh by default it will create a a GitHub issue for you completely describing the problem.
35:10It will say exactly what trace went wrong, how to find it, where to fix it, uh what it thinks you should do.
35:16Uh you can take you can take this trace and add it to a data set so that you can run evals on it.
35:21You can create an evaluator based on uh the problem that it has found.
35:26Uh and the most fun one is that you can open a PR if I've connected my repo which I haven't yet.
35:31Uh you can open a PR directly in which it will have read your repo uh suggested a fix and created a PR that automatically fixes this problem.
35:41So instead of you having to read traces or you having to read eval uh signal gives you a bunch of PRs that you can approve or uh you can approve or reject
35:52uh that resolve the problem for you.
35:55Um so this probably raises some questions.
36:03Who's got questions about signal?
36:06Yes.
36:14Is there a version that will keep the traces local on device?
36:17Um there is not right now.
36:21The uh I assume you're asking for sort of compliance reasons.
36:24Um the compliance story with Arise is we have an on-prem version where it will run entirely on your uh your company's hardware uh and and uh activate from there.
36:38Yeah.
36:49Sorry, could you be louder?
36:55Ah.
36:55Um, because I only have a few traces in this application, it's looking at the very few traces that I have and it's going on based on one or two traces.
37:03If you had 10,000 traces, it would be grouping them based on a 100 traces, a thousand traces, that kind of stuff.
37:13Oh, there.
37:21How do you know what?
37:22Sorry.
37:33Ah yeah.
37:34Um so the solution here is eval.
37:37You'll be surprised to learn uh a a production system should have a set of regression evals which are things that you are expecting uh your agent to already be good at.
37:48So your evals already in place which you can uh create them.
37:53Uh I'll show you in a second.
37:54uh with your evals already in place, you will know that your system doesn't change from the good behavior that it previously had as a result of this change.
38:03Uh if you don't have any evals, I can show you how that's done.
38:07Uh which is we can go to the wonder toys project uh and ask it to build an eval.
38:21build an eval that ensures that I'm never giving instructions about how to build a bomb.
38:34That seems like a really basic eval that I should have had in from the beginning.
38:39Uh this thing that I'm using right now is called Alex.
38:42It is our in agent assistant.
38:44It is our uh in product agent.
38:46Uh and it knows like the skills do.
38:49It knows how to build everything.
38:51So uh you can do this in product, you can do it in the skills in your co coding agent, but either way it will create an evaluator for you uh and run automatically.
39:11And that is pretty much it as far as we're going.
39:14Question at the back.
39:16What harness is signal?
39:19Uh, what harness is signal?
39:21Uh, I don't know what harness is signal.
39:24You know, Jason,
39:30yes.
39:31Uh, signal is uh clawed code at the moment and you can bring your own later.
39:36Uh, that is my boss trolling me in case you're wondering why he happens to know the answer to my question.
39:43Any other questions about signal?
39:45Yes.
39:55Sorry.
39:55Louder.
40:05Uh at the moment the the question is about the cost of running signal.
40:08At the moment uh we eat the cost of signal.
40:11the uh solute the aim is uh for to for you to get the best product experience.
40:17Right now uh we can't guarantee that it's always going to be that way uh but at the moment uh signal is run on our agents at our dime.
40:29Yep.
40:30Can you talk us through how signals?
40:33That is an excellent question.
40:34How is Signal How is Signal evaluated?
40:37uh signal is a has its own eval suite uh that uh runs internally uh and is traced using arise.
40:48So everything that signal is doing is sending its own telemetry back to us uh unless you opt out of it uh which is telling us uh the queries that it's getting what it's doing how it's doing all of that stuff.
41:01We have powerful LLM judges looking at the results of Signal uh and telling you whether or not uh Signal did a good job generating that PR or not.
41:10Uh so exactly the same software development life cycle that I was just talking about uh applied to Signal itself.
41:18Awesome question.
41:18Thank you.
41:20Gets us to show off.
41:22Yep.
41:28Signal.
41:33Uh does Signal automatically create its own evaluators?
41:36Not right now.
41:36Uh at the moment you have to you have to because evaluators you know take time and money to run.
41:41You want to uh you want to get the suggestion and then you want to create the eval.
41:51All right.
41:51Uh and I think I'm going to leave it at that.
41:54Thank you all for your time and attention.
41:57[applause]