WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=F0TNSmbo5hE

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/F0TNSmbo5hE.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.880 --> 00:00:13.440
All right.

s2
00:00:13.440 --> 00:00:14.639
Hello everybody.

s3
00:00:14.639 --> 00:00:17.440
Thank you for coming back.

s4
00:00:17.440 --> 00:00:21.119
Who here was in the 101 session this morning?

s5
00:00:21.119 --> 00:00:21.520
All right.

s6
00:00:21.520 --> 00:00:23.519
So, I didn't completely scare off everyone.

s7
00:00:23.519 --> 00:00:25.039
Thank you very much.

s8
00:00:25.039 --> 00:00:29.439
Uh, these instructions have been on the screen for the last 10 minutes.

s9
00:00:29.439 --> 00:00:32.399
I really hope that you followed them because the Wi-Fi here sucks.

s10
00:00:32.399 --> 00:00:35.040
So, it takes a long time to clone that repo.

s11
00:00:35.040 --> 00:00:43.520
Uh if not, uh these instructions will come up on the screen again and you'll have another chance to follow them.

s12
00:00:43.520 --> 00:00:44.960
Uh but let's get started.

s13
00:00:44.960 --> 00:00:46.879
This is Vive Production 2011.

s14
00:00:46.879 --> 00:00:53.120
We are going to be talking about how we close the software development life cycle loop automatically.

s15
00:00:55.440 --> 00:00:56.800
We are Arise AI.

s16
00:00:56.800 --> 00:01:01.840
We are the leading observ observation observability and evaluations platform.

s17
00:01:01.840 --> 00:01:03.680
Don't try and say both of those words at the same time.

s18
00:01:03.680 --> 00:01:05.119
It doesn't come out right.

s19
00:01:05.119 --> 00:01:09.200
Uh this is that slide that everybody puts up to be like, "Hey, we're really big.

s20
00:01:09.200 --> 00:01:09.920
We're really important.

s21
00:01:09.920 --> 00:01:11.040
You should pay attention to us.

s22
00:01:11.040 --> 00:01:11.920
We are we're really big.

s23
00:01:11.920 --> 00:01:12.640
We're really important.

s24
00:01:12.640 --> 00:01:13.520
Lots of people use this.

s25
00:01:13.520 --> 00:01:14.479
It's great."

s26
00:01:14.479 --> 00:01:17.439
Uh and that is the last of the sales pitch.

s27
00:01:17.439 --> 00:01:22.560
I am going to You can use uh any observability platform to do this.

s28
00:01:22.560 --> 00:01:23.600
We hope you use ours.

s29
00:01:23.600 --> 00:01:29.920
But the point of this is to teach you about how you can you where observability is going.

s30
00:01:29.920 --> 00:01:34.000
Uh and where it's going is continuous improvement.

s31
00:01:34.000 --> 00:01:37.040
In the 101 session this morning, I covered the basics.

s32
00:01:37.040 --> 00:01:38.400
I covered the manual stuff.

s33
00:01:38.400 --> 00:01:42.720
I covered how you do this uh on a small scale.

s34
00:01:42.720 --> 00:01:52.880
Uh how you get observability of an AI application, how you uh turn it into traces, how you turn those traces into eval, how you use your evals to improve your application.

s35
00:01:52.880 --> 00:02:04.719
Uh and today we're going to be this session we're going to be talking about how we do that at the new internet scale of things that uh we find ourselves in.

s36
00:02:04.719 --> 00:02:10.560
So uh there's going to be me talking for like 15 minutes about how this stuff works.

s37
00:02:10.560 --> 00:02:12.959
Uh and then we're going to dive into the actual workshop.

s38
00:02:12.959 --> 00:02:19.840
Unlike the 101 workshop which was a lot of me talking, uh this one is going to be extremely hands-on.

s39
00:02:19.840 --> 00:02:29.680
I am expecting you to have a working coding agent and if at all possible that repo that I had on the screen earlier uh because we are going to be asking your

s40
00:02:29.680 --> 00:02:42.160
uh agent to do things in the repo that you've installed there are a pile of agent skills already installed uh and those agent skills give your your coding agent superpowers for dealing with Arise AX.

s41
00:02:42.160 --> 00:02:48.000
Uh so your agent will suddenly be very good at doing all the things that you're about to ask it to do.

s42
00:02:48.000 --> 00:02:59.760
Uh then we're going to use those skills to uh analyze our traces and to put together uh a fix uh for a problem that we're going to find in the agent that we're working with.

s43
00:02:59.760 --> 00:03:12.640
Uh then we're going to talk about signal uh which is uh the future of this where uh we will have continuous improvement ongoing automatically at agent scale.

s44
00:03:12.800 --> 00:03:16.959
So to zoom out first let's talk about what the problem is.

s45
00:03:16.959 --> 00:03:21.120
The problem is that you is that agents are nondeterministic.

s46
00:03:21.120 --> 00:03:28.959
With traditional software, you can read the code and you can tell with some degree of reliability what exactly that code is going to do.

s47
00:03:28.959 --> 00:03:31.200
With agents, you have no idea.

s48
00:03:31.200 --> 00:03:33.280
Agents are non-deterministic.

s49
00:03:33.280 --> 00:03:36.000
Agents take multiple turns.

s50
00:03:36.000 --> 00:03:42.000
Uh they make decisions and even if you give them the exact same input, they are not going to give you the exact same output.

s51
00:03:42.000 --> 00:03:47.760
And even if they give you the exact same output, they might not have taken the same path to get there every single time.

s52
00:03:47.760 --> 00:03:52.239
So that means that your code isn't the source of truth for what does my agent do.

s53
00:03:52.239 --> 00:03:54.720
Instead, the traces are your source of truth.

s54
00:03:54.720 --> 00:03:58.400
And your traces at scale are the source of truth.

s55
00:03:58.400 --> 00:04:07.439
You have to have a whole bunch of data to tell you on average what is it that my agent does, not just this one run.

s56
00:04:08.319 --> 00:04:10.959
Because you are essentially flying blind when you are building agents.

s57
00:04:10.959 --> 00:04:15.040
This is what you get uh when you're when you download the repo.

s58
00:04:15.040 --> 00:04:18.959
You get a pile of code uh and a pile of commit messages.

s59
00:04:18.959 --> 00:04:23.280
And that used to be enough to tell you what was going to happen, but it no longer is.

s60
00:04:23.280 --> 00:04:27.120
What you get instead is uh traces.

s61
00:04:27.120 --> 00:04:38.400
Uh traces are um for those who missed the 101 session, traces very briefly are like log lines for uh AI applications.

s62
00:04:38.400 --> 00:04:47.360
every single thing that your application, your AI application does, uh, every LLM call, every tool call, every agent turn, all of those things become a trace.

s63
00:04:47.360 --> 00:04:53.680
Traces can be nested so that they hold other traces and those are what, uh, Arise works with.

s64
00:04:53.680 --> 00:05:00.479
They are sent to our servers and we give you uh, an easy visualization of what you're looking at.

s65
00:05:00.479 --> 00:05:04.240
Um, what that looks like in prog in practice is this.

s66
00:05:04.240 --> 00:05:08.720
Here are some uh live traces.

s67
00:05:08.720 --> 00:05:15.360
I'm going to pull up the specific traces from uh what we're going to be working with today, which is the Wonder Toys agent.

s68
00:05:15.360 --> 00:05:24.960
Um you can see the inputs, you can see the outputs, you can see an agent workflow happening, you can see a specific turn, and you can see an LLM response.

s69
00:05:24.960 --> 00:05:33.520
You get the inputs, you get the outputs, you get a whole bunch of metadata like how much does it cost, how long did it take, the things that you need to debug an agent.

s70
00:05:33.520 --> 00:05:42.479
Uh in practice um tracing gives you visibility for every step the agent took.

s71
00:05:42.479 --> 00:05:46.479
Uh and that is a huge leap forward just by itself.

s72
00:05:46.479 --> 00:06:00.240
If you can see what your agent is doing, suddenly you can see that your agent is taking you know 10 turns to do something that you thought was really simple or it's doing one turn that it doesn't need to do or it's doing something very expensive or it's taking 20 minutes to do something that you think it

s73
00:06:00.240 --> 00:06:02.000
should take a minute to do.

s74
00:06:02.000 --> 00:06:06.240
That is the basic uh value of observability.

s75
00:06:06.240 --> 00:06:13.039
That is why observability is so popular before you get into any of the extra stuff that we're talking about.

s76
00:06:13.039 --> 00:06:18.560
The problem with traces is that humans are a bottleneck with in the tracing loop.

s77
00:06:18.560 --> 00:06:22.400
When you put an application into production, you get a trace.

s78
00:06:22.400 --> 00:06:26.720
In fact, you get a pile of traces every single time that agent does anything.

s79
00:06:26.720 --> 00:06:32.800
When you put that into production and you have thousands of requests per second, that means that you have millions and millions of traces.

s80
00:06:32.800 --> 00:06:34.160
You can spot check those.

s81
00:06:34.160 --> 00:06:35.440
You can read your traces.

s82
00:06:35.440 --> 00:06:40.319
You can absolutely look at what individual things are doing in development in production.

s83
00:06:40.319 --> 00:06:49.440
You become a bottleneck and you can't possibly get an overall sense of what all of your traces are doing uh by reading them one at a time.

s84
00:06:49.440 --> 00:06:58.240
So the solution, the traditional solution, and by traditional I mean 2025 solution to this uh was eval.

s85
00:06:58.240 --> 00:07:13.440
Evals are either code or even better another LLM that looks at what your agent is doing, looks at the entire trace uh and says whether or not that was good or bad by some definition of good or bad that you define.

s86
00:07:13.440 --> 00:07:28.800
Uh the thing about traces, the thing about eval is that uh they come from LLMs which means that the LLMs provide not just a score of one or zero but also an explanation.

s87
00:07:28.800 --> 00:07:38.400
Uh they were like you defined good this was bad for this reason and that is the basis of an entire improvement loop.

s88
00:07:38.400 --> 00:07:49.039
Evals can give you machine readable or rather human readable explanations which machines can read these days which say this is why your eval failed.

s89
00:07:49.039 --> 00:07:55.840
This is what you could have done better and you can feed that back into a coding agent which can then make your software better automatically.

s90
00:07:55.840 --> 00:08:03.520
You can close the software development loop without you having to do anything once you've really cleverly defined what good looks like.

s91
00:08:03.520 --> 00:08:08.160
You can create this hill and your agent can climb it all by itself.

s92
00:08:08.560 --> 00:08:12.479
So that brings us to the 2025 eval loop.

s93
00:08:12.479 --> 00:08:21.599
You go from no visibility to having traces to having traces in evals to being able to do manual improvements as a result of looking at uh your traces.

s94
00:08:21.599 --> 00:08:31.599
But the problem is in 2026 that humans become a bottleneck in the loop again because your evals are now dealing with enormous levels of scale.

s95
00:08:31.599 --> 00:08:35.039
Your uh agents are pumping out code incredibly fast.

s96
00:08:35.039 --> 00:08:37.120
your applications are running at scale.

s97
00:08:37.120 --> 00:08:40.560
You've got more evals than you can possibly deal with.

s98
00:08:40.560 --> 00:08:55.920
And you've got uh so even though eval were allowed you to compress thousands of traces down to eval scores, you've now got so many evals running uh that you can't stay on top of those as a human by yourself.

s99
00:08:55.920 --> 00:09:11.200
Which brings us to the next phase of uh observability and evaluations evolution which is that instead of building in is that now that we are building at agent speed we have to improve at agent speed as well.

s100
00:09:11.200 --> 00:09:17.519
We have to find a way of closing the loop uh at scale in the same way that agents do.

s101
00:09:17.519 --> 00:09:24.640
This is a shift that is happening across the observability industry uh that everyone is feeling at the same time.

s102
00:09:24.640 --> 00:09:30.399
Observability is shifting from humans reading dashboards to agents reading traces.

s103
00:09:30.399 --> 00:09:44.160
Agents are understanding uh agents are the way that we scale our understanding of what our agents are doing uh in a way that is practical at the scale that and at the uh

s104
00:09:44.160 --> 00:09:48.800
velocity uh that we are dealing with these days.

s105
00:09:50.240 --> 00:09:56.959
So what you can do when agents are reading your agents is you can detect signals in your traces.

s106
00:09:56.959 --> 00:10:08.959
You can uh instead of having uh you know 10,000 eval results that all say you fail the eval and all of them having a different explanation that you would have to read.

s107
00:10:08.959 --> 00:10:22.000
You can have an agent read those and look at your uh eval eval failures on mass and detect signals, detect patterns inside of uh your eval failures and say

s108
00:10:22.000 --> 00:10:29.040
okay not you've you know you've got 10,000 uh successful requests today.

s109
00:10:29.040 --> 00:10:31.360
You've got a thousand failed requests today.

s110
00:10:31.360 --> 00:10:34.800
Inside those thousand there's a hundred that are the same failure.

s111
00:10:34.800 --> 00:10:37.040
That is what we're going to go and fix.

s112
00:10:37.040 --> 00:10:46.720
uh that is what you can do when you automate your uh understanding of your signals coming out of your traces coming out of your evals.

s113
00:10:48.399 --> 00:10:54.880
Uh at Arise we have built this into a product called signal uh that I'm going to talk about right at the end.

s114
00:10:54.880 --> 00:10:56.640
Uh but this is not a sales pitch.

s115
00:10:56.640 --> 00:11:04.079
This is a demonstration of what you can do when you move to this new level of abstraction.

s116
00:11:04.079 --> 00:11:17.839
uh when you jump from I was reading traces one by one you jumped up to a new level of abstraction to I am reading my evals and this is a new level of abstraction on that which is I am reading signals from my evals from my traces

s117
00:11:19.360 --> 00:11:26.880
because observability is making a big shift from being an observation to being an action.

s118
00:11:26.880 --> 00:11:38.880
We are moving from software that reports what is going on to software that improves what is going on that it improves the the software that it is observing automatically

s119
00:11:38.880 --> 00:11:41.279
uh based on your definition of good.

s120
00:11:41.279 --> 00:11:45.600
Again signals become fixes.

s121
00:11:45.600 --> 00:11:51.200
They become automated fixes and uh that brings you to a new loop.

s122
00:11:51.200 --> 00:12:00.000
This is the 2026 loop where you go from no observability to traces to eviles to signals to automated fixes and back again.

s123
00:12:00.000 --> 00:12:03.760
You create a new world where the software fixes itself.

s124
00:12:03.760 --> 00:12:05.360
The software improves itself.

s125
00:12:05.360 --> 00:12:08.320
The software climbs the hill.

s126
00:12:09.120 --> 00:12:11.120
So that's the spiel.

s127
00:12:11.120 --> 00:12:11.920
That's the vision.

s128
00:12:11.920 --> 00:12:13.920
That's where we think this is going.

s129
00:12:13.920 --> 00:12:17.040
So now we're going to get hands on.

s130
00:12:17.040 --> 00:12:20.880
Who here has already signed up for a rise?

s131
00:12:20.880 --> 00:12:22.000
Oh, that's good.

s132
00:12:22.000 --> 00:12:22.720
Okay.

s133
00:12:22.720 --> 00:12:25.519
Uh, if you haven't, now is the time.

s134
00:12:25.519 --> 00:12:28.079
Uh, you need an Arise AX account.

s135
00:12:28.079 --> 00:12:30.639
Uh, they are free to get, I promise.

s136
00:12:30.639 --> 00:12:38.000
Uh, you also need this repo, um, which was is the same one that I had right at the beginning.

s137
00:12:38.000 --> 00:12:44.399
Um, if you can't get this repo to come down on the Wi-Fi, uh, try tethering with your phone.

s138
00:12:44.399 --> 00:12:47.040
But if you can't get either of those things to work, don't worry.

s139
00:12:47.040 --> 00:12:53.600
I am going to be demonstrating it live using my own copy uh which I cloned when I still had Wi-Fi.

s140
00:12:53.600 --> 00:13:02.240
Uh is there anybody who has any questions so far about these instructions?

s141
00:13:02.240 --> 00:13:03.279
I have time.

s142
00:13:03.279 --> 00:13:04.079
We can pause.

s143
00:13:04.079 --> 00:13:05.360
We can get you involved.

s144
00:13:05.360 --> 00:13:08.480
There are arisers sitting in the room waiting to jump around to you.

s145
00:13:08.480 --> 00:13:13.120
So if you raise your hand, I'm not going to, you know, come intimidatingly down to help you.

s146
00:13:13.120 --> 00:13:15.600
Somebody is going to run to you.

s147
00:13:15.600 --> 00:13:19.839
Anybody need help with these instructions?

s148
00:13:23.120 --> 00:13:25.120
Sorry.

s149
00:13:26.560 --> 00:13:29.040
Yes, it was.

s150
00:13:30.480 --> 00:13:31.680
Uh, cool.

s151
00:13:31.680 --> 00:13:38.240
So, in this example, you're going to need an open open AI API key, uh, an arise space ID, and an Arise API key.

s152
00:13:38.240 --> 00:13:42.720
And then you're going to run npm rundev, which is going to give you the software running locally.

s153
00:13:42.720 --> 00:13:47.519
So, I've already cloned mine.

s154
00:13:48.000 --> 00:13:50.880
I'm going to follow along.

s155
00:14:00.320 --> 00:14:05.199
And address already in use because I'm running my good version.

s156
00:14:05.199 --> 00:14:07.199
All right.

s157
00:14:13.120 --> 00:14:13.600
Cool.

s158
00:14:13.600 --> 00:14:21.199
So, this is going to give me on port 3000 a web store.

s159
00:14:21.199 --> 00:14:23.040
It's called Wonder Toys.

s160
00:14:23.040 --> 00:14:23.839
I have a question.

s161
00:14:23.839 --> 00:14:25.839
Yeah.

s162
00:14:25.839 --> 00:14:37.839
I work in highly regulated billions of places and then the software fixing itself as well.

s163
00:14:37.839 --> 00:14:53.279
How operate like we still need to really know what it's doing mistakes trying to hide it own mistakes

s164
00:14:53.279 --> 00:14:56.399
how would we know so many

s165
00:14:57.360 --> 00:15:07.120
I I can't believe you're asking this question since it's such a great question to ask me specifically because on Wednesday I'm giving a talk called the death of the code review

s166
00:15:07.120 --> 00:15:10.160
which is exact actly the answer to your question.

s167
00:15:10.160 --> 00:15:13.680
Uh so I very much recommend that uh you come to that talk.

s168
00:15:13.680 --> 00:15:22.560
The the question is how do you trust uh that code that this the code written by agents who are reviewing your agents is actually going to work.

s169
00:15:22.560 --> 00:15:24.240
Uh and the answer is very complicated.

s170
00:15:24.240 --> 00:15:25.760
It's a whole 20-minute talk.

s171
00:15:25.760 --> 00:15:30.079
Uh so uh please come to that session on Wednesday.

s172
00:15:31.420 --> 00:15:33.120
[laughter]

s173
00:15:33.120 --> 00:15:37.199
No, the 20-minute talk is already the quickest possible summary I could have of that answer.

s174
00:15:37.199 --> 00:15:47.360
the end I mean the the one sentence answer is abstractions upon abstractions upon abstractions uh but uh there's a lot of nuance to it

s175
00:15:47.360 --> 00:15:53.600
so like this do you think this will be adopted in the financial industry pretty soon

s176
00:15:53.600 --> 00:16:05.680
this stuff is we have at Arise we have customers in the financial industry who are doing this right now so uh back to wonder toys so wonder toys is

s177
00:16:05.680 --> 00:16:08.399
Uh, an agentic shopping experience.

s178
00:16:08.399 --> 00:16:14.651
Have you got any toys for

s179
00:16:14.651 --> 00:16:14.800
[snorts]

s180
00:16:14.800 --> 00:16:16.160
seveny olds?

s181
00:16:16.160 --> 00:16:17.759
Let's see if it works.

s182
00:16:17.759 --> 00:16:18.639
Hopefully, it should.

s183
00:16:18.639 --> 00:16:20.160
It's all local.

s184
00:16:20.160 --> 00:16:22.160
Cool.

s185
00:16:22.320 --> 00:16:24.079
So, it presents me with this.

s186
00:16:24.079 --> 00:16:25.279
I can add them to my cart.

s187
00:16:25.279 --> 00:16:26.480
I can get more details.

s188
00:16:26.480 --> 00:16:27.759
It's a really fun little app.

s189
00:16:27.759 --> 00:16:28.800
You can add it to the cart.

s190
00:16:28.800 --> 00:16:30.160
You can ask about the products.

s191
00:16:30.160 --> 00:16:32.240
You can do all sorts of fun stuff.

s192
00:16:32.240 --> 00:16:39.680
Uh the important thing from our perspective is that uh at the moment there's no observability.

s193
00:16:39.680 --> 00:16:45.279
We can't tell what the hell is going on with this application uh because it's not sending traces anywhere.

s194
00:16:45.279 --> 00:16:49.360
So what we're going to do is we're going to ask our agent to fix that.

s195
00:16:49.360 --> 00:16:58.320
This is the most hands-on workshop that I have ever given uh in that I'm going to actually be doing this all live.

s196
00:16:58.320 --> 00:17:09.039
So, can you please add observability with arise ax to this?

s197
00:17:09.039 --> 00:17:09.679
Oh, wait.

s198
00:17:09.679 --> 00:17:10.400
Hang on.

s199
00:17:10.400 --> 00:17:11.039
Pre-erun.

s200
00:17:11.039 --> 00:17:13.679
That's the wrong one.

s201
00:17:20.799 --> 00:17:23.520
See, I had two copies.

s202
00:17:26.160 --> 00:17:28.160
All right.

s203
00:17:29.440 --> 00:17:32.000
This is the cloud code plugin for VS Code, by the way.

s204
00:17:32.000 --> 00:17:32.799
It's really great.

s205
00:17:32.799 --> 00:17:34.880
It is so much better than cloud code on the command line.

s206
00:17:34.880 --> 00:17:38.960
I don't know why anybody uses cloud code on the command line.

s207
00:17:39.147 --> 00:17:40.480
[laughter]

s208
00:17:40.480 --> 00:17:45.360
See, and you probably use Vim as well, whoever went boo just now.

s209
00:17:45.360 --> 00:17:46.799
Uh, okay.

s210
00:17:46.799 --> 00:17:57.360
Can you add observability with Arise AX to this application you've got?

s211
00:17:57.360 --> 00:18:04.160
secrets in end.local already.

s212
00:18:06.559 --> 00:18:10.799
So hopefully this will work because it worked in the initial version that I tried.

s213
00:18:10.799 --> 00:18:16.000
Uh what it's doing here is uh it's going to implement observability.

s214
00:18:16.000 --> 00:18:29.440
Implementing observability is a much simpler operation than you would imagine uh because the agents SDK, the open AAI agents SDK that this thing is built on um is already instrumented

s215
00:18:29.440 --> 00:18:30.880
uh with open inference.

s216
00:18:30.880 --> 00:18:38.880
Open inference is the open standard used by the entire observability industry uh to track what uh AI applications are doing.

s217
00:18:38.880 --> 00:18:47.600
Um, which means that all you really have to do to get observability out of an application is uh turn it on.

s218
00:18:47.600 --> 00:18:50.799
You say you've got open inference already.

s219
00:18:50.799 --> 00:18:52.160
I'm using arise.

s220
00:18:52.160 --> 00:18:53.679
This is my API key.

s221
00:18:53.679 --> 00:18:55.280
This is where my endpoint is.

s222
00:18:55.280 --> 00:19:03.520
Send all of those traces that you were already generating uh to my endpoint uh and let's look at it in the app.

s223
00:19:04.480 --> 00:19:06.000
Claude is still chugging along.

s224
00:19:06.000 --> 00:19:06.960
So, we're doing it.

s225
00:19:06.960 --> 00:19:08.480
Who is doing this right now?

s226
00:19:08.480 --> 00:19:10.559
Hands up.

s227
00:19:10.559 --> 00:19:11.600
Some of you are doing it.

s228
00:19:11.600 --> 00:19:12.720
Okay.

s229
00:19:12.720 --> 00:19:15.760
Who needs help to get here?

s230
00:19:20.080 --> 00:19:23.039
There are skills in the repo installed already.

s231
00:19:23.039 --> 00:19:25.039
So,

s232
00:19:29.120 --> 00:19:30.880
oh, that's in the top level read me.

s233
00:19:30.880 --> 00:19:37.200
The one that you want to get into is the one called uh OpenAI agents pie.

s234
00:19:38.320 --> 00:19:39.440
Open AAI agent.

s235
00:19:39.440 --> 00:19:44.400
There's also a second directory which is OpenAI agents pi pre-run which is the one where I did all of this stuff already.

s236
00:19:44.400 --> 00:19:51.440
And uh so if you are wondering what it should look like once it when it's working, that one's in there.

s237
00:19:57.360 --> 00:19:59.520
Oh, you're updating my read me and stuff.

s238
00:19:59.520 --> 00:20:02.080
You know me too well.

s239
00:20:02.080 --> 00:20:05.200
You didn't need to do all of that.

s240
00:20:06.240 --> 00:20:07.120
All right, cool.

s241
00:20:07.120 --> 00:20:10.240
It looks like it already did it.

s242
00:20:19.840 --> 00:20:24.559
Let me restart my app and see if I've got some observability going.

s243
00:20:30.240 --> 00:20:30.640
All right.

s244
00:20:30.640 --> 00:20:32.400
So, I'm going to go back to my homepage.

s245
00:20:32.400 --> 00:20:34.000
I'm going to ask it a new question.

s246
00:20:34.000 --> 00:20:43.360
I'm going to say, uh, what about colorful toys for toddlers?

s247
00:20:51.520 --> 00:20:52.720
Colorful toddler toys.

s248
00:20:52.720 --> 00:20:58.799
No. What about colorful toys in general?

s249
00:21:08.400 --> 00:21:09.280
Cashmiss.

s250
00:21:09.280 --> 00:21:18.480
Oh no. Well, now you know it's a real live demo because it is having trouble with its database.

s251
00:21:39.520 --> 00:21:40.159
Great.

s252
00:21:40.159 --> 00:21:42.159
Okay.

s253
00:21:42.640 --> 00:21:50.320
Uh so as a result of doing that, we should have some traces.

s254
00:21:50.320 --> 00:21:53.360
If I look in if I show you my m.local, local.

s255
00:21:53.360 --> 00:21:56.960
You'll see that I called uh Wonder Toys agents.

s256
00:21:56.960 --> 00:22:00.080
Wonder Toys OpenAI agents PI was my project name.

s257
00:22:00.080 --> 00:22:04.159
So that is the tracing project that I'm going to look for.

s258
00:22:04.559 --> 00:22:06.960
There it is.

s259
00:22:09.039 --> 00:22:12.960
And we can see some a we can see some traces.

s260
00:22:25.600 --> 00:22:29.039
So, this shows us exactly what we were expecting to see.

s261
00:22:29.039 --> 00:22:33.679
It shows us inputs, outputs, agent requests, all of that kind of stuff.

s262
00:22:35.440 --> 00:22:40.799
Now, I'm going to ask it deliberately a question that I know that it can't answer.

s263
00:22:42.240 --> 00:22:49.120
Show me all the toys you've got under $6.

s264
00:22:52.880 --> 00:22:54.880
Don't.

s265
00:22:56.170 --> 00:22:58.170
[sighs]

s266
00:23:08.320 --> 00:23:11.600
I was running the wrong app.

s267
00:23:11.600 --> 00:23:16.880
I was running the pre-erun where everything was working.

s268
00:23:16.960 --> 00:23:19.840
That's why I could do it.

s269
00:23:28.000 --> 00:23:30.320
Great demo everybody.

s270
00:23:30.320 --> 00:23:33.919
Super proud of myself right now.

s271
00:23:40.480 --> 00:23:41.360
Fail.

s272
00:23:41.360 --> 00:23:42.000
Yes.

s273
00:23:42.000 --> 00:23:44.000
Excellent.

s274
00:23:44.400 --> 00:23:53.120
So the version that comes in the OpenAI uh in the nonp pre-rerun directory uh doesn't have the ability to filter by price.

s275
00:23:53.120 --> 00:23:56.000
I knew that in advance which is why I asked that question.

s276
00:23:56.000 --> 00:24:08.720
Uh so if we look in our traces, we should then see coming in when I click the live button, show me old toys under $6.

s277
00:24:08.720 --> 00:24:09.360
Fantastic.

s278
00:24:09.360 --> 00:24:11.200
And you can see that it fails.

s279
00:24:11.200 --> 00:24:17.360
So what are you still doing?

s280
00:24:18.640 --> 00:24:19.360
It escaped.

s281
00:24:19.360 --> 00:24:20.080
It's fine.

s282
00:24:20.080 --> 00:24:22.559
Whatever you were doing, it was not necessary.

s283
00:24:22.559 --> 00:24:23.360
All right.

s284
00:24:23.360 --> 00:24:24.640
So that was it.

s285
00:24:24.640 --> 00:24:25.840
We added observability.

s286
00:24:25.840 --> 00:24:27.679
Suddenly traces are flowing into our application.

s287
00:24:27.679 --> 00:24:29.039
We did not write a single line of code.

s288
00:24:29.039 --> 00:24:32.480
In fact, you don't even know what agent framework I used to put this thing together.

s289
00:24:32.480 --> 00:24:34.960
Uh the whole thing is working and it's sending us traces.

s290
00:24:34.960 --> 00:24:37.600
And you can now do phase one observability.

s291
00:24:37.600 --> 00:24:40.320
you can look at your traces.

s292
00:24:41.120 --> 00:24:49.600
So, let's begin to close the loop by asking our agent to tell us how do we get to the next phase.

s293
00:24:49.600 --> 00:25:02.240
Uh, can you pull my traces and look for things that are going wrong?

s294
00:25:06.640 --> 00:25:12.320
So, a default like off-the-shelf coding agent isn't going to know this.

s295
00:25:12.320 --> 00:25:16.640
However, when you installed the repo, you installed a whole bunch of skills.

s296
00:25:16.640 --> 00:25:23.360
Uh, so it should know from those skills uh where to do that.

s297
00:25:23.360 --> 00:25:29.600
Except I'm in Oh, yeah.

s298
00:25:29.600 --> 00:25:30.559
Oh, it's working.

s299
00:25:30.559 --> 00:25:32.320
Okay, there we go.

s300
00:25:32.320 --> 00:25:34.080
Cool.

s301
00:25:34.080 --> 00:25:36.080
Uh

s302
00:25:37.302 --> 00:25:39.302
[snorts]

s303
00:25:47.679 --> 00:25:52.000
this is where I would hit yes to all and my head of security would have a conion to me.

s304
00:25:52.000 --> 00:26:01.279
Uh so what I'm asking it to do is use the skills that I gave it uh to pull down my traces uh from AX directly.

s305
00:26:01.279 --> 00:26:02.400
It has the API key.

s306
00:26:02.400 --> 00:26:05.360
It has this it has the uh endpoint.

s307
00:26:05.360 --> 00:26:16.799
So there's an AX command line uh which it's using to do all of this stuff and it's pulling down the same traces that we were looking at in the UI which it can expect uh inspect programmatically.

s308
00:26:19.120 --> 00:26:22.080
It's summarizing span kinds statuses and errors.

s309
00:26:22.080 --> 00:26:23.520
This is great.

s310
00:26:23.520 --> 00:26:34.000
Uh again, the coding agent uh knows exactly what's going on here because we've installed the skills already.

s311
00:26:34.000 --> 00:26:38.320
The skills are what are giving it all the information here.

s312
00:26:38.559 --> 00:26:46.640
It's looking into quality issues, which is exactly what I wanted it to do because there were no error spans.

s313
00:26:46.640 --> 00:26:50.880
There were just a bunch of spans where it returned 200 and said things are fine.

s314
00:26:50.880 --> 00:27:04.720
uh and it has already found this specific thing that I was uh asking it to look for the search products tool suspiciously g suspiciously returned in one millisecond.

s315
00:27:04.720 --> 00:27:10.480
So it's going to find some quality some poor quality signals and tell us about them.

s316
00:27:25.520 --> 00:27:28.880
Who's got this working so far?

s317
00:27:29.679 --> 00:27:32.080
Who's still cloning the repo?

s318
00:27:32.080 --> 00:27:34.080
Oh,

s319
00:27:37.760 --> 00:27:46.880
no. the plug the the skills are sitting in a directory called aagents/skills and your coding which coding agent are you using?

s320
00:27:49.679 --> 00:27:51.279
It's actually in the root directory.

s321
00:27:51.279 --> 00:27:57.200
So you might want to open your agent on the root directory and then cd down into that one.

s322
00:27:58.880 --> 00:28:06.320
Yes, npx skills uh install arise-ai installs all of these skills directly.

s323
00:28:06.399 --> 00:28:09.039
Sorry, what were you saying?

s324
00:28:13.200 --> 00:28:15.200
huge.

s325
00:28:19.744 --> 00:28:21.744
[laughter]

s326
00:28:22.320 --> 00:28:34.480
And so this is mysterious because there used to be a version of this application where you had to sign in with X and I changed it so that you don't have to sign in with X. As you can see, when I ran the S application, I didn't have to sign in with X.

s327
00:28:34.480 --> 00:28:42.480
Somehow it has dug up the version where you sign up in X. I am completely confused as to how that is happening.

s328
00:28:42.480 --> 00:28:43.760
Code is mysterious.

s329
00:28:43.760 --> 00:28:48.880
I will have to talk to you more afterwards about exactly how you got there.

s330
00:28:48.880 --> 00:28:54.080
All right, so Claude has finally got its together and told me what's going on in the traces.

s331
00:28:54.080 --> 00:28:57.679
Uh 42% of searches have returned zero results.

s332
00:28:57.679 --> 00:28:59.120
Degenerate all null searches.

s333
00:28:59.120 --> 00:29:05.039
The agent curled search product with every argument null producing a meaningless search that returns the first product in the catalog.

s334
00:29:05.039 --> 00:29:10.399
Min age equals max m age max age exact age filtering o over narrows.

s335
00:29:10.399 --> 00:29:13.520
Uh the trace stack calls three agent kinds.

s336
00:29:13.520 --> 00:29:16.080
The headline issue is the search products filter logic.

s337
00:29:16.080 --> 00:29:22.320
Vector search and keyword category filters frequently produce empty results when model supplies the fixes on the tool side.

s338
00:29:22.320 --> 00:29:29.760
Uh, when I tried this this morning, I only had one trace and that trace was it not finding the tra not finding the prices.

s339
00:29:29.760 --> 00:29:37.200
So, is there a problem with filtering by price?

s340
00:29:37.279 --> 00:29:41.760
I'll see if it can find that in the traces that it's already got.

s341
00:29:43.360 --> 00:29:49.760
But spoiler alert, even if it doesn't find them, we are going to get it to add the feature where it filters by prices.

s342
00:30:13.760 --> 00:30:15.840
All right.

s343
00:30:28.320 --> 00:30:29.360
Exactly.

s344
00:30:29.360 --> 00:30:30.799
doesn't have price filtering at all.

s345
00:30:30.799 --> 00:30:32.799
Cool.

s346
00:30:34.240 --> 00:30:39.039
Let's add price filtering to this app.

s347
00:30:53.016 --> 00:30:54.799
[laughter]

s348
00:30:54.799 --> 00:30:59.440
One of the things that can happen when you have agents being clever is that agents can be too clever.

s349
00:30:59.440 --> 00:31:08.720
This agent has noticed that I have a copy of this app sitting right next to it where I already solved this problem and it is just copying the code over from the pre-solved version of the app.

s350
00:31:08.720 --> 00:31:15.200
That's not what it did this morning because it didn't have the solution sitting in a directory this morning, but it's still going to work.

s351
00:31:15.200 --> 00:31:17.360
So, we're just going to let it do that.

s352
00:31:17.360 --> 00:31:24.000
We're going to let it be clever and copy the solution from the previous solution that it built this morning because why the hell not, Claude?

s353
00:31:24.000 --> 00:31:25.679
Thank you very much.

s354
00:31:25.679 --> 00:31:26.640
All right.

s355
00:31:26.640 --> 00:31:31.679
Uh, the cool thing about all of this stuff is that again, I haven't touched the code at all, right?

s356
00:31:31.679 --> 00:31:33.840
All I've done is said that hey, look for problems.

s357
00:31:33.840 --> 00:31:34.559
It has found a problem.

s358
00:31:34.559 --> 00:31:36.960
I've said, hey, what about that problem in particular?

s359
00:31:36.960 --> 00:31:38.880
Can you fix that problem in particular?

s360
00:31:38.880 --> 00:31:40.720
And it's like, yes, I'm going to do that.

s361
00:31:40.720 --> 00:31:43.039
That is the joy of agentic coding.

s362
00:31:43.039 --> 00:31:48.559
uh brilliantly because it was copying the solution from somebody else already.

s363
00:31:48.559 --> 00:31:50.000
Uh it's already done.

s364
00:31:50.000 --> 00:31:54.720
So let's go back to Wonder Toys.

s365
00:31:56.960 --> 00:32:02.000
Show me all toys under $9.

s366
00:32:07.039 --> 00:32:08.399
And tada.

s367
00:32:08.399 --> 00:32:10.559
A round of applause for me, I guess.

s368
00:32:10.559 --> 00:32:12.799
Thank you.

s369
00:32:13.582 --> 00:32:14.399
[applause]

s370
00:32:14.399 --> 00:32:16.080
or really for clone.

s371
00:32:16.080 --> 00:32:25.519
Uh so now if we look at our traces, we can see show me all toys under $9.

s372
00:32:26.880 --> 00:32:33.600
And we've got our input and our output in the response including all those toys under $9.

s373
00:32:33.600 --> 00:32:36.640
Uh this is what we were trying to get done.

s374
00:32:36.640 --> 00:32:50.320
We were trying to close the loop from uh we have no observability into this application at all to we have observability, we know what's going on to uh find the problems automatically

s375
00:32:50.320 --> 00:32:54.559
to fix the problems automatically and we have closed this loop.

s376
00:32:54.559 --> 00:33:02.399
But what if uh you could go a little bit further than that?

s377
00:33:05.360 --> 00:33:08.559
Uh what if you didn't have to do the asking?

s378
00:33:08.559 --> 00:33:17.360
What if your system just automatically detected that there were pro issues with your code, suggested fixes and decided to fix it itself?

s379
00:33:17.360 --> 00:33:20.960
That is the question that we asked ourselves.

s380
00:33:20.960 --> 00:33:25.679
Uh and that is when we built this thing which is called signal.

s381
00:33:25.679 --> 00:33:34.000
Uh signal is an agent that runs inside of Arise AX and continuously monitors all of your traces.

s382
00:33:34.000 --> 00:33:37.279
It runs every six hours by default, but that's configurable.

s383
00:33:37.279 --> 00:33:44.000
Uh, and it does what we were just doing manually in cloud code, but it does it automatically inside of AX continuously.

s384
00:33:44.000 --> 00:33:49.919
It reads your traces, it looks for patterns, it finds things that are wrong, and it suggests fixes.

s385
00:33:49.919 --> 00:33:55.039
So, luckily, I built this thing more than six hours ago, so it's already run.

s386
00:33:55.039 --> 00:34:00.880
Uh, and it has found uh a bunch of problems in previous versions of this application that I ran.

s387
00:34:00.880 --> 00:34:06.320
So, uh, the agent hallucinates an age constraint from vague user phrasing.

s388
00:34:06.320 --> 00:34:07.360
That's nice.

s389
00:34:07.360 --> 00:34:09.440
Uh, harmful query.

s390
00:34:09.440 --> 00:34:14.960
I asked it if it could give me instructions on, uh, installing a, uh, creating a bomb.

s391
00:34:14.960 --> 00:34:24.399
Um, it told me that it couldn't, but, it recommended some bomb related toys, and Signal is like, "No, just ignore the bomb queries entirely.

s392
00:34:24.399 --> 00:34:25.839
Don't try to be helpful.

s393
00:34:25.839 --> 00:34:28.320
Uh, we don't have any bomb related toys."

s394
00:34:28.320 --> 00:34:37.839
Uh, it also told me that if I was feeling violent, maybe I should take a timeout, which is again probably a domain escape, like you shouldn't be giving my psychological

s395
00:34:37.839 --> 00:34:38.879
advice.

s396
00:34:38.879 --> 00:34:42.720
Uh, we should probably have an email about that and better guard rails.

s397
00:34:42.720 --> 00:34:47.599
Uh, it also noticed a test that I gave it earlier where I just said say hello.

s398
00:34:47.599 --> 00:34:50.560
Uh, and it said hello and it's like you shouldn't do that.

s399
00:34:50.560 --> 00:34:51.359
You're a toy.

s400
00:34:51.359 --> 00:34:52.399
You're a toy store.

s401
00:34:52.399 --> 00:34:56.480
You shouldn't obey random instructions that somebody on the internet gives you.

s402
00:34:56.480 --> 00:34:58.240
uh all of which are fine.

s403
00:34:58.240 --> 00:35:02.960
Um and what it does here is it gives you uh a whole bunch of options.

s404
00:35:02.960 --> 00:35:10.720
Um it will uh by default it will create a a GitHub issue for you completely describing the problem.

s405
00:35:10.720 --> 00:35:16.160
It will say exactly what trace went wrong, how to find it, where to fix it, uh what it thinks you should do.

s406
00:35:16.160 --> 00:35:21.440
Uh you can take you can take this trace and add it to a data set so that you can run evals on it.

s407
00:35:21.440 --> 00:35:26.320
You can create an evaluator based on uh the problem that it has found.

s408
00:35:26.320 --> 00:35:31.440
Uh and the most fun one is that you can open a PR if I've connected my repo which I haven't yet.

s409
00:35:31.440 --> 00:35:41.760
Uh you can open a PR directly in which it will have read your repo uh suggested a fix and created a PR that automatically fixes this problem.

s410
00:35:41.760 --> 00:35:52.480
So instead of you having to read traces or you having to read eval uh signal gives you a bunch of PRs that you can approve or uh you can approve or reject

s411
00:35:52.480 --> 00:35:55.280
uh that resolve the problem for you.

s412
00:35:55.280 --> 00:36:03.040
Um so this probably raises some questions.

s413
00:36:03.040 --> 00:36:06.560
Who's got questions about signal?

s414
00:36:06.880 --> 00:36:08.880
Yes.

s415
00:36:14.560 --> 00:36:17.920
Is there a version that will keep the traces local on device?

s416
00:36:17.920 --> 00:36:21.119
Um there is not right now.

s417
00:36:21.119 --> 00:36:24.960
The uh I assume you're asking for sort of compliance reasons.

s418
00:36:24.960 --> 00:36:38.079
Um the compliance story with Arise is we have an on-prem version where it will run entirely on your uh your company's hardware uh and and uh activate from there.

s419
00:36:38.880 --> 00:36:40.880
Yeah.

s420
00:36:49.760 --> 00:36:52.560
Sorry, could you be louder?

s421
00:36:55.119 --> 00:36:55.839
Ah.

s422
00:36:55.839 --> 00:37:03.760
Um, because I only have a few traces in this application, it's looking at the very few traces that I have and it's going on based on one or two traces.

s423
00:37:03.760 --> 00:37:10.800
If you had 10,000 traces, it would be grouping them based on a 100 traces, a thousand traces, that kind of stuff.

s424
00:37:13.200 --> 00:37:15.839
Oh, there.

s425
00:37:21.599 --> 00:37:22.560
How do you know what?

s426
00:37:22.560 --> 00:37:24.560
Sorry.

s427
00:37:33.599 --> 00:37:34.800
Ah yeah.

s428
00:37:34.800 --> 00:37:37.280
Um so the solution here is eval.

s429
00:37:37.280 --> 00:37:48.960
You'll be surprised to learn uh a a production system should have a set of regression evals which are things that you are expecting uh your agent to already be good at.

s430
00:37:48.960 --> 00:37:53.040
So your evals already in place which you can uh create them.

s431
00:37:53.040 --> 00:37:54.880
Uh I'll show you in a second.

s432
00:37:54.880 --> 00:38:03.440
uh with your evals already in place, you will know that your system doesn't change from the good behavior that it previously had as a result of this change.

s433
00:38:03.440 --> 00:38:07.280
Uh if you don't have any evals, I can show you how that's done.

s434
00:38:07.280 --> 00:38:16.640
Uh which is we can go to the wonder toys project uh and ask it to build an eval.

s435
00:38:21.680 --> 00:38:33.760
build an eval that ensures that I'm never giving instructions about how to build a bomb.

s436
00:38:34.240 --> 00:38:39.200
That seems like a really basic eval that I should have had in from the beginning.

s437
00:38:39.200 --> 00:38:42.480
Uh this thing that I'm using right now is called Alex.

s438
00:38:42.480 --> 00:38:44.000
It is our in agent assistant.

s439
00:38:44.000 --> 00:38:46.560
It is our uh in product agent.

s440
00:38:46.560 --> 00:38:49.920
Uh and it knows like the skills do.

s441
00:38:49.920 --> 00:38:51.280
It knows how to build everything.

s442
00:38:51.280 --> 00:39:01.359
So uh you can do this in product, you can do it in the skills in your co coding agent, but either way it will create an evaluator for you uh and run automatically.

s443
00:39:11.440 --> 00:39:14.400
And that is pretty much it as far as we're going.

s444
00:39:14.400 --> 00:39:16.240
Question at the back.

s445
00:39:16.240 --> 00:39:19.280
What harness is signal?

s446
00:39:19.280 --> 00:39:21.599
Uh, what harness is signal?

s447
00:39:21.599 --> 00:39:24.400
Uh, I don't know what harness is signal.

s448
00:39:24.400 --> 00:39:26.800
You know, Jason,

s449
00:39:30.640 --> 00:39:31.280
yes.

s450
00:39:31.280 --> 00:39:36.160
Uh, signal is uh clawed code at the moment and you can bring your own later.

s451
00:39:36.160 --> 00:39:42.800
Uh, that is my boss trolling me in case you're wondering why he happens to know the answer to my question.

s452
00:39:43.599 --> 00:39:45.040
Any other questions about signal?

s453
00:39:45.040 --> 00:39:47.040
Yes.

s454
00:39:55.440 --> 00:39:55.839
Sorry.

s455
00:39:55.839 --> 00:39:57.839
Louder.

s456
00:40:05.680 --> 00:40:08.960
Uh at the moment the the question is about the cost of running signal.

s457
00:40:08.960 --> 00:40:11.359
At the moment uh we eat the cost of signal.

s458
00:40:11.359 --> 00:40:17.760
the uh solute the aim is uh for to for you to get the best product experience.

s459
00:40:17.760 --> 00:40:27.680
Right now uh we can't guarantee that it's always going to be that way uh but at the moment uh signal is run on our agents at our dime.

s460
00:40:29.119 --> 00:40:30.400
Yep.

s461
00:40:30.400 --> 00:40:33.599
Can you talk us through how signals?

s462
00:40:33.760 --> 00:40:34.800
That is an excellent question.

s463
00:40:34.800 --> 00:40:37.440
How is Signal How is Signal evaluated?

s464
00:40:37.440 --> 00:40:48.640
uh signal is a has its own eval suite uh that uh runs internally uh and is traced using arise.

s465
00:40:48.640 --> 00:41:01.359
So everything that signal is doing is sending its own telemetry back to us uh unless you opt out of it uh which is telling us uh the queries that it's getting what it's doing how it's doing all of that stuff.

s466
00:41:01.359 --> 00:41:10.480
We have powerful LLM judges looking at the results of Signal uh and telling you whether or not uh Signal did a good job generating that PR or not.

s467
00:41:10.480 --> 00:41:18.160
Uh so exactly the same software development life cycle that I was just talking about uh applied to Signal itself.

s468
00:41:18.160 --> 00:41:18.960
Awesome question.

s469
00:41:18.960 --> 00:41:20.400
Thank you.

s470
00:41:20.400 --> 00:41:22.880
Gets us to show off.

s471
00:41:22.880 --> 00:41:24.880
Yep.

s472
00:41:28.560 --> 00:41:30.560
Signal.

s473
00:41:33.280 --> 00:41:36.079
Uh does Signal automatically create its own evaluators?

s474
00:41:36.079 --> 00:41:36.800
Not right now.

s475
00:41:36.800 --> 00:41:41.920
Uh at the moment you have to you have to because evaluators you know take time and money to run.

s476
00:41:41.920 --> 00:41:48.560
You want to uh you want to get the suggestion and then you want to create the eval.

s477
00:41:51.280 --> 00:41:51.839
All right.

s478
00:41:51.839 --> 00:41:54.400
Uh and I think I'm going to leave it at that.

s479
00:41:54.400 --> 00:41:57.520
Thank you all for your time and attention.

s480
00:41:57.849 --> 00:41:59.849
[applause]