WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=_zUp6y9bDU8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/_zUp6y9bDU8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.880 --> 00:00:14.000
Good afternoon everyone.

s2
00:00:14.000 --> 00:00:15.040
My name is Encore.

s3
00:00:15.040 --> 00:00:17.760
Uh I'm going to be teaching you how to make your agent cook.

s4
00:00:17.760 --> 00:00:20.240
Again, not food related.

s5
00:00:20.240 --> 00:00:20.800
I don't know.

s6
00:00:20.800 --> 00:00:22.960
Uh but yeah, we're going to get into it.

s7
00:00:22.960 --> 00:00:29.039
Um so what I mean by that is actually before we get into that, let's go ahead and start with a quick question and a poll with everyone.

s8
00:00:29.039 --> 00:00:32.960
Um, I'm sure a lot of you are building agents or have even shipped agents.

s9
00:00:32.960 --> 00:00:36.399
I'm just curious how many of you currently are shipping based off vibes.

s10
00:00:36.399 --> 00:00:38.239
And when I say vibes, it's not a bad thing.

s11
00:00:38.239 --> 00:00:40.640
I mean like, hey, you're checking your answers.

s12
00:00:40.640 --> 00:00:41.440
You're looking at it.

s13
00:00:41.440 --> 00:00:45.440
You're making sure that it's potentially doing the right thing just based on the input and output.

s14
00:00:45.440 --> 00:00:47.280
You're making sure that it's correct or not.

s15
00:00:47.280 --> 00:00:50.160
How many of you all are doing that?

s16
00:00:50.239 --> 00:00:53.120
Okay, pretty good reasonable amount of you.

s17
00:00:53.120 --> 00:01:02.399
And then how many of you guys actually have like set eval set up like you know like oh you have LM as a judge or codebased evals or some sort of eval setup.

s18
00:01:03.280 --> 00:01:04.400
Okay, a little less.

s19
00:01:04.400 --> 00:01:05.439
Not too much less though.

s20
00:01:05.439 --> 00:01:07.119
That's pretty good actually.

s21
00:01:07.119 --> 00:01:12.159
And then how many of you that actually have your eval setup trust it enough that you know you're good enough?

s22
00:01:12.159 --> 00:01:18.400
You have a full coverage of your evals and you're ready to you're continuously ready to keep going and moving forward.

s23
00:01:18.400 --> 00:01:20.080
Wow, that's that's pretty impressive.

s24
00:01:20.080 --> 00:01:23.759
That was a solid hand up, which is kind of what I expected.

s25
00:01:23.759 --> 00:01:28.000
I mean, it's very hard to really trust your emails because they're hard to build.

s26
00:01:28.000 --> 00:01:30.000
You don't really know what you're looking for.

s27
00:01:30.000 --> 00:01:31.600
It's not a deterministic use case.

s28
00:01:31.600 --> 00:01:36.880
And it's very hard to determine what we need to actually evaluate to make sure our agents are working properly.

s29
00:01:36.880 --> 00:01:42.479
And that's kind of one of the biggest issues is a lot of people are going to start off in a very good spot, which is based off vibes.

s30
00:01:42.479 --> 00:01:43.280
It gets you going.

s31
00:01:43.280 --> 00:01:45.920
It makes sure that your agent's doing what you expect it to.

s32
00:01:45.920 --> 00:01:49.680
But realistically, you can't really ship based on this feeling of, "Yeah, it's correct."

s33
00:01:49.680 --> 00:02:02.320
Because when you're in production, you're not going to know where things are broken, what's going wrong, unless you have actual eval set up to find out where things are breaking and how you can actually directly fix it.

s34
00:02:02.880 --> 00:02:13.360
So before we get into eval, what I want to kind of take is a whole step back and talk about how you can like what your eval would be on top of.

s35
00:02:13.360 --> 00:02:15.280
And that would be on top of traces.

s36
00:02:15.280 --> 00:02:23.200
And why we need traces is if you've ever used traditional software engineering, you have something that you do call traces.

s37
00:02:23.200 --> 00:02:24.400
It's for APM monitoring.

s38
00:02:24.400 --> 00:02:27.280
It's making sure your systems are reliable and are working.

s39
00:02:27.280 --> 00:02:34.879
But realistically, what you you can actually go ahead and debug those and it's very easy to do because you have a very deterministic flow, right?

s40
00:02:34.879 --> 00:02:39.519
You can even see right here, this is very easy to follow and this is more or less what like code looks like.

s41
00:02:39.519 --> 00:02:41.680
You have a handle submit method.

s42
00:02:41.680 --> 00:02:42.959
You check if it's valid.

s43
00:02:42.959 --> 00:02:44.160
If it doesn't, it errors out.

s44
00:02:44.160 --> 00:02:47.120
If it is, then it goes down the next list and continues down that process.

s45
00:02:47.120 --> 00:02:54.800
So, it's very easy to figure out where things are b are breaking based on a just based on that sort of uh path history.

s46
00:02:54.800 --> 00:03:02.159
But in an agent, unfortunately, with your code, really all you have is that one line of code, which is just going to be agent.run and then the query.

s47
00:03:02.159 --> 00:03:05.040
And ultimately, what it's going to do is kind of up to it.

s48
00:03:05.040 --> 00:03:06.000
Will it call a tool?

s49
00:03:06.000 --> 00:03:07.840
Will it go ahead and do some analysis?

s50
00:03:07.840 --> 00:03:09.760
Will it do some internal thinking?

s51
00:03:09.760 --> 00:03:12.159
What is its next best step to do?

s52
00:03:12.159 --> 00:03:13.760
and its actual best course of action.

s53
00:03:13.760 --> 00:03:16.879
We don't know until we actually see the traces.

s54
00:03:16.879 --> 00:03:18.720
You won't know what your agent's doing.

s55
00:03:18.720 --> 00:03:22.159
So that trace kind of helps you define that.

s56
00:03:22.159 --> 00:03:28.239
And so as a kind of a decent example as to what a trace would look like would be this.

s57
00:03:28.239 --> 00:03:30.319
You have your agent call.

s58
00:03:30.319 --> 00:03:33.040
You go ahead and see the LM calls.

s59
00:03:33.040 --> 00:03:36.799
You have your documents you're retrieving, your tools that you're retrieving.

s60
00:03:36.799 --> 00:03:42.879
You basically have every step along your journey that you would see of what your agent's doing.

s61
00:03:42.879 --> 00:03:49.040
And here you can even go ahead and see, hey, we have for example this tool.get order status.

s62
00:03:49.040 --> 00:03:50.799
It's the wrong tool occult.

s63
00:03:50.799 --> 00:03:54.319
The user asked us, hey, I was supposed to get a refund here.

s64
00:03:54.319 --> 00:03:55.680
Where's my refund?

s65
00:03:55.680 --> 00:04:01.040
If we respond back, hey, your refund was issued and they know that that was their status based on their order status.

s66
00:04:01.040 --> 00:04:02.159
That's not what they were looking for.

s67
00:04:02.159 --> 00:04:05.280
They were looking to understand why hasn't their refund arrived?

s68
00:04:05.280 --> 00:04:12.319
So we need to potentially call different tools, do different actions, and there's so many different varieties even just knowing that there's an issue here.

s69
00:04:12.319 --> 00:04:15.599
But we'll dive into that in a little bit.

s70
00:04:15.680 --> 00:04:20.239
So when you actually go ahead and understand, okay, we have a trace.

s71
00:04:20.239 --> 00:04:27.840
Now I want to go ahead and debug what's going wrong with my agent, we're going to start with where first you're going to identify those issues, right?

s72
00:04:27.840 --> 00:04:29.919
Those production failures like we just did in that last slide.

s73
00:04:29.919 --> 00:04:31.759
We found out that tool was wrong.

s74
00:04:31.759 --> 00:04:33.040
But what's your next step?

s75
00:04:33.040 --> 00:04:42.560
You want to go ahead and find out, okay, not only where did it go wrong, can I get a nice sample size of that sort of information that I can then use for experimentation

s76
00:04:42.560 --> 00:04:47.520
and iteration to improve the agent when I actually want to go back and test against those cases.

s77
00:04:47.520 --> 00:04:50.400
Then what actually was the issue, right?

s78
00:04:50.400 --> 00:04:53.840
Sure, the tool call was wrong, but should the LM prompt change?

s79
00:04:53.840 --> 00:04:55.840
Should we add in guardrails?

s80
00:04:55.840 --> 00:04:57.040
Should we change up the tool?

s81
00:04:57.040 --> 00:04:58.720
Was the tool just doing the wrong things?

s82
00:04:58.720 --> 00:04:59.040
Right?

s83
00:04:59.040 --> 00:05:02.240
There's so many different areas that we need to test and go against.

s84
00:05:02.240 --> 00:05:08.880
And so we can go and test against those build out evaluations to continuously monitor those use cases.

s85
00:05:08.880 --> 00:05:14.720
So for example, if you find out, hey, we need to make sure that we set up a guardrail here.

s86
00:05:14.720 --> 00:05:16.160
How are we going to do that?

s87
00:05:16.160 --> 00:05:18.639
We want to and then we want to make sure that guardrail is doing its job.

s88
00:05:18.639 --> 00:05:27.600
So we set up a evaluator on top of that guardrail making sure is that guardrail doing its job by prohibiting certain information from leaking.

s89
00:05:27.600 --> 00:05:30.560
Finally, you actually make those changes.

s90
00:05:30.560 --> 00:05:38.400
run that experiment that I was talking about in the beginning and finally ship it to production and do that whole loop over and over again continuously.

s91
00:05:38.400 --> 00:05:43.600
Now, this is your standard go-to process for actually improving your agent.

s92
00:05:43.600 --> 00:05:47.280
It's pretty long, pretty manual, and pretty tedious.

s93
00:05:47.280 --> 00:05:51.039
It does it does work, but we want to kind of iterate and improve that.

s94
00:05:51.039 --> 00:05:52.800
How can we do that?

s95
00:05:52.800 --> 00:06:04.880
We want to make sure that we're not spending hours, days finding the issue, finding the trace, finding all these different issue points, coming up with the potential solution, going through that full eight-step process manually.

s96
00:06:04.880 --> 00:06:08.800
We want to pull all of that together and bring it into minutes.

s97
00:06:08.800 --> 00:06:11.600
The best way to do that would be right here.

s98
00:06:11.600 --> 00:06:16.400
We're going to pull in all the data from all our different sources into one specific area.

s99
00:06:16.400 --> 00:06:24.720
We're going to start with, you know, pulling our ap our arise traces so you can get all the information of what your agent did.

s100
00:06:24.720 --> 00:06:29.680
You're going to pull in your evaluations to see where things actually went wrong.

s101
00:06:29.680 --> 00:06:39.280
You're going to bring in your annotations from any humans that could theoretically tell you a little bit more information as to, you know, where it could go right, how they can improve it.

s102
00:06:39.280 --> 00:06:46.080
You bring in your actual APM traces which could be hey is the system up or down.

s103
00:06:46.080 --> 00:06:54.400
you bring in your codebase so you can understand all right based on this specific area where it failed the tool call what changes can I make in my codebase

s104
00:06:54.400 --> 00:07:09.840
so that we're c we're we're going down the right pathways we're calling the right tools we're fixing our tools that are broken and so you'll bring in all of these into one final space which is which is our coding agent whether you use cloud code cursor codeex it doesn't matter

s105
00:07:09.840 --> 00:07:17.360
all of those you can connect all that data into and it can automatically do all of that work for for you.

s106
00:07:17.360 --> 00:07:25.360
So, I know this may sound really complicated and I promise this also probably doesn't help to make it look simpler, but it is very simple to get it set up.

s107
00:07:25.360 --> 00:07:32.639
It's going to be just a couple steps of just, you know, setting up your CLI and skills and you're ready to go.

s108
00:07:32.800 --> 00:07:42.479
You get access to a bunch of information, a bunch of different tools that your agent can then do, which could be for example here, which are like, hey, you can get your code instrumented.

s109
00:07:42.479 --> 00:07:48.800
What that means is you can get all the traces from your agent into whatever platform you choose like Arise.

s110
00:07:48.800 --> 00:07:55.120
You could go ahead and pull down those traces, build out data sets, run experiments, run evaluations.

s111
00:07:55.120 --> 00:08:00.400
You have the full gamut of the experience all from within your coding agent.

s112
00:08:01.039 --> 00:08:06.960
So, what I'm going to do here is we're going to go ahead and look at a specific issue.

s113
00:08:06.960 --> 00:08:17.680
And I want a little bit of kind of uh voices from this side of the room uh where we can get some idea of like hey what do you guys think could be the actual issue.

s114
00:08:17.680 --> 00:08:19.599
So I'll give you a little bit of background.

s115
00:08:19.599 --> 00:08:22.800
This agent right here is a simple financial agent.

s116
00:08:22.800 --> 00:08:25.520
I've asked it should I invest in this company?

s117
00:08:25.520 --> 00:08:27.520
I think I chose Tata Motors.

s118
00:08:27.520 --> 00:08:33.279
Um anyways and ultimately it gave me a response which said yes you should.

s119
00:08:33.279 --> 00:08:37.599
I don't think it's directly in the screenshot, but I promise you it told me that.

s120
00:08:37.599 --> 00:08:47.360
Um, and ultimately that's not really what we want to do because I'm not registered to give any financial advice, nor is this agent, and I don't want to get in legal trouble.

s121
00:08:47.360 --> 00:08:52.240
If you guys can help me figure out maybe, hey, what do you guys think could be the issue in this agent?

s122
00:08:52.240 --> 00:09:00.480
So, what she said was, "Hey, maybe the agent needs to ask a lot more questions back to get a little bit more information before I can give a solid, concrete answer."

s123
00:09:00.480 --> 00:09:01.680
That's a very good guess.

s124
00:09:01.680 --> 00:09:03.680
Anyone else have any ideas?

s125
00:09:03.680 --> 00:09:04.160
Okay.

s126
00:09:04.160 --> 00:09:06.160
And how do you suggest we potentially improve that?

s127
00:09:06.160 --> 00:09:08.800
By the way, he said it should ask, again, pretty similar.

s128
00:09:08.800 --> 00:09:14.320
It should ask more questions before making that recommendation, but how do you suggest maybe we can go ahead and do that?

s129
00:09:14.320 --> 00:09:17.519
More context, more asking questions, more details.

s130
00:09:17.519 --> 00:09:22.240
Okay, so maybe some prompt changes to add in some context to get more context.

s131
00:09:22.240 --> 00:09:24.480
One more.

s132
00:09:24.480 --> 00:09:25.200
Sorry.

s133
00:09:25.200 --> 00:09:27.519
Where's validation?

s134
00:09:28.240 --> 00:09:29.200
Yeah, that's true.

s135
00:09:29.200 --> 00:09:31.680
We don't have any sort of ground truth to compare against.

s136
00:09:31.680 --> 00:09:33.680
We don't know if any of this is correct.

s137
00:09:33.680 --> 00:09:36.800
So there's no validation that any of the answers are even correct.

s138
00:09:36.800 --> 00:09:37.279
You're right.

s139
00:09:37.279 --> 00:09:38.640
There could be so many different issues.

s140
00:09:38.640 --> 00:09:40.880
And truthfully, oh yeah, go ahead.

s141
00:09:40.880 --> 00:09:41.200
Question.

s142
00:09:41.200 --> 00:09:45.760
Does the agent know the user's financial background?

s143
00:09:46.080 --> 00:09:46.959
It's a great question.

s144
00:09:46.959 --> 00:09:48.480
In this case, no, it does not.

s145
00:09:48.480 --> 00:09:51.279
And yeah, it should definitely get a lot more information into that.

s146
00:09:51.279 --> 00:09:58.800
Uh he asked, by the way, should the user does the does the agent know any information about the user's financial background?

s147
00:09:58.800 --> 00:10:00.320
Um and yeah, it definitely does not.

s148
00:10:00.320 --> 00:10:02.720
And there's a lot of areas that this agent could be fixed.

s149
00:10:02.720 --> 00:10:07.279
And there's genuinely genuinely not one correct answer, right?

s150
00:10:07.279 --> 00:10:12.560
But and I genuinely don't even know what the right answer is to this because this agent is still messed up right now.

s151
00:10:12.560 --> 00:10:15.839
And we're going to go ahead and try to fix it today in this demo.

s152
00:10:15.839 --> 00:10:24.240
Um, and so what we're going to do is we're going to go ahead and set up the CLI and skills, get that set up so we can actually get our coding agent

s153
00:10:24.240 --> 00:10:33.680
to go ahead and understand what's going wrong with our to go ahead and pull down the traces, understand what's going wrong with our agent, look at the evaluations we already have run,

s154
00:10:33.680 --> 00:10:43.600
look at the annotations that exist, maybe even do some of its own analysis, and potentially come back with some future suggestions just like many of you have done today

s155
00:10:43.600 --> 00:10:45.839
for how to actually improve this agent.

s156
00:10:45.839 --> 00:10:48.959
So that hey, maybe we could have it ask more questions.

s157
00:10:48.959 --> 00:10:56.720
Maybe we could add a guard rail so it only responds to certain questions and doesn't give financial advice where it's not warranted.

s158
00:10:56.720 --> 00:11:00.959
So there's lots of different ways that it can come to do those fixes.

s159
00:11:01.040 --> 00:11:07.920
To get it started, what I'm going to do is I'm going to go ahead and pull open let's see, do I have it here?

s160
00:11:07.920 --> 00:11:08.640
Yes.

s161
00:11:08.640 --> 00:11:11.440
Let's go ahead and pull open a new session.

s162
00:11:11.440 --> 00:11:13.920
Um, okay, cool.

s163
00:11:13.920 --> 00:11:15.360
I'm in the right space.

s164
00:11:15.360 --> 00:11:16.160
All right.

s165
00:11:16.160 --> 00:11:21.120
So, generally what you'd want to do whenever you want to set up the CLI and skills, super easy.

s166
00:11:21.120 --> 00:11:23.519
It would just be a couple steps to get that set up.

s167
00:11:23.519 --> 00:11:25.440
Um I apologize.

s168
00:11:25.440 --> 00:11:26.560
Oh, here we go.

s169
00:11:26.560 --> 00:11:28.320
I know what I'm doing, guys.

s170
00:11:28.320 --> 00:11:30.079
Uh here we go.

s171
00:11:30.079 --> 00:11:32.399
So, we're going to start by setting up the CLI.

s172
00:11:32.399 --> 00:11:35.440
You would just go ahead and copy this one line of code to get that set up.

s173
00:11:35.440 --> 00:11:38.800
So, I'd go into back into my terminal and install that.

s174
00:11:38.800 --> 00:11:44.480
Of course, I already had that installed right before this, but just goes just very simple and quick to set up.

s175
00:11:44.480 --> 00:11:50.320
Then what you do is you would create your actual profile, which would help you connect it to wherever your data stored.

s176
00:11:50.320 --> 00:11:51.839
In this case, it would be arise.

s177
00:11:51.839 --> 00:12:01.040
And you create that profile so it can go ahead and pull in those traces and visual and get that sort of information from those evals and annotations.

s178
00:12:01.040 --> 00:12:05.200
Once you have that profile created, you'd go ahead and set up your skills.

s179
00:12:05.200 --> 00:12:08.639
You could do this in a multitude of ways depending on your coding agent.

s180
00:12:08.639 --> 00:12:16.160
For today, we're using Cloud Code, so I use the marketplace to set it up, but it's pretty easy to set up however you want.

s181
00:12:17.279 --> 00:12:21.680
From here, let's go ahead and we're just going to ask it.

s182
00:12:21.680 --> 00:12:25.440
And again, uh just as a warning, I am using dangerously skip permissions.

s183
00:12:25.440 --> 00:12:26.240
Please don't do this.

s184
00:12:26.240 --> 00:12:28.240
This is not actually good practice.

s185
00:12:28.240 --> 00:12:31.519
This is just for a demo, so I don't have to keep pressing yes.

s186
00:12:31.519 --> 00:12:37.680
Um, but we're going to go ahead and talk to Claude and have it go ahead and take a look at these traces.

s187
00:12:37.680 --> 00:12:48.639
So, I'm going to go ahead and see if I can find the prompt I asked earlier, which was just use this skill and pull down the traces from that specific agent.

s188
00:12:48.639 --> 00:12:53.120
Find out what issues exist that we can determine from any, like I said earlier, right?

s189
00:12:53.120 --> 00:12:57.360
What any issues that exist from any of the evals, any of the annotations that we've done.

s190
00:12:57.360 --> 00:13:05.279
do your own analysis and tell me what eval maybe I can build out in the future to make sure that I don't run into any of these issues again.

s191
00:13:05.279 --> 00:13:07.920
And so this is going to take a couple minutes.

s192
00:13:07.920 --> 00:13:19.120
Um, in the meantime, I'm going to go ahead and first go and show you kind of how those traces look and how you can actually start triaging it if you want to do it manually a little bit.

s193
00:13:19.120 --> 00:13:30.079
But while this is running in the background, let's go back into the actual uh run into the actual agent that has those sort of queries that we've asked.

s194
00:13:30.079 --> 00:13:32.320
So, we'll go ahead and ask it a pretty similar question.

s195
00:13:32.320 --> 00:13:38.560
Um, you know, tell me if I should invest into, let's say, Nvidia this time.

s196
00:13:38.560 --> 00:13:41.360
It's a little bit more of a well-known stock.

s197
00:13:41.360 --> 00:13:45.519
And we're going to get that and it's going to try to tell us some sort of information.

s198
00:13:45.519 --> 00:13:48.880
And as that data comes in, looks like I've run this earlier.

s199
00:13:48.880 --> 00:13:50.959
So, and you can see it's coming in.

s200
00:13:50.959 --> 00:14:02.320
Um, but from an earlier run, we can kind of see what it's done already where you can see that query we asked, you can see the output that it'll finally generate, which we'll see how similar it is in a minute from this previous run.

s201
00:14:02.320 --> 00:14:05.920
Um, and you can see all the steps that actually happened in your agent, right?

s202
00:14:05.920 --> 00:14:14.160
So, you can understand, hey, we had a supervisor agent, a financial data agent, you can see the LLM calls, you can see all your different tool calls.

s203
00:14:14.160 --> 00:14:22.720
you get a lot of information right away that you can start using just like we done in the initial slide to see where those issues could lie.

s204
00:14:22.720 --> 00:14:23.839
Now, let's see.

s205
00:14:23.839 --> 00:14:25.120
Did Claude finish up?

s206
00:14:25.120 --> 00:14:26.320
Looks like it's still running.

s207
00:14:26.320 --> 00:14:27.040
It's doing its thing.

s208
00:14:27.040 --> 00:14:27.839
We'll let it cook.

s209
00:14:27.839 --> 00:14:30.720
You know, it's kind of the point of this.

s210
00:14:30.720 --> 00:14:41.360
Um, while it's cooking, anyone have any questions as to like, you know, how you can set up any sort form evals, any sort of traces, anything along those lines?

s211
00:14:41.360 --> 00:14:47.920
Anyone wanted to learn a little bit more about Yeah.

s212
00:14:57.360 --> 00:14:58.079
Great question.

s213
00:14:58.079 --> 00:14:58.320
Yeah.

s214
00:14:58.320 --> 00:14:59.199
So, you would go ahead.

s215
00:14:59.199 --> 00:15:06.160
So, what it does, it actually goes ahead and downloads a lot of those traces um into the into your local system.

s216
00:15:06.160 --> 00:15:13.040
So, what it'll do is actually let me go ahead and show you what it how it actually goes ahead and accesses the system itself.

s217
00:15:13.040 --> 00:15:18.480
So when you give it access to these skills, what you're actually giving it access to is the CLI underneath.

s218
00:15:18.480 --> 00:15:24.000
And what I mean by that is you're going to have all these different CLIs that it can integrate with.

s219
00:15:24.000 --> 00:15:26.959
So you can go ahead and set up and find the data sets.

s220
00:15:26.959 --> 00:15:28.639
You can pull down data sets.

s221
00:15:28.639 --> 00:15:35.279
You can have it interact with the essentially interact with the UI where it can set up evaluations on the platform on your behalf.

s222
00:15:35.279 --> 00:15:41.920
So you can monitor continuously monitor when your agents are failing, build out data sets, run experiments.

s223
00:15:41.920 --> 00:15:46.000
It can basically do anything that you that you would be able to do in the platform.

s224
00:15:46.000 --> 00:15:52.399
Sometimes it'll download, sometimes it'll make API requests, but that's generally the uh idea here.

s225
00:15:52.399 --> 00:15:54.720
Great question.

s226
00:15:55.040 --> 00:15:59.759
Any other questions in regards to CLI tracing?

s227
00:16:02.399 --> 00:16:04.720
Okay, let's go ahead and see what Claude's done.

s228
00:16:04.720 --> 00:16:06.560
Oh, still crunching away.

s229
00:16:06.560 --> 00:16:10.079
Um, so as we can see here, let's let's walk through what Claude's done so far.

s230
00:16:10.079 --> 00:16:15.839
So, we can see that we've gone ahead and gotten the spans.

s231
00:16:15.839 --> 00:16:19.600
It's digging into the evaluations to figure out where those issues are.

s232
00:16:19.600 --> 00:16:25.279
Um, it looks like it's not quite there's definitely found one very big issue with some numeric groundedness.

s233
00:16:25.279 --> 00:16:30.240
I know someone mentioned that earlier as a potential issue or something related to that.

s234
00:16:30.240 --> 00:16:34.880
Um, we're also seeing some tool calls directly fail.

s235
00:16:34.880 --> 00:16:37.680
Um, and let's see.

s236
00:16:37.680 --> 00:16:41.839
We also see, yeah, so we're getting a lot of different errors.

s237
00:16:41.839 --> 00:16:43.440
Oh, looks like it just finished up.

s238
00:16:43.440 --> 00:16:50.880
And so you can see here we get a full list as to like, hey, what's going wrong with our agent and what we can do to kind of fix it.

s239
00:16:50.880 --> 00:17:02.959
So while it's finishing up that, you can see here one of the biggest alarming things right out of the gate is about 44% of our agent interactions are resulting in a bad summary.

s240
00:17:03.199 --> 00:17:04.559
That's really bad.

s241
00:17:04.559 --> 00:17:05.760
I did not know that.

s242
00:17:05.760 --> 00:17:07.679
So, we definitely need to fix that one.

s243
00:17:07.679 --> 00:17:12.319
Another big issue is Oh, did not mean to scroll there.

s244
00:17:12.319 --> 00:17:22.480
Another big issue is, hey, we're also just getting a lot of fabricated answers, which is just as bad as this the summarization being wrong.

s245
00:17:22.480 --> 00:17:25.199
So, it sounds like it may be related, it may not be.

s246
00:17:25.199 --> 00:17:27.280
We can kind of debug that.

s247
00:17:27.280 --> 00:17:33.039
We're also seeing a relatively high rate of us just redundantly calling tools.

s248
00:17:33.039 --> 00:17:35.919
It's a waste of our time, our resources, and our money.

s249
00:17:35.919 --> 00:17:39.600
So clearly, we're doing a lot of bad things that we need to go ahead and fix.

s250
00:17:39.600 --> 00:17:45.280
And that's just from the eval from Claude's own analysis.

s251
00:17:45.280 --> 00:17:49.520
It's able to even go ahead and figure out, hey, there's other issues that are coming along.

s252
00:17:49.520 --> 00:17:51.360
We're getting silent tool failures.

s253
00:17:51.360 --> 00:17:57.120
It's telling us it succeeded, but ultimately it's not giving us any actual responses.

s254
00:17:57.120 --> 00:18:02.000
We're getting the agent hallucinates when tool fails instead of disclosing it.

s255
00:18:02.000 --> 00:18:16.160
So again related but hey that means maybe we need to not just do uh you know some eval built on top of this to understand when the tool is failing but we also need to make some prompt based changes to make sure that the the agent doesn't hallucinate here

s256
00:18:16.160 --> 00:18:22.160
and you have a multitude of like different things that it'll go ahead and tell you as well as the evals it can build out.

s257
00:18:22.160 --> 00:18:32.240
So what we could do is we can actually go ahead and tell it, hey, let's go ahead and build number one, which was the tool success silent failure eval,

s258
00:18:32.240 --> 00:18:43.280
and fix numeric groundedness so that it'll make sure that whenever it does respond, it's always grounded to some sort of actual reference point and it's not making up these numbers.

s259
00:18:43.280 --> 00:19:00.640
But we're also going to head and make sure you test this against a data set uh against the spans that caused the failure initially using and then we're going to go ahead and use another Arise skill

s260
00:19:00.640 --> 00:19:07.919
which would be here the Arise uh data set skill, the Arise Oh, there we go.

s261
00:19:07.919 --> 00:19:15.120
Arise um evaluator skill as well as the Arise experiment skill.

s262
00:19:15.120 --> 00:19:22.480
So here it's going to be able to let me just go ahead and run let this run and build out evals.

s263
00:19:22.480 --> 00:19:23.280
All right.

s264
00:19:23.280 --> 00:19:31.520
So again I'm being pretty vague here but essentially I'm just telling it go ahead and make the changes that you recommended as well as go ahead and run these experiments for us.

s265
00:19:31.520 --> 00:19:42.799
So now we can actually compare against where it failed to make sure it passes against that right and so in this scenario we can see okay it's going to go through and take its time.

s266
00:19:42.799 --> 00:19:51.919
Um I'm going to go ahead and show you a previous run because again this will take some time and while we want to let it cook I don't want to have you guys staring at a screen where Claude's just discombobulating

s267
00:19:51.919 --> 00:19:54.240
or whatever it does in its free time.

s268
00:19:54.240 --> 00:20:05.919
Um, so let's go ahead and look at where I've actually gone ahead and asked Alex or sorry Cloud Code to go ahead and answer some of our previous issue that I ran into.

s269
00:20:05.919 --> 00:20:17.679
And one of those was related to hey, I've had some times where I'm trying to get some answers about uh stocks that aren't necessarily from the United States.

s270
00:20:17.679 --> 00:20:26.720
And so we're having errors where we're talking about like South Korea or looks like they're all South Korea based, but let's assume there's other answers here.

s271
00:20:26.720 --> 00:20:28.799
Maybe Japan or China and whatever, right?

s272
00:20:28.799 --> 00:20:35.280
And you want to see and you want to be able to understand, hey, I want to be able to compare stocks across different markets.

s273
00:20:35.280 --> 00:20:39.039
And clearly people are asking that, but we don't support it.

s274
00:20:39.039 --> 00:20:44.000
And so what Claude did, it went ahead found those areas where that issue existed.

s275
00:20:44.000 --> 00:20:49.120
Like I said, it looks like it was all South Korea here, but it found those areas.

s276
00:20:49.120 --> 00:20:54.320
It built out a data set for me, which I can then go ahead and test against to make sure it works.

s277
00:20:54.320 --> 00:20:56.799
And then it went ahead and ran experiments.

s278
00:20:56.799 --> 00:20:57.840
And it made a couple changes.

s279
00:20:57.840 --> 00:21:09.679
So, it started with the initial one, which was the baseline, which was buggy, which as we expected would result in 0% uh passing rate, which is what happened when we initially got the actual productionbased

s280
00:21:09.679 --> 00:21:11.679
traces.

s281
00:21:11.679 --> 00:21:18.480
It also made a quick fix where it normalized it by adding in that uh country name.

s282
00:21:18.480 --> 00:21:19.760
It didn't work.

s283
00:21:19.760 --> 00:21:20.159
Why?

s284
00:21:20.159 --> 00:21:25.200
Because the tool I was calling was also outdated and I needed to update to the stable version.

s285
00:21:25.200 --> 00:21:26.799
So then it made a third iteration.

s286
00:21:26.799 --> 00:21:33.039
And you can see that on this chart here as well where we were able to see, hey, the first iteration didn't work.

s287
00:21:33.039 --> 00:21:39.919
It went back, made a fix, and finally got us to the final iteration where we're getting about a 90% success rate.

s288
00:21:39.919 --> 00:21:41.600
Clearly still not 100%.

s289
00:21:41.600 --> 00:21:47.360
We still need to make some changes, but hey, we're doing a lot better than we were earlier.

s290
00:21:47.360 --> 00:21:56.559
So, this is like a great way to get you going so you can actually start from, hey, I know there's an issue to here's a fix.

s291
00:21:56.559 --> 00:21:58.880
Now, I know we have quite a bit more time.

s292
00:21:58.880 --> 00:22:00.080
I want to check in.

s293
00:22:00.080 --> 00:22:01.120
How is everyone doing?

s294
00:22:01.120 --> 00:22:01.840
How do you guys feel?

s295
00:22:01.840 --> 00:22:05.280
Do you guys have any questions about how you can go ahead and get this started?

s296
00:22:05.280 --> 00:22:07.280
Yeah.

s297
00:22:23.200 --> 00:22:24.320
Great question.

s298
00:22:24.320 --> 00:22:27.520
So his question was how where do the experiments actually run?

s299
00:22:27.520 --> 00:22:29.520
Where do the evaluations run?

s300
00:22:29.520 --> 00:22:31.120
Where does everything run?

s301
00:22:31.120 --> 00:22:34.000
Um so you do have a little bit of customization there.

s302
00:22:34.000 --> 00:22:43.120
In this scenario, uh what I did is I had the experiment run on my local machine, which is I think what will be the case most of the time.

s303
00:22:43.120 --> 00:22:49.520
And then what you what I did afterwards was I went or you can host it on a cloud or wherever you'd like to experiment on.

s304
00:22:49.520 --> 00:22:53.280
And then you can go ahead and actually run the evaluations on Arise itself.

s305
00:22:53.280 --> 00:23:00.400
So in this case, I built out LM as a judge, which would then go ahead and let's see if I can see the evaluator here itself.

s306
00:23:00.400 --> 00:23:04.720
So you can see we built out two different evaluators, a codebased and an LLM based.

s307
00:23:04.720 --> 00:23:07.200
I actually had Claude go ahead and do this.

s308
00:23:07.200 --> 00:23:09.360
It built it out for me in the platform.

s309
00:23:09.360 --> 00:23:15.120
And this was the final LLM as a judge because it actually kind of failed in getting the codebased data to set up.

s310
00:23:15.120 --> 00:23:16.799
So that's why we have two.

s311
00:23:16.799 --> 00:23:18.720
And so you can see the LM as a judge.

s312
00:23:18.720 --> 00:23:24.159
It went ahead and built it out for us with this prompt and got it going so that we can actually compare.

s313
00:23:24.159 --> 00:23:27.280
Hey, did the agent work properly?

s314
00:23:28.000 --> 00:23:28.960
Any other questions?

s315
00:23:28.960 --> 00:23:31.280
That was a really good question by the way.

s316
00:23:31.280 --> 00:23:33.280
Yeah.

s317
00:23:40.880 --> 00:23:42.400
Very very good question.

s318
00:23:42.400 --> 00:23:46.480
Um so eval can be set up however you like.

s319
00:23:46.480 --> 00:23:52.640
We have of course people can people can definitely use whatever sort of models they want.

s320
00:23:52.640 --> 00:24:07.760
No one wants to be limited to just claude or just open AAI especially in today's modern age where you can see AI models literally ship every other day and then you want to be like okay I want to use that one now right and also some people have a preference for slightly cheaper models some people have a

s321
00:24:07.760 --> 00:24:21.440
preference for making sure their evals are topofthe line because ultimately they know their agent's only going to be as good as their evals so they use the highest end models for their evals even if they're not necessarily using those in production so it kind of varies you have full customizable

s322
00:24:21.440 --> 00:24:22.559
control over cut.

s323
00:24:22.559 --> 00:24:31.760
And so what you can see here for example, you have a full list of whatever sort of model provider you want to use and you can connect those even if you want to use any sort of local

s324
00:24:31.760 --> 00:24:35.120
um open source models as well.

s325
00:24:35.120 --> 00:24:38.080
Yeah, good question.

s326
00:24:38.080 --> 00:24:48.480
So her question was if we have if I spin up a smaller agent to go ahead and make some of these bug fixes, how does this compare to that?

s327
00:24:48.480 --> 00:24:51.760
Realistically, you don't have to use cloud codeex.

s328
00:24:51.760 --> 00:24:53.760
You can use whatever sort of agent you want.

s329
00:24:53.760 --> 00:25:01.679
The benefit of what we've provided is a skill that you're able to go ahead and pull down the information from your traces to make those fixes.

s330
00:25:01.679 --> 00:25:07.360
So, you're not making those fixes just based on, hey, I know this is wrong and the output's incorrect.

s331
00:25:07.360 --> 00:25:10.240
You're making those fixes based on, hey, I know this is wrong.

s332
00:25:10.240 --> 00:25:11.600
Here's where it failed.

s333
00:25:11.600 --> 00:25:12.960
Here's why it failed.

s334
00:25:12.960 --> 00:25:14.960
Here's how we can go ahead and fix it.

s335
00:25:14.960 --> 00:25:33.360
because you have not just the information of your codebase, you have the information of any of your other APM tickets, whatever or APM logs, any of your other tickets that exist in your Jira systems, etc. as well as your actual arise traces, which would be like here's the pathway the agent took, here's where it messed up, your

s336
00:25:33.360 --> 00:25:38.559
eval, your annotations, etc. Any other questions?

s337
00:25:38.559 --> 00:25:38.880
Yes.

s338
00:25:38.880 --> 00:25:41.600
Are these features available in Phoenix?

s339
00:25:41.600 --> 00:25:42.559
Great question.

s340
00:25:42.559 --> 00:25:42.960
Yes.

s341
00:25:42.960 --> 00:25:47.520
Uh we do have so he asked is are these features available in Phoenix.

s342
00:25:47.520 --> 00:25:50.320
Phoenix is Aris's open- source version.

s343
00:25:50.320 --> 00:26:00.000
We do have I believe skills and MCP as well in Phoenix which you can go ahead and hook up to your agent and it can go ahead and pull down the same sort of information.

s344
00:26:00.000 --> 00:26:08.960
Of course there are going to be some nice to haves in the enterprise edition where you have access to these online evals in a certain way that are much easier to format and set up.

s345
00:26:08.960 --> 00:26:11.200
You have a little bit more dashboarding and a few other features.

s346
00:26:11.200 --> 00:26:16.720
But yes, the the skills in CLI you can connect to your coding agent.

s347
00:26:16.720 --> 00:26:18.720
Yes.

s348
00:26:20.000 --> 00:26:22.960
Agent said that summary.

s349
00:26:22.960 --> 00:26:30.880
So how would you classify something as like a bad agent know that's a good summary or a bad?

s350
00:26:30.880 --> 00:26:31.600
Very good question.

s351
00:26:31.600 --> 00:26:39.840
So her question was in what I showed earlier there was a eval or there was a response from Claude that said hey there was a bad summary.

s352
00:26:39.840 --> 00:26:43.039
How do we know what is a bad summary or a good summary?

s353
00:26:43.039 --> 00:26:48.960
In that case, what I actually had set up, it was actually based on one of the evals that I had previously set up.

s354
00:26:48.960 --> 00:26:52.480
And so, we can actually go ahead and look at that evaluator here.

s355
00:26:52.480 --> 00:27:03.919
We can go ahead and look at let's see some I actually don't remember which one, but I'm going to go ahead and pull open this one and hope this is the right one.

s356
00:27:03.919 --> 00:27:10.000
Um, so as you can see here, this is basically the summary eval that I'd created.

s357
00:27:10.000 --> 00:27:23.039
It's basically taking a look at the information we've gathered throughout our entire agent, all the sort of context we've given to that summary agent and making sure that it's referencing the right points, it's answering the question accordingly,

s358
00:27:23.039 --> 00:27:29.200
etc. Um, and so that's how we've gotten to determine whether or not it was a good summary or not.

s359
00:27:33.200 --> 00:27:38.880
Uh, what if we set up a skill in the first place to see if it can do a good summary?

s360
00:27:38.880 --> 00:27:48.640
give it given us a good summary all the time just like we set up a skill instead instead of running.

s361
00:27:48.720 --> 00:27:49.600
Yeah, great question.

s362
00:27:49.600 --> 00:27:58.080
So, she's asking why can't we just set up skills that you know ensure that it gives a good summary and yes absolutely that is what you want to do.

s363
00:27:58.080 --> 00:28:05.039
You want to make sure that you update your prompts, update your skills, update your coding workflows so that your agents will give the best summaries.

s364
00:28:05.039 --> 00:28:11.919
The emails are only there to help you determine where there are issues, monitor those issues, make sure they don't repeat, right?

s365
00:28:11.919 --> 00:28:22.080
And so you set up that skill, great, it works for the next two months, two months later because there's a model regression because people start asking questions in a different way.

s366
00:28:22.080 --> 00:28:28.159
You end up getting different results and you don't want your e you don't want your performance to degrade because of that.

s367
00:28:28.159 --> 00:28:30.080
So you want to be able to monitor and observe that.

s368
00:28:30.080 --> 00:28:31.520
That's the point of the evals.

s369
00:28:31.520 --> 00:28:33.360
Did that help?

s370
00:28:33.360 --> 00:28:34.480
Beautiful.

s371
00:28:34.480 --> 00:28:39.440
Um I don't know for how much time we have left but go ahead

s372
00:28:39.440 --> 00:28:50.480
later agents are assessed and soc.

s373
00:28:56.080 --> 00:29:00.640
So your question was if I've run into issues with linting with the agents

s374
00:29:00.640 --> 00:29:10.320
agents seems like more

s375
00:29:11.279 --> 00:29:11.919
good question.

s376
00:29:11.919 --> 00:29:12.159
Yeah.

s377
00:29:12.159 --> 00:29:23.120
So he's basically saying hey um have you seen the issue where hey you've given access claude to all these skills and all these sort of traces and all this sort of information

s378
00:29:23.120 --> 00:29:33.120
but ultimately most of the time I've seen with cloud code or any of these other sort of coding agents they end up focusing on linting rather than any other sort of issue am I correct

s379
00:29:33.120 --> 00:29:43.360
yeah that's a very good understanding and that's kind of why we built out these skills which kind of focus on ensuring that they hyperfixate on the actual issues that have been given to them, right?

s380
00:29:43.360 --> 00:29:51.679
So they'll get all these trace information so they'll understand, hey, it's not going to be a linting based issue because you can see that it's calling the wrong tool

s381
00:29:51.679 --> 00:29:56.559
and that's not going to happen because of linting and it does have some reasoning capability to understand that.

s382
00:29:56.559 --> 00:30:02.080
Um, and so genuinely I haven't seen that so much when using the skills.

s383
00:30:03.200 --> 00:30:04.080
Awesome.

s384
00:30:04.080 --> 00:30:07.360
So as it Oh yeah.

s385
00:30:21.760 --> 00:30:23.679
Does does the system make it easy to understand?

s386
00:30:23.679 --> 00:30:25.679
Sorry.

s387
00:30:27.760 --> 00:30:28.640
Great question.

s388
00:30:28.640 --> 00:30:28.880
Yeah.

s389
00:30:28.880 --> 00:30:34.559
So, how so his question was, does it make it easy to understand how much it costs to run those evals?

s390
00:30:34.559 --> 00:30:35.440
Absolutely.

s391
00:30:35.440 --> 00:30:45.279
So, one of the best parts of running evals is in the platform is you're going to be able to get full trace history of what your emails did, which also comes with the added benefit

s392
00:30:45.279 --> 00:30:48.799
of, you can see right here, a nice little cost.

s393
00:30:48.799 --> 00:30:58.320
And so, you'll be able to understand how much your emails are actually costing you because yes, of course, anything anytime you're running another LLM, it's going to be additional cost to you.

s394
00:30:58.320 --> 00:30:59.600
So, you want to make sure it's not.

s395
00:30:59.600 --> 00:31:14.559
And of course, what's really nice about building out evals, whether it's in the UI or via the coding agent, you can have those set up basic requirements like, hey, we want to sample specific amounts so we don't run it across 100% of our traces

s396
00:31:14.559 --> 00:31:16.559
and 100% of our calls.

s397
00:31:16.559 --> 00:31:19.200
Um, you can make it focus on specific areas.

s398
00:31:19.200 --> 00:31:25.120
And of course, you can always dictate that in the coding agent or on the platform itself.

s399
00:31:25.120 --> 00:31:32.480
Um, so as a kind of refresher as to everything we kind of covered today, again, it that's the wrong one.

s400
00:31:32.480 --> 00:31:35.600
Oh, I think I have it over here.

s401
00:31:35.679 --> 00:31:46.080
Um, so essentially what I really want to give you as a takeaway, these AI agents, they're great.

s402
00:31:46.080 --> 00:31:50.880
They will do basically everything you ask them to, but again, they're not magic.

s403
00:31:50.880 --> 00:31:53.440
Like he said, they can definitely focus on linting.

s404
00:31:53.440 --> 00:32:02.080
They can definitely get caught up in small little areas, but the best part of them is they will automate your full workflow from beginning to end.

s405
00:32:02.080 --> 00:32:15.360
So whether it's that starting from observing where the issues lie, finding those issues, being able to actually pull it down, understand where the issues lie, create those data sets,

s406
00:32:15.360 --> 00:32:21.760
build out exact fixes, test them to make sure that they're working.

s407
00:32:21.760 --> 00:32:29.600
And finally, you can then go ahead and be the manual human to make sure that hey, it didn't just go back and fix the linting changes, right?

s408
00:32:29.600 --> 00:32:38.159
And we can make sure that it actually made changes that actually help improve the agent and not go for degradation.

s409
00:32:39.440 --> 00:32:41.919
That's kind of the general gist of what I wanted to show today.

s410
00:32:41.919 --> 00:32:46.559
Um, any other questions on this before, you know, I can maybe show one last thing of seeing.

s411
00:32:46.559 --> 00:32:48.960
Oh, still finishing up what I was trying to do earlier.

s412
00:32:48.960 --> 00:32:50.960
I was hoping it might be done after this.

s413
00:32:50.960 --> 00:32:52.880
took about 14 minutes.

s414
00:32:52.880 --> 00:33:04.720
But um while that's finishing up, any other questions on how we can use our agents to automate our workflows and improving your agents?

s415
00:33:04.720 --> 00:33:06.720
Yeah.

s416
00:33:19.279 --> 00:33:20.240
Great question.

s417
00:33:20.240 --> 00:33:30.000
So his question was, are we primarily relying on Claude or any of these coding agents to actually go ahead and give us the evaluation if things are working well or not?

s418
00:33:30.000 --> 00:33:34.159
Um, yes, you can definitely rely on Claude to build out everything.

s419
00:33:34.159 --> 00:33:36.159
You can also manually do it.

s420
00:33:36.159 --> 00:33:40.080
You can also ask Claude to do the specific evaluations you want.

s421
00:33:40.080 --> 00:33:47.039
So in this case, when you saw that nice clean table, that was actually defined by the evaluations me and my team had set up prior to this.

s422
00:33:47.039 --> 00:33:58.960
Um, we'd gone ahead and looked at, hey, these are specific areas that we know are potential limitations in most agents that work in this workflow and we set up those as generic ones, which is why we got a bad summary.

s423
00:33:58.960 --> 00:34:01.600
We had some, hey, is the agent healthy?

s424
00:34:01.600 --> 00:34:03.760
Is it giving the right are we getting tool calls correct?

s425
00:34:03.760 --> 00:34:04.000
Right?

s426
00:34:04.000 --> 00:34:15.200
We know those are general areas, but what Claude was able to do was look at our actual traces, understand based on those traces, what are the actual issues we're falling into today.

s427
00:34:15.200 --> 00:34:15.440
Right?

s428
00:34:15.440 --> 00:34:17.520
we have our live agent in production.

s429
00:34:17.520 --> 00:34:24.960
I'm not going to be able to go through all of those manually and find those, but Claude can or Claude cursor codeex, whatever, right?

s430
00:34:24.960 --> 00:34:32.879
And that's where like one of the biggest superpowers that you're able to give these coding agents to be able to do that for you.

s431
00:34:32.879 --> 00:34:34.960
I saw another question here.

s432
00:34:34.960 --> 00:34:35.520
Good question.

s433
00:34:35.520 --> 00:34:45.359
So his question was is there any sort of benchmark data set that we can see or any sort of like visualization that can tell us hey this is a higher quality LLM to use for evaluations

s434
00:34:45.359 --> 00:34:49.473
more expensive less expensive um realistically no

s435
00:34:49.473 --> 00:34:50.399
[laughter]

s436
00:34:50.399 --> 00:35:00.800
that would be sort of something you'd have to do for your own analysis you'd have to figure it out because ultimately one of the things that I like to say is if you're going to use out of the box eval to judge your agent

s437
00:35:00.800 --> 00:35:08.800
you're going to get out of the box results and that's not what you want when you're building customtailored AI agents which are built to be personal for you.

s438
00:35:08.800 --> 00:35:11.839
Um, and the same goes for benchmarks, right?

s439
00:35:11.839 --> 00:35:12.800
Benchmarks are great.

s440
00:35:12.800 --> 00:35:15.839
They'll show you, hey, this this is a good agent.

s441
00:35:15.839 --> 00:35:17.920
This is a good sorry, this is a good LLM.

s442
00:35:17.920 --> 00:35:21.440
But realistically, what you want is, is it a good LLM for me?

s443
00:35:21.440 --> 00:35:23.920
And that's going to be tailored to your use case.

s444
00:35:23.920 --> 00:35:26.720
So, if you're building for finance, maybe you'll use a specific one.

s445
00:35:26.720 --> 00:35:32.240
If you're building for, you know, law in finance, it might be even this slightly different model, right?

s446
00:35:32.240 --> 00:35:36.160
So, it really depends on what you're building it for.

s447
00:35:37.200 --> 00:35:37.599
Cool.

s448
00:35:37.599 --> 00:35:40.000
I think that's more or less it more or less it from me.

s449
00:35:40.000 --> 00:35:42.800
Um, if anyone has any other questions, I'm happy to answer them.

s450
00:35:42.800 --> 00:35:46.459
But yeah, thank you guys.

s451
00:35:46.459 --> 00:35:48.459
[applause]