WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=9HbzAWnKbo4 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/9HbzAWnKbo4.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.639 --> 00:00:14.080 Well, thank thank you all. s3 00:00:14.080 --> 00:00:16.880 Um, let me get set up here. s4 00:00:16.880 --> 00:00:25.760 So, not just the the founder of Arise, but but I tend to build an incredible amount of stuff. s5 00:00:25.760 --> 00:00:30.800 Um, let's see if we get this going here. s6 00:00:30.800 --> 00:00:32.480 Oh, sorry. s7 00:00:32.480 --> 00:00:33.840 One more second. s8 00:00:33.840 --> 00:00:45.200 Um, so not just a founder here, but but also a builder and I do my best to um uh to to to build agents assistance. s9 00:00:45.200 --> 00:00:48.719 Um, we have an agent in product. s10 00:00:48.800 --> 00:00:59.359 We have an agent in product called Alex and uh and a lot of I think a lot of my experience has come from actually um trying to make the stuff work and work well. s11 00:00:59.359 --> 00:01:03.120 Our first version of our our own agent frankly sucked. s12 00:01:03.120 --> 00:01:07.840 Uh it was many years ago uh probably two years ago when the first in the space to do it. s13 00:01:07.840 --> 00:01:21.280 Um and a lot of what we have built uh has come out of that our own experience in building building this agent and and signal is kind of our our next generation of this which is trying to automate s14 00:01:21.280 --> 00:01:26.560 a bunch of things which we do every day uh and build it into products that that people can use. s15 00:01:26.560 --> 00:01:29.920 Um so I'm going to try to I'm going to go through this this materials here. s16 00:01:29.920 --> 00:01:32.320 I'll try to go fast and try to show you a lot of product too. s17 00:01:32.320 --> 00:01:33.439 I'm a product person. s18 00:01:33.439 --> 00:01:44.000 Um, so if you've built a startup before, um, you you've experienced this, your platform's down, it's it's late at night and and you and you want to go fix it. s19 00:01:44.000 --> 00:01:49.439 Um, and and really the the it takes a lot of energy to go do that. s20 00:01:49.439 --> 00:02:00.960 And we're going to talk about like the automation we've built a little bit and and what what the future looks like and and I truly believe um that the future of the the observability space is is actually s21 00:02:00.960 --> 00:02:02.799 changing massively right now. s22 00:02:02.799 --> 00:02:03.439 Why is that? s23 00:02:03.439 --> 00:02:06.799 Well, observability used to be for humans. s24 00:02:06.799 --> 00:02:10.479 Used to be a UI you click, a graph you click, something you look at. s25 00:02:10.479 --> 00:02:17.040 Um and and today it's I would argue it's a lot of 2.0 which is like this combination of coding agent. s26 00:02:17.040 --> 00:02:27.440 Those of you who built skills skills for um Pyroscope, Google Cloud or or whatnot, that these these skills help you with your your human debugging these systems. s27 00:02:27.440 --> 00:02:38.400 Um and and really telemetry is like this smoke uh thrown off of your system that can allow these agents to go make fixes. s28 00:02:38.400 --> 00:02:41.840 It tells you what path in the code it took. s29 00:02:41.840 --> 00:02:45.040 Without that, you're guessing and there's a million paths it could have taken. s30 00:02:45.040 --> 00:02:55.040 the the data thrown off by your system allows um allows you to to go go use agents to go debug your software. s31 00:02:55.040 --> 00:02:57.360 Evals add another layer to this. s32 00:02:57.360 --> 00:03:05.280 Um but really what we're at here is is how do I build systems that autonomously fix themselves? s33 00:03:05.280 --> 00:03:06.800 Really that that is what we're after. s34 00:03:06.800 --> 00:03:10.400 Both both AI agents, I put AI into my my my system. s35 00:03:10.400 --> 00:03:12.959 How do I have this thing just improve itself? s36 00:03:12.959 --> 00:03:18.159 and and today we're kind of in the 2.0 which is a human making fixes and reviewing things. s37 00:03:18.159 --> 00:03:30.959 Um but there's a future we're all driving towards and throwing off traces, throwing off logs, throwing off way more than you normally would and having agents run at this for a continuous loop is where we're going. s38 00:03:30.959 --> 00:03:36.319 You can build at agent speed, but today you can't improve your systems really at this agent speed. s39 00:03:36.319 --> 00:03:42.080 So those of us feel this this kind of governor happening within our our our our products. s40 00:03:42.080 --> 00:03:46.239 um and and the bottleneck is actually not the fix anymore. s41 00:03:46.239 --> 00:03:57.920 So those of us who've used these systems and and use used um coding agents with with skills, the the the bottlenecks, a lot of the the confidence in in do I have it right? s42 00:03:57.920 --> 00:04:02.560 You know, a lot of this is is about is this fix the right one to push? s43 00:04:02.560 --> 00:04:06.159 Um and and so these are kind of the challenges here. s44 00:04:06.159 --> 00:04:10.720 And then how do you how do you build this loop in a way that just moves faster? s45 00:04:10.720 --> 00:04:20.880 Um and and a little bit of the way we we've kind of come to do it and we do it in our system is we've kind of inverted this this loop which is like a human s46 00:04:20.880 --> 00:04:30.800 you know looks at things and an agent uh fixes it to a person now can wake up with with an idea of the issues based upon the errors occurred in their system. s47 00:04:30.800 --> 00:04:36.800 So so the agent is actually you know maybe it's not a a fix itself but it's putting up an issue. s48 00:04:36.800 --> 00:04:39.600 It's looking at the data before a human even looks at it. s49 00:04:39.600 --> 00:04:55.840 Um and and what you move from there is is kind of humans grabbing tickets to to having some amount of evidence um some deep evidence relative to whatever you're looking at already sitting in front of you by the time you actually even look at it. s50 00:04:55.840 --> 00:05:03.120 And and human review is kind of one thing but but a lot of times maybe you're driving this little investigation a bit from where it started. s51 00:05:03.120 --> 00:05:09.840 So that's the reality of where we are today is there's still maybe it's not human reviewing but human driving the the step two and three. s52 00:05:09.840 --> 00:05:13.440 Um but but this is kind of what what we view the loop as. s53 00:05:13.440 --> 00:05:19.840 And really what it is is there's you know there's an event that occurs that you're kind of kicking things off on or you're looking at periodically. s54 00:05:19.840 --> 00:05:24.479 Um and then there's some context around that which is really driven by skills. s55 00:05:24.479 --> 00:05:26.960 Um, I guess a question for all of you. s56 00:05:26.960 --> 00:05:29.919 Who's created skills in this room? s57 00:05:29.919 --> 00:05:35.120 Who's created a skill that that that interfaces to an observability platform? s58 00:05:35.120 --> 00:05:36.000 Okay, handful. s59 00:05:36.000 --> 00:05:36.479 Okay, cool. s60 00:05:36.479 --> 00:05:37.440 Awesome. s61 00:05:37.440 --> 00:05:45.440 Um, so the magic of of of skills that that that connect to observability platforms um is it can gather the context. s62 00:05:45.440 --> 00:05:51.520 The agent can decide what it needs, what it needs to look at um to to start to troubleshoot what you have there. s63 00:05:51.520 --> 00:05:57.919 Um, and then there's idea of triggers which are like periodic and and um and uh and event based. s64 00:05:57.919 --> 00:06:05.360 And so the future observability actually looks a lot more like this than it does clicking around graphana UI. s65 00:06:05.360 --> 00:06:12.479 So first off evidence well normally these like or are or what do you start with what do you look at? s66 00:06:12.479 --> 00:06:15.280 Uh traces are are pretty nice logs as well. s67 00:06:15.280 --> 00:06:21.280 uh but but you know most of the AI systems these days have like traces at the core of of the agent framework. s68 00:06:21.280 --> 00:06:29.120 So, so you kind of start with with looking at traces and this is this could be periodic you know every five minutes this could be based upon an event an error s69 00:06:29.120 --> 00:06:41.120 and normally there's some combinations of these which is um you know some some like uh context and log you know context and skills used to put together logs maybe there's the repo s70 00:06:41.120 --> 00:06:50.240 uh you want kind of a combination of all this together um to understand what to go fix the repo tells you the code path that you know the you know all s71 00:06:50.240 --> 00:06:55.600 tells you everything that's there, the the production logs or traces that the agent pulls down. s72 00:06:55.600 --> 00:07:00.240 Um, normally our skills actually pull pull little temp files down into the the repo. s73 00:07:00.240 --> 00:07:09.520 Um, so that you kind of have this this this idea of what actually happened, what the code is there enough and and all that together to put up a fix. s74 00:07:09.520 --> 00:07:15.199 Um, so it's this combination of the right data and file format in the repo along with your code in the repo. s75 00:07:15.199 --> 00:07:21.120 That's kind of the magic of this skills which are composable for the agent to go actually put a fix. s76 00:07:21.120 --> 00:07:25.919 Um, and a lot of this, some of you, a lot of you probably do this locally today. s77 00:07:25.919 --> 00:07:27.199 You you run this locally. s78 00:07:27.199 --> 00:07:31.599 You have an agent that you you kick up, maybe you're spinning up, but it's on your laptop. s79 00:07:31.599 --> 00:07:39.759 And I think we all feel this this this move from from this laptop um to to maybe to to basically sandboxes. s80 00:07:39.759 --> 00:07:53.520 Um and and really the sandbox is this this running environment where um based upon an event or a periodic you know a periodic event you can kick this thing off and it does the same thing you were doing locally s81 00:07:53.520 --> 00:07:59.759 get it working locally first locally on your laptop and then event based based upon the observability platforms like ourselves. s82 00:07:59.759 --> 00:08:05.759 Um you can trigger these on a schedule or or kind of you know every every error that comes up. s83 00:08:05.759 --> 00:08:15.199 Um, and generally, you know, you generally it's kind of put putting the loop together to do this and and um, and I want to kind of give you one example. s84 00:08:15.199 --> 00:08:16.560 So, this is Alex, our agent. s85 00:08:16.560 --> 00:08:18.240 This is a a real example. s86 00:08:18.240 --> 00:08:19.280 It's a very simple one. s87 00:08:19.280 --> 00:08:22.000 And then I'm going to show you what it looks like in product. s88 00:08:22.000 --> 00:08:24.400 Um, this what we use every day. s89 00:08:24.400 --> 00:08:29.440 Um, but this is just an example where um, we had a stream canceled event. s90 00:08:29.440 --> 00:08:37.279 So um so Alex is is basically um Alex is is basically our our inproduct assistant. s91 00:08:37.279 --> 00:08:43.760 Um to-do update is is a uh is a is a way of of managing kind of its its task list. s92 00:08:43.760 --> 00:08:55.680 Um and it was trying to you know I'll walk you through the the error in a second but basically um it's calling a bunch of these two to-do updates and kind of um errors out and and so for us it's it's s93 00:08:55.680 --> 00:08:58.640 you know how do I put the data together um to debug this? s94 00:08:58.640 --> 00:09:05.200 how do we do it automatically and and signal is just something that's running in the background for us that's putting up like issues relative to these things. s95 00:09:05.200 --> 00:09:10.720 Um this was uh a kind of one or two line fix that it comes up with. s96 00:09:10.720 --> 00:09:14.000 These are these are ideal but a lot of times the fixes are bigger. s97 00:09:14.000 --> 00:09:19.600 Um and and the bigger it is the more likely a human's involved in kind of like spearheading it over the line. s98 00:09:19.600 --> 00:09:22.399 But again it's about that that cold start. s99 00:09:22.399 --> 00:09:27.839 Can I start with like all this information on the issue and guide it the rest of the way is kind of where we are right now. s100 00:09:27.839 --> 00:09:32.560 Um, and for us, your job kind of moves from responder to reviewer. s101 00:09:32.560 --> 00:09:38.240 Um, and and and the view is like traces and evals don't go away in any way, shape, or form. s102 00:09:38.240 --> 00:09:40.560 They're just they're they're a key part of the loop. s103 00:09:40.560 --> 00:09:42.800 Now, you're going to trace 10 times more. s104 00:09:42.800 --> 00:09:48.399 You're going to log 10 times more because that helps you know what path your software took. s105 00:09:48.399 --> 00:09:53.600 Before, you wouldn't do that because because humans can't dig through all the logs. s106 00:09:53.600 --> 00:09:55.120 It's just noise. s107 00:09:55.120 --> 00:10:00.480 But by logging and tracing more of your like is it every inch of your software? s108 00:10:00.480 --> 00:10:01.680 Maybe in some places. s109 00:10:01.680 --> 00:10:11.519 Um by logging and tracing orders and orders of magnitude more than we do today, we can actually create these continuous loops that know what path was taking your software s110 00:10:11.519 --> 00:10:13.760 and and and actually have it fix itself. s111 00:10:13.760 --> 00:10:17.440 So this is kind of my vision for where I think things are going. s112 00:10:17.440 --> 00:10:29.200 um in a way and and for us I'll show you signal in a second and you'll see all these I mean I feel like there's there's this think of this as an an SR you know something that helps you debug maybe SR for for AI s113 00:10:29.200 --> 00:10:37.920 um but I feel like there's a lot of black boxes out there like oh there's a SR agent that does this or S agent that does this all we're really trying to do ourselves s114 00:10:37.920 --> 00:10:50.399 is take your local debugging experience with cloud code cursor and run it periodically so pick your sandbox pick your harness, pick your skills, we'll pre-bank a bunch of things with you. s115 00:10:50.399 --> 00:10:57.519 So, we're just trying to again take the things we were doing locally and actually run them um uh you know, run them in a system. s116 00:10:57.519 --> 00:11:07.600 So, we believe in you know an open approach um to this and um I I'll give you a demo of what this looks like um from a product perspective. s117 00:11:07.600 --> 00:11:14.720 So, so this is um this is a a financial trading agent. s118 00:11:14.720 --> 00:11:21.440 Um given what you saw in the previous uh presentation, I would not recommend doing a financial trading agent. s119 00:11:21.440 --> 00:11:24.000 Um they they they're unlikely to make you money. s120 00:11:24.000 --> 00:11:25.680 Uh at least not not yet. s121 00:11:25.680 --> 00:11:28.160 Um maybe there's some people uh doing it good. s122 00:11:28.160 --> 00:11:34.959 But long story short is this one's, you know, uh people asking questions about stock trading right now and it's giving giving answers. s123 00:11:34.959 --> 00:11:37.760 Um there's a lot of ways this this can fail. s124 00:11:37.760 --> 00:11:41.120 And so this this gives you this is a rise. s125 00:11:41.120 --> 00:11:41.760 It's a platform. s126 00:11:41.760 --> 00:11:45.200 So first off from let me describe the products we have. s127 00:11:45.200 --> 00:11:48.560 Uh this is AX which is our our SAS platform. s128 00:11:48.560 --> 00:11:52.320 Um we also have Phoenix which is open source if you just want to start tomorrow. s129 00:11:52.320 --> 00:11:56.560 Um Signal right now is is just available in in our AX SAS platform. s130 00:11:56.560 --> 00:12:02.959 Um which also could be deployed VPC but but this so so give you an idea of our product lines. s131 00:12:02.959 --> 00:12:05.519 Uh if you want to try out Signal, it's it's an AX. s132 00:12:05.519 --> 00:12:16.560 Um and what it looks like is something like this, which is it's just periodically running and and kind of coming up with like issues and you can hook it up to s133 00:12:16.560 --> 00:12:17.920 your GitHub repo. s134 00:12:17.920 --> 00:12:20.000 It can create an issue in your repo. s135 00:12:20.000 --> 00:12:21.839 You can create an evaluator from this. s136 00:12:21.839 --> 00:12:26.560 Maybe u maybe there's a a specific problem by which you want to catch again. s137 00:12:26.560 --> 00:12:28.399 You can add these to a data set. s138 00:12:28.399 --> 00:12:38.720 So if you want to add these and and it has evidence associated with this like traces um in this case um in this one it has skills like for Google cloud and some other logging systems. s139 00:12:38.720 --> 00:12:48.000 So we can frontend a bunch of places the data we're we're pretty good at building I think these skills to debug issues again uh but you can add your own skills. s140 00:12:48.000 --> 00:12:52.320 So these examples here you know traces running out without a guardrail. s141 00:12:52.320 --> 00:13:02.480 Um there's there's um uh you know safety safety issues and intent issues and a lot of this too is like you know how does this work? s142 00:13:02.480 --> 00:13:05.200 How do I you know it feels a little too black box to me. s143 00:13:05.200 --> 00:13:16.800 Well all this is open and open box um in the sense that um I can set up you know I can set up the harness that I want it to run on. s144 00:13:16.800 --> 00:13:17.920 This one's cloud code. s145 00:13:17.920 --> 00:13:20.720 I can pick my sandboxes and sandbox systems. s146 00:13:20.720 --> 00:13:23.120 Um, I can use cloud managed agents if I want. s147 00:13:23.120 --> 00:13:25.040 I can use Arise sandbox. s148 00:13:25.040 --> 00:13:29.120 Uh, why would I want to use Arise sandboxes versus cloud managed agents? s149 00:13:29.120 --> 00:13:34.800 Well, a lot of our customers um don't want to connect their production systems to Tanthropic. s150 00:13:34.800 --> 00:13:40.880 You know, you you you want your s you want these these sandboxes to debug your database or connect to it. s151 00:13:40.880 --> 00:13:56.160 So we install in the VPC of a lot of you know bigname companies out there um from from Uber to um to bookings to you name it and and these people don't want to send their connections out but they'll they'll use a s you know s152 00:13:56.160 --> 00:14:02.560 many many companies um are very comfortable installing a V into a VPC and actually connecting it up. s153 00:14:02.560 --> 00:14:09.920 So you can use Arise sandboxes or you can use Daytona or any of any of these that you're comfortable with um that you built relationships with. s154 00:14:09.920 --> 00:14:24.240 Um and and then from a a platform perspective, you know, we support um running we support, you know, tracking the different agents that you're running. s155 00:14:24.240 --> 00:14:25.519 So you have this swarm of agents. s156 00:14:25.519 --> 00:14:30.160 Maybe you've kicked off um maybe you're kicking off a signal which which is our agent. s157 00:14:30.160 --> 00:14:31.600 It's running periodically. s158 00:14:31.600 --> 00:14:36.720 Maybe you're kicking off your you know you've named another agent um in the system. s159 00:14:36.720 --> 00:14:46.399 And these all support, you know, viewing the session that that ran, downloading the transcript, and you can resume a clouded session locally, too. s160 00:14:46.399 --> 00:14:48.399 So, the idea is that this thing's constantly running. s161 00:14:48.399 --> 00:14:50.240 You're picking the harness, the sandbox. s162 00:14:50.240 --> 00:14:56.240 You're deciding the prompt if you want, hey, don't be aggressive or look for, you know, look for security issues. s163 00:14:56.240 --> 00:15:02.560 So, you're deciding the prompts that drive this and you're also deciding the skills that go along with this. s164 00:15:02.560 --> 00:15:10.160 So um in a preset here um I can add you know add different skills. s165 00:15:10.160 --> 00:15:11.519 I can add my own skills. s166 00:15:11.519 --> 00:15:12.720 I can link repos. s167 00:15:12.720 --> 00:15:14.800 I can I have pre-baked skills too. s168 00:15:14.800 --> 00:15:25.760 Um so the ideas observability platforms are really starting to get are becoming tied to the continuous loop to the the fix not just the the signal. s169 00:15:25.760 --> 00:15:32.240 Um, and and you you want to take your local experience you have debugging the stuff locally. s170 00:15:32.240 --> 00:15:43.279 You want to take the evals that are running and and actually have these all work in something that puts up a fix or at least gets you a cold start and then I can take it over locally if I want to continue debugging s171 00:15:43.279 --> 00:15:44.399 uh from from here. s172 00:15:44.399 --> 00:15:52.160 Um, so this gives you um a rough idea of of kind of of of signal to PR um what we're doing. s173 00:15:52.160 --> 00:15:59.759 Um, I did want to offer, you know, questions if people people have any questions on what we're doing or how we see uh the industry evolving. s174 00:15:59.759 --> 00:16:01.120 Happy to happy to answer. s175 00:16:01.120 --> 00:16:03.360 Thank you. s176 00:16:04.826 --> 00:16:06.826 [applause] s177 00:16:13.279 --> 00:16:13.519 Yeah. s178 00:16:13.519 --> 00:16:15.680 Go ahead. s179 00:16:20.560 --> 00:16:27.920 question of like why can't we just connect cloud code to your data and have cloud code do all these things. s180 00:16:27.920 --> 00:16:32.079 I think there's like a version of that question that can probably be asked for these autofixes, right? s181 00:16:32.079 --> 00:16:36.160 Like why not have cloud code read the traces and push the PR itself. s182 00:16:36.160 --> 00:16:38.800 I'm curious how you would respond to that question. s183 00:16:38.800 --> 00:16:39.440 Yeah. s184 00:16:39.440 --> 00:16:45.040 Uh so so why wouldn't have cloud code kind of hook to your your data and just just do it? s185 00:16:45.040 --> 00:16:53.120 Um the answer is like you should um like like the vision and what we do actually at at Arise is we have uh a lot of skills. s186 00:16:53.120 --> 00:16:59.120 I think first off to to make that really work well you have to do a bit of well-designed skills in the data space. s187 00:16:59.120 --> 00:17:08.160 The skill like the the important things of designing these skills are are around really around getting data you know finding the right data first. s188 00:17:08.160 --> 00:17:12.000 So I want to find a group of traces relative to a session or something. s189 00:17:12.000 --> 00:17:15.039 Getting that data into the repo in a file format. s190 00:17:15.039 --> 00:17:17.280 These harnesses are magical with files. s191 00:17:17.280 --> 00:17:18.959 So you get the file what happened. s192 00:17:18.959 --> 00:17:22.079 In some cases we have 10meg files like sitting in the repo. s193 00:17:22.079 --> 00:17:32.000 Um so it's designing the skill to be really really well done with the um with this data and and then giving claude enough skills to be composable to find issues. s194 00:17:32.000 --> 00:17:33.600 So the answer is absolutely yes. s195 00:17:33.600 --> 00:17:37.840 like we have Pyroscope skills that will find memory issues. s196 00:17:37.840 --> 00:17:40.480 We have facets in Pyroscope that the skill knows how to use. s197 00:17:40.480 --> 00:17:43.520 I can cohort by customer to see if a customer is causing an issue. s198 00:17:43.520 --> 00:17:54.720 Um but but you've got to kind of design the skill surface area in a way that Claude can really really work well and and and it's not just like point Claude at the data. s199 00:17:54.720 --> 00:17:55.360 I see. s200 00:17:55.360 --> 00:17:57.039 Thank you. s201 00:17:57.039 --> 00:17:59.280 Any other questions? s202 00:17:59.280 --> 00:18:01.679 Anyone else? s203 00:18:01.760 --> 00:18:02.240 Oh yeah. s204 00:18:02.240 --> 00:18:02.480 Okay. s205 00:18:02.480 --> 00:18:04.160 One more. s206 00:18:04.160 --> 00:18:05.120 Thanks for the talk. s207 00:18:05.120 --> 00:18:17.120 Um, there was quite a few mention of eval but you know I'm looking at the traces so you know I understand the concept of traces but where where did the evals come in when you have that signal that says hey something broke in production. s208 00:18:17.120 --> 00:18:17.440 Yeah. s209 00:18:17.440 --> 00:18:26.880 So so the so the eval typically will the um the eval essentially are running and being layered on typically to the production traces something we call online evals. s210 00:18:26.880 --> 00:18:32.400 Um let me see if this one has an example here of it. s211 00:18:32.400 --> 00:18:50.799 Um so so eval actually are data on the trace itself and so the agent knows how to uh grab the data from traces knows how to um visualize and you know the skills to basically pull data for for the aggregate values of evals across the traces so that so the the skills that you give s212 00:18:50.799 --> 00:18:55.600 uh the harness allow it to get the data on the eval from from the traces. s213 00:18:55.600 --> 00:19:08.400 Um so eval are kind of like I view them as le at least a first generation eval evals which are elements a judge um as a as a AI layer that allows you to run periodically s214 00:19:08.400 --> 00:19:21.919 and and assess your system but it's like it but it's adding a little bit more you know pre-processed information on the data that that and then as signal is running it's using data from the evals that were layered on s215 00:19:21.919 --> 00:19:30.320 um in addition to all the raw data that it has there Um it but it tends to be like you build an eval for a failure you've seen before a lot of times. s216 00:19:30.320 --> 00:19:38.960 So I have these prompt injection things that I'm trying to catch or something or um or or a failure in the way it's responded maybe to to something before. s217 00:19:38.960 --> 00:19:49.360 So they they tend to be this like you know u at least the LM as a judge is tends to be like this this thing you um preset up and then you can actually create evaluators for failure. s218 00:19:49.360 --> 00:19:52.480 Say you find this failure that's pretty common and happening all the time. s219 00:19:52.480 --> 00:19:54.480 I can create an eval so I can catch it next time. s220 00:19:54.480 --> 00:20:00.559 I just you think of it as like almost a an AI um assessment that's always running. s221 00:20:00.559 --> 00:20:04.000 Uh the other note is the element as a judge can run really at scale. s222 00:20:04.000 --> 00:20:13.520 Well, every you know I have customers who who lay you know layer element as a judge across um their full data set uh where where this tends to be like s223 00:20:13.520 --> 00:20:16.320 you know uh more periodic on a lot of data. s224 00:20:16.320 --> 00:20:17.600 So cool. s225 00:20:17.600 --> 00:20:19.840 Thank you.