WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=EUsPvBeIx70 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/EUsPvBeIx70.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.440 --> 00:00:01.360 Welcome everyone. s2 00:00:01.360 --> 00:00:02.560 My name is Rahul Singh. s3 00:00:02.560 --> 00:00:05.840 I'm a staff AI research engineer at Fedra. s4 00:00:05.840 --> 00:00:06.840 And I'm Vance Kulistik. s5 00:00:06.840 --> 00:00:09.760 I'm a senior engineering manager, also at Fedra. s6 00:00:09.760 --> 00:00:15.560 And today we want to talk about the time when we gave an LM 500,000 sensor names and it got confused. s7 00:00:15.560 --> 00:00:18.240 We call this problem semantic blindness. s8 00:00:18.240 --> 00:00:21.920 At Fedra we build AI agents for AI factories. s9 00:00:21.920 --> 00:00:34.720 This includes agents which allow our customers to talk about their data centers and explain to themselves and understand how the data centers are working and what problems they are facing on a day-to-day basis. s10 00:00:34.720 --> 00:00:47.640 User queries can be anything from what chiller is running hot to analyze the distribution of temperatures across my data halls to is any of my GPUs facing any problems? s11 00:00:47.640 --> 00:01:00.880 Now from this variety of queries you can see that these include talking what specific equipments is chiller six all right to talking about a groups of equipments GPUs in data hall one. s12 00:01:00.880 --> 00:01:18.960 The industry has not really figured out a common naming pattern yet and every single customer can have their own things from simple names like racks with GPUs with data halls which give you an exact idea of where different things are to something that is more difficult to comprehend like s13 00:01:18.960 --> 00:01:21.080 CH3 something something six. s14 00:01:21.080 --> 00:01:24.800 We've seen all kind of names in the industry going forward. s15 00:01:24.800 --> 00:01:30.000 Now when you're building a demo system this works because you're working at a small scale. s16 00:01:30.000 --> 00:01:36.760 A a simple LM can look at all the names of your equipment and figure out what user is talking about. s17 00:01:36.760 --> 00:01:41.920 But this problem really becomes intractable as you go to for scale. s18 00:01:41.920 --> 00:01:53.000 For example at 1 gigawatt scale factories you will see 400,000 plus GPUs and to support those GPUs you have power meters, you have chillers, you have other equipments. s19 00:01:53.000 --> 00:01:59.080 LM context windows are finite and you will very quickly saturate uh and it just becomes a problem. s20 00:01:59.080 --> 00:02:03.840 Like we say, a product is something that works for all scenarios and does not fail silently. s21 00:02:03.840 --> 00:02:07.240 A demo just has to work for one. s22 00:02:07.400 --> 00:02:16.240 In addition to having the LLM figure out these names, uh we could also have uh embedded them in a rack database, a vector embedding approach. s23 00:02:16.240 --> 00:02:21.240 But, the problem is often times the names are so similar that semantic search just fails. s24 00:02:21.240 --> 00:02:29.520 Uh there's very little difference between a vector of Sorry, a string name of 20 characters long which just differs by, let's say, one character. s25 00:02:29.520 --> 00:02:35.800 Chiller 6 instead of Chiller 7. Or uh CDO something uh versus, well, something else. s26 00:02:35.800 --> 00:02:36.640 It's very small. s27 00:02:36.640 --> 00:02:41.880 So, you get a lot of uh problems with getting accurate uh recall. s28 00:02:41.880 --> 00:02:46.800 Also, LLMs suffer from what we call a frequency uh penalty. s29 00:02:46.800 --> 00:02:56.680 If you keep on outputting very similar names over and over again, or very similar tokens, uh more accurately, over and over again, there are internal penalties in the LLM which just shut off their output. s30 00:02:56.680 --> 00:03:02.800 So, if the user says, "Can list all the names of uh let's say GPUs in i7?" s31 00:03:02.800 --> 00:03:04.640 There are, let's say, 100. s32 00:03:04.640 --> 00:03:13.000 Just by listing through them, the LLM's guardrails would see uh think that well, it's spiraling, and it would just shut the system up. s33 00:03:13.000 --> 00:03:15.040 So, we can't really have these two approaches. s34 00:03:15.040 --> 00:03:16.880 RAG would not work. s35 00:03:16.880 --> 00:03:19.400 Just naive LLMs would not work. s36 00:03:19.400 --> 00:03:23.920 As we move into production systems, we need something that can scale as the system scales. s37 00:03:23.920 --> 00:03:27.720 Now, there are naive solutions which we shall discuss going forward. s38 00:03:27.720 --> 00:03:32.720 A naive approach to solve this problem would be just to well, divide and conquer. s39 00:03:32.720 --> 00:03:38.239 Take your different uh equipments, branch them in different shards, and pass them through LLMs. s40 00:03:38.239 --> 00:03:40.560 Uh parallel calls should work, right? s41 00:03:40.560 --> 00:03:42.640 Well, that's what we thought. s42 00:03:42.640 --> 00:03:46.640 The problem is you get horrible recall and hallucinations. s43 00:03:46.640 --> 00:03:53.920 You will see LLMs invent phantom equipment that do not exist, and also silently drop things that do exist. s44 00:03:53.920 --> 00:04:00.080 Now, this creates a problem for mission-critical systems where the users need to know exactly what is happening with their systems. s45 00:04:00.080 --> 00:04:02.920 Any problems there will quickly erode user trust. s46 00:04:02.920 --> 00:04:10.160 At the same time, you will miss on specific problems which can cascade into bigger, bigger problems going forward. s47 00:04:10.160 --> 00:04:25.240 Anyhow, something that we realized here is that as the size of these physical infrastructure grows, LLMs can not the LLM-based solutions can not grow with the size of individual components or instances or nodes. s48 00:04:25.240 --> 00:04:30.080 They We have to find something that grows sub-linearly with increasing equipment count. s49 00:04:30.080 --> 00:04:32.280 And this is what we figured out. s50 00:04:32.280 --> 00:04:36.000 So, we should not grow with instances, we should grow with tree depth. s51 00:04:36.000 --> 00:04:42.240 Now, what do we mean by tree depth is we realized that there's a hierarchical structure in which an AI factory is arranged. s52 00:04:42.240 --> 00:04:51.320 You will have data centers, then each data centers will have different data halls, you have each of them will have different aisles, and you will have different rows and then racks and then GPUs. s53 00:04:51.320 --> 00:04:56.560 Similarly, a chiller plant will have rooms where chillers are arranged, then you will have pumps. s54 00:04:56.560 --> 00:04:58.400 There will be a separate cooling tower unit. s55 00:04:58.400 --> 00:05:00.360 It's kind of like a tree. s56 00:05:00.360 --> 00:05:05.200 The depth of the tree grows very slowly, the width grows extremely fast. s57 00:05:05.200 --> 00:05:14.120 In other words, you will have a hierarchy that only adds new equipments very rarely, but it adds a lot of them when it does. s58 00:05:14.120 --> 00:05:18.360 You will have a lot of GPUs, but without GPUs, you won't have a lot of things. s59 00:05:18.360 --> 00:05:23.680 Now, we realized that this could be used to solve our problems. s60 00:05:23.680 --> 00:05:27.000 And there are four insights that really come into the picture for this. s61 00:05:27.000 --> 00:05:29.720 One is a linearizer. s62 00:05:29.720 --> 00:05:41.120 What I mean by the linearizer is the LLM has to figure out where each of these things are arranged and how to map from a vague user query to specific equipment or groups of equipments. s63 00:05:41.120 --> 00:05:46.960 Here a summarized representation of our system's graph can be really, really useful. s64 00:05:46.960 --> 00:05:57.440 For example, uh if you're uh a 1 gigawatt scale factory can have over a million nodes and each node represents a unique equipment. s65 00:05:57.440 --> 00:06:06.320 But, because you want to go from the root to the leaf, all you have to do is describe all the paths and that's a very small finite list. s66 00:06:06.320 --> 00:06:13.680 For example, here you can see that to get to a GPU, you can just you just need four different layers to get to it. s67 00:06:13.680 --> 00:06:19.120 To get to a chiller, similarly four layers to get to switches, similarly four layers. s68 00:06:19.120 --> 00:06:31.880 With this, 64 GPU layer sorry 64 GPU system and a 460,000 GPU system produce roughly the same size of summaries. s69 00:06:31.880 --> 00:06:43.840 This consolidated context gives your LLM all it needs to know to figure out how the distribute how a plant is arranged and how different equipment are distributed in that plant. s70 00:06:43.840 --> 00:06:49.960 The second insight that we had was LLMs are good for planning but not good for searching. s71 00:06:49.960 --> 00:06:54.880 This is what we realized when we saw very poor recall with our sharded solutions. s72 00:06:54.880 --> 00:07:06.040 So, instead of making the LLM sift through all the different fuzzy names that we can get from our users, we asked the LLM to give us exactly how to look for them. s73 00:07:06.040 --> 00:07:11.360 So, for example, here the query says, "Give me all the GPUs that are running hot in data hall 11." s74 00:07:11.360 --> 00:07:20.680 The LLM does not need to go through all these names of all our GPUs to figure out which one of those are in data hall 11 and then figure out how they are running hot. s75 00:07:20.680 --> 00:07:26.480 All it needs to do is structured outputs which tells us that well, we need to collect GPUs. s76 00:07:26.480 --> 00:07:37.320 The scope under which we need to collect is a subtree which is just data hall 11 and the filter that we need to apply to finally figure out what exactly we need is GPUs that are running hot. s77 00:07:37.320 --> 00:07:41.919 Now, this can be different things depending on the context and you can implement different filters. s78 00:07:41.919 --> 00:07:47.800 All we need to know is how do we create the set that we want to know get to. s79 00:07:48.000 --> 00:07:59.440 The third thing that we realized is that once you have a structured output to figure out what exactly you need, uh building the back end for it is relatively simple and straightforward. s80 00:07:59.440 --> 00:08:14.360 Now, all we need to do is create different uh subsets of uh our equipment or pre-index trees as we like to call them based on their location and based on how they interact with other uh equipment around to get an idea of what you need to look at. s81 00:08:14.360 --> 00:08:19.640 So, for example, from the previous uh slide we saw that we wanted to look at GPUs in data hall 11. s82 00:08:19.640 --> 00:08:27.720 We could just get all the GPUs in data hall 11 in one pre-index subtree or other data halls, for example, or other racks, for example. s83 00:08:27.720 --> 00:08:35.800 And then to get to the final result, all we need to do is uh run a query to find all GPUs that are running hot and take their intersection. s84 00:08:35.800 --> 00:08:48.560 Set operations ensure that we have perfect recall and accuracy irrespective of uh how we want to filter uh the queries and what uh you know, fuzziness the user may have in their query. s85 00:08:48.839 --> 00:08:50.040 Next slide, please. s86 00:08:50.040 --> 00:08:50.440 Yes. s87 00:08:50.440 --> 00:08:55.520 And finally, uh what if you have something that is very, very vague? s88 00:08:55.520 --> 00:09:07.600 In which case, uh you have to get the LLMs to get to searching, but instead of searching directly over names, we figured out that getting the LLMs to uh give us patterns to look for. s89 00:09:07.600 --> 00:09:17.160 This could be uh patterns in the data but in the names can be much more useful to figure out uh what the user is talking about instead of just passing the entire s90 00:09:17.160 --> 00:09:19.680 name list uh to the LLM. s91 00:09:19.680 --> 00:09:23.600 The LLM never has to see uh very large number of tokens. s92 00:09:23.600 --> 00:09:35.200 All it needs to do is uh see some patterns in the naming convention, figure out what exactly the user is talking about, create a pattern, and then we can execute it uh on the back end on our side. s93 00:09:35.200 --> 00:09:42.160 This makes sure that the LLMs has a constant or relatively constant uh cost of operation. s94 00:09:42.160 --> 00:09:53.560 Whereas if we had to read through everything, we would have been scaling linearly with increasing equipment count, which itself grows exponentially as the size of the system increases. s95 00:09:54.360 --> 00:10:10.200 Finally, to process a user query end-to-end, we go from the user's query, which could be anything from one equipment or a group of equipment, to a planner LLM, which figures out user's intent and gives us a search pattern. s96 00:10:10.200 --> 00:10:25.360 Based on that search plan, we have a deterministic resolver, which indexes, does set operations, figures out exactly what we need to do, and creates the final set that maps to the user's query, and this is what we call the result set. s97 00:10:25.360 --> 00:10:33.800 All of this is a two or three-step process instead of a multi-step agentic loop, which can keep on running over and over again. s98 00:10:33.800 --> 00:10:39.120 And this keeps our total cost also relatively flat and constant. s99 00:10:40.000 --> 00:10:40.760 Cool. s100 00:10:40.760 --> 00:10:46.480 Yeah, if Rahul's job is to design the architecture, my job is production readiness. s101 00:10:46.480 --> 00:10:59.000 So, we need to make sure that whatever we designed, the elegant solutions we came up, actually hold up under real load, real customer data, and all the messy edge cases you only ever see production. s102 00:10:59.000 --> 00:11:04.800 So, before any of this went near a customer, we put it through some extensive tests and evals. s103 00:11:04.800 --> 00:11:13.400 We measured the new system head-to-head against the old ones with same LLM model, same data, and we did three runs per case just to make sure. s104 00:11:13.400 --> 00:11:18.120 Our goal was simple, prove that Rahul's solutions are ready for production. s105 00:11:18.120 --> 00:11:19.680 And they definitely are. s106 00:11:19.680 --> 00:11:24.320 If you look at some stats here, the old approach degraded pretty fast at scale. s107 00:11:24.320 --> 00:11:33.480 So, we got 80% correctness at 64 GPUs, and that dropped to about 30% when the GPU grew to 400 460,000. s108 00:11:33.480 --> 00:11:42.400 On the other hand, the our new approach has maintained correctness with 100% accuracy across all of those um tests that we give it. s109 00:11:42.400 --> 00:11:44.120 And this is not just synthetic test. s110 00:11:44.120 --> 00:11:46.280 Uh this is also a real data. s111 00:11:46.280 --> 00:11:55.600 So, we we had 66 cases on six real production systems and they product they produce zero failures as well. s112 00:11:56.080 --> 00:11:59.760 And not just correctness, it's also dramatically lighter. s113 00:11:59.760 --> 00:12:11.280 So, when we talk about 1 GW scale uh data center, the old approach burned 116 million tokens for just a single validation pass while still having a lot of errors. s114 00:12:11.280 --> 00:12:19.440 If you look at the new one, it's a 390,000 tokens, which comes to around 300 less tokens fewer tokens. s115 00:12:19.440 --> 00:12:22.840 But the part that matters the most, the cost is flat. s116 00:12:22.840 --> 00:12:33.240 As as Rahul was talking about before, the the system grows in size, but we the cost of the query was 9,000 tokens a query where the system was 64 GPUs s117 00:12:33.240 --> 00:12:35.360 or 460,000. s118 00:12:35.520 --> 00:12:42.760 So, instead of getting the cost grow exponentially w- when the customers grow their systems, it stays the same. s119 00:12:43.600 --> 00:12:44.720 That's the impact. s120 00:12:44.720 --> 00:12:50.520 Now, there's something about uh what we learned and the part I want you to take back to your own work. s121 00:12:50.520 --> 00:12:53.040 It starts with a lens from Karpathy. s122 00:12:53.040 --> 00:12:57.600 Most of you have probably seen his presentation or his framing talk about this before. s123 00:12:57.600 --> 00:12:59.760 He talks about different kinds of software. s124 00:12:59.760 --> 00:13:02.720 Software 1.0 is a deterministic code you write. s125 00:13:02.720 --> 00:13:06.520 It's predictable, exact, but a lot less flexible. s126 00:13:06.520 --> 00:13:12.600 On the other hand, we have software 3.0, which is basically a behavior you prompt out of an LLM. s127 00:13:12.600 --> 00:13:17.800 In our case, that might be show me the GPUs in data hall 11 that are running hot. s128 00:13:17.800 --> 00:13:22.320 The This is very flexible, smart, but it's a bit fuzzy. s129 00:13:22.320 --> 00:13:32.720 Now, his observation was that in legacy software, the 3.0 is steadily eating 1.0. More of what used to be deterministic code becomes a prompt. s130 00:13:32.720 --> 00:13:40.120 Just hold that picture because the lesson for new built AI native systems runs a little bit the other way. s131 00:13:41.480 --> 00:13:46.320 But the real skill is knowing which work is the LM not best for. s132 00:13:46.320 --> 00:13:52.480 And you always want to keep the thing the L LM should be used for the things that it does well. s133 00:13:52.480 --> 00:14:02.720 It's great at parsing ambiguous requests, judging where to look for data and what to look for, handling phrasing we've never seen from a new user that has a different query, s134 00:14:02.720 --> 00:14:08.080 and at the end also synthesizing and writing the final human-readable answer. s135 00:14:08.080 --> 00:14:12.720 But But everything you you can data model, you should move into code. s136 00:14:12.720 --> 00:14:15.360 This is the key one especially for large systems. s137 00:14:15.360 --> 00:14:24.240 If your data has structure, call it a hierarchy, graph, or a schema, a language model scanning it token by token is definitely the wrong tool. s138 00:14:24.240 --> 00:14:37.920 Both retrieval, exact set logic, counting, dedup across near identical names, which happens a lot in the data center land, anything that must be 100% reproducible, it should be a deterministic code. s139 00:14:37.920 --> 00:14:45.160 The simple heuristic that usually works, if you can write down the structure or the rules, it's a 1.0 job. s140 00:14:45.160 --> 00:14:54.720 And pure LM is weakest exactly when the system is large and well structured, which is precisely where we operate and our customers. s141 00:14:55.000 --> 00:14:58.400 So, in a sense, we ran Carpathia's trend backwards. s142 00:14:58.400 --> 00:15:06.440 We started almost pure 3.0. We threw everything in the context window because that is the fastest way to find out what's even worth building. s143 00:15:06.440 --> 00:15:10.600 And as we said, it worked pretty well on a simple demo to start. s144 00:15:10.600 --> 00:15:21.240 Then, as it had to really scale, we moved the parts that can be treated as known engineering problems into 1.0. And we still kept the hard judgment in the LM. s145 00:15:21.240 --> 00:15:22.600 So, that's the inversion. s146 00:15:22.600 --> 00:15:34.480 Legacy software drifts from 1.0 towards 3.0 and new AI native software starts at 3.0 and matures towards 1.0 for the use cases that earn it, of course. s147 00:15:34.480 --> 00:15:36.680 We are not here to replace model judgment. s148 00:15:36.680 --> 00:15:41.680 We want to feed it with the 1.1.0 tools, as we call it. s149 00:15:41.680 --> 00:15:48.360 So every 1.0 function you add is more reliable ground for the LLM to stand on. s150 00:15:48.360 --> 00:15:52.120 So to finish up, let the LLM keep the hard decisions. s151 00:15:52.120 --> 00:16:00.720 Everything it shouldn't be guessing on, add add it in as code and hand that structure back to the model to work with. s152 00:16:00.720 --> 00:16:09.440 So develop with software 3.0, productionize by adding in software 3.0 1.0. Thanks for watching everyone. s153 00:16:09.440 --> 00:16:20.800 Uh any follow-up questions at all, you can reach us by email Rahul Russell Google at fidra.ai, Lanza at least to get fidra.ai, or you can drop them in the video comments below. s154 00:16:20.800 --> 00:16:22.560 Thank you for watching. s155 00:16:22.560 --> 00:16:24.560 Thanks.