WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=RdQCY_-SXbA NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/RdQCY_-SXbA.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.640 --> 00:00:13.440 Hi everyone. s2 00:00:13.440 --> 00:00:15.160 My name is Venu Ganesh. s3 00:00:15.160 --> 00:00:16.960 Um, excited to be here today. s4 00:00:16.960 --> 00:00:20.360 Uh, I'm the CEO and co-founder of Kepler. s5 00:00:20.360 --> 00:00:26.320 Um, this is going to be kind of an interesting talk because it comes from what happened 3 months ago. s6 00:00:26.320 --> 00:00:35.520 So, 3 months ago, uh, I think there's a little bit of an overlap, but I gave a talk at AIE World Fair, uh, uh, really around verification, um, and why that s7 00:00:35.520 --> 00:00:42.320 most institutions need verification in some form of the trust ecosystem of AI. s8 00:00:42.320 --> 00:00:52.840 Well, yesterday most of you saw Jack and I were on stage talking about why Jack, who ran Citadel's biggest book, and started Candlestick Capital, still doesn't trust AI. s9 00:00:52.840 --> 00:01:03.320 Since that 3-month period, there's been about a 270 million dollars of VC money that got poured into formal verification layers. s10 00:01:03.320 --> 00:01:10.680 A VC actually took Kepler's own pitch deck and gave it to a competitor, and produced something that has our same tagline. s11 00:01:10.680 --> 00:01:14.480 I'm not bitter, just a little, but the funny thing is they copied the wrong part. s12 00:01:14.480 --> 00:01:21.840 And so, what I want to do is actually give you, the practitioners today, you can't just tack verification on at the end. s13 00:01:21.840 --> 00:01:33.720 And so, uh, when I gave that chat or that talk a few, um, months ago, uh, Anthropic had just published an article on how Kepler does what it does, and why it's really powerful and and really unique. s14 00:01:33.720 --> 00:01:42.200 So, what I want to do is get into the technical side of how we do the thing that we do, and ultimately this is driven by one very simple insight, s15 00:01:42.200 --> 00:01:45.360 Wall Street still doesn't trust AI. s16 00:01:45.360 --> 00:01:47.240 Most people think it's because of this. s17 00:01:47.240 --> 00:01:49.320 Uh, AI will estimate things. s18 00:01:49.320 --> 00:01:50.720 It will hallucinate things. s19 00:01:50.720 --> 00:01:53.560 A frontier model will make stuff up. s20 00:01:53.560 --> 00:01:56.160 And ultimately it is, and that's a component of it. s21 00:01:56.160 --> 00:02:03.972 But the actual core underlying problem is we have a structural problem in this technology that we're all so desperate to use. s22 00:02:03.972 --> 00:02:14.412 Meaning, if you actually chat with people that are building rag platforms, most of what they'll say is, "Well, the models will eventually get good or get better to the point they will eliminate hallucination." s23 00:02:14.412 --> 00:02:16.732 That is a nonsensical term. s24 00:02:16.732 --> 00:02:18.812 You know another word for hallucination? s25 00:02:18.812 --> 00:02:20.132 Alpha. s26 00:02:20.132 --> 00:02:21.812 Alpha is how a firm makes money. s27 00:02:21.812 --> 00:02:28.932 The creativity that you have and that your analysts have is a mechanism through which you can be a unique firm. s28 00:02:28.932 --> 00:02:31.332 Otherwise, there should be only one company on Earth. s29 00:02:31.332 --> 00:02:42.692 So, when you ask these types of simple questions around where data comes from, there is a piece of this that the model will just make things up, and there's a piece of this that your firm's internal data is never actually reflected s30 00:02:42.692 --> 00:02:45.372 in this setup. s31 00:02:45.372 --> 00:02:46.972 So, what are we seeing right now? s32 00:02:46.972 --> 00:02:50.212 We're seeing everyone with a checkbook buy everything they can. s33 00:02:50.212 --> 00:02:56.092 Every rag platform, some firms are trying to build, some firm some firms are running agent pilots. s34 00:02:56.092 --> 00:03:04.612 I had a a customer meeting this morning where someone said they were actually an AI-first firm, which means they have one co-pilot agent running all the stuff. s35 00:03:04.612 --> 00:03:12.332 So, why are we not seeing the complete switch of fundamental investing becoming effectively, why are we not all Jane Street? s36 00:03:12.332 --> 00:03:14.572 Why Why is that not happened yet? s37 00:03:14.572 --> 00:03:26.052 And so, the challenge really comes from this idea of in order for you to produce work product that you believe in, you have to trust it, meaning you have to blame somebody who gets it wrong. s38 00:03:26.052 --> 00:03:33.572 So, we see numbers like this all the time, where 80% of finance execs trust numbers, but then 31% find those errors. s39 00:03:33.572 --> 00:03:36.012 So, what do we actually need? s40 00:03:36.012 --> 00:03:45.252 We need a situation where the reviews that we're getting for AI apps, as builders ourselves, do more than just ragify things. s41 00:03:45.252 --> 00:03:55.852 And And these are quotes we've actually pulled from specific uh I don't I don't want to say our competitors, but other people, where things demo super well and they behave like a really smart intern. s42 00:03:55.852 --> 00:04:03.092 Or things are right if you just hook up the right structured information, but that last 10% still gets things wrong. s43 00:04:03.092 --> 00:04:11.734 So, what I want to do is we're lucky, we have great partners and folks like Turbo puffer and I gave a similar chat yesterday about this, but how do we actually do the verification? s44 00:04:11.734 --> 00:04:17.374 How do we architecturally build a system that does not hallucinate numerically? s45 00:04:17.374 --> 00:04:18.854 So, let's talk about it. s46 00:04:18.854 --> 00:04:23.174 Now, wrong numbers or wrong facts come from three sources. s47 00:04:23.174 --> 00:04:26.334 Either one, you retrieved the wrong document. s48 00:04:26.334 --> 00:04:30.094 So, I pulled a document, some random thing from somewhere else. s49 00:04:30.094 --> 00:04:31.134 How do you fix this? s50 00:04:31.134 --> 00:04:33.854 Well, by retrieving the right document. s51 00:04:33.854 --> 00:04:35.854 But, that in itself is complicated. s52 00:04:35.854 --> 00:04:40.494 Is the right document my own firm's internal analysis of what this stock is going to be worth? s53 00:04:40.494 --> 00:04:45.974 Is it, let's say, uh some analyst who I really trust more than anyone else? s54 00:04:45.974 --> 00:04:47.374 That's piece one. s55 00:04:47.374 --> 00:04:48.654 Now, failure two. s56 00:04:48.654 --> 00:04:52.334 We get the right source and we pull the wrong number out of that source. s57 00:04:52.334 --> 00:04:58.694 Let's say we're trying to figure out what the revenue for, I don't know, Palantir was back in 2023. s58 00:04:58.694 --> 00:05:08.894 Well, if I look at the 2025 10K that says Palantir's 2023 revenue was this, I better make sure that that number corresponds to 2023 and not 2025. s59 00:05:08.894 --> 00:05:13.614 So, the fix here is really we have a filter that actually checks each citation. s60 00:05:13.614 --> 00:05:18.254 But, this is still a problem because that filter only applies after the fact. s61 00:05:18.254 --> 00:05:23.294 And formal verification doesn't actually help here because the thing has already been produced. s62 00:05:23.294 --> 00:05:24.974 Three is the mental math. s63 00:05:24.974 --> 00:05:26.534 Models don't do math. s64 00:05:26.534 --> 00:05:28.414 They predict the next token. s65 00:05:28.414 --> 00:05:32.134 Now, for those of you who are going to say we have deterministic models, great, we do. s66 00:05:32.134 --> 00:05:36.334 We also have one CPU cycle that's good at adding two numbers together. s67 00:05:36.334 --> 00:05:38.414 Why don't I use that CPU cycle? s68 00:05:38.414 --> 00:05:49.334 So, third failure is let's use deterministic tools that we already have to actually compute numbers and entities rather than relying on some external tool. s69 00:05:49.454 --> 00:05:50.254 So, here's what we do. s70 00:05:50.254 --> 00:05:54.614 This is the magic behind Kepler, at least for piece one, there's a whole bunch of other stuff. s71 00:05:54.614 --> 00:05:57.254 We are architected fundamentally differently. s72 00:05:57.254 --> 00:06:03.934 Code writes every number and what code does in pre-processing is we assign pointers to unique identifiers. s73 00:06:03.934 --> 00:06:05.534 Meaning facts. s74 00:06:05.534 --> 00:06:08.575 Meaning something like a number in a 10K. s75 00:06:08.575 --> 00:06:12.535 This operates in a giant pre-processing and contextualization step. s76 00:06:12.535 --> 00:06:17.655 Such that all of this exists, going from unstructured data to a structured data store. s77 00:06:17.655 --> 00:06:20.695 Then, the model is effectively a smart analyst. s78 00:06:20.695 --> 00:06:27.575 It figures out how to go about solving the problem, but not actually being the thing to solve the problem itself. s79 00:06:27.575 --> 00:06:29.215 So, it plans. s80 00:06:29.215 --> 00:06:34.455 It says, "I'm going to go here and retrieve this source and retrieve this entity and do this thing." s81 00:06:34.455 --> 00:06:36.735 Code and math do the retrieval. s82 00:06:36.735 --> 00:06:43.615 By actually pulling a number out of a 10K at like a search time, that's a hugely expensive token cost. s83 00:06:43.615 --> 00:06:49.615 And Anthropic and OpenAI should thank you for paying all those tokens in token maxing, but most of your firms don't like that, at least anymore. s84 00:06:49.615 --> 00:06:52.815 I know there were people getting bonuses from token maxing for a while. s85 00:06:52.815 --> 00:07:02.935 Um Now, the next piece of this is really once the model's actually read both these pointers, it can't structurally generate numbers. s86 00:07:02.935 --> 00:07:12.775 Meaning unless the model actually has a mechanism of dereferencing that pointer, that number or that fact cannot be generated in Kepler's output. s87 00:07:12.775 --> 00:07:16.215 Now, we do run a post-processing step, which is the verification step. s88 00:07:16.215 --> 00:07:24.335 And this is the piece that was kind of funny where the last company just copied the verification piece, but all the structural stuff beforehand didn't make it. s89 00:07:24.335 --> 00:07:34.855 So, what you as a consumer actually see is you see something like the revenue is this amount to this amount, and you physically can click on these chips and see exactly how we got the information. s90 00:07:35.015 --> 00:07:37.215 Now, in my past life, I had an opportunity. s91 00:07:37.215 --> 00:07:39.295 I I was head of business engineering at Citadel. s92 00:07:39.295 --> 00:07:46.695 So, I worked across um the firm to figure out effectively what do we need to produce software that helps us generate alpha. s93 00:07:46.695 --> 00:07:48.415 You know the crazy thing with Citadel? s94 00:07:48.415 --> 00:07:55.455 PMs are so talented and so smart, but they work when they're paired with deterministic tools and technology. s95 00:07:55.455 --> 00:07:58.655 And that's kind of the magical insight behind fundamental investing. s96 00:07:58.655 --> 00:08:04.935 A fundamental investor so good at understanding data, but that data has to be right. s97 00:08:04.935 --> 00:08:14.577 And so, when you build a system like Kepler, what you effectively have to do is look up the right information and then be clever about the information you don't have. s98 00:08:14.577 --> 00:08:16.097 So, let me give you the parallel. s99 00:08:16.097 --> 00:08:27.097 Most folks and most buy-side researchers or buy-side investors try to use tooling or build DCFs on their own and then they find and search for information they don't have. s100 00:08:27.097 --> 00:08:29.337 They schedule GLG or Guidepoint calls. s101 00:08:29.337 --> 00:08:30.497 They find experts. s102 00:08:30.497 --> 00:08:32.457 They speak to management. s103 00:08:32.457 --> 00:08:34.977 The same system actually applies here. s104 00:08:34.977 --> 00:08:41.977 So, what we do is we pick the path and we store the information deterministically that we can actually store deterministically. s105 00:08:41.977 --> 00:08:53.217 When we have information we can't store deterministically, we run similar preprocessing steps to grab information, store that information in a hybrid storage mechanism, which is a combination of Turbo Buffer, s106 00:08:53.217 --> 00:08:56.457 which is awesome by the way, and a bunch of other um mechanisms. s107 00:08:56.457 --> 00:09:07.857 And then we ultimately use a combination of re-ranking with literally old-school like pointer dereferencing like dot get methods to pull things into our system. s108 00:09:07.857 --> 00:09:13.097 Now, why this matters is a lookup cannot get it wrong. s109 00:09:13.097 --> 00:09:14.337 Search can. s110 00:09:14.337 --> 00:09:21.457 If you're running the same 10 searches with the same query, the probabilistic nature of models means there's inherent entropy. s111 00:09:21.457 --> 00:09:23.537 That's actually an opportunity. s112 00:09:23.537 --> 00:09:26.097 And so, that's what we've changed here. s113 00:09:26.097 --> 00:09:31.777 Now, how we do this is we break this down and this occurs across of the SEC world as well. s114 00:09:31.777 --> 00:09:36.177 Um we are frequently throwing candidate documents away. s115 00:09:36.177 --> 00:09:45.897 Meaning we consistently have to look at not only what source is the right source for what information, but also which documents should we stop looking at kind of globally. s116 00:09:45.897 --> 00:09:50.297 So, uh we run ANN plus BM25 to get back a set of documents. s117 00:09:50.297 --> 00:09:51.937 It's usually up to 200. s118 00:09:51.937 --> 00:09:55.417 Uh we do reciprocal uh ranking We effectively get 60 documents back. s119 00:09:55.417 --> 00:09:57.257 It's probably the best way to think about this. s120 00:09:57.257 --> 00:10:00.337 Um then we do cross-encoders. s121 00:10:00.337 --> 00:10:08.137 We try to figure out exactly the query and the actual document chunked together and get those results that way. s122 00:10:08.137 --> 00:10:11.858 What that allows us to do is we can search really wide. s123 00:10:11.858 --> 00:10:13.738 You know what LLMs aren't good at doing? s124 00:10:13.738 --> 00:10:15.498 Searching really wide. s125 00:10:15.498 --> 00:10:26.818 What they try to do is create a plan and find either the fastest or easiest breadth-first search mechanism actually made depth-first search mechanism of actually like getting the first document and moving forward. s126 00:10:26.818 --> 00:10:31.498 So we allow this to search really wide and then narrow from there. s127 00:10:31.498 --> 00:10:35.818 The reason LLMs don't like searching wide is because it's expensive. s128 00:10:35.818 --> 00:10:45.938 It's because looking across a bunch of different sources is significantly worse than getting the reward function back with your first hit and going from there. s129 00:10:46.258 --> 00:10:51.578 Okay, so um I want to talk a little bit about security here cuz it's super important. s130 00:10:51.578 --> 00:10:55.898 Kepler was built with a security-first mindset and security-first mentality. s131 00:10:55.898 --> 00:11:02.098 Now client documents, and this is specifically targeted towards the builders here, this is a thing that no one's chatting about. s132 00:11:02.098 --> 00:11:11.098 No one is actually talking about the security of what OAuth token is used and what user-scoped mechanism of retrieving documents, which is actually pretty scary. s133 00:11:11.098 --> 00:11:16.738 So let's talk about how things go wrong in these systems that people are trying to build and how you can fix it. s134 00:11:16.738 --> 00:11:19.378 First and foremost, you forget to filter the tenant. s135 00:11:19.378 --> 00:11:23.818 All of a sudden Citadel can search through all the Millennium's documents, you will be in a lot of trouble. s136 00:11:23.818 --> 00:11:26.618 Second, a caller claims to be another firm. s137 00:11:26.618 --> 00:11:32.138 Meaning I can claim that I'm impersonating this other entity and querying documents that way. s138 00:11:32.138 --> 00:11:35.538 Third, search returns information that's out of date. s139 00:11:35.538 --> 00:11:39.378 We've had this problem kind of globally and even Google search results. s140 00:11:39.378 --> 00:11:41.618 Fourth, direct read skips the search rules. s141 00:11:41.618 --> 00:11:48.938 Meaning you're not actually querying the full corpus of information or like there's not an allow list, you only get back the single document that you did the search. s142 00:11:48.938 --> 00:11:52.178 Five, you hit an index that isn't populated. s143 00:11:52.178 --> 00:12:01.178 The way to actually solve all these problems, and there's individual things listed here, is actually scoping searches with security built in. s144 00:12:01.178 --> 00:12:06.538 That's kind of another critical piece that I think a lot of people miss when they're trying to build these systems themselves. s145 00:12:06.538 --> 00:12:11.778 Inevitably, people will start caring about the security associated with querying this information. s146 00:12:11.778 --> 00:12:15.699 Alpha is the most elusive thing out there, especially for folks on Wall Street. s147 00:12:15.699 --> 00:12:24.979 So, making sure you have a good mechanism of actually constraining the set of documents that come back with each query is really important. s148 00:12:25.379 --> 00:12:26.139 Okay. s149 00:12:26.139 --> 00:12:29.099 Now, let's talk about verification, which is the last piece here. s150 00:12:29.099 --> 00:12:33.499 This is probably what we're known most for, but we do the other stuff I mentioned as well. s151 00:12:33.499 --> 00:12:40.579 Um how we do this is by pre-processing information and storing it deterministically. s152 00:12:40.579 --> 00:12:45.059 We have systems like code that actually write individual pointers. s153 00:12:45.059 --> 00:12:50.939 The model can only then read those pointers and grab information back if they exist in the database. s154 00:12:50.939 --> 00:12:55.299 If I grab something out of a hash map that doesn't exist, I don't get anything back. s155 00:12:55.299 --> 00:13:03.139 And so, this combination is what allows us to actually make sure the information that we retrieved actually came from the source that it came from. s156 00:13:03.139 --> 00:13:18.939 Now, this is a kind of hidden version of what our source IDs look like, but these are hashes, and these are very simple things that you can do in your own system and your own setup to ensure that you're pulling back facts that in fact do exist. s157 00:13:19.059 --> 00:13:20.059 Cool. s158 00:13:20.059 --> 00:13:22.619 So, that's a little bit about how we do verification. s159 00:13:22.619 --> 00:13:27.979 Now, I want to chat a little bit about what this enables kind of globally. s160 00:13:28.019 --> 00:13:34.299 This enables us to verify facts that come out of our grounded corpus of data. s161 00:13:34.299 --> 00:13:43.299 Now, you as a builder, or you as a startup creator, or you as someone who's trying to figure out how do I trust my outputs, really just need you to do a couple of things. s162 00:13:43.299 --> 00:13:52.619 You need to show people that as you're creating these agents that are theoretically looking for alpha, or whatever they'd be looking for, you're using information that is actually relevant s163 00:13:52.619 --> 00:13:58.579 and concrete, and most importantly accurate, to the set of people that you're trying to service. s164 00:13:58.579 --> 00:14:08.259 So, we end up with a contract where citations are either verified, corrected at search time, or flat out refused. s165 00:14:08.259 --> 00:14:11.139 This is crazy if you think about it. s166 00:14:11.139 --> 00:14:17.461 If a model cannot deresolve that pointer, it cannot print that number out. s167 00:14:17.461 --> 00:14:19.861 So, that's kind of an insane thing. s168 00:14:19.861 --> 00:14:23.781 It means that the whole and not just a number, let's talk about facts. s169 00:14:23.781 --> 00:14:28.341 Like we see these court cases all over of precedents being hallucinated. s170 00:14:28.341 --> 00:14:39.301 Well, why can't we create a system that just has the precedent stored deterministically and allows us to search through that list and not have a random court case that never existed all of a sudden start existing? s171 00:14:39.301 --> 00:14:44.541 So, this tech and this this mechanism and this technology actually becomes fairly globally applicable. s172 00:14:44.541 --> 00:14:56.501 And the reason I'm on stage telling you how we do what we do is because I'm terrified that Wall Street never trusts this technology because of the fear that so many other companies or so many build-it-at-home s173 00:14:56.501 --> 00:15:01.061 folks have been uh instituting in these organizations. s174 00:15:01.061 --> 00:15:08.781 It's It's gotten to the point where the vast majority of finance AI tools right now are being used to format PowerPoint presentations. s175 00:15:08.781 --> 00:15:13.261 We still haven't actually built a DCF cradle to grave despite what most people are saying. s176 00:15:13.261 --> 00:15:18.261 And if we had, every single company on Wall Street would have adopted that overnight. s177 00:15:18.261 --> 00:15:19.221 So, this is my fear. s178 00:15:19.221 --> 00:15:21.461 My fear comes from a very simple fact. s179 00:15:21.461 --> 00:15:29.781 For us to actually get value out of this technology, we need to be able to build technology that people actually can trust and can verify. s180 00:15:29.781 --> 00:15:32.661 That isn't just true in finance, it's true in a bunch of different industries. s181 00:15:32.661 --> 00:15:40.541 So, please take what you need to from this presentation and get people using this technology, otherwise we're all going to be out of a job. s182 00:15:40.541 --> 00:15:42.941 Now, I want to try through what happened when we saw this. s183 00:15:42.941 --> 00:15:49.221 So, when we went through this whole process, um this is actually some of our our real production data here. s184 00:15:49.221 --> 00:15:55.781 Um refusals are when the model does not is not able to produce a number. s185 00:15:55.781 --> 00:16:02.221 Meaning, the model will say, "I couldn't dereference this pointer, therefore I could not produce this number." s186 00:16:02.221 --> 00:16:04.741 We saw that drop super drastically. s187 00:16:04.741 --> 00:16:10.461 Now, the reason that it didn't drop to zero is because models are still probabilistic. s188 00:16:10.461 --> 00:16:14.381 Meaning, it can still make up numbers after the fact during inference. s189 00:16:14.381 --> 00:16:18.862 So, there was another pre- a post-processing step after this that we we have as well. s190 00:16:18.862 --> 00:16:29.422 But my point with this slide is just by creating a system where the models have to dereference their own pointers, you can actually create a very powerful, very verifiable s191 00:16:29.422 --> 00:16:31.262 AI platform. s192 00:16:31.262 --> 00:16:33.102 And that to me is my ask to all of you. s193 00:16:33.102 --> 00:16:40.382 As you go through the building process, as you go through the scoping process, any tool that you use, ask them if you can actually verify the numbers. s194 00:16:40.382 --> 00:16:50.062 Ask them if you can see where the sources are, not in terms of some after-the-fact annotation, but click on the number, whether it's a calculation, whether it's an extraction, s195 00:16:50.062 --> 00:16:54.662 or some combination of the two, to make sure you trust the insight. s196 00:16:54.662 --> 00:16:58.022 And all of this ultimately is going to be the unlock. s197 00:16:58.022 --> 00:17:03.062 So, you as a vendor or you as a builder, ask these questions. s198 00:17:03.062 --> 00:17:04.542 Does the model actually write pointers? s199 00:17:04.542 --> 00:17:07.902 Does it actually create a mechanism for verifying the output? s200 00:17:07.902 --> 00:17:10.262 Is every citation source checked? s201 00:17:10.262 --> 00:17:12.582 Is arithmetic source checked? s202 00:17:12.582 --> 00:17:18.182 Do you have a mechanism for stopping files that reach clients or customers that it shouldn't reach? s203 00:17:18.182 --> 00:17:22.262 And most importantly, what happens to a number that came from nowhere? s204 00:17:22.262 --> 00:17:25.182 What happens when the when a fact comes from nowhere? s205 00:17:25.182 --> 00:17:29.102 And how do I actually handle these kinds of of hallucinations? s206 00:17:29.422 --> 00:17:39.422 So, this whole platform is actually to make a very simple statement very obvious, which is we can have the smartest models in the world. s207 00:17:39.422 --> 00:17:51.702 We can hit some crazy cool piece of technology, but we still have the same database problems that we've had from the beginning, which are I cannot if I search through a s208 00:17:51.702 --> 00:17:56.422 a row and that column does not value doesn't exist, I won't get a retrieval back. s209 00:17:56.422 --> 00:17:58.622 We're now out of the search phase of AI. s210 00:17:58.622 --> 00:18:00.822 We need to be building infrastructure. s211 00:18:00.822 --> 00:18:09.182 We need to be building scaffolding and harnesses and substrates or whatever the cool VC term is these days that actually enable us to make use of this technology. s212 00:18:09.182 --> 00:18:11.822 So, this is my challenge and my ask to all of you. s213 00:18:11.822 --> 00:18:18.022 Um I hope you go out there and build something compelling, but I hope you go build something compelling that you can trust. s214 00:18:18.022 --> 00:18:19.757 Thank you. s215 00:18:19.757 --> 00:18:21.757 [applause] s216 00:18:27.543 --> 00:18:29.543 Hey.