How Kepler Built Verifiable AI for Financial Services — Vinoo Ganesh

AI Engineer · 22 min · 260 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hey everyone, thank you for being here.
00:14My name is Venu Ganesh and I'm the CEO and co-founder of Kepler.
00:18Today, I'm going to talk to you about how we built verifiable AI for financial services.
00:24First, a little about me.
00:25My career has been working in fairly difficult places to work in terms of numerical accuracy and verifiability.
00:33Began my career at Palantir where I led the compute platform as well as a lot of our USG engagements.
00:38Built and sold a alt data startup.
00:40Then was head of business engineering at Citadel.
00:43I have some Citadel colleagues here in the audience as well.
00:46I've advised a bunch of startups and been lucky that all of them have reached pretty positive outcomes.
00:51So this whole talk is a distilled version of a case study Anthropic did on Kepler.
00:56If you scan that QR code, we're the only company they've ever done a case study of, which is kind of cool.
01:03And so this is a distilled version of that.
01:05That has a lot more information about how we were able to do what we do, why it's been pretty impactful in financial services, and really where we go from here.
01:15So my only goal with this whole talk is to convince you that AI is going to start doing some very powerful things in terms of producing work product.
01:24So everything that I tell you is inevitable.
01:27Whether it's Kepler or whether it's anyone else, we are already on this journey and this trajectory.
01:33So we all better be ready.
01:36So I first want to observe every talk that we've seen in the financial services space so far has been about producing more.
01:43Like how do I token max?
01:45How do I get AI to do more?
01:47Very little of it is about how to actually make AI trustworthy or produce trustworthy products.
01:54And what's interesting is even terms like trust and verify-ability have actually been largely abused.
01:59Evals are not verifiable.
02:01You cannot take a non-deterministic LLM and eval your way to something deterministic.
02:06These are probability machines.
02:08And the kind of underlying reason for this is that AI has made a writing problem a reading problem.
02:15We can produce insane amounts of content, whether that's code, whether it's marketing, whether it's like a DCF in record time.
02:26But we can't easily verify this.
02:27And that's because for years the hardest part about this whole process was actually producing the work.
02:33Edge and alpha came from people like Citadel being able to hire hundreds of analysts who could scour the internet and understand where any source of alpha could exist.
02:43So it really came from this idea of being able to consume content.
02:48The problem is when a model reads everything, you have the most real version of alpha decay that you possibly can.
02:54There is no edge if everyone can look at Tegus and get all the same information.
02:59So the hard part now is trusting what actually got produced by the model.
03:04And this is kind of funny.
03:05This is not necessarily a finance problem.
03:08Every system that exists has some form of this.
03:12Software, we run CICD.
03:15We do unit tests.
03:16We do integration tests.
03:17We have code reviews.
03:19When a doctor writes a prescription, a pharmacist fills that prescription.
03:23So if it says 10,000 milligrams of a medication, someone catches that.
03:28We have a pilot and a co-pilot.
03:30We have an EMT that's a primary EMT and a secondary.
03:33In finance, we have maybe an overworked VP as a verification layer, but that concept doesn't really exist.
03:40And now I'm going to say something even more aggressive.
03:42Uh the reason that people buy products like Bloomberg and FactSet is to displace culpability.
03:49When you buy a tool like that, you know that information free.
03:52It exists in SEC filings.
03:54But, you believe that because a bunch of contractors or folks overseas vetted this data and stuck it in a central instance, at least if it's wrong, everyone on Wall Street has the same incorrect information.
04:06And that's interesting in certain ways.
04:08It's also kind of a scary proposition.
04:11And that's because every one of these tools is read-only.
04:14You as an analyst look at Bloomberg, you look at FactSet, you consume information, and you produce the work product.
04:21And that's where the biggest gap and biggest wall to real meaningful adoption across Wall Street has actually been.
04:28And so, why does this actually matter?
04:30We can't use AI properly in this ecosystem.
04:33There is a reason that analysts are still working till 4:00 a.m., and there's a reason folks in investment banks are actually dying because of the hours they're putting in.
04:42Because AI can produce a very confident answer, but when it comes to producing any kind of meaningful work product, we're totally lacking.
04:51Not only that, we have the SEC, we have the OCC, we have a number of these regulatory agencies who exist to make sure that you are not insider trading,
05:00or you're not producing a trading decision that can't be justified or backed by some set of primitives or some set of information that you can reliably say, "I made this decision because of these sources of information."
05:13And so, in this ecosystem, the biggest challenge is how do I get AI to jump from producing the search technologies that it's doing right now to producing work product.
05:26That can mean a fairness opinion, it can mean a DCF, it can mean an investment memo, it can mean looking at every SIM that your firm had 5 years ago and figuring out why your IRR number wasn't what it should have been, or anything else.
05:39And so, the whole industry has solved this by citing things.
05:43When you search the internet with Claude or ChatGPT, it gives you a list of sources that pulled information from.
05:50What's ironic is you can't easily curate those sources.
05:54So, if you find my random Substack that says Palantir is going to be $3,000 a share, and you trade off of that, please go do that because it'll be very helpful for me.
06:02But, that's not a real vetted source.
06:05Seeking Alpha and some of these blogs or Reddit posts, they're data points, but they're not real vetted sources.
06:11So, showing where you got the information from is only half the battle.
06:16And that's where we're limited right now.
06:18The citation is effectively an after-the-fact audit.
06:21Now, a verification is a deterministic, repeatable, numerically verifiable mechanism that we can use to produce validity that a number is right.
06:32That was a lot of buzzwords.
06:34A verification just means I can prove deterministically this number is right.
06:39So, when I extract a revenue number from a 10-K, I can deterministically prove that is the correct number from that 10-K.
06:47So, these are two sides of kind of the same game.
06:50One is showing where you got the sources from, and one is verifying that the information that you pulled out is actually correct.
06:57This becomes challenging.
06:59It becomes challenging because verification is not a outcome.
07:05It lives in the path or the set of steps that are required for you to produce information that is valid for your individual firm.
07:15Finance is one of the rare industries where two people can be have the same information, and one can be long a stock, and one can be short that stock with the exact same data.
07:24And so, the idea of verification is not actually ground truth.
07:28It is verifying that you got an output that respects the nouns and verbs or the rules of your organization.
07:35If a desk at Citadel, like a TMT desk at Citadel, believes that a particular stock is going to go up into the right, another TMT desk at Citadel may have the exact opposite belief,
07:45and they may have the same verification mechanisms that produce vastly different outcomes.
07:50And so, this really comes on to something simple.
07:53What are the sources that we trust?
07:55What are the transformations that we apply on those sources to produce information that we care about?
08:00And how do we make sure that's codified in a way that makes logical sense?
08:04And so, the whole point of this is simple.
08:07This came from a 10K is not the the validation as a whole.
08:12Your job now is to figure out as an individual, how do I use AI in a verifiable way?
08:18And here's the honest answer.
08:19You don't.
08:21AI is great at doing non-deterministic tasks.
08:24It can solve problems in a way that's novel.
08:28It can figure out exactly how to do EBITDA adjustments, but it can't be the one responsible for doing the mathematical adjustments.
08:35Because it's a probability machine.
08:37It is great at next token prediction.
08:39So, the second contention of this whole talk is that you cannot use AI to produce verifiable work product in finance without augmenting it with a deterministic substrate.
08:51Which effectively means, if you're a portfolio manager at Citadel, you have access to a number of deterministic tools that you use to make your trading decisions.
09:01We need to model AI like that PM.
09:03It needs to exist in grounding, needs to exist with a certain risk threshold and verifiability.
09:10So, here's how we did it.
09:11Here's what Anthropic was excited about.
09:13We have three tenants that we use to ensure that numerical accuracy is a tenant of the Kepler platform, and that allows us to produce pretty powerful, pretty verifiably reliable information.
09:26The first is atomic provenance, and I'll talk through all of these.
09:29The second is scope determinism, and the third is derivation chains, uh which we'll all talk about.
09:36But the core premise is this.
09:38There are certain things humans should never do.
09:40And I will tell you right now, I don't believe a human should sit there and look at a PDF 10K, 10Q, 8K, or earnings call transcript and on one screen take that number and put it into an Excel model on the other screen.
09:54Humans were not built for that.
09:55That's not like there's a reason we don't have databases just written in our heads.
10:00And so, let's break this down for how you can actually use AI to produce work product.
10:06Let's talk about provenance.
10:07Provenance as a whole just means writing down exactly where you got the information.
10:12Now, uh have a lot of respect for everyone else who got here on stage, but we did a model we trained a model that was really good at extracting information.
10:20It outperformed foundation models, it's 94% great.
10:23It's in the article.
10:25Who here would trade off of something that's 94% accurate?
10:29Right.
10:29So, fine-tuning your way on probabilistic solutions still is it's really cool and like TechCrunch will be really excited about it, but like no one else really cares about it.
10:38And that's because a wrong number is still wrong if you're in that unfortunate 6%.
10:45So, with atomic provenance, what we do is the model writes effectively a reference to the number.
10:51It cannot write the number or manipulate the number in any way.
10:54It doesn't even understand what that number is.
10:56We have tools that are really good at understanding numbers.
10:59They're databases.
11:00They are systems that can codify information and read them and write them with appropriate fidelity.
11:07So, that's the first piece of how we do things.
11:09Anytime a model makes a decision, it makes the decision to figure out exactly where it got the number from and hand off to something that can write that number.
11:19We then run it through a deterministic check where any kind of a wrong number, if we can't verify it independently, we strip it out.
11:28That number will never make it to someone if it doesn't follow the deterministic check, the Providence ledger, and most importantly, uh the whole cycle kind of repeating at least a couple of times.
11:39So, this is not me saying, "Have 10 models and each individual have OpenAI check chat check Anthropic and have Anthropic check XAI."
11:46These are not probabilistic systems evaluating each other's work.
11:49There's a core canonical process of extracting a number, persisting it, and making sure that process actually occurred properly.
11:58And this is when I say atomic traditional atomicity in like database land.
12:02Second, scope determinism.
12:04This is This was a super controversial idea like a year ago, and VCs were like, "This is crazy."
12:10Now, this half of this talk has been This session has been about this.
12:13Um the model is really good at reasoning and planning.
12:16Intelligence is commoditized.
12:18GLM 52 shows it.
12:19You can download it off of Hugging Face right now.
12:22You have something as powerful as Opus 48.
12:24Now, what the model cannot do is math.
12:28And why would it?
12:29Why would I run 1 + 1 through a multi-billion parameter model instead of one CPU cycle?
12:34Unless you're companies that are giving bonuses on people token maxing, which is another Polarys thing.
12:39[snorts]
12:39Um and so, what the model does is the model decides what to compute.
12:44It never does the computation itself.
12:47And so, from the Kepler platform perspective, what we do is we split the deterministic pieces of the model, which are none, from the non-deterministic pieces of the model, which are all of them.
12:58And we give the model the right tooling and technology to calculate the deterministic pieces.
13:02Now, let's get really concrete.
13:04A model can read something like, "Okay, I need to understand what net margin is.
13:09I know the right pieces of information to go to to get that data, but I can't be the entity that's actually running the code behind the scenes to pull that number out of a PDF
13:18or parsing the XBRL behind the scenes to pull that information out.
13:22That is code."
13:23Now, the deterministic pieces of the platform pull that information out and persist it outside of anything the model understands.
13:30And with those two together, we can actually produce a numerically accurate answer to the question of like, what was the net What was the net margin of this stock or this company last quarter?
13:43This also is a lot cheaper because again, I don't need the model to do a bunch of stuff it shouldn't be doing.
13:49Now, the last piece here is how do we actually do reconciliation?
13:54When someone asks about a ratio like a gross margin or anything else that doesn't exist in a filing, therefore, I can't just go look up what the gross margin is or what the set of EBITDA adjustments were.
14:06The other thing that's really complicated is everyone calculates these ratios and these multiples differently.
14:11Everyone does enterprise value calculations differently.
14:14Things that are considered recurring or non-recurring may be unique.
14:18So, not only do we have to codify that in the processes that are run, but we need some kind of a chain of events to figure out what went into producing an individual number and what went into producing an outcome.
14:29This is not any different than the chain of events that an analyst does on a desk at a hedge fund to make a risk-reward or a trading decision.
14:37It's just done by the model in a way that can be replayed and rewound.
14:41And so, that leads us to something fairly simple here.
14:44Uh we have a system that knows which data points it's allowed to produce, meaning from structured filings, from numerical data, from any other ecosystem, it knows what it's allowed to produce,
14:57and it knows what it's never allowed to produce.
14:59So, if it's pulling things out of prose, raw tables, or anything else, it doesn't do that extraction.
15:05What this allows us to do is this allows us to do things like consolidate financial statements in seconds with every number tied back to its individual source.
15:15Which means not picking any companies, but the CapIQ's, the Deloopa's that are all using contractors for this, we don't need to do that anymore.
15:24We can actually create a financial model in a numerically accurate way that allows you to build work product.
15:31We can build a DCF in the format that you actually want.
15:33In knowledge, the model will not hallucinate that a row exists that shouldn't exist.
15:38Now, the really crazy thing and why I say this is inevitable is everything that I'm telling you generally generalizes past finance.
15:46We're picking numbers here because finance cares about numbers.
15:50But you can imagine a world where every court case, if you're a Harvey or a Legora, runs through the same process.
15:56A pre-processing step that understands that we can extract entities like case A versus case B and store that deterministically so we don't hallucinate citations.
16:07Or every drug discovery formulation that exists in NIH white papers such that we never miss a compound or anything else.
16:15And so, the kind of uh interesting dimension that we're entering is we're an ecosystem right now where we're almost like pre-SSL in the e-commerce ecosystem.
16:27Where like what's the TAM of e-commerce?
16:30Trillions, but how many people were comfortable putting their credit card number on the internet before there was security?
16:35Zero.
16:37So, we're in the last step.
16:39AI can now produce verifiable work product across a number of industries.
16:44Meaning the rag platforms of the past are really, really helpful and really cool in codifying workflows, but there's a reason that the foundational labs are going after every one of these.
16:53There's a reason that Claude for science is not a deterministic system, but still a rag-based system.
17:00And so, the piece that should be really exciting about this whole thing is there is a piece of this that no one has built yet in a variety of disciplines and a variety of industries.
17:12We're really good at consuming tokens.
17:14In fact, there's a club here for people that consumed a billion plus tokens.
17:18They're walking around with gold cards.
17:19It's kind of funny, actually.
17:21But, like, that's kind of hilarious, right?
17:24Like, in what time in history has an employee been rewarded for your company to pay another vendor for how much money you're spending?
17:31And so, token maxing, I think, is now being thought of as not the right approach here to actually solve your problems.
17:39So, the natural thing will become a rush to the bottom, which is an optimization problem, which we've seen over and over again.
17:46If you rewind time, I actually sat on this stage 4 years ago, maybe not this particular room, talking about like Snowflake Summit and Databricks Summit, where the idea was all of a sudden people wanted ROI
17:59on top of like to understand the ROI of their Snowflake investment or their Databricks investment.
18:04And we started doing cost optimization.
18:06We started figuring out how to make sure every dollar of capital we put into Snowflake and into Databricks went to actually producing a pipeline that people were using.
18:15That same trend is about to start.
18:17And we're figuring out right now, how do we use the right tool for the right job?
18:21And sometimes you don't need a multi-billion parameter model when one CPU cycle will just work.
18:27So, kind of wrapping this up, AI has made producing work completely, I say nearly free, but like, thank you VCs, heavily subsidized.
18:35The reading problem is still very open.
18:39And you verifying a data point is not enough anymore.
18:44The system has to be able to track or track its own provenance and actually ensure that the numbers that you're pulling represent your own unique company philosophies.
18:55Citations got us like 50% of the way there, but the next half of the verifiability and provability is going to be how we start using this in real like valuable,
19:05verifiable work.
19:07And the interesting thing here is the work product itself is the proof.
19:12In code, we have I mean unit tests, we have every single pull request and every commit and every code review on that pull request stored in perpetuity.
19:21There are companies here trying to mine that information to create a representation of your on your ontology right now on a company specific basis.
19:29We need that same ecosystem in finance.
19:32So, let's just say this, all these problems are solved at this point.
19:35The last remaining mile is going to be that personalization.
19:39So, the second version of this in 2027 hopefully one of us will be on stage talking about how we're now able to build verifiable ontologies that actually proxy our investment processes
19:49instead of saying how do I not spend a trillion tokens to solve this individual problem.
19:56Um so, we're also I mean Susanna, we're both from Kepler.
20:00We're growing pretty quickly.
20:02Obligatory come join us if these problems are interesting to you.
20:05And yeah, I think we're starting in finance now.
20:07We'll be in a lot of different dimensions pretty quickly.
20:10Um so, thank you so much and happy to answer any questions.
20:12[applause]
20:17[applause]
20:19Yeah, go ahead.
20:27Yes, your question is about provenance and from the provenance ledger, where does the number actually come from?
20:31At its core, the number comes from three different things.
20:34It comes from extracted information from the filings.
20:37It comes from either a mathematical calculation that we do to derivative production, so like a ratio or something else.
20:43Or it comes from your internal documents or other internal pieces of information that operate in conjunction with that external data to produce that individual data point.
20:52So, anytime the model effectively is responsible for telling some entity to do an IO operation, that's part of the provenance chain.
21:02Yeah.
21:08What are your customers most excited about in your product?
21:10Yeah, so the question is um what are our customers most excited about?
21:13It's funny.
21:13Uh everyone wants AI like the dream is the AI portfolio manager.
21:19The portfolio managers don't want the AI portfolio manager.
21:22Like they want the AI analyst.
21:24And so the thing they're most excited about is a way of rapidly producing, rapidly doing kind of the repeatable painful tasks that their analysts are doing.
21:33Things like uh analyst actually sits there and listens to an earnings call transcript.
21:38If we didn't have to have an analyst do that, it would be amazing.
21:41Or an analyst sits there with like 15 tabs and they're opening every 8-K and 10-K over the last, you know, whatever years to create the V0 of a financial model.
21:51So they're most excited about getting their analyst time back.
21:53That's the honest answer.
21:56And these are expensive analysts.
21:57Like these Some of these folks make 6-700k a year to do this.
22:05Cool.
22:05I'm happy to answer any more questions outside as actually Yeah, outside as well if you want to defer it time.
22:09Yeah, the gentleman in the back.
22:26[music]