Every AI Company Is Accidentally Building a Bank — Dor Sasson, Stigg

AI Engineer · 20 min · 186 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Yeah, we're good.
00:13Okay.
00:14So, hi everyone.
00:16Given there's I hear there's a match right now as we speak.
00:19So, given that you are here means a lot to me.
00:23So, at least I'm I'm winning the World Cup right now.
00:25So, that's that's a that's a big thing.
00:27Thank you for joining me.
00:28I flew all the way over on co-founder CEO of Stig.
00:31I flew all the way over to make a kind of a funky statement on an AI conference which is literally every AI company right now is accidentally building a bank.
00:41And when I mean bank, I'm going to try to kind of unpack this thing not as like a weird marketing metaphor, but effectively what what is really happening right now with the AI economy
00:53and why a lot of the constructs and the things that we are seeing using consuming and paying for in AI are actually really behaving like constructs that we know from the banking financial systems.
01:06So, cool.
01:07Thank you for for for having me.
01:09Um and let's get it going.
01:13By the way, I'm the first talk today.
01:14So, the clicker is not working.
01:15It's going to be interesting here and out as they move for this.
01:18So, let me take you through a little bit of background.
01:23As I was working through this talk back in April really every single code generating product platform out there just broke under this new type of economy and under this new type of
01:39consumer behavior.
01:41And in just a you know, matters like 5 weeks everything felt like pricing emergency one after the other.
01:48And the case I want to make here is that these pricing emergencies go way beyond just emergencies from a financial commercial set of the house.
01:56These really are infrastructure emergencies that are emerging because of how those systems were built to begin with.
02:04So, just to give you some perspective, we had in in April we had Anthropic basically eliminating access for Open Clo and other third-party agents to basically use the subscriptions plans.
02:15Then immediately we had OpenAI basically changing how the product here is being priced, bumping the price overnight 5 5x, which is quite intense.
02:26And then we had GitHub freezing and then completely eliminating access trial access free access to their We have some echo here.
02:35To their free plans for for for Copilot.
02:39And effectively like all these companies all at once in a single month experienced what it means to actually scale on scale AI and what actually happens when you don't have the fundamentals in place.
02:53So, let me take one one step farther.
02:56So, let's zoom into even just one of the use cases, right?
02:59So, if you think about Anthropic and you think about the Open Clo use case, right?
03:06What happened was effectively Anthropic was subsidizing every consumption of every user using Open Clo within the Clo Clo Max subscription.
03:15So, effectively what the customers were paying daily was, you know, dollars and cents.
03:20But on the on the cost side for Anthropic, it was like 150s to even 750 subsidized, which is effectively not an economics that you can scale even for the company at at the size at the scale of Anthropic,
03:35right?
03:35So, what it seemed to be a business problem really was an infrastructure problem because Anthropic didn't have a an effective way to delineate between users who are using their APIs
03:47and users who are using their subscription um, uh, programs.
03:51And so, they had to immediately cease and stop access, right?
03:54They had to to to react.
03:56Um, cool.
03:58So, and this is not unique to Anthropic, guys.
04:04Like, uh, we're seeing this like, I don't know if you all heard read the news most recently with a company, uh, undisclosed company burning through like half a billion
04:13of their AI, uh, cloud credits, um, just merely having like a few employees burning through the entire organization contract.
04:21Uh, we had, uh, Uber saying they're basically torched down their entire, uh, AI usage consumption budget for the year in just a few weeks into the year.
04:32And then we had Rep- Replit giving an example of what could happen if, uh, a single org has three users that basically burning down the entire credit pool for Replit for the entire org.
04:44So, really what happening here is, uh, this is not just an an an Anthropic problem.
04:50And I think what's what's mutual and kind of like switching gears from the examples to the actual what's really happening here, um, the problem basically with with all these three examples
05:01is that the checks for what you're entitled, what you're allowed to do, whether you're an agent or a user, we are AI product happened, uh, after, uh, the invoice and not before.
05:14So, basically with each and every one of those examples, you were allowed to consume, you were allowed to use the product, you were allowed to burn tokens, and checks happen only after the invoice.
05:23So, basically the spend and the aftereffect gets reconciled only after it's too late, only after the invoice is is sent, and this is effectively bad for business, it's sticker shock, it's bad for
05:36practice of how we use these systems.
05:38These systems are now mission critical, and we're basically none of these companies had in place the financial infrastructure that allows, exactly like banks, guys, to check before we draw down, before we consume, right?
05:52So, I think what what we're seeing is effectively uh really is a change in how we architect, and how we ship, and how we build software.
06:02AI is not just changing what is value, is not just changing how we interact, who is the users.
06:08It also effectively changing what exactly is getting paid, and how we actually architect for systems that can uh handle runtime, handle the complexity of usage happening before the the the invoice, before the checks, and how we actually can architect for that.
06:27So, I think what's interesting, and what you see in front of you um is basically on the left side, what we're seeing is how most teams ship this architecture today,
06:36and you can see that basically the check and the settlements are happening after after the inference, after the usage is being logged.
06:44And that's typically too late.
06:46We see situations like spikes, overspends, overages, like experiences that are bad for business, bad for your users, and quite frankly, in many cases, also bad for the vendor.
06:56Because if you didn't allocate enough resources and compute to actually allow for this to happen, you you also have a margin problem potentially, right?
07:05On the right side, we're basically saying, or I'm trying to make the case for you all that with AI, with the current infrastructure, and how AI is being built and sold,
07:16we actually need to enforce and and do this synchronously before we actually update the balances, and reconcile and settle in in async way after after effects.
07:28So, hot path actually get checked synchronously, and everything that comes after uh is basically reconciled after.
07:34I think the best example, guys, is if imagine you go to the ATM, right?
07:39And you draw cash.
07:40When you draw the cash, it that the check if you are allowed to draw it, doesn't happen after you get the cash.
07:46It happens before you draw the cash.
07:47And so, that's exactly the same thing.
07:50Our industry effectively misses this layer, this financial infrastructure that allows for this type of behavior to happen naturally, right?
08:01And so, obviously this sounds like a great idea, though, right?
08:04Like you're you're pitching this infrastructure, sounds like you have something to do with this.
08:09Um But guys, this is not just me.
08:11Uh in February, OpenAI, they have a quite a great team.
08:15They're team called financial engineering.
08:17And these guys basically published how they think about the architecture and what they had to build for this to work at the scale of OpenAI.
08:27And what they had to say reads very similarly to how transactional systems of banks work.
08:35Basically, you have a decision waterfall that needs to consider in runtime what is the client, the user, the agent is allowed to do.
08:43And it needs to take into account many different conditions and policies of what is really allowed, right?
08:49So, are you allowed or you have access to a certain feature or product?
08:55What are the certain rate limitations under your plan?
08:58Whether you are in a trial or in a certain promotional program.
09:02Like you need all these decisions to be calculated immediately at front time.
09:07And then it ultimately decide, can I give this user access?
09:10Can they draw down?
09:11Can they consume or not?
09:12And all those decisions really can happen after the invoice.
09:16They can't because you need to actually be able to calculate and prioritize this.
09:21So, I think we have three things to notice from here, right?
09:24The first thing is this is a single synchronous evaluation.
09:28It happens at the request.
09:31It doesn't happen after effect.
09:33I think the second thing is the priority is deterministic.
09:36So, we know in advance what are all the different rules that apply for certain things to be consumed, right?
09:43And I think lastly, and here you guys, this is my opinionated view, you can disagree.
09:47I don't think this is billing.
09:49I don't think this is necessarily this entire architecture is just related to the idea of invoicing and billing.
09:56This is a financial system that actually make decisions ahead of any invoice, ahead of any billing element.
10:02And I think we're going to see more and more of this becoming a standard for other products and other AI systems as well.
10:10Really just nerding just a little bit on I'm sorry, I just skipped one slide.
10:16So, yeah, just just nerding just a little bit about what other trades or or elements we see and are used to in banks and we take them for granted,
10:27but in AI they're not that granted, in software they're not that granted.
10:31So, just one example, right?
10:32Like if two separate parties have access to the same banking account, and that banking account has like $10, and they are entitled to draw down those $10.
10:42What happens and when they both of them try to draw down at same time, right?
10:46So, banking systems has the idea of holding, of settling.
10:51You can really the concurrency is effectively solved with banks, but with AI and agents, it's not that case.
10:58You could have whatever, like 10,000 agents trying to reach out to the same dollar pool, and they're going to try to do that all at once, and superficially it might look like they should be able to draw down because there is balance in the pool.
11:12But, what happens when they all draw down from the same pool simultaneously, concurrency become an issue.
11:17So, how do you solve for that?
11:20This entire idea in financial systems actually called like it's a classic double double spend problem.
11:26It comes from the accounting world.
11:28It's a double entry bookkeeping, and as I'm sure you all are building shipping today with AI as the revenue scales, as your company scale, the idea of double entry accounting, double entry billing, the idea that you can solve concurrency,
11:43these things would matter as you scale and they're not so easily solved down the line.
11:49So, think about things like you know, can you basically introduce rates like hold and settle, think about idempotency and auditability of those uh requests and etc. Another thing that I think is very very
12:09known in in you know, how banking systems works but not not necessarily enough established in AI is the idea that all the different pools have different sources, right?
12:19So, in bank you have your debit account, you have your cash account, you have your savings, right?
12:25And if you draw down, the bank actually knows how to calculate where you drawing down from and what does it mean?
12:32Here, you know, say I have a pool of you know, one you know, 1,700 credits but they're not the same, they're not alike.
12:40Each one of them is sourced from a different source.
12:43So, how do you decide which which one of them do you draw down against?
12:47Do you start with the ones that were you know, given granted earlier?
12:51Do you start with the ones that were promotional?
12:53So, how do you really calculate what's getting drawn down to the users?
12:57I think that's that's a real problem.
12:59Most companies today try to solve it like a single integer databases and that's not really going to work as you think forward.
13:08Cool.
13:10Um so really tying the knot the knot here, I think if you think about how we interact with software, it used to be flat.
13:19So, it used to be just users and organizations and there wasn't a lot of meat in between them, right?
13:25The the the hierarchy of you consume software was quite flat.
13:29It was quite straightforward in terms of who see the value.
13:32But today, I think effectively credit pools are no longer flat.
13:36Uh you have a lot of different hierarchies in between who consumes different credits, and also they demand different governance, different logics, allocations, budgets, spend caps.
13:47And effectively, what happens is that budgets are set at the top, where for instance, if you're selling to the enterprise, you're maybe selling like pre-commit annual contract.
13:56Uh C-suite or somebody who approved that seven-digits contract, they expect to see how that credit are getting drawdown along the contract life.
14:05But they don't just want to see that they were used.
14:08They want to know who actually used them, which team, which users, uh which agents, uh to what extent, and they want to have fine-grain control over that.
14:17That's literally the bare bones of how more and more companies thinking about selling AI workloads and how they they effectively work in production.
14:27I'm going to run through some of this.
14:29Um I think this idea of like what does it mean like high cardinality graphs and dimensions, being able to slice and dice against model types, users, like all this type of uh dimensionality
14:41of uh of uh consumption is is probably interesting to to talk about.
14:46But what I actually want to expand on is ultimately those four patterns.
14:50So as you build AI and start selling AI at scale, you're effectively like four patterns that keep repeating themself, and they are very very much alike and and very similar to the idea of, you know, banking systems.
15:01I think the first one we talked about earlier is basically being able to reserve before inference and settle actuals after.
15:09Um this is like going to to be like a pattern that again and again we're seeing more and more often, and it has ramifications on how you do accounting, how you do
15:19uh basically handle all those uh different concurrencies workloads and what have you.
15:23I think the second thing is we're seeing more and more AI companies unexcited about the idea of sending all this usage data to to somewhere else over the cloud.
15:32For latency reasons, for cost reasons, for data sovereignty reasons, there's a lot of good reasons for why these companies actually rather to keep the events data inside their VPC,
15:43and I think we're going to see more and more AI companies seriously considering deploying this this architecture, this ledger, this metering within their VPC rather the the contrary of sending it over the internet.
15:54I think for agentic checks, so think about how agents spawn, the idea of checking balances when agent does something you don't know in advance what they're ultimately going to do, how that how that task is going to basically cost.
16:09So, if you don't know the cost, you need to be able to check those balances as the agent continues to get work.
16:16And you need to be able to reserve an account for them in advance and then asynchronously after effect reconcile.
16:25Quickly moving forward, so ultimately if you have those systems in place, what you really gain out of it is visibility.
16:33So, it used to be given that for usage-based products you need visibility, but I think what happens today, you know, it goes beyond just plain visibility.
16:42It it means that effectively like the ability to see for CFO, for CIO consumption of AI workloads across different models, across different features, across different products is not just, you know, a nice to have.
16:56It's becoming a table stakes.
16:57It becomes something that you can't do business without, and we're going to see more and more companies expect to have this off the bat before they actually do business with you.
17:07And so really just giving some examples to how some of the best frontier AI companies are approaching some of this, I think you're you're seeing the patterns already there.
17:18So, I think a lot of these companies already appreciate that you're no longer just selling software in the you know, commodity way that we used to think about subscriptions
17:28and usage.
17:29It's becoming more and more of a financial systems and financial systems require some of these complex ideas that weren't we're not there before AI.
17:38Um and I think just to give like one small example that I talked about earlier, but I think might have gone a little bit uh hidden.
17:47Um One of the changes in April that OpenAI did is really they didn't change the pricing.
17:53All they did was change the pace in which tokens get burned.
17:57So, effectively you use the same model, we pay the same cost uh per unit, but we burn more tokens.
18:02So, that effectively was a pricing change.
18:05How do you execute such a change?
18:07Um because it's not just financial change, it's an infrastructure change.
18:09You're basically change the rate of how the software gets consumed.
18:13So, there's a lot of complexity not in terms of just how we change pricing, but also how the infrastructure underneath allows for more complex ideas into how value is getting
18:22uh uh introduced to the users.
18:24And really just, you know, before wrapping up, you know, there our team likes to say, and this is something we started to say more and more often than before,
18:33it really feels like you know, there was in the in the early 2010s there was this Stripe moment where every company wanted to ship product into the internet and you need a checkout
18:43and you need it to be secure and it and the developers need to to to to to make it easy for them to quickly get them up and running.
18:49I feel like where we are today is those ideas of like building a banks, it's okay if you feel like, I don't know, you like wearing suit and tie, maybe you want to build a bank,
18:59but if you are a building a bank, you probably want to be aware to all of these traits and elements that has to be there so you don't meet them down the line where where it's get really, really more complex to fix in a hindsight.
19:11And so, feels like the industry really misses its Stripe moment where all these constructs just exist and it's they're so easy to use.
19:19And you know, there's a lot of excitement from our team when it comes to the potential of having such systems ready and help AI companies keep shipping.
19:28Um if any of that was interesting, really come by our booth.
19:32We are in a quite of a weird location right at the start, but still there.
19:37I know there's a game right now.
19:38Come visit.
19:39We're demoing the product.
19:40Feel free to ask any questions and thank you for coming today.
19:43It was awesome to to have you have you with me.
19:45And yeah, love to nerd some more by our booth.