How to avoid disaster when vibe-coding a billing engine — Andrew Garvin, Stripe

AI Engineer · 17 min · 179 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Hi everyone.
00:14My name is Andrew Garvin.
00:15I'm one of the co-founders of Metronome.
00:17Metronome, if you're not familiar, is the top platform for usage billing, which as you might imagine is taking off right now.
00:23Um, so much so that earlier this year we were acquired by Stripe in the largest deal that Stripe has ever done.
00:28Um, and what I'm going to show you today is a fun project that we've been cooking on with the Stripe team.
00:33Um, that is a demonstration of some of the things that we could work on together.
00:38Um, this is going to combine two very different uh products within Stripe.
00:43Um, if you haven't seen it already, I recommend looking at Stripe Projects.
00:46Just open up open that up on your phone while I'm going through this.
00:50Um, this is something that actually was launched literally the week that Metronome was acquired.
00:54And so, as you might imagine, the velocity of development at Stripe and at Metronome is pretty high.
00:59Basically, what Stripe Projects is is it's an orchestrator to allow you to operate and build your business as fast as possible.
01:06Um, and so in essence, what it does is it provisions a Stripe account for you as well as backend services that you may need, think like Versel, Postgress, and in this case, a metronome billing agent in order to launch a product quickly or an application quickly all through the CLI.
01:22Um, so the demonstration that we're going to do today is a very simple demonstration of how to get set up through Stripe projects, but you can imagine all the sorts of things that people are building on Stripe today.
01:31Um, one of the really cool aspects of being inside Stripe is their scale and data.
01:36Um, and one of the things that we've observed inside of Stripe is that the use of Stripe's CLI has exponentially increased over the course of the past five six months.
01:45We're thinking about all of the implications of the coding agents operating systems which we'll get into uh a greater topic on later in this demonstration.
01:54Um so the beginning part was sort of the thesis for this is how to avoid dis disaster when vibe coding a billing engine.
02:00Uh as you might imagine being in the billing space for multiple years now we've seen all sorts of crazy things happen and it's even getting crazier now that people are expecting to operate metronome
02:10a very complicated and deep product with a coding agent.
02:14Um, and so as a result, we are working from a developer experience standpoint to make this a more seamless experience for people and to have them avoid disaster.
02:23Um, and so we're going to see in a second what that de a demonstration of what that might look like and how we're guiding things.
02:29Um, but just to frame it the what's happening today in with launching agent products can go crazy and it can go sideways in all sorts of ways.
02:38So as just a couple of the types of problems that metronome helps solve for folks.
02:41Uh first we are for example and have for many years now taken in all of the API calls to OpenAI and Enthropic and metered that for those companies.
02:50We've worked with those companies since before they had any revenue.
02:53Uh and so obviously we've operated at global scale operating a metering service with a number of different data impacts.
02:59In addition to that we also operate credit models.
03:02Obviously today with usage based pricing it's not just about having a pay go uh metered business model but also all sorts of different forms of credits commits salesled uh discounts and offers.
03:13And then finally now in in the last 6 months especially the impact of failures here is growing in importance in particular because agents can run away with spend and so we're thinking about how to give more
03:26controls to our customers that they can offer to theirs.
03:29So think like for example having agents have a wallet um that they only they can spend from and having controls at that level.
03:37Okay.
03:37So that's what we're going to go into today.
03:39Let's actually get into doing this demonstration.
03:42You're going to see how simple it is.
03:43Um so I'm going to initialize Stripe projects right now.
03:51[sighs]
03:52Oops.
03:54And this is in fact a live demo.
03:56So you should expect all sorts of different things to happen here.
03:59So what this is doing is again initializing stripe projects.
04:02Um we are going to I think this should work.
04:07Um so we've selected that we're going to use claude here and we have a very simple thing.
04:12What I I go around the world basically talking with companies about their business model.
04:15It's not initialized.
04:17Aha.
04:18I see.
04:21All right.
04:22Come on up.
04:25Cool.
04:35[snorts]
04:37I got it.
04:38All right.
04:38Thanks.
04:46All right.
04:47Sweet.
04:48Um, yes.
04:49Let's proceed.
04:52Cool.
04:56All right.
04:56So, our product, our project is in fact ready now.
05:01And so, I go around the world talking with companies about how to set up their businesses.
05:05One of the things that's really top of mind right now is um replicating certain business models.
05:10People want to get off the ground without having to think about it too deeply.
05:13One of the key topics right now is replicating Lovable's pricing model.
05:17And so I'm going to enter a prompt in natural language that allows us to guide the metronome billing agent and Stripe to set up a demo account that has that
05:28um that has that element to it and create a demo billing engine uh in Metronome mimicking the lovable pricing model.
05:41Oops.
05:42Just make sure I don't misspell that.
05:45Um, and so if you want to on the side you can see you can open up Levable's pricing and you can see all the different elements to it.
05:51But in particular they have a prepaid credit auto recharge model which is very common in um in sort of a self-s served motion today.
05:58Um so let's get this going.
06:01Um so while this is going I want you to pay attention to a couple of things that will happen.
06:06Um and in particular if you look at the obviously the slide on the left there's a couple different points that I want to hit on.
06:11So first um again metronome is a very complicated and deep product and uh there's a lot of different ways to hit foot guns etc uh if you're not guided and so what we've invested in is building an extensible set of skills files that can provide context to the agent that's implementing metronome and working with our API.
06:29Um these skills files are also portable and easy to install so you can use them on your own side.
06:34Um and what it does is it allows us to essentially remove the friction associated with getting started.
06:40Um and so here uh that and that's very important because you want to be able to test and work with the product and evolve over time.
06:47You're seeing here in error code um from a developer experience standpoint.
06:51You know this is nothing new but our our perspective is to have much more verbose and clear errors so that the agent can self-correct.
07:00Um and uh and so again our developer experience teams are working on finding more failure cases like that um and being able to help guide especially in the initialization
07:11and setup.
07:12Um one thing that's that's important here I think keying off of the last talk that was in this room.
07:18Um the goal that we have from a product development standpoint is not to have a customer operate the entire system without a human in the loop.
07:27This is a type of system that is both business critical, has deep business logic behind it.
07:31And so instead, what we are recommending and building toward is to use the use your coding agent as a way to accelerate your work and get into a test mode and test environment.
07:43Um, and so again, what we're doing here, we're not expecting to ship into production.
07:46We're not pushing it into production.
07:48In fact, when we go into the metronome environment, you'll see that basically what we've done is built a sandbox experience.
07:53Um and in the metronome context there's what it means to sort of test your initial setup is not just that you can see uh a contract or something like that or see a customer provision but also you need to see usage and so in in on the back end here our skills files are directing
08:09um the agent to actually flow usage into the metronome platform so that you can see what a live uh in what a live customer would look like.
08:18Um I'm going to do one more beat on stripe projects here.
08:21I think in this case what's happening is that stripe projects so so using the stripe CLI it's it's engaging with metronome which is an external vendor here you could also imagine
08:31also provisioning a bunch of other applications uh and using natural language to call for those as well um and so what again what's nice about this is that it basically removes the friction associated with setting up a test application or a test environment
08:44um and we again have seen a ton of uh of usage of this form um and we expect to see even more.
08:52Um, this is sort of like from our perspective coming into Stripe, this is one of the things that's been really amazing is that Stripe is on the forefront of thinking about agentic commerce and preparing primitives for the moment that we're in right now where in fact this is exactly what's happening.
09:06Companies that are launching new um new applications and new businesses that we've seen an exponential increase in new business formation at Stripe.
09:14Um and then in addition to that an exponential increase in customers that are using Stripe and using Metronome through the coding agents themselves.
09:24Okay.
09:25Um we are almost done here I believe.
09:30Um, while this is going, I I like another sort of like thing to sit back and think about when I go around to product teams today.
09:38They're obviously thinking about building for agents, but I think one of the things that's important to do is to de decode what exactly does that mean?
09:44And so I what I like about this sort of framework for thinking about the coding agents today is thinking about the different roles that they play.
09:53So obviously companies are launching agents as a product and therefore that's one of the reasons why they need to have a usage based pricing model because if the agent can be the product and run up token a token bill it's important for you to be able to meter on that.
10:06Um what we're talking about here with the Stripe project CLI is the agent as a buyer.
10:11So literally procuring their initial Stripe instance as well as additional backend services.
10:17That's important to basically make your services discoverable to agents that may be building an application or working in the open web.
10:25Um, on the Stripe side, that means both in a B2C environment, so they're working on a gentommerce, but then in the metronome environment, we're talking about in a B2B context.
10:32Um, and uh, and so there's sort of like multiple different levels to play out there.
10:36And then finally, one of the reasons why Metronome is really taking off right now is because of the agents as a user.
10:42Um, and so for example, we've been working with HubSpot for the past couple of years.
10:46They are currently on a path to transform their entire business from a seatbased model to a creditspbased model.
10:51Um, if you've seen some things in the news, that's starting in in EMIA where they have dramatically lowered their seats based price and added on a creditspbased model.
10:59The fundamental reason behind that is because what they need to be concerned about is a world in which an agent can operate their entire system.
11:06Um, and in that world essentially paying for a seat level access to the product to perform your work is no longer important in some sense.
11:14Um, we've been talking about this in uh as sort of headlessness.
11:18You know, Salesforce, various others have have talked about this.
11:20This is what Metronome is literally seeing today.
11:23Um, and so as like one example, last week I was at a uh I was at Andre's demo day where all five of the demoing companies were salesled agents
11:33meant to operate platforms like SAP or operate um operate invoicing platforms, etc., etc. Um, and again in that world, it's important for you to have a usagebased pricing model because you have the possibility of essentially all of the value acrewing to essentially one
11:50user of your platform, which in this case would be an agent.
11:54If you guys are thinking about pricing models, so not just developing in the agentic space, I'm a good person to talk to.
12:00I'll be out here in a second.
12:01Some of the things that we're um thinking about here are basically um not only having a creditsbased model which has been uh dominant on the market since openai launched their prepaid credit auto recharge model a couple of years ago through metronome
12:15but in addition to that offering more and like more and extended offers including in a salesled in an enterprise environment um and so for example what's happening with all the coding agency in the enterprise think like cognition or cursor or openAI anthropic themselves is that they are starting to adopt more
12:32um uh commit structures like the CSPs have done for the past 10 years.
12:37Think having prepaid commitments, postpaid commitments, and specific types of offers for specific types of customers.
12:44Okay, I think that we should be good to go now.
12:47And let's see what it looks like when we open up Metronome.
13:01Okay.
13:11What's up?
13:15Uhhuh.
13:17We've done multiple different versions of this, as you might imagine.
13:28Okay.
13:29So, as we open this up, uh, pull this.
13:45It's fun.
13:45I love all these demos where you're just looking at people logging in.
13:48Um, so as we open this up, so the general pain that Metronome has is an onboarding wizard meant for a human that needs to set up their environment.
13:56We've we don't need this now because we had an agent set up this environment.
13:59And as I come in, you're going to see some of the core metronome primitives here.
14:03Let's start by looking at the customer that was set up.
14:05Again, this is for testing purposes.
14:07Um, up at the top, you're seeing the customer with a certain lifetime spend.
14:11This was auto again populated by um by the agent for the demo environment.
14:16Um I'm immediately going to go into their invoice and we'll come back to this in a second.
14:21Um and so what you what you see here is a draft invoice that was created associated with um sort of replicating the lovable pricing model again.
14:30Um so if you have this on the side you can see all the different elements of that.
14:33But the core aspect of lovable's pricing model is a creditonly pricing model where you autorecharge on a monthly basis.
14:39Um and then in add in addition to that they have multiple different types of credits that are scoped to different types of usage beyond the the use of those credits.
14:47Then if you go over and if you overspend then you have an invoice at the end of the period.
14:52Um so a couple of the different like concepts there that are relatively complicated to administer is the credit itself.
14:58And so Metronome has a first class uh credit object here.
15:01What you're seeing is that there was a credit credit created for the initial period that we're testing for.
15:05we had usage that that draw that drew down from that entire credit balance.
15:10And then finally, if we go back to the customer pane, um in addition to that, you can see the usage that we that we plopped in.
15:18All obviously in a production environment, you would be seeing this in against real usage that you have.
15:23The core reason again to show it in this manner is to just see what it would look like if you adopted the pricing model and then had real usage against it.
15:31Again, I'm going to come back to the invoice.
15:33And so here you can click into each of these different components.
15:36Um, build credits, plan mode credits, cloud credits, AI gateway credits.
15:39This is exactly what the lovable pricing model looks like.
15:41And again, the way that we coached the agent to be able to do to to build this was just describing a natural language to replicate lovable pricing model.
15:49It was nothing more difficult than that.
15:51Um, so without going into Metronome's platform to too great an extent, um, the what we just did here was we initialized uh and created a stripe instance.
16:02We then in through Stripe projects uh we also created a metronome instance.
16:07Then we coached the agent to be able to uh build a demo instance of metronome that had a real pricing model live in production.
16:16And so you could imagine basically testing then from there the exact testing and tweaking from there exactly what you wanted before bringing that into production.
16:24Um this sort of framework for thinking about development both applies to Stripe where we are working very very hard to make it easier to uh run a complicated business model and get off the ground but also I think it bears lessons for how we might pursue agentic development more generally outside of Stripe.
16:41So again, think about some of the primitives that we talked about here today.
16:45Agent as a buyer, agent as your product, agent as your user and disambiguating what the different the different um modes and uh and like implications of those are.
16:54And then in addition to that um having having ways in which we coach the agent to operate more effectively in including in a in a in a difficult environment.
17:03You can try this for yourself now.
17:04So uh the easiest way to get started is with the commands that are that are listed here and you can see everything that is available through Stripe Projects through Stripe Projects online.
17:14Um as there are a number of different providers that are onboarding every day.
17:18Um so companies like Versell um like hugging face etc are basically like working in Stripe projects environment to be able to make their own products more discoverable to agents that are operating Stripe system.
17:46[music]