Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio

AI Engineer · 16 min · 161 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13I'm Kanesh Manuja.
00:14I'm a principal engineer at Twilio.
00:18Let's start with a quick show of hands.
00:20Who here has seen the message, something went wrong.
00:24Please try again.
00:27Well, we have a few lucky ones and a few that have had a good lunch.
00:32Um, so behind that simple message is actually a system that is very complex that serves you that message despite the model providers being down.
00:45And that's what we're going to productionize today or discuss productionizing today.
00:50So what is an LM gateway?
00:52An LLM gateway is an entry point or a middleware between your apps and the model providers behind them.
00:59It does a bunch of things.
01:01Routing, authentication, fallback, rate limits, all kinds of governance that you can think of.
01:08And right at the heart of the gateway is a fight between four things.
01:13It's availability, latency, your guardrails and costs.
01:18In case of a degradation, you cannot maximize all four.
01:23You need to pick what you want.
01:26So with this talk, if you use an LLM gateway, I want you I want to help you to make that trade-off for your use case.
01:35And if you design a gateway, I want you to design or provide those levers to your callers and customers u so that your customers are happy.
01:46Let's start with availability.
01:50If you have a single model provider, their ceiling is your ceiling.
01:57Their outage is your outage.
02:03So in typical software engineering, the way you tackle unreliable dependency is by retrying.
02:11Retrying with exponential backoffs, with jitters.
02:16And when all of that fails, you have a circuit breaker that trips after you've seen sufficient failures and you stop calling the damn thing.
02:24This is not enough for LLMs.
02:26LLMs are very different compared to your fast cheap APIs that you retry on.
02:32Retrying an LLM API eats into your latency budget really fast.
02:38And also tripping over a circuit breaker when you have another perfectly fine model provider to route to doesn't make sense.
02:47You should use the second model provider.
02:50And third, as I said, the calls are slow and expensive.
02:54So blind retries just multiply your cost and your tail latencies.
03:00So what is a better idea here?
03:02It is actually a per request fallback.
03:05What that means is you can actually try model provider A and then in sequence try model provider B if your request to model provider A fails.
03:15Another option to consider here is you can fire requests to both the providers in parallel.
03:20But that's only if you're highly highly obsessed with latencies because that's just going to double your cost.
03:28Some of the similar circuit breaking patterns apply here to LMS as well.
03:33If you know that your primary has been failing for some time, it doesn't make sense to try it again.
03:42You put it, you take it out of the load balancer or your request path and put it in a cool down and then after a few minutes have passed, try putting that back again.
03:53One interesting choice that you have to make here is where your failure counts live.
04:00You can decide to have the failure counts live in memory on the instances that are serving your traffic or you can have shared infra where your failure counts are shared across the fleet.
04:13There are trade-offs.
04:15If you want quick failovers, then fleetwide helps.
04:20And with instance uh with local state counters the issue that you run into is whenever you change your deployment size your configuration and your expectations change.
04:31So something to consider.
04:35What that clean diagram did not really show you are some of the other gotchas that I'm going to discuss.
04:40So fallbacks are not transparent.
04:43While the industry is converging on an OpenAI API compatible format, I would say there are still nuances.
04:50So you need to really test your fallbacks well.
04:53They can have differences in your tool calling schemas, token limits, stop reasons and what have you.
04:58So with LM gateways, you can have a normalization layer that can ensure that you can do cross provider fallbacks as well.
05:08Another thing is streaming it.
05:16So essentially nobody wants to wait for 30 seconds to have a wall of text appear in front of them.
05:22So there are use cases where streaming is absolutely required.
05:27But it comes as at a cost.
05:28You trade away your levers.
05:30You cannot once you have decided to go with provider A, you have to continue going with provider A. You cannot mid-stream change the providers.
05:40Whatever has been sent to the client, it's done.
05:43And that's where the something uh went wrong message, that's the one that you see.
05:49It's not because of laziness.
05:50It's by design uh that you see that and it's one of the trade-offs.
05:55I would like to call out one other thing where I've seen teams trip over and over again.
06:01They really provision and test their primary providers really well, but they the second provider, the fallback provider doesn't necessarily get the same level of love.
06:12And I would argue that your throughputs or your capacity or your headroom should be even higher for the second provider or the fallback provider because that's your last line of defense.
06:24If that goes down, your application goes down.
06:30Let's discuss latencies.
06:32Availability failures are right in your face.
06:35They fail.
06:37You get alarmed.
06:38You get paged.
06:39But high latencies can be the quiet ones.
06:43And they need to receive more love um than I would say tuning your services for just availability.
06:54One thing to call out, a gateway may run mixed workloads and you can have embedding embedding requests that takes just less than a second.
07:04You can have classification requests that take less than a second.
07:08Uh you have chat requests taking 3 seconds and reasoning requests taking a long time.
07:15Quick show of hands.
07:16If you measure your aggregate latency for your entire service.
07:22Well, that was a trick question.
07:24Sorry.
07:24You shouldn't.
07:25It doesn't make sense.
07:25It's a lie.
07:27You should be tracking your P99 per model per route, not a gateway wide number.
07:32Gateway wide number doesn't make sense, especially if you're running mixed workloads.
07:37And I hope you're not u for those who raise your hand.
07:41Another thing that can really I cannot emphasize this enough is for you to set timeouts on per model class per route.
07:50That's where that's the number one root cause of your silent outage.
07:54If you don't have a timeout, your gateway thinks you're hap your request is being happily served while it is not.
08:01And I'll leave you with this message for for latencies.
08:04Um, specifically a reasoning models normal is actually a chat models outage.
08:10So you definitely need to track latency per route.
08:17Okay, this is the most painful or this the slide that has given me the most scarse which is reasoning and router models.
08:25So this is where truly the latency is unpredictable and reasoning models they do not give you they they're highly undeterministic more deterministic undeterministic than your normal models.
08:41You cannot set the temperature to zero in many cases and the same prompt can take somewhere from 2 seconds to 60 seconds and we've seen that in production where P99
08:51suddenly popped to 60 seconds for no good reason.
08:54So that's while there's no magical solution to it.
08:59I would recommend that you at least start with fixing the reasoning level per route.
09:04So with router models, they hide that abstraction behind you.
09:09Like they pick which models to run and I would highly recommend that you at least make as much uh you make requests as determinist deterministic as possible with an undeterministic
09:22system.
09:24Another idea is hedging the tail.
09:27You can have a you can fire another request if your primary request actually consumed let's say P90 of your latency budget.
09:37This can hedge the t this can really hedge the P99 tail u for for your services.
09:45All right.
09:46This is one of my favorite ones.
09:48Um to keep your model secure you need to have guardrails.
09:54And with that, guardrails are necessary for preventing your services from prompt injection attacks, keeping PII filters in place, having toxicity filters, keeping the LMS to stop swearing at your customers,
10:09all those good things.
10:11But just like a model provider, there are trade-offs, too.
10:15Guardrails are just like another service that can go down that can be unreliable and that's where you need to choose do you fail open or do you fail close when I say fail open
10:29you can still serve the request even if your guardrails are down fail close you block the request and say hey I'm not available that's the trade-off between availability and security to certain extent
10:41while there's a no universal answer it really depends on your use case you can decide like for example a toxicity filter if it's not up and running you can still serve that request.
10:54So the default choice should be the worst case that you can live with.
11:03There are a few things that you can actually do to improve the behavior of your systems in face of uh you know guardrails being down and and managing just unreliability of the guardrails themselves.
11:18So the first is time budget.
11:21Your request should never be bound by your guardrail timing.
11:26It should always be the LM that is the rate determining step.
11:30So make sure that you have timeouts in place and those guardrails run with a specific time budget.
11:38Another important thing is fallback.
11:41You've heard, you probably know and I've talked about it.
11:44We always discuss fallbacks with regards to model providers, but guardrails are critical services too where you can consider fallbacks, have secondary provider, secondary checks, cache decisions uh to keep your service available
11:59when a guardrail provider is down.
12:04Another interesting choice that pops up with regards to guardrails is the placement of the guardrails.
12:11Typically, you can place the guardrail in three ways.
12:16You can have a pre- hook that runs where the guardrail actually runs on the input.
12:20You can and that's probably the safest uh but it does add serial latency uh to your requests.
12:28Another one is in parallel.
12:29This is one of my favorites, but just to call out, streaming wouldn't work well here with with parallel.
12:36So if you're specially producing structured output, please don't stream them.
12:40Uh try to save your latencies and run run these guardrails concurrently for your structured outputs.
12:46Another one is post hooks.
12:48The these are best for um output monitoring, auditing your outputs and and so forth.
12:58So, so far we've all I've discussed all the things that can go wrong with regards to our dependencies.
13:07We haven't discussed that we are actually adding another dependency in the request path itself which is the central or which is the LM gateway itself.
13:16There are a few things where we have been bitten by u and we've learned some lessons that I want to share with you.
13:22If you're working on an LLM gateway or using one, one is shared limits.
13:29Make sure that your API keys are segregated per route, per use case to the most granular possible uh to the most granular thing that you can imagine.
13:41U having a noisy tenant can be one of the biggest problems here.
13:49Another thing is load shedding.
13:52This is a feature that you should uh as part of your runbooks, game days, uh make sure that the gateway that you're using supports load shedding because when you have a retry storm, it becomes really hard to just scale out.
14:04You cannot simply scale out services that is under a retry storm and all these web servers they have an internal queue and they're configurable.
14:14Make sure that they're bounded and they cannot request they cannot accept requests that are unbounded.
14:20And if you want to have some custom logic, you can even have traffic prioritization here as well to make sure under load your most important use cases get served.
14:29Well, last thing that I wanted to discuss is the whole idea of a central gateway itself.
14:39It is a single point of failure.
14:41So if you're thinking of having a central gateway for your entire company for to LLMs, I would recommend rethink that and see what are the reasons that you want it.
14:52What I've noticed is that in most scenarios, it's not the central gateway that they want.
14:58They want centralized governance.
15:00And there is a path forward where you can actually decentralize the gateway and still centralize government governance.
15:08So do not try to centralize your traffic but you can have plugins, you can have custom code that can centralize your governance.
15:18Uh governance can be in the form of cost tracking, rate limit managing management and there are other solutions possible.
15:25So explore those before you chart on having one central gateway for your entire company.
15:32It can be managed by a single team, but I wouldn't recommend deploying it as a single deployment for the entire company even though it's distributed.
15:43With that said, I want to end this talk on a personal note.
15:47So, it is my son's birthday today and I'm here talking to strangers about circuit breaking.
15:55So the least you can do for me is please go and prevent one incident for me and for your customers.
16:02Thank you.
16:03If you have any questions.
16:05Yeah.
16:21[music]