WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=zrZ1amZBSPw

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/zrZ1amZBSPw.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.200 --> 00:00:14.880
I'm Kanesh Manuja.

s3
00:00:14.880 --> 00:00:18.240
I'm a principal engineer at Twilio.

s4
00:00:18.240 --> 00:00:20.880
Let's start with a quick show of hands.

s5
00:00:20.880 --> 00:00:24.560
Who here has seen the message, something went wrong.

s6
00:00:24.560 --> 00:00:27.119
Please try again.

s7
00:00:27.439 --> 00:00:32.719
Well, we have a few lucky ones and a few that have had a good lunch.

s8
00:00:32.719 --> 00:00:45.200
Um, so behind that simple message is actually a system that is very complex that serves you that message despite the model providers being down.

s9
00:00:45.200 --> 00:00:50.399
And that's what we're going to productionize today or discuss productionizing today.

s10
00:00:50.399 --> 00:00:52.640
So what is an LM gateway?

s11
00:00:52.640 --> 00:00:59.600
An LLM gateway is an entry point or a middleware between your apps and the model providers behind them.

s12
00:00:59.600 --> 00:01:01.039
It does a bunch of things.

s13
00:01:01.039 --> 00:01:08.000
Routing, authentication, fallback, rate limits, all kinds of governance that you can think of.

s14
00:01:08.560 --> 00:01:13.680
And right at the heart of the gateway is a fight between four things.

s15
00:01:13.680 --> 00:01:18.960
It's availability, latency, your guardrails and costs.

s16
00:01:18.960 --> 00:01:23.040
In case of a degradation, you cannot maximize all four.

s17
00:01:23.040 --> 00:01:26.159
You need to pick what you want.

s18
00:01:26.159 --> 00:01:35.439
So with this talk, if you use an LLM gateway, I want you I want to help you to make that trade-off for your use case.

s19
00:01:35.439 --> 00:01:46.720
And if you design a gateway, I want you to design or provide those levers to your callers and customers u so that your customers are happy.

s20
00:01:46.720 --> 00:01:49.360
Let's start with availability.

s21
00:01:50.560 --> 00:01:57.280
If you have a single model provider, their ceiling is your ceiling.

s22
00:01:57.280 --> 00:02:01.200
Their outage is your outage.

s23
00:02:03.439 --> 00:02:11.360
So in typical software engineering, the way you tackle unreliable dependency is by retrying.

s24
00:02:11.360 --> 00:02:16.000
Retrying with exponential backoffs, with jitters.

s25
00:02:16.000 --> 00:02:24.080
And when all of that fails, you have a circuit breaker that trips after you've seen sufficient failures and you stop calling the damn thing.

s26
00:02:24.400 --> 00:02:26.560
This is not enough for LLMs.

s27
00:02:26.560 --> 00:02:32.879
LLMs are very different compared to your fast cheap APIs that you retry on.

s28
00:02:32.879 --> 00:02:38.800
Retrying an LLM API eats into your latency budget really fast.

s29
00:02:38.800 --> 00:02:47.280
And also tripping over a circuit breaker when you have another perfectly fine model provider to route to doesn't make sense.

s30
00:02:47.280 --> 00:02:50.480
You should use the second model provider.

s31
00:02:50.480 --> 00:02:54.400
And third, as I said, the calls are slow and expensive.

s32
00:02:54.400 --> 00:02:59.760
So blind retries just multiply your cost and your tail latencies.

s33
00:03:00.800 --> 00:03:02.560
So what is a better idea here?

s34
00:03:02.560 --> 00:03:05.760
It is actually a per request fallback.

s35
00:03:05.760 --> 00:03:15.599
What that means is you can actually try model provider A and then in sequence try model provider B if your request to model provider A fails.

s36
00:03:15.599 --> 00:03:20.640
Another option to consider here is you can fire requests to both the providers in parallel.

s37
00:03:20.640 --> 00:03:28.480
But that's only if you're highly highly obsessed with latencies because that's just going to double your cost.

s38
00:03:28.560 --> 00:03:33.760
Some of the similar circuit breaking patterns apply here to LMS as well.

s39
00:03:33.760 --> 00:03:42.080
If you know that your primary has been failing for some time, it doesn't make sense to try it again.

s40
00:03:42.080 --> 00:03:53.840
You put it, you take it out of the load balancer or your request path and put it in a cool down and then after a few minutes have passed, try putting that back again.

s41
00:03:53.840 --> 00:04:00.400
One interesting choice that you have to make here is where your failure counts live.

s42
00:04:00.400 --> 00:04:13.680
You can decide to have the failure counts live in memory on the instances that are serving your traffic or you can have shared infra where your failure counts are shared across the fleet.

s43
00:04:13.680 --> 00:04:15.840
There are trade-offs.

s44
00:04:15.840 --> 00:04:20.479
If you want quick failovers, then fleetwide helps.

s45
00:04:20.479 --> 00:04:31.680
And with instance uh with local state counters the issue that you run into is whenever you change your deployment size your configuration and your expectations change.

s46
00:04:31.680 --> 00:04:34.479
So something to consider.

s47
00:04:35.360 --> 00:04:40.639
What that clean diagram did not really show you are some of the other gotchas that I'm going to discuss.

s48
00:04:40.639 --> 00:04:43.360
So fallbacks are not transparent.

s49
00:04:43.360 --> 00:04:50.479
While the industry is converging on an OpenAI API compatible format, I would say there are still nuances.

s50
00:04:50.479 --> 00:04:53.440
So you need to really test your fallbacks well.

s51
00:04:53.440 --> 00:04:58.720
They can have differences in your tool calling schemas, token limits, stop reasons and what have you.

s52
00:04:58.720 --> 00:05:08.960
So with LM gateways, you can have a normalization layer that can ensure that you can do cross provider fallbacks as well.

s53
00:05:08.960 --> 00:05:16.320
Another thing is streaming it.

s54
00:05:16.320 --> 00:05:22.960
So essentially nobody wants to wait for 30 seconds to have a wall of text appear in front of them.

s55
00:05:22.960 --> 00:05:27.280
So there are use cases where streaming is absolutely required.

s56
00:05:27.280 --> 00:05:28.639
But it comes as at a cost.

s57
00:05:28.639 --> 00:05:30.639
You trade away your levers.

s58
00:05:30.639 --> 00:05:40.720
You cannot once you have decided to go with provider A, you have to continue going with provider A. You cannot mid-stream change the providers.

s59
00:05:40.720 --> 00:05:43.919
Whatever has been sent to the client, it's done.

s60
00:05:43.919 --> 00:05:49.440
And that's where the something uh went wrong message, that's the one that you see.

s61
00:05:49.440 --> 00:05:50.800
It's not because of laziness.

s62
00:05:50.800 --> 00:05:55.680
It's by design uh that you see that and it's one of the trade-offs.

s63
00:05:55.680 --> 00:06:01.520
I would like to call out one other thing where I've seen teams trip over and over again.

s64
00:06:01.520 --> 00:06:12.479
They really provision and test their primary providers really well, but they the second provider, the fallback provider doesn't necessarily get the same level of love.

s65
00:06:12.479 --> 00:06:24.560
And I would argue that your throughputs or your capacity or your headroom should be even higher for the second provider or the fallback provider because that's your last line of defense.

s66
00:06:24.560 --> 00:06:28.639
If that goes down, your application goes down.

s67
00:06:30.479 --> 00:06:32.639
Let's discuss latencies.

s68
00:06:32.639 --> 00:06:35.759
Availability failures are right in your face.

s69
00:06:35.759 --> 00:06:37.120
They fail.

s70
00:06:37.120 --> 00:06:38.160
You get alarmed.

s71
00:06:38.160 --> 00:06:39.759
You get paged.

s72
00:06:39.759 --> 00:06:43.280
But high latencies can be the quiet ones.

s73
00:06:43.280 --> 00:06:51.199
And they need to receive more love um than I would say tuning your services for just availability.

s74
00:06:54.319 --> 00:07:04.960
One thing to call out, a gateway may run mixed workloads and you can have embedding embedding requests that takes just less than a second.

s75
00:07:04.960 --> 00:07:08.479
You can have classification requests that take less than a second.

s76
00:07:08.479 --> 00:07:15.120
Uh you have chat requests taking 3 seconds and reasoning requests taking a long time.

s77
00:07:15.120 --> 00:07:16.560
Quick show of hands.

s78
00:07:16.560 --> 00:07:22.639
If you measure your aggregate latency for your entire service.

s79
00:07:22.880 --> 00:07:24.080
Well, that was a trick question.

s80
00:07:24.080 --> 00:07:24.479
Sorry.

s81
00:07:24.479 --> 00:07:25.039
You shouldn't.

s82
00:07:25.039 --> 00:07:25.919
It doesn't make sense.

s83
00:07:25.919 --> 00:07:27.759
It's a lie.

s84
00:07:27.759 --> 00:07:32.960
You should be tracking your P99 per model per route, not a gateway wide number.

s85
00:07:32.960 --> 00:07:37.039
Gateway wide number doesn't make sense, especially if you're running mixed workloads.

s86
00:07:37.039 --> 00:07:41.280
And I hope you're not u for those who raise your hand.

s87
00:07:41.280 --> 00:07:50.880
Another thing that can really I cannot emphasize this enough is for you to set timeouts on per model class per route.

s88
00:07:50.880 --> 00:07:54.639
That's where that's the number one root cause of your silent outage.

s89
00:07:54.639 --> 00:08:01.680
If you don't have a timeout, your gateway thinks you're hap your request is being happily served while it is not.

s90
00:08:01.680 --> 00:08:04.560
And I'll leave you with this message for for latencies.

s91
00:08:04.560 --> 00:08:10.639
Um, specifically a reasoning models normal is actually a chat models outage.

s92
00:08:10.639 --> 00:08:15.919
So you definitely need to track latency per route.

s93
00:08:17.120 --> 00:08:25.520
Okay, this is the most painful or this the slide that has given me the most scarse which is reasoning and router models.

s94
00:08:25.520 --> 00:08:41.200
So this is where truly the latency is unpredictable and reasoning models they do not give you they they're highly undeterministic more deterministic undeterministic than your normal models.

s95
00:08:41.200 --> 00:08:51.600
You cannot set the temperature to zero in many cases and the same prompt can take somewhere from 2 seconds to 60 seconds and we've seen that in production where P99

s96
00:08:51.600 --> 00:08:54.720
suddenly popped to 60 seconds for no good reason.

s97
00:08:54.720 --> 00:08:59.200
So that's while there's no magical solution to it.

s98
00:08:59.200 --> 00:09:04.800
I would recommend that you at least start with fixing the reasoning level per route.

s99
00:09:04.800 --> 00:09:09.279
So with router models, they hide that abstraction behind you.

s100
00:09:09.279 --> 00:09:22.640
Like they pick which models to run and I would highly recommend that you at least make as much uh you make requests as determinist deterministic as possible with an undeterministic

s101
00:09:22.640 --> 00:09:24.320
system.

s102
00:09:24.320 --> 00:09:27.760
Another idea is hedging the tail.

s103
00:09:27.760 --> 00:09:37.360
You can have a you can fire another request if your primary request actually consumed let's say P90 of your latency budget.

s104
00:09:37.920 --> 00:09:45.279
This can hedge the t this can really hedge the P99 tail u for for your services.

s105
00:09:45.680 --> 00:09:46.240
All right.

s106
00:09:46.240 --> 00:09:48.320
This is one of my favorite ones.

s107
00:09:48.320 --> 00:09:54.640
Um to keep your model secure you need to have guardrails.

s108
00:09:54.640 --> 00:10:09.839
And with that, guardrails are necessary for preventing your services from prompt injection attacks, keeping PII filters in place, having toxicity filters, keeping the LMS to stop swearing at your customers,

s109
00:10:09.839 --> 00:10:11.839
all those good things.

s110
00:10:11.839 --> 00:10:15.920
But just like a model provider, there are trade-offs, too.

s111
00:10:15.920 --> 00:10:29.680
Guardrails are just like another service that can go down that can be unreliable and that's where you need to choose do you fail open or do you fail close when I say fail open

s112
00:10:29.680 --> 00:10:41.680
you can still serve the request even if your guardrails are down fail close you block the request and say hey I'm not available that's the trade-off between availability and security to certain extent

s113
00:10:41.680 --> 00:10:54.399
while there's a no universal answer it really depends on your use case you can decide like for example a toxicity filter if it's not up and running you can still serve that request.

s114
00:10:54.399 --> 00:11:00.880
So the default choice should be the worst case that you can live with.

s115
00:11:03.680 --> 00:11:18.000
There are a few things that you can actually do to improve the behavior of your systems in face of uh you know guardrails being down and and managing just unreliability of the guardrails themselves.

s116
00:11:18.000 --> 00:11:20.959
So the first is time budget.

s117
00:11:21.360 --> 00:11:26.399
Your request should never be bound by your guardrail timing.

s118
00:11:26.399 --> 00:11:30.959
It should always be the LM that is the rate determining step.

s119
00:11:30.959 --> 00:11:38.880
So make sure that you have timeouts in place and those guardrails run with a specific time budget.

s120
00:11:38.880 --> 00:11:41.120
Another important thing is fallback.

s121
00:11:41.120 --> 00:11:44.880
You've heard, you probably know and I've talked about it.

s122
00:11:44.880 --> 00:11:59.920
We always discuss fallbacks with regards to model providers, but guardrails are critical services too where you can consider fallbacks, have secondary provider, secondary checks, cache decisions uh to keep your service available

s123
00:11:59.920 --> 00:12:04.160
when a guardrail provider is down.

s124
00:12:04.160 --> 00:12:11.600
Another interesting choice that pops up with regards to guardrails is the placement of the guardrails.

s125
00:12:11.839 --> 00:12:16.079
Typically, you can place the guardrail in three ways.

s126
00:12:16.079 --> 00:12:20.880
You can have a pre- hook that runs where the guardrail actually runs on the input.

s127
00:12:20.880 --> 00:12:28.000
You can and that's probably the safest uh but it does add serial latency uh to your requests.

s128
00:12:28.000 --> 00:12:29.920
Another one is in parallel.

s129
00:12:29.920 --> 00:12:36.880
This is one of my favorites, but just to call out, streaming wouldn't work well here with with parallel.

s130
00:12:36.880 --> 00:12:40.720
So if you're specially producing structured output, please don't stream them.

s131
00:12:40.720 --> 00:12:46.959
Uh try to save your latencies and run run these guardrails concurrently for your structured outputs.

s132
00:12:46.959 --> 00:12:48.560
Another one is post hooks.

s133
00:12:48.560 --> 00:12:56.560
The these are best for um output monitoring, auditing your outputs and and so forth.

s134
00:12:58.959 --> 00:13:07.920
So, so far we've all I've discussed all the things that can go wrong with regards to our dependencies.

s135
00:13:07.920 --> 00:13:16.320
We haven't discussed that we are actually adding another dependency in the request path itself which is the central or which is the LM gateway itself.

s136
00:13:16.320 --> 00:13:22.720
There are a few things where we have been bitten by u and we've learned some lessons that I want to share with you.

s137
00:13:22.720 --> 00:13:29.120
If you're working on an LLM gateway or using one, one is shared limits.

s138
00:13:29.120 --> 00:13:41.680
Make sure that your API keys are segregated per route, per use case to the most granular possible uh to the most granular thing that you can imagine.

s139
00:13:41.680 --> 00:13:49.839
U having a noisy tenant can be one of the biggest problems here.

s140
00:13:49.839 --> 00:13:52.000
Another thing is load shedding.

s141
00:13:52.000 --> 00:14:04.480
This is a feature that you should uh as part of your runbooks, game days, uh make sure that the gateway that you're using supports load shedding because when you have a retry storm, it becomes really hard to just scale out.

s142
00:14:04.480 --> 00:14:14.560
You cannot simply scale out services that is under a retry storm and all these web servers they have an internal queue and they're configurable.

s143
00:14:14.560 --> 00:14:20.880
Make sure that they're bounded and they cannot request they cannot accept requests that are unbounded.

s144
00:14:20.880 --> 00:14:29.920
And if you want to have some custom logic, you can even have traffic prioritization here as well to make sure under load your most important use cases get served.

s145
00:14:29.920 --> 00:14:39.519
Well, last thing that I wanted to discuss is the whole idea of a central gateway itself.

s146
00:14:39.519 --> 00:14:41.440
It is a single point of failure.

s147
00:14:41.440 --> 00:14:52.800
So if you're thinking of having a central gateway for your entire company for to LLMs, I would recommend rethink that and see what are the reasons that you want it.

s148
00:14:52.800 --> 00:14:58.000
What I've noticed is that in most scenarios, it's not the central gateway that they want.

s149
00:14:58.000 --> 00:15:00.800
They want centralized governance.

s150
00:15:00.800 --> 00:15:08.320
And there is a path forward where you can actually decentralize the gateway and still centralize government governance.

s151
00:15:08.320 --> 00:15:18.240
So do not try to centralize your traffic but you can have plugins, you can have custom code that can centralize your governance.

s152
00:15:18.240 --> 00:15:25.360
Uh governance can be in the form of cost tracking, rate limit managing management and there are other solutions possible.

s153
00:15:25.360 --> 00:15:32.160
So explore those before you chart on having one central gateway for your entire company.

s154
00:15:32.160 --> 00:15:43.199
It can be managed by a single team, but I wouldn't recommend deploying it as a single deployment for the entire company even though it's distributed.

s155
00:15:43.680 --> 00:15:47.920
With that said, I want to end this talk on a personal note.

s156
00:15:47.920 --> 00:15:55.279
So, it is my son's birthday today and I'm here talking to strangers about circuit breaking.

s157
00:15:55.279 --> 00:16:02.880
So the least you can do for me is please go and prevent one incident for me and for your customers.

s158
00:16:02.880 --> 00:16:03.440
Thank you.

s159
00:16:03.440 --> 00:16:05.519
If you have any questions.

s160
00:16:05.519 --> 00:16:07.519
Yeah.

s161
00:16:21.201 --> 00:16:23.201
[music]