WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=y-OVWZD4j6U

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/y-OVWZD4j6U.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.559 --> 00:00:13.759
Hey everybody.

s3
00:00:13.759 --> 00:00:15.360
Um, my name is Eric.

s4
00:00:15.360 --> 00:00:21.199
I'm a product manager at Traversal and today we're going to talk about self-driving production.

s5
00:00:21.199 --> 00:00:24.080
So, just a quick primer on what I'll cover.

s6
00:00:24.080 --> 00:00:30.560
Uh I'm going to talk a little bit about how AI agents are changing the software development life cycle.

s7
00:00:30.560 --> 00:00:40.719
Uh the problems that that introduces for engineers and how AI for site reliability engineering fits in uh which is where traversal comes into play.

s8
00:00:40.719 --> 00:00:53.039
And then I'll talk about a few examples of um what this looks like in practice for enterprises that we work with at Traversal uh and a bit about how we've built our AIS surre

s9
00:00:53.039 --> 00:00:57.920
uh to deliver the results that we're seeing with Fortune 500 companies.

s10
00:00:58.320 --> 00:01:06.400
So uh I guess to start some quick context on uh kind of the different phases of software engineering.

s11
00:01:06.400 --> 00:01:12.479
Uh so sure you're all familiar but at a high level you could think about software engineering as kind of three buckets.

s12
00:01:12.479 --> 00:01:15.280
There's system design like what is it that I want to build?

s13
00:01:15.280 --> 00:01:17.520
What's the architecture that I want to build?

s14
00:01:17.520 --> 00:01:21.439
There's development which is actually writing the code building the thing.

s15
00:01:21.439 --> 00:01:22.720
And then there's troubleshooting.

s16
00:01:22.720 --> 00:01:29.360
So once you build something and it's interacting with the real world, it starts to break uh and you need to fix it.

s17
00:01:29.360 --> 00:01:34.799
And so with the advent of coding agents, the development portion has shrunk.

s18
00:01:34.799 --> 00:01:36.479
development is a lot faster.

s19
00:01:36.479 --> 00:01:42.720
Folks like me without any formal computer science training can now contribute u production code.

s20
00:01:42.720 --> 00:01:43.920
And so this is great.

s21
00:01:43.920 --> 00:01:45.600
Development's a lot faster.

s22
00:01:45.600 --> 00:01:47.759
Teams can do 10x more.

s23
00:01:47.759 --> 00:01:50.000
But then what happens to either end of the spectrum?

s24
00:01:50.000 --> 00:01:54.000
What happens to system design and what happens to troubleshooting?

s25
00:01:54.000 --> 00:02:00.880
The hope is that you spend more time on the design part thinking creatively about what it is that you want to build.

s26
00:02:00.880 --> 00:02:08.479
But what we actually see with the enterprises that we work with is that um more and more time is being spent on troubleshooting.

s27
00:02:08.479 --> 00:02:11.120
Uh much more code is being written.

s28
00:02:11.120 --> 00:02:14.400
People have less understanding of the code that's being pushed into production.

s29
00:02:14.400 --> 00:02:20.000
And so what you end up with is more issues, more comp uh more complex environments.

s30
00:02:20.000 --> 00:02:29.040
Uh, and instead of spending all your time on the creative architecture and design, your teams are bogged down on troubleshooting.

s31
00:02:29.680 --> 00:02:32.480
And the the data kind of bears this out.

s32
00:02:32.480 --> 00:02:33.760
This is a big problem.

s33
00:02:33.760 --> 00:02:38.959
Um, by some accounts, enterprises are spending upwards of $400 billion a year.

s34
00:02:38.959 --> 00:02:43.040
Uh, 40% of executives say that this is a problem that their teams face.

s35
00:02:43.040 --> 00:02:51.120
And on average, engineers are losing seven plus hours each week uh just troubleshooting when they're on call.

s36
00:02:51.200 --> 00:03:00.959
And this is going to become an increasing issue that we'll all hear more about, that we'll all experience more over time as tools like cloud code and codeex and cursor,

s37
00:03:00.959 --> 00:03:08.400
which are fantastic, um uh they are fantastic, but they result in more code and more complexity.

s38
00:03:08.400 --> 00:03:10.800
And so what are we supposed to do about it?

s39
00:03:10.800 --> 00:03:14.560
Um there are all kinds of great observability tools.

s40
00:03:14.560 --> 00:03:21.120
Tools like data dog, elastic, splunk, service now where I worked for several years before joining traversal.

s41
00:03:21.120 --> 00:03:28.000
Um these are great but uh there's only so much that they can do like these tools will tell you that something is broken.

s42
00:03:28.000 --> 00:03:30.239
They'll point out all the things that have broken.

s43
00:03:30.239 --> 00:03:35.519
Um they'll point out things that are maybe correlated but they won't tell you the root cause of the issue.

s44
00:03:35.519 --> 00:03:47.040
And when I worked on observability products at Service Now, this was something I heard time and again from the customers that I served, which was like, you're telling me what's broken, you're not telling me why, and you're not telling me what to do about it.

s45
00:03:47.040 --> 00:03:58.959
And so, no matter how many dashboards you create in these tools, it's not really helping teams keep up with the pace of innovation uh and complexity that's being added to their environments.

s46
00:03:59.840 --> 00:04:02.640
And this is not just something we believe at Traversal.

s47
00:04:02.640 --> 00:04:04.640
This is kind of a widely held view.

s48
00:04:04.640 --> 00:04:16.239
um Google who kind of wrote the S sur handbook like the bible of site reliability engineering uh has commented on this directly and we also heard recently from anthropic

s49
00:04:16.239 --> 00:04:21.919
u quotes where they described in their own words that LLM are not great at this problem.

s50
00:04:21.919 --> 00:04:27.040
They're not great at pointing out what is the root cause of an issue.

s51
00:04:27.919 --> 00:04:29.840
Now like why is this so hard?

s52
00:04:29.840 --> 00:04:31.520
What makes this so challenging?

s53
00:04:31.520 --> 00:04:38.800
Um in this particular example that we have on the screen like which was based on something we saw with one of our customers.

s54
00:04:38.800 --> 00:04:50.080
Um if a checkout API is failing uh you might need to make five to 10 hops across like dozens of services and pabytes of data to actually get to the root cause.

s55
00:04:50.080 --> 00:05:02.800
And so in a very small contained environment, perhaps there's a single seasoned SR at your company that knows the full stack and can debug everything and has all the tribal knowledge to sift through all your data.

s56
00:05:02.800 --> 00:05:09.759
But if you're a Fortune50 enterprise or a Fortune 100 enterprise, there's no single engineer that has all of that context.

s57
00:05:09.759 --> 00:05:19.680
And so what you end up with is a war room with 50 engineers like pulled in uh and many many hours trying to debug this issue because everyone has

s58
00:05:19.680 --> 00:05:22.880
their own particular slice of context.

s59
00:05:22.880 --> 00:05:25.759
No one has kind of the full scope and breadth of this.

s60
00:05:25.759 --> 00:05:38.479
And so no, it's really challenging for a human or a human paired with cloud code to get kind of the full spectrum of context needed to jump from a checkout API is failing over here

s61
00:05:38.479 --> 00:05:44.560
five hops away to the expired TLS certificate that maybe caused the issue.

s62
00:05:46.479 --> 00:05:52.639
And so that's kind of what motivated u our founders to to launch traversal.

s63
00:05:52.639 --> 00:06:00.720
Fundamentally the belief that we have at traversal is that uh root cause analysis is not an observability problem.

s64
00:06:00.720 --> 00:06:02.560
It's a it's a causal problem.

s65
00:06:02.560 --> 00:06:06.800
And so that's why our four founders, three of them come from academia.

s66
00:06:06.800 --> 00:06:08.880
One comes from quant finance.

s67
00:06:08.880 --> 00:06:18.160
Uh and they've dedicated like decades of their lives towards causal machine learning which is how do you find cause and effect?

s68
00:06:18.160 --> 00:06:24.720
uh and have applied it to this problem because it's a massive one that enterprises are really struggling with.

s69
00:06:25.680 --> 00:06:30.960
And so that kind of brings us to the topic of this talk which is self-driving production.

s70
00:06:30.960 --> 00:06:39.759
Um what we're building at Traversal is the ability for an enterprise to basically have a self-driving production environment.

s71
00:06:39.759 --> 00:07:00.160
What that means is a closed loop system where uh as issues break the AI system finds the issue the finds the issue or the incident or the alert finds the root cause of what caused that issue in the first place puts up a fix verifies that it's been fixed and the loop is closed with hopefully without ever

s72
00:07:00.160 --> 00:07:05.599
having to page or disrupt any of your team members so that they can keep focusing on building.

s73
00:07:05.599 --> 00:07:07.360
And this is a spectrum.

s74
00:07:07.360 --> 00:07:10.479
U a lot of companies are not fully there yet.

s75
00:07:10.479 --> 00:07:16.240
We like to think of it in terms of levels almost like self-driving cars uh or autonomous driving.

s76
00:07:16.240 --> 00:07:18.479
We think about it in the same frame.

s77
00:07:18.479 --> 00:07:23.599
So level zero is where most companies are today where everything is fully manual.

s78
00:07:23.599 --> 00:07:27.199
You're pulling a bunch of people into a Slack war room or onto a Zoom call.

s79
00:07:27.199 --> 00:07:30.080
You're spending hours debugging things.

s80
00:07:30.080 --> 00:07:34.400
Level one might be where you have some rules and automations.

s81
00:07:34.400 --> 00:07:40.639
And we're hearing a lot about like loops that teams are building now with tools like cloud code or cursor or codeex.

s82
00:07:40.639 --> 00:07:49.599
Uh and so these might be like rulebased automations where you can build out a very structured specific workflow in response to an alert.

s83
00:07:49.599 --> 00:07:53.599
Those rule-based automations kind of break down when it's a novel situation.

s84
00:07:53.599 --> 00:07:56.400
And usually a sub one is something you haven't seen before.

s85
00:07:56.400 --> 00:07:57.919
There's not really a runbook for it.

s86
00:07:57.919 --> 00:08:01.520
And so the rule-based automation breaks down there.

s87
00:08:01.520 --> 00:08:06.879
And level two and level three is kind of where you start to get a little more automation built in.

s88
00:08:06.879 --> 00:08:17.280
In level three, perhaps you have like a a homegrown agent that is really good at debugging a issues for a specific service or a specific set of alerts.

s89
00:08:17.280 --> 00:08:24.639
But really where we see the leap is going from level three to level four, which is cutting across an entire environment.

s90
00:08:24.639 --> 00:08:30.639
So hundreds of services like hundreds of repos, thousands of lo log indexes.

s91
00:08:30.639 --> 00:08:34.880
It's quite challenging to homegrow a solution that can handle that.

s92
00:08:34.880 --> 00:08:47.120
And level five is kind of the holy grail, which is not only can we diagnose issues across a full production environment for a large enterprise, but also put up the fixes and verify the fixes as well.

s93
00:08:47.839 --> 00:08:55.200
And so, uh, we're really proud that at Traversal, we get to work with and serve some of the biggest companies in the world.

s94
00:08:55.200 --> 00:09:03.519
Companies like American Express, who I spend a lot of time with personally, uh, Pepsi, Digital Ocean, Capital One.

s95
00:09:03.519 --> 00:09:10.000
Um, and it's really cool to see the volume and scale of data that Traversal is able to handle.

s96
00:09:10.000 --> 00:09:26.640
So, you know, trillions of logs that are generated, trillions of spans, tens of billions of metrics and events, um, and hitting 80% plus root cause uh, for these high severity incidents at this scale is a technical feat that we're pretty proud of.

s97
00:09:26.640 --> 00:09:36.560
And so, beyond just these kind of results at a high level, um, as a product manager, I like to think about the use cases that we build for our customers.

s98
00:09:36.560 --> 00:09:42.480
And so there's there's a few different types of scenarios and use cases that we found that can deliver a lot of value.

s99
00:09:42.480 --> 00:09:52.399
I'll dig into a few in more detail, but at a high level, there's alert intelligence, a product that I spend a lot of time on, which is helping on call teams

s100
00:09:52.399 --> 00:09:59.600
triage and prioritize hundreds or thousands of alerts that they may be receiving in a single day, helping you deal with alert fatigue.

s101
00:09:59.600 --> 00:10:03.440
There is incident root cause analysis, which is where we got started.

s102
00:10:03.440 --> 00:10:11.680
How do we help you find the root cause, pinpoint the root cause of an incident in minutes when it otherwise would have taken hours and dozens of engineers?

s103
00:10:11.680 --> 00:10:19.440
There is self-healing, which is not only telling you what the root cause was, but putting up a fix to address and mitigate that issue.

s104
00:10:19.440 --> 00:10:30.079
There's production support, which is the the variety of questions and issues that your team might be dealing with in any given day, wanting to chat with their production environment,

s105
00:10:30.079 --> 00:10:38.079
get specific data points that maybe aren't part of a a broader incident, but are still taking up a lot of time and toil for your teams.

s106
00:10:38.079 --> 00:10:40.640
And then pre-production, there is code resilience.

s107
00:10:40.640 --> 00:10:47.600
Like when you're putting up changes, how do you make sure that this code is making your system more resilient and reliable over time?

s108
00:10:47.600 --> 00:10:51.920
And so at Traversal, we cover kind of the whole gamut of use cases.

s109
00:10:51.920 --> 00:11:05.920
Um, but I'll dig into alert intelligence and incident root cause just to give you a bit of a more detailed sense of what that might look like, what that looks like, and I'm happy to answer any questions after about the others.

s110
00:11:06.560 --> 00:11:13.920
So for alert intelligence, the example that I have here is uh based on our engagement with Pepsi.

s111
00:11:13.920 --> 00:11:21.600
And so Pepsi uses traversal across a bunch of their different applications, but a very core one is for their supply chain.

s112
00:11:21.600 --> 00:11:30.480
They have a lot of internal tooling and software to make sure that um the raw materials and the finished product can move through their supply chain.

s113
00:11:30.480 --> 00:11:39.839
And a very critical piece of that is making sure that finished product gets from their warehouse to their trucks to their retailers at the end.

s114
00:11:39.839 --> 00:11:47.120
And um before traversal their team would get thousands or tens of thousands of alerts every single week.

s115
00:11:47.120 --> 00:11:52.240
Uh and at any given time a single engineer might have a backlog of 700 alerts.

s116
00:11:52.240 --> 00:11:55.200
This is kind of a crippling state to be in.

s117
00:11:55.200 --> 00:11:59.040
Like you have so much noise that you don't know where to look.

s118
00:11:59.040 --> 00:12:02.000
You don't know what's broken and things slip through the cracks.

s119
00:12:02.000 --> 00:12:04.880
Uh or you just try to keep up and burn out.

s120
00:12:04.880 --> 00:12:15.760
And so this was a big problem that Pepsi was facing and they used traversal uh to help parse the signal from the noise of these tens of thousands of alerts.

s121
00:12:15.760 --> 00:12:27.360
And so now instead of each engineer having a backlog of 700 alerts, they get a very pri a prioritized and filtered set of alerts that have already been pre-investigated.

s122
00:12:27.360 --> 00:12:31.040
We flag to them the alerts that are actually worth digging into.

s123
00:12:31.040 --> 00:12:41.519
We flagged to them opportunities to reduce alert noise, whether that's updating the underlying alert rules in their code, uh dismissing alerts that don't need to be addressed right now,

s124
00:12:41.519 --> 00:12:50.079
uh or creating tickets for alerts that are maybe technical debt that can be handled later on but aren't super important to deal with at this moment.

s125
00:12:50.079 --> 00:12:57.920
And so we've seen a lot of great results with Pepsi uh on this alert use case and many of our other customers as well.

s126
00:12:57.920 --> 00:13:08.560
Um the next example is around incident root cause analysis and and this is a case study from our engagement with annex um where I've personally spent a lot of time myself.

s127
00:13:08.560 --> 00:13:27.360
Um, so before working with traversal, what an incident looked like at American Express is anywhere from five to 10 teams get paged, anywhere from 20 to 50 engineers, and it could take, in this particular example, 60 minutes, but it could take hours or days

s128
00:13:27.360 --> 00:13:30.160
to get resolution to to an incident.

s129
00:13:30.160 --> 00:13:34.399
You can imagine if customers aren't able to pay their credit card bill, like that's a big problem.

s130
00:13:34.399 --> 00:13:38.639
Or if they're not able to log into their mobile app, that's a big problem.

s131
00:13:38.639 --> 00:13:46.000
And so now uh traversal is the first responder to every single incident that's created at American Express.

s132
00:13:46.000 --> 00:13:50.160
And so what that looks like is the second the bridge is declared.

s133
00:13:50.160 --> 00:13:51.760
That's the term they use internally.

s134
00:13:51.760 --> 00:14:02.320
The second the incident is declared traversal is dispatched within 3 minutes we'll post a very detailed root cause analysis in the Slack channel where that incident is being managed.

s135
00:14:02.320 --> 00:14:06.240
We can post updates to Service Now tickets if they want to as well.

s136
00:14:06.240 --> 00:14:15.519
And rather than paging five teams and 58 engineers because you have that initial analysis, it's a lot less chaotic, you have a sense of what actually broke.

s137
00:14:15.519 --> 00:14:23.440
And either no teams get paged or maybe you page the one or two teams just to verify the findings that traversal posted.

s138
00:14:23.440 --> 00:14:32.320
Uh and so we save in this case like 50 plus engineers uh from the time and headache of being paged uh into this incident.

s139
00:14:32.320 --> 00:14:36.480
Um, and that's especially appreciated when these incidents are happening in the middle of the night.

s140
00:14:36.480 --> 00:14:41.839
Like this is a a good night of sleep that 53 engineers can have now.

s141
00:14:42.560 --> 00:14:48.639
And so those are those are two of the use cases that we're especially proud of at Traversal, but like I mentioned, there's a bunch of others.

s142
00:14:48.639 --> 00:14:55.920
Maybe in the last couple minutes I'll just highlight like AI site reliability engineering is a very hot space.

s143
00:14:55.920 --> 00:15:03.279
You've probably, if you're familiar, you've probably heard of a lot of companies in the space or claiming that they're solving these problems.

s144
00:15:03.279 --> 00:15:10.160
Um, we think that there's a few really key questions you should be asking uh that are critical to getting this right.

s145
00:15:10.160 --> 00:15:16.639
So, the first is can your AI surre see all of your production data?

s146
00:15:16.639 --> 00:15:24.160
If you're only giving it a slice or if it has gaps in its understanding, it's going to be very hard to get to a very detailed root cause.

s147
00:15:24.160 --> 00:15:35.839
The second is can it search through this data can be it's often pabytes of data hundreds of billions of logs um without blowing up costs or taking down your observability

s148
00:15:35.839 --> 00:15:37.040
infrastructure.

s149
00:15:37.040 --> 00:15:44.320
Um this is really not trivial to do at scale for a Fortune 100 enterprise.

s150
00:15:44.320 --> 00:15:50.720
The third is can it map out the relationship relationships between all of these entities in your data.

s151
00:15:50.720 --> 00:15:57.440
even if you're able to ingest and read through all this data, if you don't know what to do with it and if you're lost, it's kind of worthless.

s152
00:15:57.440 --> 00:16:14.320
And then four, does it get better and smarter over time autonomously without having to dedicate engineering resources to maintaining a bunch of markdown files or having a bunch of forward deployed engineers taking up your your team's time to maintain the system knowledge?

s153
00:16:14.320 --> 00:16:17.360
And finally, u can it make multiple hops?

s154
00:16:17.360 --> 00:16:24.480
Can it find the non-trivial, non-obvious root cause that's far away from the initial symptom in a matter of minutes?

s155
00:16:24.480 --> 00:16:29.680
What we've seen is like anything more than 5 minutes and you've kind of lost the plot.

s156
00:16:29.680 --> 00:16:34.399
You've lost the patience of the on call team and people will fall back onto their own habits.

s157
00:16:34.399 --> 00:16:35.920
So, it needs to be really good.

s158
00:16:35.920 --> 00:16:38.800
It needs to be really fast.

s159
00:16:39.120 --> 00:16:47.120
And I won't go into this in too much detail, but this is kind of how we think about these five questions at traversal.

s160
00:16:47.120 --> 00:16:56.639
So at the bottom of the stack is how we think about kind of a analyzing all your data uh without increasing cost.

s161
00:16:56.639 --> 00:17:03.920
So basically how do we ingest and map all of the data uh that we're connected to and integrate with all of that data.

s162
00:17:03.920 --> 00:17:06.559
In the middle of the stack is how we make sense of that data.

s163
00:17:06.559 --> 00:17:09.520
And so we call that our production world model.

s164
00:17:09.520 --> 00:17:14.880
How do we map out all the relationships between all the dent the data that we're ingesting?

s165
00:17:14.880 --> 00:17:24.079
And then finally, how do we surface that and package that up in either use cases or user experiences that are valuable to your team?

s166
00:17:24.079 --> 00:17:31.600
And so there's a lot of components to getting this right u especially at Fortune 500 scale.

s167
00:17:32.320 --> 00:17:38.000
And so um that's a bit about what we've been building at Traversal, what we're delivering at Traversal.

s168
00:17:38.000 --> 00:17:41.280
The core pieces like I mentioned are the production world model.

s169
00:17:41.280 --> 00:17:49.440
So how we map and make sense of all the data that we integrate with and then what we call uh what we call refer to as our causal search engine.

s170
00:17:49.440 --> 00:17:57.600
Basically the the harness and mechanism that we give our agents to search through all of that data uh that we've mapped out.

s171
00:17:57.600 --> 00:18:08.640
And so the production world model and the causal search engine are how we believe we are going to help Fortune 500 enterprises get to the level five of autonomy

s172
00:18:08.640 --> 00:18:11.760
fully self-driving production.

s173
00:18:12.640 --> 00:18:13.120
Cool.

s174
00:18:13.120 --> 00:18:15.919
Thank you for your time.

s175
00:18:30.892 --> 00:18:32.892
[music]