WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=vSx5IULvBns

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/vSx5IULvBns.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:13.560
Hello.

s3
00:00:13.560 --> 00:00:15.000
Hello.

s4
00:00:15.000 --> 00:00:23.600
Everybody, welcome to this talk always on agents run production without the on-call tax.

s5
00:00:23.600 --> 00:00:25.080
My name is Justin Smith.

s6
00:00:25.080 --> 00:00:28.400
One of the founding product engineers at Resolve AI.

s7
00:00:28.400 --> 00:00:37.560
Been in the space for about 15 plus years in the sort of monitoring, observability, how do you kind of operate production systems space.

s8
00:00:37.560 --> 00:00:39.440
Was at Splunk for a while.

s9
00:00:39.440 --> 00:00:42.360
Was one of the architects on the observability suite there.

s10
00:00:42.360 --> 00:00:45.120
Spent a good 10 year at VMware.

s11
00:00:45.120 --> 00:00:48.520
Really really enjoy like product design and front-end architecture.

s12
00:00:48.520 --> 00:00:54.680
How do you How do you How do people experience a product or use case or something like that?

s13
00:00:54.680 --> 00:00:58.720
That's the stuff I like to dabble in.

s14
00:00:58.720 --> 00:01:06.280
Um but I want to talk a little bit about the first wave of AI and it's it's been a fun one.

s15
00:01:06.280 --> 00:01:11.680
I think the first big wave and I'm sure we've all experienced this is just how we build software.

s16
00:01:11.680 --> 00:01:15.640
Um but there's some sort of net effects of that.

s17
00:01:15.640 --> 00:01:18.680
It's a lot of bigger PRs that are coming through.

s18
00:01:18.680 --> 00:01:21.280
We definitely see a lot of this a lot more frequently.

s19
00:01:21.280 --> 00:01:35.560
So people are shipping code at a much, you know, faster rate from developers and we're beginning to see maybe from even non-developers that maybe don't actually know the code or what it's doing or sort of like operating principles behind it.

s20
00:01:35.560 --> 00:01:39.240
But we're getting developer productivity.

s21
00:01:39.240 --> 00:01:40.400
And that's good, right?

s22
00:01:40.400 --> 00:01:45.080
That's that's a good thing that we're all able to sort of produce more and faster.

s23
00:01:45.080 --> 00:01:56.120
Kind of sort of what we actually found out and this was a survey study done is that 70% of the time from an engineer is actually not just like is not focused just on writing code.

s24
00:01:56.120 --> 00:02:00.560
It's actually spent on actually running the code that is actually shipped into production.

s25
00:02:00.560 --> 00:02:08.000
Um maintaining all the the platforms, scaling the infrastructure, debugging all the incidents, and being on call, um shipping hot fixes, right?

s26
00:02:08.000 --> 00:02:20.120
Dealing with alerts, um updating all the sort of run books and operating procedures, restoring services, dealing with escalations, dealing with sort of like um questions from other, you know, teams and things like that.

s27
00:02:20.120 --> 00:02:24.840
So, really coding was never the the the big bottleneck, right?

s28
00:02:24.840 --> 00:02:28.560
Um a lot of it was really around, uh thank you, granola.

s29
00:02:28.560 --> 00:02:33.360
Um a lot of it was really around like how do we actually run these things sort of in production.

s30
00:02:34.320 --> 00:02:36.760
And that's getting harder and harder and harder.

s31
00:02:36.760 --> 00:02:42.840
Um AI is creating a lot more issues in production as, you know, AI code sort of goes through.

s32
00:02:42.840 --> 00:02:48.840
Um it's not clear we have the right sort of um structures in place to deal with the amount of kind of changes that are coming through.

s33
00:02:48.840 --> 00:02:52.840
Um unlimited tokens is is sort of coming to an end, the the token max, right?

s34
00:02:52.840 --> 00:02:54.160
They're starting to clamp down.

s35
00:02:54.160 --> 00:02:55.160
Prices are going up.

s36
00:02:55.160 --> 00:02:59.800
Companies are getting a lot more stringent on, you know, what's being used um for AI.

s37
00:02:59.800 --> 00:03:01.600
Um you know, we need full stack AI.

s38
00:03:01.600 --> 00:03:07.959
It's not just about the models anymore, it's about the context around the models and what the models can do inside of a specific domain.

s39
00:03:07.959 --> 00:03:13.400
These become the the problem areas that we need to sort of uh focus in and tackle on.

s40
00:03:13.400 --> 00:03:15.320
And this is true today, right?

s41
00:03:15.320 --> 00:03:19.800
So, it's it's creating more sort of uh complexity inside of our environment.

s42
00:03:19.800 --> 00:03:22.680
But, I mean the the reality is that systems have always been complex.

s43
00:03:22.680 --> 00:03:28.000
That's why we have, you know, these big tools that can, you know, try to give us insights into these systems.

s44
00:03:28.000 --> 00:03:37.080
Um there are multiple teams, there's multiple systems that are all having to work together, and they all have their own, you know, goals that they're trying to deliver towards, but you have organizational goals.

s45
00:03:37.080 --> 00:03:43.480
And how do you keep all of this sort of uh you know, um in Get rid of this.

s46
00:03:43.480 --> 00:03:45.880
Um how do you keep all of this in balance, right?

s47
00:03:45.880 --> 00:03:54.880
Um how do you, you know, pull all of this stuff together in a way that actually uh actually helps you and and facilitates your uh your organization.

s48
00:03:54.880 --> 00:04:07.920
Um and the answer is, well, you got to use AI inside of production to deal with um sort of the the amount of increase of complexity that AI is kind of putting into your product or into your system.

s49
00:04:08.120 --> 00:04:15.680
Um and so that's where Resolve, you know, this was kind of our sort of hypothesis from the beginning was um you know, we're going to see an influx in

s50
00:04:15.680 --> 00:04:21.920
um you know, issues uh coming out of coding um just the increase in coding uh velocity.

s51
00:04:21.920 --> 00:04:26.400
Um there's going to be more need for kind of AI to actually operate and run these run these systems.

s52
00:04:26.400 --> 00:04:35.080
Um we're, you know, lucky to work with, you know, some uh world-class engineering teams that are solving like really difficult problems at, you know, crazy scale.

s53
00:04:35.080 --> 00:04:49.800
Um and you know, that that gives us insight into a lot of how bigger organizations are having to deal with the influx of AI, etc. Um Resolve itself uh hosts a bunch of different sort of capabilities.

s54
00:04:49.800 --> 00:04:53.960
Um we have a number of agents that sort of you know, you get to kind of experience.

s55
00:04:53.960 --> 00:04:55.480
One of them is just an on-call agent.

s56
00:04:55.480 --> 00:04:57.040
This is kind of where we started, right?

s57
00:04:57.040 --> 00:05:01.600
Um so for every alert that comes in, um we can do a triage of that alert.

s58
00:05:01.600 --> 00:05:05.200
We can do kind of a full root cause investigation of that alert.

s59
00:05:05.200 --> 00:05:12.160
Um and you know, this is for anybody that's had to be on call before, um you know, on call is a it's a nightmare, right?

s60
00:05:12.160 --> 00:05:15.440
Um you're often uh only going on call every few weeks.

s61
00:05:15.440 --> 00:05:18.960
Uh you you maybe don't fully understand all the changes that have come in.

s62
00:05:18.960 --> 00:05:22.760
You don't fully understand maybe all the different systems that you're having to interact with.

s63
00:05:22.760 --> 00:05:30.520
Um and so, you know, the complexity is already there and having an AI agent that's able to come support you and pull that context together is incredibly incredibly valuable.

s64
00:05:30.520 --> 00:05:40.440
Um and so, uh that can often grow from just sort of getting a single page into a much larger incident across many different teams um across an organization.

s65
00:05:40.440 --> 00:05:55.000
Um and we have agents there to come support that much larger activity of, you know, all all kind of cross-collaboration, etc. Keeping everybody uh in sync and aligned on uh um where the where the incident is happening, what the impact of that is, um etc.

s66
00:05:55.000 --> 00:05:57.200
And then we also focus a lot on background agents.

s67
00:05:57.200 --> 00:06:04.840
And this is sort of covering the long tail of, you know, what happens when there's not a fire brewing um or or going on at any one, you know, point.

s68
00:06:04.840 --> 00:06:19.480
Um there's still lots of operational work that you as an engineer or an engineering team have to do and a lot of ceremonies of, you know, passing contact context off or dealing with kind of one-off issues or kind of having to scratch that itch in the back of your head of like, is that part of the system

s69
00:06:19.480 --> 00:06:20.919
okay or not okay?

s70
00:06:20.919 --> 00:06:24.360
Um and you're constantly having to sort of balance across all the these different things.

s71
00:06:24.360 --> 00:06:31.919
Um underneath all of that, you know, the we have an agent uh architecture um deals with models and context and reasoning and and actions.

s72
00:06:31.919 --> 00:06:35.520
Um learning is a a I'll sort of like half pause on that one.

s73
00:06:35.520 --> 00:06:43.480
I think, you know, some of the biggest issues that we've seen, it's it's not that a a model by itself is is not smart or whatever.

s74
00:06:43.480 --> 00:06:48.400
I mean, models have gotten incredi- incredibly capable over the last um year, let's say, right?

s75
00:06:48.400 --> 00:06:50.560
But especially over the last like 6 months or so.

s76
00:06:50.560 --> 00:07:01.240
Um but the idea of understanding, like truly understanding your environment um and the way that your services interact and where the hotspots are, keeping track of all of that sort of understanding is incredibly

s77
00:07:01.240 --> 00:07:02.040
difficult.

s78
00:07:02.040 --> 00:07:06.560
But it's incredibly important for any model to be successful at, you know, the task that it needs to do.

s79
00:07:06.560 --> 00:07:13.800
It has to have an underlying sort of learning system to be able to capture that knowledge um and that sort of understanding of how your system operates.

s80
00:07:13.800 --> 00:07:22.320
Um so we spend a lot of time thinking about how do we have systems that not just can understand your environment at any one point, but grow as as your system evolves?

s81
00:07:22.320 --> 00:07:25.240
Because again, your system is evolving faster and faster.

s82
00:07:25.240 --> 00:07:32.520
We need to keep up with learning about what's the current state, um what's the current sort of causal chains that we need to be sort of keeping keeping an eye on.

s83
00:07:32.520 --> 00:07:37.400
Um and then of course all the enterprise enterprise stuff sort of underneath.

s84
00:07:37.400 --> 00:07:41.720
Um and so this is that same view kind of uh packed out.

s85
00:07:41.720 --> 00:07:43.800
Um Come on.

s86
00:07:43.800 --> 00:07:44.320
Okay.

s87
00:07:44.320 --> 00:07:47.800
Um, but so today so we do a lot at Resolve um, the on-call and the incident stuff.

s88
00:07:47.800 --> 00:07:51.240
I'm going to focus a lot uh, more just on the background agent um, stuff.

s89
00:07:51.240 --> 00:07:54.680
So how do we deal with the things that maybe aren't sort of immediate fires.

s90
00:07:54.680 --> 00:07:59.480
Um, if you have questions about the immediate fire stuff, um, we have a booth down in the expo.

s91
00:07:59.480 --> 00:08:00.560
Please come check it out.

s92
00:08:00.560 --> 00:08:05.560
Uh, our team would love to demo to you etc. Uh, but today we're going to focus on the background agent.

s93
00:08:05.560 --> 00:08:07.360
So kind of a little pop quiz.

s94
00:08:07.360 --> 00:08:09.080
Um, feel free to raise your hands.

s95
00:08:09.080 --> 00:08:11.960
Is anybody using agents in part as part of your daily workflow?

s96
00:08:11.960 --> 00:08:13.360
Maybe outside of the coding.

s97
00:08:13.360 --> 00:08:15.600
I'm assuming everybody's doing coding agents these days.

s98
00:08:15.600 --> 00:08:20.360
Is anybody doing like actually running sort of um, agents that are sort of helping in other ways?

s99
00:08:20.360 --> 00:08:21.120
Okay.

s100
00:08:21.120 --> 00:08:21.680
Uh, decent amount.

s101
00:08:21.680 --> 00:08:22.760
Any good examples?

s102
00:08:22.760 --> 00:08:24.320
Any fun stuff that anybody has?

s103
00:08:24.320 --> 00:08:25.800
You can just yell it out.

s104
00:08:25.800 --> 00:08:27.080
Meeting review.

s105
00:08:27.080 --> 00:08:28.120
Meeting reviews?

s106
00:08:28.120 --> 00:08:28.760
Yeah, meeting reviews.

s107
00:08:28.760 --> 00:08:30.520
I just had my granola show up and

s108
00:08:30.520 --> 00:08:31.440
Market research.

s109
00:08:31.440 --> 00:08:32.320
Market research.

s110
00:08:32.320 --> 00:08:33.525
I do a lot of.

s111
00:08:33.525 --> 00:08:34.039
[laughter]

s112
00:08:34.039 --> 00:08:35.719
So let's let's have a good conversation about.

s113
00:08:35.719 --> 00:08:36.880
Yeah, yeah, yeah, I do that all the time.

s114
00:08:36.880 --> 00:08:37.520
Any other ones?

s115
00:08:37.520 --> 00:08:38.320
Maybe one more?

s116
00:08:38.320 --> 00:08:39.440
Therapy.

s117
00:08:39.440 --> 00:08:41.719
What's the one?

s118
00:08:41.880 --> 00:08:43.599
I can't hear it.

s119
00:08:43.599 --> 00:08:44.520
Therapy.

s120
00:08:44.520 --> 00:08:46.000
That's a fantastic one actually.

s121
00:08:46.000 --> 00:08:47.880
We're we we are humans here today.

s122
00:08:47.880 --> 00:08:49.440
This is very important.

s123
00:08:49.440 --> 00:08:51.600
Um, that's actually that's a very good one.

s124
00:08:51.600 --> 00:08:52.320
Um, okay.

s125
00:08:52.320 --> 00:08:55.120
So so people are having some stuff uh, going on.

s126
00:08:55.120 --> 00:08:59.680
Um, So, you know, and this is kind of recaps a little bit uh, again.

s127
00:08:59.680 --> 00:09:07.280
A lot of production work is not about is not there's not a sort of big ceremony that everyone is focused on for the the type of work that we have to do.

s128
00:09:07.280 --> 00:09:09.320
Um, on-call you you've got a page that goes off.

s129
00:09:09.320 --> 00:09:10.720
You know somebody's going to receive that.

s130
00:09:10.720 --> 00:09:13.120
Incidents you create a bridge, you invite people in.

s131
00:09:13.120 --> 00:09:13.920
That's great.

s132
00:09:13.920 --> 00:09:25.400
Um, but there's just a long tail of other things that we are accountable for that doesn't have sort of a thing that's going to show up in your sort of job description of like this is what you're going to be, you know, responsible for.

s133
00:09:25.400 --> 00:09:29.600
Um, watching deploys that go out and make sure that they're actually getting out um, healthy.

s134
00:09:29.600 --> 00:09:36.680
Um, a morning report uh, incident digest of just like what's the state of my system today so that we're all on the same page.

s135
00:09:36.680 --> 00:09:39.400
Um, hey that P99 drift kind of came back.

s136
00:09:39.400 --> 00:09:42.160
Is somebody looking at that or not?

s137
00:09:42.160 --> 00:09:46.080
And you know, this is pulling people in to to try to like figure out what's going on.

s138
00:09:46.080 --> 00:09:47.240
This may not be paging, right?

s139
00:09:47.240 --> 00:09:50.880
Because we don't we're not going to alert on everything.

s140
00:09:50.880 --> 00:09:52.960
Produce the capacity report, right?

s141
00:09:52.960 --> 00:09:54.880
Like are we tracking okay, right?

s142
00:09:54.880 --> 00:09:58.000
This is maybe a company goal this this quarter.

s143
00:09:58.000 --> 00:09:59.240
Are we tracking against that?

s144
00:09:59.240 --> 00:10:01.600
Somebody's going to have to be responsible for doing that.

s145
00:10:01.600 --> 00:10:07.720
The recurring health check and just kind of checking and making sure things are kind of running okay and not waiting for a customer to come complain first.

s146
00:10:07.720 --> 00:10:13.920
So this work doesn't have like an obvious like, oh, this you know, this now needs to go be done.

s147
00:10:13.920 --> 00:10:17.160
But it's work that we end up having to do.

s148
00:10:17.440 --> 00:10:18.600
So what is a task?

s149
00:10:18.600 --> 00:10:24.360
A task is just execution and the context to understand how to actually execute the task.

s150
00:10:24.360 --> 00:10:28.080
Execution is very very important.

s151
00:10:28.080 --> 00:10:32.320
It's understanding what to do and being able to execute that.

s152
00:10:32.320 --> 00:10:35.240
Maybe having access to the tools, etc., right?

s153
00:10:35.240 --> 00:10:37.120
Obviously very important to do.

s154
00:10:37.120 --> 00:10:43.160
But we think the production context is just way more important because it's one thing to go check a dashboard.

s155
00:10:43.160 --> 00:10:46.600
It's another thing to say that metric smells off.

s156
00:10:46.600 --> 00:10:49.360
And the execution is can load the dashboard.

s157
00:10:49.360 --> 00:10:53.240
It's the production context that's going to say, this feels wrong.

s158
00:10:53.240 --> 00:10:55.960
And I don't know if I can even explain why it feels wrong.

s159
00:10:55.960 --> 00:11:00.880
It just feels wrong and I want to dig into the next layer of sort of understanding of that.

s160
00:11:00.880 --> 00:11:06.200
And so really if if we start talking about background agents and being able to kind of perform task, you need both of these.

s161
00:11:06.200 --> 00:11:14.320
You need the execution engine, that's great, but you really need that production context that tells you is this important or not important.

s162
00:11:15.640 --> 00:11:22.120
So every background agent, you know, there's a a few different principles that we like to think about with our background agents.

s163
00:11:22.120 --> 00:11:22.680
When does it work?

s164
00:11:22.680 --> 00:11:23.560
How does it work?

s165
00:11:23.560 --> 00:11:25.320
How does it know what to go do?

s166
00:11:25.320 --> 00:11:27.120
When does the agent work?

s167
00:11:27.120 --> 00:11:28.440
It can work in a bunch of different ways.

s168
00:11:28.440 --> 00:11:29.480
It can just do it on a schedule.

s169
00:11:29.480 --> 00:11:34.800
Maybe this is the the report, etc. Just kind of do some summarization for me kind of on an ongoing basis.

s170
00:11:34.800 --> 00:11:36.280
Maybe it's a weekly event, right?

s171
00:11:36.280 --> 00:11:48.640
We do an on-call handover uh every Thursday, and so that a lot of um the work that our agent does is sort of prepare like what what are the kind of interesting trends from the last week that the next on-caller needs to sort of understand as they pick up the rotation.

s172
00:11:48.640 --> 00:11:55.240
Um, event streams, so you know, there's lots of systems that will sort of push events as kind of key things happen.

s173
00:11:55.240 --> 00:11:58.680
Um, so deployments go through a CI ICD pipeline.

s174
00:11:58.680 --> 00:12:01.240
Um, there's other sort of uh Slack-based, right?

s175
00:12:01.240 --> 00:12:10.160
We get a lot of uh Slack things messages coming through, um etc. Um, and these are things that we can sort of pick up and trigger and say, "Oh, if this event happens,

s176
00:12:10.160 --> 00:12:13.160
um let me sort of understand what that event is and go do some work."

s177
00:12:13.160 --> 00:12:16.080
Um, and then message-based, so I can just tell it, "Hey, go do some work."

s178
00:12:16.080 --> 00:12:16.920
And it will go do some work.

s179
00:12:16.920 --> 00:12:19.080
That's fantastic.

s180
00:12:19.080 --> 00:12:20.560
Um, how does it run?

s181
00:12:20.560 --> 00:12:21.280
Always runs.

s182
00:12:21.280 --> 00:12:22.120
It's in the cloud.

s183
00:12:22.120 --> 00:12:24.480
Um, so if you close your laptop, it's okay.

s184
00:12:24.480 --> 00:12:27.680
Um, runs inside of a sandbox, so it has kind of a file system underneath it.

s185
00:12:27.680 --> 00:12:33.400
Um, this allows it to sort of self-organize a lot of its work, etc. as it's doing uh doing things.

s186
00:12:33.400 --> 00:12:36.200
Um, and then obviously back to the learning loop, right?

s187
00:12:36.200 --> 00:12:48.200
So, that idea of knowledge and sort of a memory system underneath that um to really understand your systems and as it's doing a task, able to sort of reflect on that task and uh you know, do a better job next time.

s188
00:12:48.200 --> 00:12:52.320
Or the things that it learned from one task, it can sort of apply into a different task.

s189
00:12:52.320 --> 00:12:59.480
Um, because again, this this sort of shared uh sort of knowledge system um works across all the different tasks that we have.

s190
00:12:59.480 --> 00:13:02.920
Um, so how does the agent know what to do?

s191
00:13:02.920 --> 00:13:04.760
Um, it has a task system.

s192
00:13:04.760 --> 00:13:07.839
It can pull in all the skills that you have in other systems, that's fine.

s193
00:13:07.839 --> 00:13:08.880
You can connect those.

s194
00:13:08.880 --> 00:13:11.880
Um, and it's got obviously the integrations that it's going to plug into.

s195
00:13:11.880 --> 00:13:16.600
Um, so let's talk a little bit about what types of things you uh can hand over.

s196
00:13:16.600 --> 00:13:26.640
Um, we've got four sort of workloads that we're going to talk about, but if you think about the previous couple slides, these are sort of very basic primitives that we've built into the system.

s197
00:13:26.640 --> 00:13:28.120
You can get very creative.

s198
00:13:28.120 --> 00:13:36.000
We've We have a number of people inside of Resolve that have gotten very creative with the type of sort of background activities that things that that they that they have.

s199
00:13:36.000 --> 00:13:44.400
So, I want you to use these as kind of These are things we've seen be very successful inside of Resolve, but also with you know, a number of our customers.

s200
00:13:44.400 --> 00:13:47.680
But, you know, sky's the limit and and you can get really creative.

s201
00:13:47.680 --> 00:13:48.600
So, deployment monitoring.

s202
00:13:48.600 --> 00:13:55.480
So, this is a big one, you know, any change inside of your environment is an opportunity for something to go wrong.

s203
00:13:55.480 --> 00:14:07.120
Um and so, you know, having an agent that's able to watch as all these change events come in just to do a sanity check of is everything stable is incredibly incredibly important.

s204
00:14:07.120 --> 00:14:11.000
And, you know, a lot of people have decent CICD system.

s205
00:14:11.000 --> 00:14:15.800
I mean, this is like tried and true stuff that we've had as an industry for quite a while.

s206
00:14:15.800 --> 00:14:20.360
But, we we noticed a couple gaps, you know, from in most of our customers.

s207
00:14:20.360 --> 00:14:23.440
You know, typically the checks that it does are good.

s208
00:14:23.440 --> 00:14:31.760
They're good baselines, but it's not exhaustive based on the type of changes that are going in etc. There's certain signals you'd want to watch or not want to watch.

s209
00:14:31.760 --> 00:14:35.200
And so, every rollout is a bit unique.

s210
00:14:35.200 --> 00:14:46.000
Often times you have change systems that you're not piping through a CICD system like a feature flag or maybe some infra changes that might happen which maybe don't get any monitoring at all.

s211
00:14:46.000 --> 00:14:51.720
And you're sort of just trusting that an alert might fire and an on-caller will wake up and say, "Who changed what?"

s212
00:14:51.720 --> 00:14:53.440
right?

s213
00:14:53.440 --> 00:15:01.040
Deployment monitoring is is actually a really big use case that we suggest people sort of go through and I'll show some examples of that in a second.

s214
00:15:01.040 --> 00:15:02.280
Schedule health and anomaly checks.

s215
00:15:02.280 --> 00:15:07.680
So, this is just sort of the ongoing periodic checking of some of your systems.

s216
00:15:07.680 --> 00:15:21.640
And, you know, this is maybe something where it's like go check, you know, sort of my general dashboards on a routine basis maybe every morning just kind of do a casual check just to make sure there's nothing kind of weird from last night that I might need to be be of.

s217
00:15:21.640 --> 00:15:24.320
But, uh, this can also just be sort of a time-based thing.

s218
00:15:24.320 --> 00:15:27.000
Like I made a change in part of our system.

s219
00:15:27.000 --> 00:15:31.520
I'm worried about this, you know, uh, you know, third-party service that I'm kind of interacting with.

s220
00:15:31.520 --> 00:15:40.400
Let me just kind of set an agent to kind of watch that maybe for the next week just to make sure everything is kind of stable and then that agent can sort of, um, you know, stop his job.

s221
00:15:40.400 --> 00:15:43.160
Uh, operational reports and handoffs, I talked a little bit about this.

s222
00:15:43.160 --> 00:15:49.800
Uh, these are the sort of ceremonial things that we might want to do just to, you know, spread information, summarize things, kind of bring things to the fore.

s223
00:15:49.800 --> 00:15:52.040
Um, and then a first responder to engineering questions.

s224
00:15:52.040 --> 00:16:01.080
And this one's kind of fun a little bit, um, because the trigger for this is actually just a Slack message and I will say, um, one of my biggest,

s225
00:16:01.080 --> 00:16:09.200
like, let's call it responsibilities, uh, as an engineer is watching all my Slack channels and trying to make sure everyone's kind of happy.

s226
00:16:09.200 --> 00:16:12.640
Um, and like that nobody has any burning questions or anything like that.

s227
00:16:12.640 --> 00:16:23.840
Um, and so I can be sort of heads down trying to build something, um, and then, you know, the, you know, uh, eventual sort of Slack notification comes in that like this channel somebody asked this sort of kind of important question,

s228
00:16:23.840 --> 00:16:28.800
um, and I just need to jump in there and and try to provide context, etc. It's not hard work.

s229
00:16:28.800 --> 00:16:36.200
It's not hard for me to go answer questions, but it's disrupting me and if I don't go answer it, um, they won't get an answer for a while.

s230
00:16:36.200 --> 00:16:40.880
And what we found is like our agent actually has access to a lot of information that people ask questions about.

s231
00:16:40.880 --> 00:16:42.520
At least this is true internally.

s232
00:16:42.520 --> 00:16:51.680
Um, so we actually have an agent that can watch all of these sort of critical channels, um, and determine whether it has enough sort of confidence to answer the question or not.

s233
00:16:51.680 --> 00:16:57.000
Um, and one of the fun things is like it, uh, the our agents have access to like Slack DMs and things like that.

s234
00:16:57.000 --> 00:17:02.120
Um, and so you can have an agent that basically will DM you to say, "I think I know the answer to this, but I'm not sure.

s235
00:17:02.120 --> 00:17:05.079
Can you confirm this for me before I, you know, respond back?"

s236
00:17:05.079 --> 00:17:11.079
Um, so this kind of emergent behaviors gets kind of fun and interesting as you just kind of build these things out.

s237
00:17:11.079 --> 00:17:15.400
Okay, so I'm going to flip over and hope all of this works.

s238
00:17:16.079 --> 00:17:21.120
Um, Cool.

s239
00:17:21.120 --> 00:17:26.360
Um So, uh let's see if I can find the one that I wanted to show.

s240
00:17:26.360 --> 00:17:31.280
So, this is our uh sort of demo application running in our sort of demo sort of Slack environment.

s241
00:17:31.280 --> 00:17:38.560
Um and uh what I wanted to show off was some of our deployment stuff and talk a little bit more about um what's kind of going on under the hood.

s242
00:17:38.560 --> 00:17:43.120
Um so, this is, you know, sort of fake environment um just to kind of showcase some things.

s243
00:17:43.120 --> 00:17:51.000
Um so, here um anytime somebody posts a sort of GitHub tag, um our agent's going to sort of see that and say, "Oh, I I should That's a release.

s244
00:17:51.000 --> 00:17:51.600
Is that a release?

s245
00:17:51.600 --> 00:17:52.960
Yes, that is a release.

s246
00:17:52.960 --> 00:17:54.280
Um let me go watch that."

s247
00:17:54.280 --> 00:17:56.040
Um but, it's not just going to watch it.

s248
00:17:56.040 --> 00:17:58.600
Um it's going to do something that's slightly more intelligent.

s249
00:17:58.600 --> 00:18:01.360
Uh cuz like I said before, everybody kind of has a CI/CD system.

s250
00:18:01.360 --> 00:18:04.680
It will do the sort of standard checks on on, you know, certain KPIs.

s251
00:18:04.680 --> 00:18:18.680
Um but, what the agent is able to do is actually look at the changes that are going in, understand what telemetry might help us evaluate whether those changes are, you know, good or not good or like are putting the system in an abnormal state,

s252
00:18:18.680 --> 00:18:25.080
um and build a sort of customized plan that it's going to check uh just for this specific release.

s253
00:18:25.080 --> 00:18:32.000
And this is why I kind of go back to like, you know, our goal is not to sit here and say, "We're going to replace an entire CI/CD pipeline.

s254
00:18:32.000 --> 00:18:34.160
You've spent time organizing that."

s255
00:18:34.160 --> 00:18:40.120
But, this can sort of patch a lot of, you know, uh parts of your system that may not be as robust as they should be.

s256
00:18:40.120 --> 00:18:46.640
And it would be great if you had a single engineer just focused on like watching all the things on every release, but that's really expensive.

s257
00:18:46.640 --> 00:18:48.080
There's a lot of cognitive load.

s258
00:18:48.080 --> 00:18:49.880
You'd rather have them doing other things.

s259
00:18:49.880 --> 00:18:56.640
Um so, now the agent can come and actually do a lot of that sort of dynamic understanding of this is the change, um so, I'm going to sort of check for these things.

s260
00:18:56.640 --> 00:19:02.840
And so, here, um the checkout replaces currency service, um you know, we're monitoring the checkout latency and the error rates.

s261
00:19:02.840 --> 00:19:05.560
Well, let's take a look at the Kafka pipeline cuz that's sort of involved.

s262
00:19:05.560 --> 00:19:10.280
This is the sort of causal chain I want to sort of say, I want to make sure is is healthy.

s263
00:19:10.280 --> 00:19:15.560
And it'll check that, and it can check it not just once, um but sort of on an ongoing basis.

s264
00:19:15.560 --> 00:19:17.360
And you know, none of this is hard-coded in.

s265
00:19:17.360 --> 00:19:22.240
It's not like, oh, let's just wait for 15 minutes and then try this again, and then we'll be done.

s266
00:19:22.240 --> 00:19:35.000
Again, the agent has a bit more autonomy, and and you get to guide it a bit on how you how much autonomy you want it to have, but it could decide, I want to wait for another hour cuz this type of issue might only hit every you know, every so often, so I really want to spend a

s267
00:19:35.000 --> 00:19:36.560
little bit more time focused on this.

s268
00:19:36.560 --> 00:19:40.560
Maybe I'll come back in 3 days and say, is this deploy still kind of healthy?

s269
00:19:40.560 --> 00:19:45.560
Are we seeing the the change in the effect that I expected to see out of this?

s270
00:19:45.560 --> 00:19:54.120
So this is a type of thing that we can bring, and and this again works for feature flags, infra changes, sort of any sort of eventing system that you can think of.

s271
00:19:54.120 --> 00:19:59.560
The sort of on-call handoff reports, let's scroll up just a little bit.

s272
00:19:59.560 --> 00:20:09.440
Um So this is just a summarization of all of the work that was done, you know, over the last day that the agent is kind of just summarizing up.

s273
00:20:09.440 --> 00:20:20.960
And this one's a little bit verbose, but you can see a bunch of different sort of investigation summaries that we did, some notable changes, etc. Work work completed.

s274
00:20:20.960 --> 00:20:23.320
Um So there's a critical open.

s275
00:20:23.320 --> 00:20:26.720
You know, I guess that's the on-call handoff.

s276
00:20:26.720 --> 00:20:30.680
But one of the nice things I don't know if I'll be able to watch it go all the way.

s277
00:20:30.680 --> 00:20:36.360
But you can always just come back in this thread and, you know, this is too verbose.

s278
00:20:36.360 --> 00:20:37.360
Verbose.

s279
00:20:37.360 --> 00:20:38.760
Make it shorter.

s280
00:20:38.760 --> 00:20:43.480
Um and I'm not going to be able to unfortunately no time to watch this actually go, but this works.

s281
00:20:43.480 --> 00:20:53.560
The the agent is able to update its task underneath, and able to sort of give you the answer like update so that the next time it fires, it's not going to

s282
00:20:53.560 --> 00:20:55.360
uh be as verbose.

s283
00:20:55.360 --> 00:20:59.920
And I can tell it explicitly what I want, etc. I was just kind of giving an example.

s284
00:20:59.920 --> 00:21:01.560
This is kind of the more fun one.

s285
00:21:01.560 --> 00:21:05.000
So, you know, here I'm just like posting different problems.

s286
00:21:05.000 --> 00:21:08.480
Um, I'm not having to know that resolve exists.

s287
00:21:08.480 --> 00:21:10.960
Um, I don't have to like at mention resolve, whatever.

s288
00:21:10.960 --> 00:21:13.880
Um, I've set up this agent to sort of passively watch this channel.

s289
00:21:13.880 --> 00:21:21.200
Um, if you see something that you think you have an answer for that somebody's kind of, you know, digging into, um, go ahead and respond.

s290
00:21:21.200 --> 00:21:22.040
Um, otherwise don't.

s291
00:21:22.040 --> 00:21:27.040
So, you know, here's a message that I posted um, that it's decided I don't need to respond to this.

s292
00:21:27.040 --> 00:21:32.560
Um, so again, very kind of flexible system um, that can kind of adapt to a bunch of different things.

s293
00:21:34.880 --> 00:21:36.600
Oh, where are my slides?

s294
00:21:36.600 --> 00:21:38.600
Oh.

s295
00:21:40.720 --> 00:21:43.520
We have uh, in the UI there's a bunch of stuff that you can do.

s296
00:21:43.520 --> 00:21:50.160
You know, you can always go down and inspect all the different tasks that you have and and view their reports, view previous runs.

s297
00:21:50.160 --> 00:21:55.080
You can see all the work that the that the agent has done underneath um, to sort of accomplish that task.

s298
00:21:55.080 --> 00:22:00.160
So, you get a lot of visibility into what the agent is doing, but we think the surface area being where you live, right?

s299
00:22:00.160 --> 00:22:08.360
So, Slack is this like kind of um, or or MS Teams if you're on MS Teams, as this kind of first party experience to sort of integrate the the agent into

s300
00:22:08.360 --> 00:22:10.160
um, is incredibly important.

s301
00:22:10.160 --> 00:22:14.160
Um, and so, you know, how would you get this stuff uh, sort of set up?

s302
00:22:14.160 --> 00:22:18.200
Um, it's really just through talking with the agent.

s303
00:22:18.200 --> 00:22:23.480
Um, and so, here this is me sort of saying, "Hey, I want to do a new recurring health summary for my team."

s304
00:22:23.480 --> 00:22:32.240
Um, so the agent's going to uh, take a look at my environment, it's going to explore my environment a little bit um, and eventually likely come back and ask me a couple questions about

s305
00:22:32.240 --> 00:22:43.560
um, what I want to see, what kind of reports do I want, how how verbose do I want it, um, etc. Uh, the agent's going to go ahead and do all that and it's going to set up that sort of initial thing for me so that I can test it out, make sure it's working, um, and then

s306
00:22:43.560 --> 00:22:47.480
share it with the rest of uh, with the rest of my team.

s307
00:22:48.320 --> 00:22:53.760
In the interest of time, I don't think we'll get to this, but come by the booth and you can see more.

s308
00:22:53.760 --> 00:22:58.080
Um, cool.

s309
00:23:01.120 --> 00:23:03.160
Um, so that that's background agents.

s310
00:23:03.160 --> 00:23:06.880
And again, I I sort of lean back on be creative, right?

s311
00:23:06.880 --> 00:23:13.720
Like everyone has unique work, I mean as as a company we believe that every, you know, every company is a unique place.

s312
00:23:13.720 --> 00:23:27.280
That's why we spend so much time on our knowledge system, etc. Um, truly understanding your, you know, what your environment looks like, what your needs are, etc. Um, that begins to kind of tell you where the where the biggest benefit from having these agents begin to pick up work um, would be.

s313
00:23:27.280 --> 00:23:38.240
Um, also just to call out, if you have an agent harness, if you, you know, internally if you're building your own, um, everything that I showed is accessible uh, through kind of MCP servers, etc.

s314
00:23:38.240 --> 00:23:48.360
Um, so you can really graft resolve into kind of any um, system that you have um, as just kind of an extension of learning to kind of do deeper work or to sort of pull production context

s315
00:23:48.360 --> 00:23:52.240
uh, a bit more efficiently or you know, even augmented with all that learning stuff that we've done.

s316
00:23:52.240 --> 00:23:56.080
Um, and then obviously bring your own skills um, along for the ride.

s317
00:23:56.080 --> 00:23:57.840
Um, don't go duplicate a bunch of stuff.

s318
00:23:57.840 --> 00:24:08.000
Um, so really the biggest things to take away, um, cost of operational work, it's not navigating uh, you know, it's it's it's not just in the task execution, it's in the environment complexity,

s319
00:24:08.000 --> 00:24:08.440
right?

s320
00:24:08.440 --> 00:24:11.000
Um, that's where the biggest issue is going to happen.

s321
00:24:11.000 --> 00:24:17.960
Um, background agents, they run on schedules, they run on triggers, they're very composable, you can sort of graft them into lots of different use cases.

s322
00:24:17.960 --> 00:24:20.280
Um, it's fun to see people explore that.

s323
00:24:20.280 --> 00:24:22.840
Um, and uh, yeah.

s324
00:24:22.840 --> 00:24:31.080
Um, you open resolve just to kind of see what the top findings are, etc. Um, but you know, ideally a lot of your interaction is kind of in the places that you're already kind of doing work.

s325
00:24:31.080 --> 00:24:36.680
Um, so um, if you have any questions, uh, you can find me down at the booth or you know, just meet me out in the hall.

s326
00:24:36.680 --> 00:24:37.960
Um, but thanks for coming.

s327
00:24:37.960 --> 00:24:39.007
Appreciate it.

s328
00:24:39.007 --> 00:24:41.007
[applause]

s329
00:24:53.417 --> 00:24:55.417
[music]