WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=ImnZ7s68jw8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/ImnZ7s68jw8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.760 --> 00:00:13.360
Awesome.

s3
00:00:13.360 --> 00:00:15.880
Well, welcome to AI engineer.

s4
00:00:15.880 --> 00:00:24.080
Um uh thought given where at AI engineers New York, we'd kind of revisit the question of what does an AI engineer do?

s5
00:00:24.080 --> 00:00:31.320
Feel like every year we have an idea of what an AI engineer does and that definition kind of changes.

s6
00:00:31.320 --> 00:00:41.360
Um And as I was putting this talk together, it was super timely because Karpathy actually just tweeted something this weekend where he kind of he kind of talks about this growing

s7
00:00:41.360 --> 00:00:57.040
gap between basically folks at the frontier who are running thousands of agents and swarms of always-on agents and he really kind of drops this funnel where 6 billion plus users have maybe touched or interfaced

s8
00:00:57.040 --> 00:00:58.520
with an LLM.

s9
00:00:58.520 --> 00:01:02.320
1 to 2 billion or maybe like casual free tier users.

s10
00:01:02.320 --> 00:01:11.280
20 million are professionals who have maybe started using it for more taking some piece of work that would have taken weeks down to to days.

s11
00:01:11.280 --> 00:01:16.640
But really those at the frontier labs are which is a super small percentage.

s12
00:01:16.640 --> 00:01:19.040
It's like 5,000 users.

s13
00:01:19.040 --> 00:01:25.640
Uh are the ones who are running thousands and thousands of agents at once.

s14
00:01:25.640 --> 00:01:30.160
And as I look at this, it really is I mean he's kind of dropped a bit of the funnel.

s15
00:01:30.160 --> 00:01:33.520
I mean the first question anyone looks at this is that where am I on this?

s16
00:01:33.520 --> 00:01:42.680
And you know, he's kind of drawn out a bit of a road map of how does everyone get to in the next couple of years one of those 5,000

s17
00:01:42.680 --> 00:01:45.840
users who are running thousands of agents.

s18
00:01:45.840 --> 00:01:52.280
And this has been happening I'd say for for a couple of decades now where the abstraction layer has gotten higher and higher.

s19
00:01:52.280 --> 00:02:07.440
I mean definitely faster in the last couple of years, but we've gone from engineers writing higher level abstraction languages to I mean just copy-pasting code, using local agents to using

s20
00:02:07.440 --> 00:02:18.861
cloud agents to really the expectation now is how how do all of us start running a team of agents or factories of agents internally and build toward that.

s21
00:02:18.861 --> 00:02:18.880
[snorts]

s22
00:02:18.880 --> 00:02:24.000
And what this has led to is essentially what is a then and now of what an AI engineer is.

s23
00:02:24.000 --> 00:02:34.600
And this is is this is a bit uh I I think of a of a roadmap for some, but I think internally this is what we've already started to kind of push towards.

s24
00:02:34.600 --> 00:02:42.280
As an AI engineer, your job's no longer to just write the feature, write an AI feature, it's about how do you build the loop to write that feature.

s25
00:02:42.280 --> 00:02:53.800
It's no longer about you know, just reading the diff and actually doing code review, it's actually how do you set up systems and agents to do the code reviews, and how do you write evals to actually do

s26
00:02:53.800 --> 00:02:58.640
the non-deterministic um to review those non-deterministic tasks.

s27
00:02:58.640 --> 00:03:07.920
It's actually no longer even about just setting up metrics and and software kind of telemetry, but it's actually how do you have agents go through and read the observability

s28
00:03:07.920 --> 00:03:13.760
data and have agents actually consume that telemetry to find patterns.

s29
00:03:13.760 --> 00:03:13.900
And so,

s30
00:03:13.900 --> 00:03:14.440
[snorts]

s31
00:03:14.440 --> 00:03:25.200
this loop that we've kind of been talking about, which is the build, the eval, ship, you monitor, you improve it, well, the AI engineer is now actually responsible not just for doing each of those steps,

s32
00:03:25.200 --> 00:03:31.160
but how do you build the loop that does the that that actually does each of those steps.

s33
00:03:31.160 --> 00:03:38.560
And at Arize, one of the things we've rolled out in our product, and I'm going to talk about a bit of what we've learned in building this, is we built signal

s34
00:03:38.560 --> 00:03:41.120
to actually run this loop for AI engineers.

s35
00:03:41.120 --> 00:03:53.440
What signal does is that signal actually looks at all of the traces, it reads the telemetry, it identifies patterns in that telemetry, and it can actually put up a fix or a PR

s36
00:03:53.440 --> 00:03:56.120
if it has access to the repo.

s37
00:03:56.120 --> 00:03:59.920
And signal signal's an always-on agent.

s38
00:03:59.920 --> 00:04:06.520
So, this is an example of how signal starts, but it actually starts off by just reading the traces.

s39
00:04:06.520 --> 00:04:16.680
Super simple example, but it goes through, reads um and all of this configurable, but and certain volume of traces that came in in some cadence.

s40
00:04:16.680 --> 00:04:22.720
It reads it and then the goal is to actually identify issues from that telemetry data.

s41
00:04:22.720 --> 00:04:25.840
It has to keep track of issues that it's previously seen.

s42
00:04:25.840 --> 00:04:32.200
So, we have to consider the memory of did see something before that it thought was an issue?

s43
00:04:32.200 --> 00:04:40.680
Did something seem like an issue, but it wasn't actually captured and now as it collects more data, it actually thinks it's much more of a critical critical issue.

s44
00:04:40.680 --> 00:04:46.000
And so, signal actually starts off by reading the traces to identify problems.

s45
00:04:46.000 --> 00:04:55.400
And so, one of the first things that we realized was that the future of who's actually consuming telemetry data has really changed.

s46
00:04:55.400 --> 00:05:00.000
It's no longer about humans consuming telemetry data.

s47
00:05:00.000 --> 00:05:11.040
It's not even about humans consuming it and agents kind of taking on the task, but it really is how do you set up your telemetry data so that agents are the primary consumers of that data.

s48
00:05:11.040 --> 00:05:19.760
And what that does is that that observability data can become the trigger for actually running and triggering these always-on agents.

s49
00:05:19.760 --> 00:05:29.400
And so, the first thing that if you're building a team of of you know, a factory of really always-on agents is what triggers these agents to wake up or start.

s50
00:05:29.400 --> 00:05:31.880
And there can be a variety of triggers.

s51
00:05:31.880 --> 00:05:36.800
For signal, for us it's you know, it's configurable to run on an ongoing cadence.

s52
00:05:36.800 --> 00:05:45.040
So, every 3 hours, 6 hours, go pull the last, you know, X amount of traces, read them, identify problems.

s53
00:05:45.040 --> 00:05:48.200
It can also be triggered by events.

s54
00:05:48.200 --> 00:06:01.160
So, Slack messages, um if some user complains about something, it can be a metric-based trigger, but these are essentially triggers to kick up the agent actually starting.

s55
00:06:01.160 --> 00:06:07.360
Once the agent actually starts, we have to spend some time figuring out, well, what is what is an agent run actually look like?

s56
00:06:07.360 --> 00:06:23.800
For us, we actually pull in all of the context of the traces, um any additional third-party data, so we allow collecting to APM logs, APM metrics, um but all of the third-party telemetry data that we want to pull in, um

s57
00:06:23.800 --> 00:06:37.040
access to the repo, um access to the actual um the the access to any keys that are needed to actually run run this agent, but all of this information

s58
00:06:37.040 --> 00:06:44.919
is um put inside of a sandbox, and this is actually what's started up every time the agent wakes up.

s59
00:06:44.919 --> 00:06:55.320
And so, the agent's triggered, we build a sandbox, we put everything that we need inside of that sandbox, there's a proxy, the only way to get information in and out of the sandbox is by this single proxy,

s60
00:06:55.320 --> 00:07:07.720
and everything that happens inside of that sandbox is actually already getting traced, and every output, so the issues that are generated and created are all getting saved to some kind of memory.

s61
00:07:07.720 --> 00:07:11.520
And so, this happens every single time the agent actually runs.

s62
00:07:11.520 --> 00:07:24.360
Every time the agent finishes running, so it swept through all of the traces, um it either found something or it didn't, there's what actually stays and what actually gets thrown away.

s63
00:07:24.360 --> 00:07:34.640
What stays is that if it found any issues, all of the output is actually saved off in some kind of memory, so that the next time an agent gets spun up, it doesn't have to start from ground zero.

s64
00:07:34.640 --> 00:07:38.040
It has a history of what are the issues that it actually found.

s65
00:07:38.040 --> 00:07:49.680
Um the agents are traced, so all of the telemetry data of what the agent called, um you know, if it actually put up a PR, all of those things are actually traced.

s66
00:07:49.680 --> 00:07:56.480
And any skills that it may have learned while actually doing a run can also get saved to the actual repo.

s67
00:07:56.480 --> 00:08:13.080
But, the things that go away is the sandbox gets torn down, any credentials, um you know, obviously get get removed, um the repo access is removed, and so all of the stuff that doesn't need to be saved kind of gets torn down after the run of signal.

s68
00:08:13.120 --> 00:08:18.120
And we've designed signal in a way to actually keep this agent configure durable.

s69
00:08:18.120 --> 00:08:29.040
So, that means that either if the if the sandbox fails, or if the harness fails, then it shouldn't have to start everything from scratch in the middle of the run.

s70
00:08:29.040 --> 00:08:41.039
It should actually just be able to pick up from the step that it failed, and all of the state, so-called, is kind of saved off in kind of a a memory outside of the sandbox.

s71
00:08:41.039 --> 00:08:48.920
And so, we've designed this to actually be a very durable agent, so that if it fails, it can spin it back up.

s72
00:08:48.920 --> 00:08:54.360
Um you can launch a new sandbox, launch a new harness to kind of pick back up where it started.

s73
00:08:54.360 --> 00:09:00.520
So, all of this is essentially the underpinnings of how we built signal to run every single run.

s74
00:09:00.520 --> 00:09:10.760
Agent gets triggered, agent gets started, loaded with all the contexts, the evidence, uh it sweeps through the telemetry data, it either finds something or it doesn't, all of that gets saved to memory.

s75
00:09:10.760 --> 00:09:14.200
If something fails in the middle, the agent can get reset up.

s76
00:09:14.200 --> 00:09:19.960
And we built signal to also be you know, agnostic of the model or the harness.

s77
00:09:19.960 --> 00:09:26.680
So, today signal supports not just Cloud Code, which is what we started with, but it also supports Cursor and Codex.

s78
00:09:26.680 --> 00:09:31.480
And the goal is to actually support all of the the major, even the open source harnesses out there.

s79
00:09:31.480 --> 00:09:39.640
Um and signal actually also supports um multiple uh it will support kind of multiple models.

s80
00:09:39.640 --> 00:09:45.560
So, even if you're using Claude code, uh you can pick any of the models inside the Anthropic family.

s81
00:09:45.560 --> 00:09:55.960
Um and this is really important to us because one, it's an awesome way for us to evaluate the quality of the agent um every time we spin it up.

s82
00:09:55.960 --> 00:10:10.680
It helps us control costs, so we can see, you know, as we use more powerful models, the better quality maybe the PRs are, but can we reduce the quality of the the you know, can we reduce the cost and actually still maintain the rigor

s83
00:10:10.680 --> 00:10:15.160
um and maintain the quality of issues that are actually detected by Signal.

s84
00:10:15.160 --> 00:10:21.720
So, this is something we deeply thought about from from first principles when we were designing Signal.

s85
00:10:21.720 --> 00:10:26.120
And this is actually a pretty good segue into what does an AI engineer need to do?

s86
00:10:26.120 --> 00:10:30.400
Well, an AI engineer really needs to kind of learn how to write evals.

s87
00:10:30.400 --> 00:10:35.240
And so, we actually every time Signal runs, Signal finds an issue, it puts up a PR.

s88
00:10:35.240 --> 00:10:40.880
Well, there's evals that are run on top of the agent to test that PR.

s89
00:10:40.880 --> 00:10:47.120
Um and an eval can be as simple as um you know, testing something as part of your development workflow.

s90
00:10:47.120 --> 00:10:55.320
You have a dataset, you can be a simple prompt um it with with outputs that you're testing to make sure that there's no regressions.

s91
00:10:55.320 --> 00:11:02.200
Um but these evals can also they need to be kind of aligned with human judgment.

s92
00:11:02.200 --> 00:11:11.200
And I'd argue that this is actually probably where an AI engineer should be spending a lot of their time as they're building the system to actually run the loop.

s93
00:11:11.200 --> 00:11:18.760
They should be actually spending time looking at the traces with their agents, identifying what the common errors are.

s94
00:11:18.760 --> 00:11:23.880
Um and there's a lot of skills I think I was hearing Lance yesterday talk about the Claude skill to build evals.

s95
00:11:23.880 --> 00:11:33.600
We have a skill with the arise um evaluator to actually help you build and write evals as well as align those evals with the human judgment.

s96
00:11:33.600 --> 00:11:42.840
But, this is probably where the shift to defining what is good becomes really, really important for an AI engineer to actually do.

s97
00:11:42.840 --> 00:11:49.280
And for a lot of teams I I think we speak with and we work with, their question becomes, "Well, how do I trust the evaluator?"

s98
00:11:49.280 --> 00:12:01.360
And do I do I really trust that the eval is um you know, is going to capture you know, how how do I trust that it's as good as my judgment?

s99
00:12:01.360 --> 00:12:08.760
And I'd say what we spend a lot of our own time doing for our own products is that we actually evaluate the evaluators.

s100
00:12:08.760 --> 00:12:16.320
Um we spend a decent amount of time on actually building these data sets, golden truth data sets.

s101
00:12:16.320 --> 00:12:30.040
We run e-vals, verify them against the human output, and then try to align to make sure that the judge actually agrees with the human um and and they're in agreement before we actually ship out

s102
00:12:30.040 --> 00:12:31.400
uh a new evaluator.

s103
00:12:31.400 --> 00:12:38.280
And these e-vals can then be used both in an offline testing as well as in an online testing manner.

s104
00:12:38.280 --> 00:12:47.080
Um and this this is probably what I would say as an AI engineer is more and more of what the job is becoming.

s105
00:12:47.080 --> 00:12:56.839
Um and so, what I'm going to leave you guys with is as an AI engineer, you've built this agent, you've built this agent to actually run, be always on,

s106
00:12:56.839 --> 00:13:01.839
you run e-vals before you actually ship it into production.

s107
00:13:01.839 --> 00:13:07.880
You can ship it, and then once it goes into production, the goal is to actually trace everything.

s108
00:13:07.880 --> 00:13:16.000
And tracing everything now means not just tracing the agents, but also tracing the coding agents behind the agents that built this.

s109
00:13:16.000 --> 00:13:31.080
So, you can trace um cloud code, you can trace cursor, you can trace codex, but all these coding agents can be traced so you can understand what happened um in the factory of agents that you all are building.

s110
00:13:31.080 --> 00:13:35.720
And then these agents that all the traces can then also be evaluated online.

s111
00:13:35.720 --> 00:13:48.960
So, there's an offline evaluation when you're actually testing what you're what a new change is, but as it goes into production, these become your online evals that you're running on top of the telemetry data.

s112
00:13:48.960 --> 00:14:01.400
This for us ends up becoming the new data that then gets pulled back into the agents to read the telemetry data, the evals, and surface up kind of problems

s113
00:14:01.400 --> 00:14:02.560
inside of the traces.

s114
00:14:02.560 --> 00:14:16.080
So, the the system really becomes how do you go from collect data, eval that data, run agents on top of it so that you can find problems and figure out where the gaps are to actually improve

s115
00:14:16.080 --> 00:14:18.040
improve your systems.

s116
00:14:18.040 --> 00:14:24.520
So, if I go back to kind of the question that we started this whole thing with, it's what does an AI engineer do?

s117
00:14:24.520 --> 00:14:32.240
I mean, the gap between I think there's a pretty big gap between what's happening in the labs today and what's happening with a lot of businesses.

s118
00:14:32.240 --> 00:14:40.960
And some of it could just be the data inside of businesses are a lot more messy than what is very constrained and testable inside of a lab.

s119
00:14:40.960 --> 00:14:54.480
But, AI engineers are kind of the the role to close that gap, and I would think it really starts with tracing everything, tracing the agents you're actually building, running evals, so being able to learn how to test these

s120
00:14:54.480 --> 00:15:07.680
test these products that we're really building, and then how do you build a system so that agents can actually improve from the telemetry that's actually being collected so that you're no longer just building individual features,

s121
00:15:07.680 --> 00:15:13.200
but you're actually building the system to improve to improve your products.

s122
00:15:13.200 --> 00:15:15.600
If you want to learn more, come check out we have a booth.

s123
00:15:15.600 --> 00:15:20.285
You can check out Signal, learn more, but thank you so much for listening to us.

s124
00:15:20.285 --> 00:15:20.940
[applause]

s125
00:15:20.940 --> 00:15:22.940
[music]

s126
00:15:26.960 --> 00:15:28.960
Hey.