WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=XIHZpZtYCzo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/XIHZpZtYCzo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.480 --> 00:00:15.160
Uh good afternoon, everybody.

s2
00:00:15.160 --> 00:00:20.680
Um I know this is one of the the last talks of the afternoon, so so hopefully people got some energy left.

s3
00:00:20.680 --> 00:00:21.760
Um my name is Nikhil.

s4
00:00:21.760 --> 00:00:24.480
Uh I lead forward deployed engineering here at Factory.

s5
00:00:24.480 --> 00:00:33.440
Um and you know, uh this is this is this episode of Agents and Infrastructure is uh basically came out of like notes from the field.

s6
00:00:33.440 --> 00:00:42.960
And you know, so we um we really believe that uh deployed engineering is um you know, part of the sensory nervous system of an engineering org.

s7
00:00:42.960 --> 00:00:54.120
And essentially, it's trying to figure out uh if for our most complex customers, uh what's difficult, what we need to build, and use that kind of inform our product strategy.

s8
00:00:54.120 --> 00:00:58.640
And the customer in question uh today is going to be Blackstone.

s9
00:00:58.640 --> 00:00:59.960
Uh some of you may have heard of them.

s10
00:00:59.960 --> 00:01:09.440
They're uh you know, tricky little large financial institution, uh trillion dollars in assets, and kind of represent this kind of like fractally complex enterprise, right?

s11
00:01:09.440 --> 00:01:11.280
Where they have a lot Whoa.

s12
00:01:11.280 --> 00:01:23.640
Have a lot of uh have a lot of regulations, have a lot of security constraints, and are really really opinionated about how to deploy uh non-deterministic software in uh in their environments.

s13
00:01:23.640 --> 00:01:28.840
And it's you really one of those uh you know, scenarios where like none of our defaults like work.

s14
00:01:28.840 --> 00:01:33.360
It's kind of like, you know, they couldn't run agents in uh their their laptop.

s15
00:01:33.360 --> 00:01:39.560
They couldn't uh they they wanted to bring their own compute and their own inference, uh their own identity, and and single tenant everywhere, right?

s16
00:01:39.560 --> 00:01:49.200
Um and I think um I I realize now that I'm I should probably pause and probably explain a little bit about what we do in just to make sure that uh in case nobody else nobody knows.

s17
00:01:49.200 --> 00:01:51.920
But um we we bring autonomy to software engineering.

s18
00:01:51.920 --> 00:02:00.160
Uh we're in the business of um you know, doing that by uh our coding agents named Droid, and using that to compose software factories.

s19
00:02:00.160 --> 00:02:03.812
And so you can imagine this requires a lot of permissions.

s20
00:02:03.812 --> 00:02:11.532
And that is a particularly tricky thing to do when the thing that we were trying to do with Blackstone is basically doing delegation.

s21
00:02:11.532 --> 00:02:27.772
So you can imagine through Slack or through Jira or through kind of your kind of client of choice basically saying you know assigning something to Droid and Droid is the name of our agent and so Droid going off and can basically completing a task saying okay, I want to assign the Jira ticket to Droid and it'll come

s22
00:02:27.772 --> 00:02:29.412
back to you with like a PR.

s23
00:02:29.412 --> 00:02:36.612
But I think that the but things get complicated when you have all these constraints of an enterprise, right?

s24
00:02:36.612 --> 00:02:44.732
The two fundamental questions that kept coming back again and again is who is an agent acting on behalf of and where does it run?

s25
00:02:44.732 --> 00:02:56.012
It is you know especially because so many of agentic products that say over the last kind of year 18 months were able to piggyback off of our individual identities.

s26
00:02:56.012 --> 00:03:05.532
So like maybe you spun up a a sandbox and it just it had like your identity attached to it and it was able to use like the the permissions that were attached to that sandbox.

s27
00:03:05.532 --> 00:03:17.092
But I mean ultimately like we really think that um that is like was like a kind of kind of temporary idea because ultimately agents can't just keep borrowing your identity

s28
00:03:17.092 --> 00:03:19.252
and they're ultimately they're just applications.

s29
00:03:19.252 --> 00:03:25.772
Agents are are basically just like slightly non-deterministic applications and you know we don't need to reinvent the wheel here.

s30
00:03:25.772 --> 00:03:35.772
We were doing this for maybe decades of software engineering and it's about using the same systems of the past but kind of maybe like slightly repositioning them for the future.

s31
00:03:35.772 --> 00:03:44.652
And so you know another you know at this conference there's um plenty of people offering you know agent sandboxes.

s32
00:03:44.652 --> 00:03:55.452
Maybe some of you want to run them in EC2 fleets and we just found that this problem of combination of where does an agent run and who is it

s33
00:03:55.452 --> 00:03:59.612
who is running as gets com- was coming up again and again and again.

s34
00:03:59.612 --> 00:04:06.094
And we found that even in the middle of our engagement with Blackstone like you know there was we were going to go with one of these one of these providers,

s35
00:04:06.094 --> 00:04:10.694
but then they kind of changed some like key constraints that meant that we wanted to maybe move to an EC2 machine.

s36
00:04:10.694 --> 00:04:19.974
And and this just kept coming up to our to our customers again and again, where it is like somebody wants to run it on Kubernetes, somebody wants to run on EC2 or G Cloud or whatever it is.

s37
00:04:19.974 --> 00:04:25.614
And we just needed a way to kind of organically move our product to each one of these things.

s38
00:04:25.614 --> 00:04:32.214
And and in the past we had a bring your own compute um product.

s39
00:04:32.214 --> 00:04:42.174
But what we found is that just installing an agent on a piece of on a computer was insufficient because that means that our customer still needed to deal with the compute life cycle.

s40
00:04:42.174 --> 00:04:44.454
It turns out that's a like pretty complex problem.

s41
00:04:44.454 --> 00:04:47.174
So what we came up with was was Open Compute.

s42
00:04:47.174 --> 00:04:56.894
Open Compute is a to be open sourced project that is basically trying to abstract the compute life cycle for any agent.

s43
00:04:56.894 --> 00:05:06.134
That is we obviously built this for Droid, but it is the idea is that this should be able to work for any arbitrary agent in your customer's environment.

s44
00:05:06.134 --> 00:05:16.894
And this is I mean this is a really important idea for us because um one of our, you know, key differentiators is that we care deeply about building you a custom fit suit

s45
00:05:16.894 --> 00:05:20.454
to deploy our product in your environment on your terms.

s46
00:05:20.454 --> 00:05:28.654
So that means, you know, your security constraints, you know, on the compute of choice, and like totally in your VPC without it ever kind of like leaving your environment, right?

s47
00:05:28.654 --> 00:05:32.054
Where as much as possible is on your terms.

s48
00:05:32.054 --> 00:05:41.934
Um so this gets you a lot of the advantages of what maybe an on-prem deployment could give you without also having like the pain of also having to port over your control layer.

s49
00:05:41.934 --> 00:05:51.454
Like if you think about like in the previous world you would have your data plane and your control plane, and this is the kind of an emergent idea of like a compute plane really needs to be kind of spun out into its own thing for agents.

s50
00:05:51.454 --> 00:05:55.054
So what you end up with is this kind of pattern, right?

s51
00:05:55.054 --> 00:06:07.654
I mean this is just an an example simplified pattern of you have maybe a client of some kind, you have maybe you're delegating from you know, a third party of some kind and you can you basically

s52
00:06:07.654 --> 00:06:20.775
uh talk to our back end which has these two concepts of a compute gateway and a relay and the compute gateway kind of tells this open our open compute driver

s53
00:06:20.775 --> 00:06:31.135
to say somebody wants to start a new session which spins up compute in their VPC which could be kind of your micro VM of choice it could be in EKS it could be EC2

s54
00:06:31.135 --> 00:06:38.215
and then that sandbox uh is the is the machine that actually talks to the relay which talks to the client.

s55
00:06:38.215 --> 00:06:41.815
That was a lot of words so I'm going to let's let's do that real slow.

s56
00:06:41.815 --> 00:06:52.695
So let's say you're an agent and so let's say you're a client which could be our CLI which could be our product it could be on frankly one of your products

s57
00:06:52.695 --> 00:06:57.935
and you have something that says I want to start a new agent session.

s58
00:06:57.935 --> 00:07:10.175
And so that session talks to our back end and says so that's that request talks to our back end and says I want to get compute to start this new session.

s59
00:07:10.175 --> 00:07:18.815
And so we go to our compute gateway which is also an abstraction of like you know maybe we're using open compute to get compute but maybe we're also using a sandbox

s60
00:07:18.815 --> 00:07:27.295
you know so it's an it's an it's an it's another abstraction to make this kind of even clearer to say okay and now this this gateway is going to say I'm going to talk to open compute

s61
00:07:27.295 --> 00:07:40.695
which was and this is like a which already has an open connection to the control plane of the open compute application in the customer's VPC and says hey I want to create a new

s62
00:07:40.695 --> 00:07:42.295
I want to create a new computer.

s63
00:07:42.295 --> 00:07:50.375
And so what open compute is going to do this control controller is going to do and say is going to say I'm going to talk to this idea of a driver.

s64
00:07:50.375 --> 00:07:59.815
A driver is a kind of like a wrapper for a given kind of compute so this could be you can imagine like one driver for EC2 one driver for Kubernetes

s65
00:07:59.815 --> 00:08:05.295
you know where your pod lives it could also be you know maybe for a sandbox provider like Daytona or Modal.

s66
00:08:05.295 --> 00:08:12.057
So like it is like totally um like it is an abstraction that could wrap around your compute layer of choice.

s67
00:08:12.057 --> 00:08:20.817
So, Open Compute would talk to your driver of choice in that in this example on EC2 machine, an EC2 driver, and say, "I want a new you know, I want a new EC2 machine."

s68
00:08:20.817 --> 00:08:33.617
So, then we spin that up, and then and then we we connect that EC2 machine that reaches out with to a to what we call our relay, which is something in our control plane, and keeps an open web socket.

s69
00:08:33.617 --> 00:08:43.297
So, and this is kind of actually an important idea because one thing that we hear from customers all the time is that they're very kind of like nervous about ingress, but less nervous about egress.

s70
00:08:43.297 --> 00:08:49.817
So, basically, what we promise in the system is that once set up, we never need to make an inward connection.

s71
00:08:49.817 --> 00:08:56.177
It's always the customer that's making an outward connection and establishing web socket connection.

s72
00:08:56.177 --> 00:08:56.857
Excuse me.

s73
00:08:56.857 --> 00:09:09.737
So, essentially, so once um so once and there's a new like EC2 machine that is going to be like where our agent daemon starts, it's going to establish a open connection

s74
00:09:09.737 --> 00:09:11.857
to our relay.

s75
00:09:11.857 --> 00:09:21.017
And then when you have, you know, our CLI or or in Slack or or our, you know, our web application, it's when you send a hello world, it's going to

s76
00:09:21.017 --> 00:09:28.657
go to our relay, which is able to relay this like literally relay this information to this this request to the sandbox.

s77
00:09:28.657 --> 00:09:31.137
And they're able to kind of have a back and forth.

s78
00:09:31.137 --> 00:09:42.537
And to kind of like just kind of close out this idea, in this particular example, we can assume that the the actual LLM call could be established like a BYOK situation where

s79
00:09:42.537 --> 00:09:56.137
let's say a company, I mean, this is kind of only relevant for I'd say fairly mature engineering organizations, which may already have a deal with, you know, one of the labs, so you can they can like bring their own key and you can imagine like a bedrock situation or something like that to do your LLM calls.

s80
00:09:56.137 --> 00:09:58.857
So, even that stays within sight of VPC.

s81
00:09:58.857 --> 00:10:14.298
So, in this way, without kind of having like a deploying factory entirely on prem, like where you need to have each one of these pieces kind of like loaded in your environment, you're able to get the most of of what you get from that, of like the control of the non-deterministic

s82
00:10:14.298 --> 00:10:28.098
part of the agent loop, and you're going to feel like you have full observability and controllability, and make sure that like the blast radius of anything that you think is like too spicy for an agent to do is going to be contained in inside a customer's VPC.

s83
00:10:28.098 --> 00:10:39.378
And this is I mean, we've seen like I'd say pretty significant success with this, just because this kind of gets you really far in in kind of gaining trust in in

s84
00:10:39.378 --> 00:10:42.178
kind of a customer relationship.

s85
00:10:42.178 --> 00:10:48.618
We believe that also kind of like long-term that like owning your sovereign is going to become a more and more important idea.

s86
00:10:48.618 --> 00:10:56.698
And I think that that's often talked about with respect to models, but I mean, I think that that the entire stack is something that eventually large enterprises are going to want to own.

s87
00:10:56.698 --> 00:11:06.858
And so, you can imagine um you know, us building out components to make it easy for a company to do all of that on their own terms, which is everything from

s88
00:11:06.858 --> 00:11:16.098
owning their own eval, owning their own model, to owning their own compute layer, to also hosting like the product in in in their VPC, in their in their environments,

s89
00:11:16.098 --> 00:11:19.938
and being kind of being able to tell the entire story on their terms.

s90
00:11:19.938 --> 00:11:26.447
So, so basically that you know, in a few weeks, we plan to open source this, and

s91
00:11:26.447 --> 00:11:26.698
[snorts]

s92
00:11:26.698 --> 00:11:32.098
I think that it's going to be useful for people just because the abstractions like have nothing to do with factory.

s93
00:11:32.098 --> 00:11:39.858
They're to do with managing compute, and like being friendly for an agent to kind of interact, right?

s94
00:11:39.858 --> 00:11:43.778
So, it is the the unit of work is is a computer.

s95
00:11:43.778 --> 00:11:51.418
So, you can imagine all the kind of classical things that you'd want to do in a compute life cycle, which is, you know, being able to pause and resume,

s96
00:11:51.418 --> 00:12:02.858
destroy and start, and all of this is based off of like templates that you can control in terms of you know, what you install, and what sha you can have like basically snapshots

s97
00:12:02.858 --> 00:12:08.498
of you know, the exact requirements of your build, your your containers of choice.

s98
00:12:08.498 --> 00:12:16.659
And and that So, that's So, the what you see here is kind of like the spec of like what our factory compute gateway calls.

s99
00:12:16.659 --> 00:12:23.659
Like this is like the abstraction that you need to kind of um kind of convert to if if you're interested in this pattern.

s100
00:12:23.659 --> 00:12:31.539
And and again a fairly standard pattern of what you do in a sandbox is to you know suspend or create a snapshot or to fork.

s101
00:12:31.539 --> 00:12:40.499
And especially the fork is is like a really important idea because you can imagine especially as fleets of agents and swarms of agents are are kind of gaining steam.

s102
00:12:40.499 --> 00:12:49.899
I think that one session delegating to another and having sub agents and sub agents and sub agents this is like fractal idea means that being able to you know save a snapshot,

s103
00:12:49.899 --> 00:12:57.139
fork it, and hand off like a subtask of your problem to another agent is going to like be a really important pattern.

s104
00:12:57.139 --> 00:13:00.659
So, yeah snapshotting is also going to be really important.

s105
00:13:00.659 --> 00:13:05.299
And we also plan for this to be another thing that a customer can own.

s106
00:13:05.299 --> 00:13:18.539
So, we've talked a lot about compute, but you can also imagine like being able to save snapshot on S3 or your kind of like database of choice and eventually also being able to hydrate it with secrets in your secret store of choice.

s107
00:13:18.539 --> 00:13:28.499
So, being giving customers like the optionality to kind of like hydrate, you know, their compute layer however they want and wherever they want.

s108
00:13:28.499 --> 00:13:34.699
Um and so yeah, drivers are basically this idea of um the different compute stories.

s109
00:13:34.699 --> 00:13:35.099
Let's see.

s110
00:13:35.099 --> 00:13:35.699
Yeah.

s111
00:13:35.699 --> 00:13:45.299
And I think that um I think an important idea here is that we are that this is a project to manage computers and not sessions.

s112
00:13:45.299 --> 00:13:52.259
And I think that was a really important idea for us because the moment you tie in the idea of a session and by session I don't mean like a a web session.

s113
00:13:52.259 --> 00:14:06.939
I mean like a factory back and forth like a factory session is that you get tied into it it becomes like factory specific where I think that that makes it harder for for a customer to feel comfortable with with hosting that software.

s114
00:14:06.939 --> 00:14:15.659
Whereas this is like eventually going to be open source and a totally like abstract concept that they could use for for you know for products beyond factory.

s115
00:14:15.659 --> 00:14:21.334
Um and I think just to give an idea of like the kind of complexity that this solves, like the

s116
00:14:21.334 --> 00:14:22.181
[snorts]

s117
00:14:22.181 --> 00:14:37.261
in in dealing with uh like compute life cycle, you might, you know, fire off a request saying, um I want to start a new session, and maybe the maybe the AWS says that uh you know, I received your request,

s118
00:14:37.261 --> 00:14:40.621
uh but it never acts with like the new sandbox of sorts.

s119
00:14:40.621 --> 00:14:46.261
Um you maybe your request is maybe uh mid-session, one of your requests is dropped, and you don't get an ack back.

s120
00:14:46.261 --> 00:14:52.221
Just to come something to manage like the bookkeeping of uh did I start a sandbox for you?

s121
00:14:52.221 --> 00:14:53.381
Uh what was it called?

s122
00:14:53.381 --> 00:14:57.981
Where how do I route requests from a given user to a given uh sandbox?

s123
00:14:57.981 --> 00:15:09.301
And um and being just being able to retry to make sure that like the client uh and the sandbox understand where they are in a given conversation, and to just like do that bookkeeping for you.

s124
00:15:09.301 --> 00:15:17.621
Um and I think that you can imagine that this is um I mean, managing arbitrary compute is like a fairly complex problem.

s125
00:15:17.621 --> 00:15:24.261
I mean, one example is uh you know, the fact that uh you know, micro VMs in AWS have like have an have an 8-hour session cap.

s126
00:15:24.261 --> 00:15:31.221
Um and one of our products is uh is missions, which can run for like multiple days of like multi-agent orchestration.

s127
00:15:31.221 --> 00:15:41.861
And so, you know, how do you uh save uh a sandbox uh with a snapshot and rehydrate a new one and kind of like swap in place as if nothing happened?

s128
00:15:41.861 --> 00:15:43.901
It's like, you know, pretty difficult problems.

s129
00:15:43.901 --> 00:15:48.981
And uh you know, some of the stuff that we've had to kind kind of uh slog through.

s130
00:15:48.981 --> 00:15:56.621
And and additionally, just kind of like, you know, while this has been in kind of like private preview, um how do we actually communicate this idea like while it's not open source?

s131
00:15:56.621 --> 00:16:04.581
Um so, like versioning, uh Terraform, and sharing you know, these Terraform packages to help folks stand it up has been has been interesting.

s132
00:16:04.581 --> 00:16:17.701
Um and yeah, I mean, I think that um another um kind of interesting thing that came out of like the way this problem was defined is that if you uh you can just kind of provide the spec to to Droid,

s133
00:16:17.701 --> 00:16:27.182
uh and just say just make me a driver that that fits the spec and it and it kind of like is its own test like if like if something can it's almost like a

s134
00:16:27.182 --> 00:16:34.222
it's a very good version of test driven and spectrum development where if it satisfies a given this given spec it's it's basically ready to go.

s135
00:16:34.222 --> 00:16:44.182
So it's very easy to kind of contribute new drivers and add new kind of compute types so this can just kind of keep getting flex more and more flexible over time.

s136
00:16:44.182 --> 00:16:50.302
But yeah, I mean ultimately just to kind of close out agents shouldn't care where they're on.

s137
00:16:50.302 --> 00:17:02.622
This is something that comes up for people again and again and this is the way we solved it at factory and and how we are able to give a really high level flexibility

s138
00:17:02.622 --> 00:17:05.702
to our deployments for our customers around the world and that's all.

s139
00:17:05.702 --> 00:17:07.862
Thank you.