Your AI Agents Shouldn't Care Where They Run - Nikhil Harithas, Factory

AI Engineer · 17 min · 139 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Uh good afternoon, everybody.
00:15Um I know this is one of the the last talks of the afternoon, so so hopefully people got some energy left.
00:20Um my name is Nikhil.
00:21Uh I lead forward deployed engineering here at Factory.
00:24Um and you know, uh this is this is this episode of Agents and Infrastructure is uh basically came out of like notes from the field.
00:33And you know, so we um we really believe that uh deployed engineering is um you know, part of the sensory nervous system of an engineering org.
00:42And essentially, it's trying to figure out uh if for our most complex customers, uh what's difficult, what we need to build, and use that kind of inform our product strategy.
00:54And the customer in question uh today is going to be Blackstone.
00:58Uh some of you may have heard of them.
00:59They're uh you know, tricky little large financial institution, uh trillion dollars in assets, and kind of represent this kind of like fractally complex enterprise, right?
01:09Where they have a lot Whoa.
01:11Have a lot of uh have a lot of regulations, have a lot of security constraints, and are really really opinionated about how to deploy uh non-deterministic software in uh in their environments.
01:23And it's you really one of those uh you know, scenarios where like none of our defaults like work.
01:28It's kind of like, you know, they couldn't run agents in uh their their laptop.
01:33They couldn't uh they they wanted to bring their own compute and their own inference, uh their own identity, and and single tenant everywhere, right?
01:39Um and I think um I I realize now that I'm I should probably pause and probably explain a little bit about what we do in just to make sure that uh in case nobody else nobody knows.
01:49But um we we bring autonomy to software engineering.
01:51Uh we're in the business of um you know, doing that by uh our coding agents named Droid, and using that to compose software factories.
02:00And so you can imagine this requires a lot of permissions.
02:03And that is a particularly tricky thing to do when the thing that we were trying to do with Blackstone is basically doing delegation.
02:11So you can imagine through Slack or through Jira or through kind of your kind of client of choice basically saying you know assigning something to Droid and Droid is the name of our agent and so Droid going off and can basically completing a task saying okay, I want to assign the Jira ticket to Droid and it'll come
02:27back to you with like a PR.
02:29But I think that the but things get complicated when you have all these constraints of an enterprise, right?
02:36The two fundamental questions that kept coming back again and again is who is an agent acting on behalf of and where does it run?
02:44It is you know especially because so many of agentic products that say over the last kind of year 18 months were able to piggyback off of our individual identities.
02:56So like maybe you spun up a a sandbox and it just it had like your identity attached to it and it was able to use like the the permissions that were attached to that sandbox.
03:05But I mean ultimately like we really think that um that is like was like a kind of kind of temporary idea because ultimately agents can't just keep borrowing your identity
03:17and they're ultimately they're just applications.
03:19Agents are are basically just like slightly non-deterministic applications and you know we don't need to reinvent the wheel here.
03:25We were doing this for maybe decades of software engineering and it's about using the same systems of the past but kind of maybe like slightly repositioning them for the future.
03:35And so you know another you know at this conference there's um plenty of people offering you know agent sandboxes.
03:44Maybe some of you want to run them in EC2 fleets and we just found that this problem of combination of where does an agent run and who is it
03:55who is running as gets com- was coming up again and again and again.
03:59And we found that even in the middle of our engagement with Blackstone like you know there was we were going to go with one of these one of these providers,
04:06but then they kind of changed some like key constraints that meant that we wanted to maybe move to an EC2 machine.
04:10And and this just kept coming up to our to our customers again and again, where it is like somebody wants to run it on Kubernetes, somebody wants to run on EC2 or G Cloud or whatever it is.
04:19And we just needed a way to kind of organically move our product to each one of these things.
04:25And and in the past we had a bring your own compute um product.
04:32But what we found is that just installing an agent on a piece of on a computer was insufficient because that means that our customer still needed to deal with the compute life cycle.
04:42It turns out that's a like pretty complex problem.
04:44So what we came up with was was Open Compute.
04:47Open Compute is a to be open sourced project that is basically trying to abstract the compute life cycle for any agent.
04:56That is we obviously built this for Droid, but it is the idea is that this should be able to work for any arbitrary agent in your customer's environment.
05:06And this is I mean this is a really important idea for us because um one of our, you know, key differentiators is that we care deeply about building you a custom fit suit
05:16to deploy our product in your environment on your terms.
05:20So that means, you know, your security constraints, you know, on the compute of choice, and like totally in your VPC without it ever kind of like leaving your environment, right?
05:28Where as much as possible is on your terms.
05:32Um so this gets you a lot of the advantages of what maybe an on-prem deployment could give you without also having like the pain of also having to port over your control layer.
05:41Like if you think about like in the previous world you would have your data plane and your control plane, and this is the kind of an emergent idea of like a compute plane really needs to be kind of spun out into its own thing for agents.
05:51So what you end up with is this kind of pattern, right?
05:55I mean this is just an an example simplified pattern of you have maybe a client of some kind, you have maybe you're delegating from you know, a third party of some kind and you can you basically
06:07uh talk to our back end which has these two concepts of a compute gateway and a relay and the compute gateway kind of tells this open our open compute driver
06:20to say somebody wants to start a new session which spins up compute in their VPC which could be kind of your micro VM of choice it could be in EKS it could be EC2
06:31and then that sandbox uh is the is the machine that actually talks to the relay which talks to the client.
06:38That was a lot of words so I'm going to let's let's do that real slow.
06:41So let's say you're an agent and so let's say you're a client which could be our CLI which could be our product it could be on frankly one of your products
06:52and you have something that says I want to start a new agent session.
06:57And so that session talks to our back end and says so that's that request talks to our back end and says I want to get compute to start this new session.
07:10And so we go to our compute gateway which is also an abstraction of like you know maybe we're using open compute to get compute but maybe we're also using a sandbox
07:18you know so it's an it's an it's an it's another abstraction to make this kind of even clearer to say okay and now this this gateway is going to say I'm going to talk to open compute
07:27which was and this is like a which already has an open connection to the control plane of the open compute application in the customer's VPC and says hey I want to create a new
07:40I want to create a new computer.
07:42And so what open compute is going to do this control controller is going to do and say is going to say I'm going to talk to this idea of a driver.
07:50A driver is a kind of like a wrapper for a given kind of compute so this could be you can imagine like one driver for EC2 one driver for Kubernetes
07:59you know where your pod lives it could also be you know maybe for a sandbox provider like Daytona or Modal.
08:05So like it is like totally um like it is an abstraction that could wrap around your compute layer of choice.
08:12So, Open Compute would talk to your driver of choice in that in this example on EC2 machine, an EC2 driver, and say, "I want a new you know, I want a new EC2 machine."
08:20So, then we spin that up, and then and then we we connect that EC2 machine that reaches out with to a to what we call our relay, which is something in our control plane, and keeps an open web socket.
08:33So, and this is kind of actually an important idea because one thing that we hear from customers all the time is that they're very kind of like nervous about ingress, but less nervous about egress.
08:43So, basically, what we promise in the system is that once set up, we never need to make an inward connection.
08:49It's always the customer that's making an outward connection and establishing web socket connection.
08:56Excuse me.
08:56So, essentially, so once um so once and there's a new like EC2 machine that is going to be like where our agent daemon starts, it's going to establish a open connection
09:09to our relay.
09:11And then when you have, you know, our CLI or or in Slack or or our, you know, our web application, it's when you send a hello world, it's going to
09:21go to our relay, which is able to relay this like literally relay this information to this this request to the sandbox.
09:28And they're able to kind of have a back and forth.
09:31And to kind of like just kind of close out this idea, in this particular example, we can assume that the the actual LLM call could be established like a BYOK situation where
09:42let's say a company, I mean, this is kind of only relevant for I'd say fairly mature engineering organizations, which may already have a deal with, you know, one of the labs, so you can they can like bring their own key and you can imagine like a bedrock situation or something like that to do your LLM calls.
09:56So, even that stays within sight of VPC.
09:58So, in this way, without kind of having like a deploying factory entirely on prem, like where you need to have each one of these pieces kind of like loaded in your environment, you're able to get the most of of what you get from that, of like the control of the non-deterministic
10:14part of the agent loop, and you're going to feel like you have full observability and controllability, and make sure that like the blast radius of anything that you think is like too spicy for an agent to do is going to be contained in inside a customer's VPC.
10:28And this is I mean, we've seen like I'd say pretty significant success with this, just because this kind of gets you really far in in kind of gaining trust in in
10:39kind of a customer relationship.
10:42We believe that also kind of like long-term that like owning your sovereign is going to become a more and more important idea.
10:48And I think that that's often talked about with respect to models, but I mean, I think that that the entire stack is something that eventually large enterprises are going to want to own.
10:56And so, you can imagine um you know, us building out components to make it easy for a company to do all of that on their own terms, which is everything from
11:06owning their own eval, owning their own model, to owning their own compute layer, to also hosting like the product in in in their VPC, in their in their environments,
11:16and being kind of being able to tell the entire story on their terms.
11:19So, so basically that you know, in a few weeks, we plan to open source this, and
11:26[snorts]
11:26I think that it's going to be useful for people just because the abstractions like have nothing to do with factory.
11:32They're to do with managing compute, and like being friendly for an agent to kind of interact, right?
11:39So, it is the the unit of work is is a computer.
11:43So, you can imagine all the kind of classical things that you'd want to do in a compute life cycle, which is, you know, being able to pause and resume,
11:51destroy and start, and all of this is based off of like templates that you can control in terms of you know, what you install, and what sha you can have like basically snapshots
12:02of you know, the exact requirements of your build, your your containers of choice.
12:08And and that So, that's So, the what you see here is kind of like the spec of like what our factory compute gateway calls.
12:16Like this is like the abstraction that you need to kind of um kind of convert to if if you're interested in this pattern.
12:23And and again a fairly standard pattern of what you do in a sandbox is to you know suspend or create a snapshot or to fork.
12:31And especially the fork is is like a really important idea because you can imagine especially as fleets of agents and swarms of agents are are kind of gaining steam.
12:40I think that one session delegating to another and having sub agents and sub agents and sub agents this is like fractal idea means that being able to you know save a snapshot,
12:49fork it, and hand off like a subtask of your problem to another agent is going to like be a really important pattern.
12:57So, yeah snapshotting is also going to be really important.
13:00And we also plan for this to be another thing that a customer can own.
13:05So, we've talked a lot about compute, but you can also imagine like being able to save snapshot on S3 or your kind of like database of choice and eventually also being able to hydrate it with secrets in your secret store of choice.
13:18So, being giving customers like the optionality to kind of like hydrate, you know, their compute layer however they want and wherever they want.
13:28Um and so yeah, drivers are basically this idea of um the different compute stories.
13:34Let's see.
13:35Yeah.
13:35And I think that um I think an important idea here is that we are that this is a project to manage computers and not sessions.
13:45And I think that was a really important idea for us because the moment you tie in the idea of a session and by session I don't mean like a a web session.
13:52I mean like a factory back and forth like a factory session is that you get tied into it it becomes like factory specific where I think that that makes it harder for for a customer to feel comfortable with with hosting that software.
14:06Whereas this is like eventually going to be open source and a totally like abstract concept that they could use for for you know for products beyond factory.
14:15Um and I think just to give an idea of like the kind of complexity that this solves, like the
14:21[snorts]
14:22in in dealing with uh like compute life cycle, you might, you know, fire off a request saying, um I want to start a new session, and maybe the maybe the AWS says that uh you know, I received your request,
14:37uh but it never acts with like the new sandbox of sorts.
14:40Um you maybe your request is maybe uh mid-session, one of your requests is dropped, and you don't get an ack back.
14:46Just to come something to manage like the bookkeeping of uh did I start a sandbox for you?
14:52Uh what was it called?
14:53Where how do I route requests from a given user to a given uh sandbox?
14:57And um and being just being able to retry to make sure that like the client uh and the sandbox understand where they are in a given conversation, and to just like do that bookkeeping for you.
15:09Um and I think that you can imagine that this is um I mean, managing arbitrary compute is like a fairly complex problem.
15:17I mean, one example is uh you know, the fact that uh you know, micro VMs in AWS have like have an have an 8-hour session cap.
15:24Um and one of our products is uh is missions, which can run for like multiple days of like multi-agent orchestration.
15:31And so, you know, how do you uh save uh a sandbox uh with a snapshot and rehydrate a new one and kind of like swap in place as if nothing happened?
15:41It's like, you know, pretty difficult problems.
15:43And uh you know, some of the stuff that we've had to kind kind of uh slog through.
15:48And and additionally, just kind of like, you know, while this has been in kind of like private preview, um how do we actually communicate this idea like while it's not open source?
15:56Um so, like versioning, uh Terraform, and sharing you know, these Terraform packages to help folks stand it up has been has been interesting.
16:04Um and yeah, I mean, I think that um another um kind of interesting thing that came out of like the way this problem was defined is that if you uh you can just kind of provide the spec to to Droid,
16:17uh and just say just make me a driver that that fits the spec and it and it kind of like is its own test like if like if something can it's almost like a
16:27it's a very good version of test driven and spectrum development where if it satisfies a given this given spec it's it's basically ready to go.
16:34So it's very easy to kind of contribute new drivers and add new kind of compute types so this can just kind of keep getting flex more and more flexible over time.
16:44But yeah, I mean ultimately just to kind of close out agents shouldn't care where they're on.
16:50This is something that comes up for people again and again and this is the way we solved it at factory and and how we are able to give a really high level flexibility
17:02to our deployments for our customers around the world and that's all.
17:05Thank you.