Why AI Makes You 100x Faster but Not Your Company - Mitchell Troyanovsky, Basis

AI Engineer · 19 min · 215 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12I'm Mitch.
00:12I'm one of the two co-founders of Basis.
00:15We are based right here in New York City and the thing that we focus on is how can we get agents to do production-scale work reliably enough so that we are trusted
00:30and used end-to-end by accounting firms across the country.
00:35So, a lot of our research really focuses on this.
00:37How can you get these agents to do real work in the real economy?
00:41So, I want to maybe start with a question, which is if you're in the at this conference, honestly, if you're watching this talk, you've probably increased your own productivity a lot.
00:54You're using co-work or uh Codex or cloud code, whatever it is.
00:59And yet, if you go and look at the average company, the average enterprise, the average firm, their productivity hasn't gone up 10x, 100x.
01:10I know my productivity has gone up maybe 100x.
01:13But, if I were working inside of a company, it wouldn't be anywhere near that order of magnitude.
01:18And so, why is that?
01:22At Basis, one of the things we have is that humans are already used to working with non-deterministic systems.
01:28It's just those systems are normally their co-workers, not their computers.
01:32And so, I actually think to answer this question, it's useful to maybe look at the existing way we organize intelligent systems, which are companies.
01:41So, how do organizations scale their own productivity, right?
01:47Organizations need to trust that the people doing the work at every level are doing it in a way that is going to produce good outputs consistently at scale, right?
02:00If look at this image here, the uh here I'm showing an accounting firm, but this could honestly be any company, traditional company.
02:07The person who is the head of the firm, they have literally no idea about the exact day-to-day of the people at the very bottom.
02:15And yet somehow they trust the work is going to be done.
02:19How does that happen?
02:21It doesn't happen through deterministic systems, right?
02:24People are not deterministic.
02:26They're intelligent entities.
02:28And instead there are processes that are put in place, and those processes, if we believe the people who, you know, put them in place, think that those processes will generalize
02:38to good outcomes, then they can be comfortable that even though the people executing the processes are not deterministic, you will generally get to good outcomes if you follow the process.
02:49This is true, by the way, even in the smallest steps.
02:51So, in this example, earlier I was showing the process of maybe doing the monthly accounting work for a client at an accounting firm, but even within that process, say you wanted to see, how do I reconcile the cash?
03:03Well, I need to get the bank data, and even that has a process.
03:09The process could be really basic.
03:10It could just be use the approved software to extract the data, but there's still some process that is put in place.
03:17You wouldn't expect the firm to just let anyone Google, you know, extract PDF and then just use whatever they want to go extract it, because now you wouldn't have process.
03:29Now let's talk about agents.
03:33As I said earlier, everyone already has co-work, ChatGPT, Codex, Cloud Code, but where is that productivity?
03:42Our opinion is there are two main problems that come with what I'll call coding-style agent products, where you're working with an agent and you just tell it to go and do something.
03:52The first problem is that there is a competency gap, because agents, as you think of them as you think of them today, they outsource the complexity to the user.
04:04Right, if I'm the user, I now have to manage my skill and memories and think about how to when I should use the agent.
04:14I have to have like theory of mind over it.
04:16This is really hard.
04:18And for those who are here, maybe you guys find it easy, but I promise for the wide majority of knowledge workers, they have no idea how to do any of this.
04:29And there's a second problem.
04:31Which is there is a process gap.
04:33Right, if you are saying, "Hey user, here's your agent, do what you want."
04:39Well, now suddenly the way the work is being done is essentially up to whoever the user is to customize the way the agent is behaving.
04:47Which goes entirely against the way that organizations scale.
04:51The process should be coming from senior leadership down, not from any individual employee deciding how they're going to do their job.
05:00Right, if you going back to my example, if every employee just made up their own way of extracting data, you know, even back in 2018, that wouldn't be a good outcome.
05:10And so, we believe strongly that the key
05:13[snorts]
05:13to building production agents that actually do work end-to-end is solving these two problems.
05:17You have to solve the consistency gap by making it simple.
05:22To the user, you shouldn't even have to know it's an agent.
05:25Or you probably know, but you don't have to really feel it.
05:28It should be as simple as just working with a co-worker.
05:31You just tell it what you want it to do and all the things you expect to work should work.
05:35And the second is the process gap.
05:37You need agents to be reliable.
05:40Not having a stroke of genius one day and forgetting something the other day.
05:44It's not about being perfect.
05:47Humans are not perfect.
05:49Right, nobody is looking for perfect.
05:51What you're looking for is reliability and consistency and trustworthiness.
05:57So, what does that actually look like?
06:00We think that to be simple and reliable where you know it's you don't have to think about what to do.
06:08The agent can just do it for you without the user knowing the internals.
06:13To do it the same way consistently, you have to be you have to be opinionated.
06:21Let me just show you actually one slide back here.
06:23I want to point out that getting it right isn't actually the main task, because again, humans are imperfect themselves.
06:30The point is to do it in a similar way.
06:32It's okay if you you can't do something.
06:35But you should be consistently able to not do something, and then you get better, just like a person does.
06:40If you suddenly one day are an expert at something, and then you miss something very basic that would be expected, now you're not reliable.
06:48So you have to be opinionated.
06:51What's an example of an opinion?
06:54Let's say we have an agent, and we need to go and read a PDF.
06:58So in this case, what is the best way to read a PDF?
07:02There's many ways.
07:02You can throw it into a VLM, you can go write some Python code and use an open-source OCR library, you can work with one of the many OCR-specific model providers.
07:13You could do an OCR, and then you could have the VLM check its work.
07:17You could have five VLMs check its work.
07:19You could do 100 VLMs and take the best of them.
07:21There's literally infinite possibilities here.
07:24And so if you just said, "Agent, extract Excel extract PDF," you might get good outcomes, but they're not going to be reliable.
07:33Right?
07:34Not because the process it did was bad, but because the process it did was not opinionated.
07:39So the way we develop opinions is you use initial intuition, right?
07:50Some things you know what the opinions are from expertise.
07:53For example, if I'm trying to get an agent to do tax work well, I know that if the agent is going goes and reads the primary source before returning to the user,
08:04it is more likely to be accurate.
08:07I don't need a set of evals to tell me that.
08:09I don't need to divine that from first principles.
08:11We know that, right?
08:13I know that if I have an agent make a PowerPoint, if it takes a picture of the PowerPoint before it returns to user, it is less likely to have a formatting issue.
08:22That is obvious.
08:24And so, there are lots of different ways to develop opinions.
08:27And one of the key points here, I'll just make before I move on to talking about how to use the opinions, is that the opinions are not just about accuracy,
08:34right?
08:34We just talked about the example of taking a picture of a PowerPoint.
08:37Well, that adds latency, right?
08:40You may not want that.
08:41Maybe the agent only makes formatting mistakes one out of a hundred times, and you're okay with that.
08:46And so, therefore, you don't have to take a picture of the PowerPoint.
08:50The point isn't exactly what the opinion is.
08:52And this talk, we're going to focus not on how to develop opinions, but why opinions matter, and then how to leverage those opinions as you scale in production.
09:02So, say that you have a bunch of outcome-based evals.
09:09And when I say outcome-based evals, I mean what we traditionally think of as evals.
09:13It has a right answer.
09:15We have some rubric we're trying to judge what the agent did.
09:18How do you know that the evals generalize to production?
09:22Right?
09:23You might have a thousand cases or a hundred cases.
09:25But in production, if you're doing real work, you're going to have hundreds of thousands of situations.
09:30And that's only going to increase as the trajectories get much longer, because the amount of decisions that are made in a trajectory uh just grow exponentially as the like long horizon abilities of agents increase.
09:47And so, I can't prove if I If you're looking at all these distribution of uh trajectories, you can't prove every one of them is correct.
09:54You don't have a ground truth for every trajectory.
09:57But you can potentially look at the trajectories to decide if the behavior, the opinion that you have, is actually happening.
10:06And you can then infer that if it's actually happening, it's more likely to generalize to production.
10:12It's not perfect.
10:13You don't know for sure, but at least you are having an opinion.
10:18Let's run through a real example.
10:20So, let's say that I uh for a behavior, this is a an accounting example where if there is a uh mistake for uh in in the bank reconciliation, um
10:34uh I don't want the agent to like post an entry to make it tie.
10:40I want it to uh it sort of explain the difference instead of putting a plug.
10:44Again, this is subjective.
10:46You might want the other way.
10:48But, what you don't want is a customer getting one way one time and another way the other time.
10:53And so, what would you do?
10:54Well, first, let's understand what governs the behavior.
10:58Right?
10:58The behavior's not just you write the two sentences into your system prompt and it just works.
11:03That might work for very simple narrow agents.
11:05But, when you're building these big agent systems, the behavior's actually a combination of the system prompt, your progressive disclosures, uh potentially tool descriptions, obviously the training of the model.
11:16There's a lot of things that govern the behavior.
11:19And one tip, by the way, I have for when you think about building production agents is really think of the context the agent is going to see as it's working as runtime training data.
11:30The word prompt, I think, trivializes it a bit, right?
11:32At the end of the day, this is the data that the agent is being trained on at runtime.
11:37Not literally trained on, but effectively trained on.
11:40And so, you want to care about every single token and understand how that data causes the agent to behave in certain ways.
11:48So, when you're putting uh behaviors into production, it it might be very difficult for one behavior to put it in exactly one, you know, paragraph that you know exactly because again, the behavior is all of the data at once,
12:01not just one thing.
12:03And one of the things that we developed at Basis was this concept of a behavior spec.
12:11And the idea of a behavior spec is essentially saying before we update the context or write the context, uh what do we actually want?
12:20What is our intent?
12:22You can kind of think of this a little bit as like a product spec for the agent era.
12:26And you can't write You shouldn't be writing every detail down.
12:29You should be taking the most important opinions and putting it because if you put everything, you'll be duplicating a lot of state.
12:34So, you define the ones that matter the most.
12:37And then you can use them as a way to align your team from a spec perspective, but then you can also use them as a rubric.
12:46And you can say in this particular trajectory, whether that is in any trajectory that doesn't have a ground truth, did the agent get into a situation in which this behavior was relevant?
12:58And then if so, did it actually exhibit the behavior?
13:02And this way you can scale the amount of signal that you get on your trajectories far larger than trying to increase the amount of outcome-based evals, ground truths, and rubrics that you have.
13:13Those are really important, but you cannot scale those anywhere near as fast as you can scale the ability to supervise what the behaviors of the agent are.
13:24Let's just take a quick example here.
13:26Um this is so this is actually the second concept, um which is called forking, which we can combine here with behavior specs to think about.
13:35Forking, if you've ever used uh cloud code or or Codex the web apps, there's a little little fork icon uh under each message, uh where you can essentially create a copy of the agent using the same thing, which works really well on your local computer.
13:48It's hard to build a system that does this in the cloud perfectly with recreating the database environment and recreating the uh the prompts, uh all the all the different environments.
13:59But forking is another tool that we use to check the behavior.
14:02So, a flow that you can imagine is saying we had a trajectory, could be in an eval set, it could be in testing or in production, and it failed the behavior eval.
14:14And now you're trying to understand why.
14:16Why did it fail failed the behavior eval?
14:19Why did it not exhibit the behavior?
14:20And then you can go into the trajectory, go right before the point in which the agent made the decision that was in contrast to the behavior, assuming again your behavior is good, and
14:31it's actually producing signal.
14:33And then you can fork it.
14:35And when you fork it, you can ask the agent uh or you would fork it right after it made the decision.
14:41You could ask it why did you do that?
14:44Or you can go and update the uh context of the agent and see if that improves the behavior.
14:50And so with forking you can start to actually take the signal and start to figure out, okay, how can we then improve the harness to now uh leverage the signal?
15:00Whereas if you're just looking at signal from an outcome-based eval, you can't do that anywhere near as easily because it failed the outcome, how do you know why?
15:09You have potentially a thousand inference calls in your 5-hour long agent trajectory run.
15:15And then we have the last tool that I'd like to talk about.
15:17This is actually my favorite, which is called simulations.
15:20Uh this is going to sound silly, but I promise it's very effective, which is that you can, when working with coding agents, ask the coding agent to make a sub agent that pretends to be your production agent.
15:33And you have to tell it to not cheat, and you give it the system prompt and the skills, and you render everything perfectly.
15:38And what you can do is you can ask it what it would do in some scenario.
15:41You can give it the situation, and see if you can recreate the situation that you saw in your eval or in production.
15:46And if you can recreate it, now you can tweak your context to see if the sub agent would change its behavior based on your tweaks.
15:54And the reason to do this is obviously far worse than forking, but you do this because this is much cheaper, much faster, and allows you as an agent builder
16:03to really easily iterate on ideas without needing to go through the expensive process of actually forking an entire environment.
16:12And you can do this today without building anything.
16:14Um I did this for literally year year and a half plus before we uh built forking into our internal systems.
16:23And the main idea here is understanding that if you look at the the distribution from earlier, you're trying to understand is the is the uh agent generalizing to that distribution.
16:36And so you don't know purely based on the outcome eval.
16:38So you can try to choose the best outcome evals that should represent the curve, but you don't really know.
16:43And so supervising the process through some of these methods gives you some additional signal in order to understand if it's generalizing properly.
16:53And what's really cool is as you start to get more signal, you can actually start to close the loop.
17:01And an opinion is what lets you do that, right?
17:03If you don't have opinions about your system, you're not going to be able to develop that signal.
17:10And now if we go back to the firm, we talk about talk about supervision, we talked about how the agent uh if he's our opinionated, he can be more reliable and he actually can be simpler.
17:21What that means is the agent maybe it starts doing something narrow, but because people will work with it, they see that it's easy to use, and they see that it's reliable.
17:31As it gets better and better, they will trust it more.
17:33It's just like a co-worker.
17:35You have a co-worker day one.
17:36Imagine someone sends you a PR.
17:39You're not going to if it's you're looking at two axes whenever someone sends you a PR.
17:43It's these two axes.
17:45What's the black greatest of the PR, and what is your perception of how competent that person is?
17:50Right?
17:51If the least competent person in your company is sending you a PR that's editing a core service, you're going to look at it.
17:58If the most competent person is sending you a PR that's editing an internal document, you may not look at it as much.
18:04And so this idea is true with humans working with agents just like it is with humans working with humans.
18:10And I think I just wanted to make one last point to close here, which is that one of the things that people say when they hear this is, "Well, wait a second.
18:17If you let the agents just do what they want, won't they find a better way?
18:21Won't they move 37 your work?
18:23And the answer is they might.
18:26But that's not how you scale GDP and scale output.
18:30Nobody wants their agent to move 37 their tax return.
18:35They want it done reliably, at scale, in a way that they can trust, and in a way that their people know how to work with.
18:42And so the thing I will leave you with is for agents to diffuse, they need to be simple.
18:47For teams to trust them, they need to be reliable.
18:50The way you get to simple and reliable is you develop opinions on how the agent behaves, and then opinions allow you to build that additional signal that helps you start to close the loop so that you can achieve self-improving
19:03agents in production.
19:04Thank you very much.
19:06[applause]