WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=OL7kfezynJM

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/OL7kfezynJM.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:12.560 --> 00:00:13.120
All right.

s3
00:00:13.120 --> 00:00:15.240
Hey everyone, I'm Arjun Singh.

s4
00:00:15.240 --> 00:00:22.000
Today I'm going to talk to you about multiplayer agentic engineering or how to enable your whole team and your best agents to work together.

s5
00:00:22.000 --> 00:00:27.000
If you go to the talks or go around the expo, you're going to see that a lot of people are talking about putting the agents at the center of everything.

s6
00:00:27.000 --> 00:00:29.480
Makes sense, they're really powerful, they're really cool.

s7
00:00:29.480 --> 00:00:32.279
But you don't see a lot of people talking about the people.

s8
00:00:32.279 --> 00:00:36.880
Like this is all for us to make us our lives better, our more our more productive, whatever.

s9
00:00:36.880 --> 00:00:41.520
And so we're going to really focus on how the people fit into these agentic workflows.

s10
00:00:41.520 --> 00:00:44.640
Just a little bit about us first.

s11
00:00:44.640 --> 00:00:48.400
So our team has worked together building software for over a decade.

s12
00:00:48.400 --> 00:00:51.640
Um my co-founder Sergey and I, we met in the PhD program at Berkeley.

s13
00:00:51.640 --> 00:00:54.480
I worked on robotics, he worked on computer vision.

s14
00:00:54.480 --> 00:00:56.880
And during that we co-founded a company called GradeScope.

s15
00:00:56.880 --> 00:00:58.160
Some of you may have used it.

s16
00:00:58.160 --> 00:01:04.800
It's used by millions of students worldwide at thousands of universities, helping instructors grade their students' work.

s17
00:01:04.800 --> 00:01:09.320
And um pretty much the entire team working on Superconductor used to work together on GradeScope.

s18
00:01:09.320 --> 00:01:15.320
And so we've had a team that's worked together productively from first user to acquisition, working on something new together again.

s19
00:01:15.320 --> 00:01:31.160
And I think it's kind of an interesting experiment because you know, over the past year we've very aggressively integrated agents in our workflows and we've kind of surfaced all the different bottlenecks and friction points that come up and how to do that productively and keep collaborating the way we used to but with the new power of agents.

s20
00:01:31.160 --> 00:01:38.080
So today I'm going to talk to you about how, you know, we we the lessons we learned from kind of solving those friction points and solving those bottlenecks.

s21
00:01:38.200 --> 00:01:44.480
And in the talk description I mentioned five lessons and I'm going to be an engineer and start from zero and add a sixth one in there.

s22
00:01:44.480 --> 00:01:49.680
Um the first one I'm going to start with is just to be model and harness agnostic.

s23
00:01:49.680 --> 00:01:50.880
So there's a few reasons for that.

s24
00:01:50.880 --> 00:01:53.840
The best model and harness can change weekly.

s25
00:01:53.840 --> 00:01:55.480
It could change cuz a new one comes out.

s26
00:01:55.480 --> 00:01:58.000
It could change cuz the best one got taken away.

s27
00:01:58.000 --> 00:02:03.840
Um things happen and you don't want that to disrupt your entire team's flow.

s28
00:02:03.840 --> 00:02:06.160
The other thing is that open weight models are actually pretty good now.

s29
00:02:06.160 --> 00:02:10.479
We've been really happy with GLM 5.2. They're much cheaper.

s30
00:02:10.479 --> 00:02:16.400
Um and you want to be able to kind of explore with them and integrate them without again having to change your entire workflow.

s31
00:02:16.560 --> 00:02:23.000
The last thing I'll mention on this is that the incentives of the people selling you tokens aren't really aligned with yours.

s32
00:02:23.000 --> 00:02:24.200
You're here for a reason.

s33
00:02:24.200 --> 00:02:25.760
You're you're you're So you're doing things for a reason.

s34
00:02:25.760 --> 00:02:29.840
You're trying to make your customers' lives better, make your product better, delight your customers.

s35
00:02:29.840 --> 00:02:31.640
They want to sell you more tokens.

s36
00:02:31.640 --> 00:02:36.840
And you might be happy to pay for as many tokens as it takes, but you don't want to pay for more than that.

s37
00:02:36.840 --> 00:02:43.080
And so again, kind of being able to switch between things lets you stay in control of all of that.

s38
00:02:43.480 --> 00:02:53.480
And so, um you know, as I go through the talk, I'll mention a couple of places where our product makes it easy for us, but whether you use us or not, I'm just going to leave things with you that I think are really important

s39
00:02:53.480 --> 00:02:57.680
um for you to be able to work collaboratively effectively.

s40
00:02:57.880 --> 00:03:03.600
So, the next one is to turn every human interface into an agent and human interface.

s41
00:03:03.600 --> 00:03:09.800
So, you know, typically when people are working with coding agents, they're on their laptop, kind of stuck on that laptop.

s42
00:03:09.800 --> 00:03:12.680
Nobody else can talk to that agent.

s43
00:03:12.680 --> 00:03:16.959
So, the first place people go to kind of expose more interfaces for them is Slack.

s44
00:03:16.959 --> 00:03:18.000
Cloud has a Slack bot.

s45
00:03:18.000 --> 00:03:18.959
Coda has a Slack bot.

s46
00:03:18.959 --> 00:03:20.000
We have a Slack bot.

s47
00:03:20.000 --> 00:03:20.640
It's really cool.

s48
00:03:20.640 --> 00:03:22.280
You can say, "Hey, Coda bot, do XYZ."

s49
00:03:22.280 --> 00:03:22.720
It does it.

s50
00:03:22.720 --> 00:03:24.519
Somebody else can talk to it.

s51
00:03:24.519 --> 00:03:25.800
But it's not enough.

s52
00:03:25.800 --> 00:03:29.959
Cuz now we've taken it from trapped on somebody's laptop to kind of trapped in Slack.

s53
00:03:29.959 --> 00:03:36.560
And a lot of work happened in Slack, so that's better than than than nothing, but certainly not all work happens in Slack.

s54
00:03:36.560 --> 00:03:41.920
So, what we really wanted was to be able to work with the same session from every relevant interface.

s55
00:03:41.920 --> 00:03:45.600
Could be Slack, could be our app, could be GitHub, could be elsewhere.

s56
00:03:45.600 --> 00:03:51.040
And so, one possible flow is you start and collaborate on a session in Slack.

s57
00:03:51.040 --> 00:03:56.840
And then maybe you continue in a kind of more engineer-focused environment in the desktop app or the mobile app.

s58
00:03:56.840 --> 00:03:58.640
And then, you can finish it up in GitHub.

s59
00:03:58.640 --> 00:04:01.800
And the important thing here is the exact same agent session.

s60
00:04:01.800 --> 00:04:07.040
So, it's like the agent didn't forget what you did in one place in Slack when you go and talk to it from GitHub, it's the same session.

s61
00:04:07.040 --> 00:04:09.560
It's got the same context.

s62
00:04:10.480 --> 00:04:16.320
And the second lesson builds on top of that, which is to make the agent work visible and collaborative across the team.

s63
00:04:16.320 --> 00:04:18.880
And so, obviously Slack makes it more collaborative.

s64
00:04:18.880 --> 00:04:22.480
Um But here, we've got that kind of app view.

s65
00:04:22.480 --> 00:04:24.840
And you know, Sergey made this ticket.

s66
00:04:24.840 --> 00:04:26.640
I've been talking to the same ticket.

s67
00:04:26.640 --> 00:04:28.400
Our growth person hopped in as well.

s68
00:04:28.400 --> 00:04:31.440
So, you can kind of see at the top here all the different people that interacted with this.

s69
00:04:31.440 --> 00:04:35.200
So, I can see who's getting notified about this session, who's seen it.

s70
00:04:35.200 --> 00:04:39.600
Um that's especially important when you have work triggered by non-technical people.

s71
00:04:39.600 --> 00:04:39.800
Right?

s72
00:04:39.800 --> 00:04:42.320
So, it's like, you know, our customer support person created a ticket.

s73
00:04:42.320 --> 00:04:43.560
It's working really well.

s74
00:04:43.560 --> 00:04:46.160
I want to understand like has this been vetted by an engineer or not?

s75
00:04:46.160 --> 00:04:49.120
You can kind of see who's involved really easily.

s76
00:04:49.120 --> 00:04:52.680
And then, if I'm reviewing something, I can just pop in and say like, "Hey, why did you do it this way?"

s77
00:04:52.680 --> 00:04:59.760
And again, because it's the same agent session, I don't need to wait for Sergey to kind of get my notification on GitHub and respond to me.

s78
00:04:59.760 --> 00:05:03.840
The answer to the question is almost certainly in this thread.

s79
00:05:03.840 --> 00:05:05.360
I don't also don't want to read the entire thread.

s80
00:05:05.360 --> 00:05:08.120
So, I can just ask ask the agent.

s81
00:05:08.120 --> 00:05:14.080
Or how we most often kind of make the work visible is with artifacts.

s82
00:05:14.080 --> 00:05:21.360
So, it doesn't matter where the work started or where it's finishing, the agent can show you the work it's doing as screenshot or video or other.

s83
00:05:21.360 --> 00:05:23.160
And you can see it from everywhere.

s84
00:05:23.160 --> 00:05:25.480
So, again, you don't have to worry about like, "Oh, where is that thing?

s85
00:05:25.480 --> 00:05:28.240
I got to go to GitHub to see the image or got to go to Slack to see the image."

s86
00:05:28.240 --> 00:05:29.360
It's just everywhere.

s87
00:05:29.360 --> 00:05:30.720
Work is visible everywhere.

s88
00:05:30.720 --> 00:05:33.280
Collaborate from anywhere.

s89
00:05:35.120 --> 00:05:36.673
The third lesson

s90
00:05:36.673 --> 00:05:36.919
[clears throat]

s91
00:05:36.919 --> 00:05:44.760
um I'm going to talk about here is to turn every external signal into code that your team can quickly evaluate.

s92
00:05:44.760 --> 00:05:48.320
And I I to show this live, but the Wi-Fi is not quite there.

s93
00:05:48.320 --> 00:05:51.000
So, I'm going to show you something from yesterday.

s94
00:05:51.000 --> 00:05:53.400
But, what do I mean by external signal?

s95
00:05:53.400 --> 00:05:59.120
So, it could be Slack conversation, could be a meeting you have with a customer, an onboarding call, a sales call.

s96
00:05:59.120 --> 00:06:06.560
Could be an internal team meeting, could be something from Sentry or bug tracker, a bug report from a customer, an email, feature request.

s97
00:06:06.560 --> 00:06:08.960
And right now what's happening is like all that stuff is already exists.

s98
00:06:08.960 --> 00:06:11.440
It's in all those different systems.

s99
00:06:11.440 --> 00:06:13.040
People hook them together with MCPs.

s100
00:06:13.040 --> 00:06:16.400
So, now your coding agent can check the email or check Notion or whatever it might be.

s101
00:06:16.400 --> 00:06:19.560
But, like how do you how how does it know what to work on, right?

s102
00:06:19.560 --> 00:06:21.320
It's like it's still kind of stuck everywhere.

s103
00:06:21.320 --> 00:06:30.200
And so, like some humans are involved in like kind of taking stuff from one place and telling it solve email number 48 or ticket number 6,000.

s104
00:06:30.200 --> 00:06:32.919
But, that's still a lot of coordination.

s105
00:06:32.919 --> 00:06:41.280
And so, what we do is we we have several different ways to automatically ingest these signals, prioritize what to what to do with it and and kind of act on them.

s106
00:06:41.280 --> 00:06:45.600
And my favorite one, the most fun one, is what we call our meeting bot.

s107
00:06:45.600 --> 00:06:49.880
And so, I'm going to switch switch over to my browser here for a second.

s108
00:06:49.880 --> 00:06:52.680
And uh Okay.

s109
00:06:52.680 --> 00:06:57.080
So, we've got a booth at the expo and we had the meeting bot running all day yesterday.

s110
00:06:57.080 --> 00:06:59.520
So, this is a 4-hour meeting of a Google Meet.

s111
00:06:59.520 --> 00:07:04.080
You just kind of invite the bot to Meet or Zoom or Teams or whatever it might be.

s112
00:07:04.080 --> 00:07:06.720
And it listens all day.

s113
00:07:06.720 --> 00:07:10.240
And it created all sorts of stuff as it was listening.

s114
00:07:10.240 --> 00:07:12.520
If it finds existing work, it'll link to it.

s115
00:07:12.520 --> 00:07:12.680
All right?

s116
00:07:12.680 --> 00:07:15.800
So, it's not going to just like create new work if it's something already working on.

s117
00:07:15.800 --> 00:07:22.480
Some of this is, you know, people testing the meeting bot out and telling it to do some weird things, um or interesting things, or just creative ideas.

s118
00:07:22.480 --> 00:07:29.960
But, a lot of it's actually just like really good ideas that come out of people looking at what we're doing, asking questions, having new ideas of what to do with it.

s119
00:07:29.960 --> 00:07:36.800
And so, it's kind of nice cuz the last idea that was here was someone saying, "Hey, like when I work with coding agents, I want to make sure that

s120
00:07:36.800 --> 00:07:43.680
the agent has clear criteria to evaluate whether it did a good job on the work before it tells me that it's done.

s121
00:07:43.680 --> 00:07:47.360
And um so they had that idea, the bot just picked up on it.

s122
00:07:47.360 --> 00:07:49.320
None of us did anything manually.

s123
00:07:49.320 --> 00:07:54.000
It created this ticket and started working on it.

s124
00:07:54.000 --> 00:07:57.040
And then I was able to just say, "Hey, take a screenshot of what what you did."

s125
00:07:57.040 --> 00:07:58.920
And here's that screenshot.

s126
00:07:58.920 --> 00:08:03.280
And it kind of modified our ticket form to add these two new fields of acceptance criteria.

s127
00:08:03.280 --> 00:08:05.760
Now, I'm going to Am I going to ship this one exactly how it is?

s128
00:08:05.760 --> 00:08:06.880
Like, no, probably not.

s129
00:08:06.880 --> 00:08:14.280
But it's a new idea, it's concrete, I can play with it, I can go and actually like use the live preview and like see if this improves performance.

s130
00:08:14.280 --> 00:08:22.880
And so it takes like this you know, you know, your hundreds or thousands of ideas that are everywhere and it helps you kind of move with the speed of what your customers are asking you for and what they're thinking.

s131
00:08:22.880 --> 00:08:33.520
And um it's really fun because every time we have an onboarding your customer call or or team meeting, we almost always have dozens of new ideas that are prototyped,

s132
00:08:33.520 --> 00:08:38.240
but more importantly, at least a few shippable PRs with a very minimal intervention.

s133
00:08:38.240 --> 00:08:42.479
So we talk, stuff comes out, we look at it, we ship it.

s134
00:08:42.479 --> 00:08:44.919
It's so much fun.

s135
00:08:45.960 --> 00:08:48.160
Put this back.

s136
00:08:49.480 --> 00:09:00.960
So the next thing I'm going to mention is that, you know, the these three things that I've talked to you about really rely on having your workflow, your code base, your project

s137
00:09:00.960 --> 00:09:04.720
set up to work in an isolated cloud environment.

s138
00:09:04.720 --> 00:09:09.440
So that way the agents aren't trapped on an individual's machine.

s139
00:09:10.280 --> 00:09:12.760
So there's several reasons why this is important.

s140
00:09:12.760 --> 00:09:16.480
So the first one is to eliminate what some people are calling lid anxiety.

s141
00:09:16.480 --> 00:09:17.920
You want to be able to close your laptop.

s142
00:09:17.920 --> 00:09:26.440
You've probably seen people running around the office with their laptops open while stuff is working or at the airport or you know, there's some posts on Twitter or whatever about um you know, like people having their

s143
00:09:26.440 --> 00:09:29.640
laptop tethered to their phone in their cars as they're driving home.

s144
00:09:29.640 --> 00:09:35.280
This was actually probably the impetus for for me and for a few people on our team to even start working on this.

s145
00:09:35.280 --> 00:09:39.200
You You last year I started working on Cloud code with cloud code a lot.

s146
00:09:39.200 --> 00:09:41.440
I had a I think at the time like 6-month-old.

s147
00:09:41.440 --> 00:09:45.080
I like didn't want to like be tied to my laptop or have that stress.

s148
00:09:45.080 --> 00:09:50.680
I was like, I I don't I I don't ever want to think about whether I can like step away from a laptop or not.

s149
00:09:50.680 --> 00:09:52.720
And so, we moved everything to the cloud.

s150
00:09:52.720 --> 00:09:54.840
Things are working always.

s151
00:09:54.840 --> 00:09:59.880
It eliminated that problem for us that people have been talking about for the past year.

s152
00:09:59.920 --> 00:10:00.680
It's really helpful.

s153
00:10:00.680 --> 00:10:02.200
It's important to me.

s154
00:10:02.200 --> 00:10:06.520
But I don't think that's the most important reason to do this.

s155
00:10:06.520 --> 00:10:12.400
I think the most important reason to do this actually was was touched on in the previous talk, if you were here for it.

s156
00:10:12.440 --> 00:10:17.800
I think you should only give your access give your agents access to only what they need.

s157
00:10:17.800 --> 00:10:17.960
Right?

s158
00:10:17.960 --> 00:10:23.840
So, if you think about what's happening, you have a bunch of developers with these agents running on their laptop.

s159
00:10:23.840 --> 00:10:31.360
Their laptops, unless you have like impeccable hygiene, probably have a bunch of stuff on it that you don't want the LLMs or agents to have access to.

s160
00:10:31.360 --> 00:10:34.800
And yeah, like everybody's working on these sandboxes and approval flows.

s161
00:10:34.800 --> 00:10:37.720
And so, you but but but really you're in one of two camps.

s162
00:10:37.720 --> 00:10:46.080
You're either approving a bunch of stuff or you're hoping that your auto approval flow or your YOLO mode or whatever is configured properly and your sandbox is configured properly

s163
00:10:46.080 --> 00:10:50.040
and doesn't read a bunch of stuff on your laptop that it shouldn't have.

s164
00:10:50.040 --> 00:10:53.520
And, you know, as as the previous talk mentioned, like these agents are getting more autonomous.

s165
00:10:53.520 --> 00:10:54.800
They're getting really resourceful.

s166
00:10:54.800 --> 00:10:57.560
They They're trying to please you and do what you said.

s167
00:10:57.560 --> 00:11:00.120
And so, when you say, "Hey, you know, wipe this agent database."

s168
00:11:00.120 --> 00:11:07.280
and it finds a token on your laptop that it can use and it thinks it's working with staging, but actually it's production and now it just deleted everything.

s169
00:11:07.280 --> 00:11:10.600
I'm not trying to say this is happening constantly, but it still happens.

s170
00:11:10.600 --> 00:11:23.760
And for us, the peace of mind of just like letting anybody run with these experiments and ideas and prototypes and and and real code without having to worry about this is is is is really worthwhile.

s171
00:11:24.200 --> 00:11:31.880
To go one step further on that, um it's not just, "Hey, make sure they don't have the credentials that they shouldn't have."

s172
00:11:31.880 --> 00:11:38.520
It's also make sure they can't exfiltrate your code or your projects or your secrets or your content to somewhere they shouldn't be able to.

s173
00:11:38.520 --> 00:11:44.760
And so you have a configurable network sandbox and you say, "Look, these are the places you're allowed to access, these are the ones you can't access."

s174
00:11:44.760 --> 00:11:49.680
And anytime it tries to access something that it shouldn't, it just pops up and says, "Hey, tried to access something.

s175
00:11:49.680 --> 00:11:50.600
Do you want to give it access?

s176
00:11:50.600 --> 00:11:54.000
Maybe you're trying to integrate a new vendor and you need documentation."

s177
00:11:54.000 --> 00:11:57.200
And you can do it on a per ticket basis or for the whole project.

s178
00:11:57.200 --> 00:12:00.840
And so again, that peace of mind of like people can do things.

s179
00:12:00.840 --> 00:12:03.400
If they need new access, it's easy to grant it.

s180
00:12:03.400 --> 00:12:12.600
And um we're not going to leak a bunch of important data uh by uh running agents in yellow mode.

s181
00:12:13.760 --> 00:12:21.320
And the last thing I'll mention about that is that this is the key for allowing your non-technical team members to trigger real work.

s182
00:12:21.320 --> 00:12:21.880
Right?

s183
00:12:21.880 --> 00:12:26.240
Your non-technical people don't have development environments set up on their computers.

s184
00:12:26.440 --> 00:12:40.040
But we've gotten our support people or growth people to actually be meaningfully impacting the product by just talking to the users, seeing bugs, experiencing themselves, and just go to Slack or the rabbit cell and say, "Hey, fix this."

s185
00:12:40.040 --> 00:12:41.160
They fix it.

s186
00:12:41.160 --> 00:12:42.280
Screenshots are shown.

s187
00:12:42.280 --> 00:12:45.440
Engineer gets it, gets merged.

s188
00:12:45.440 --> 00:12:50.880
Without that, they'd have to put it in linear and linear would eventually pick it up and a PM would triage it or whatever.

s189
00:12:50.880 --> 00:12:51.440
None of that here.

s190
00:12:51.440 --> 00:12:54.720
You just ask for it and it's done.

s191
00:12:55.200 --> 00:13:00.160
Now, the reason people didn't do this you know, up until somewhat recently, like this was really painful.

s192
00:13:00.160 --> 00:13:05.560
Getting your full thing set up in this like kind of sandbox environment used to be really, really painful.

s193
00:13:05.560 --> 00:13:07.280
But agents have gotten better.

s194
00:13:07.280 --> 00:13:13.120
We have our own environment setup assistant that kind of takes your project and gets it to work in one of these sandboxes.

s195
00:13:13.120 --> 00:13:19.440
But honestly, whether you use this or not, I highly recommend you get your project working this way and you can just get Cloud Coder or CodeX to do this for you.

s196
00:13:19.440 --> 00:13:23.640
You don't have to use us, but you know, we think it's the best way.

s197
00:13:25.240 --> 00:13:29.720
And the last lesson is to benchmark agents on your code base.

s198
00:13:29.720 --> 00:13:35.040
So, um the way we do this, we select pull requests that represent great engineering work.

s199
00:13:35.040 --> 00:13:38.920
It could be agent created, could be human created, could be a hybrid, doesn't matter.

s200
00:13:38.920 --> 00:13:41.360
You pick the agents you want to use and benchmark.

s201
00:13:41.360 --> 00:13:46.560
And then you get a quality versus cost and time breakdown on your code base.

s202
00:13:46.560 --> 00:13:47.560
Now, why do you want to do this?

s203
00:13:47.560 --> 00:13:55.720
There's There's many reasons, but one is that if you're kind of going off the public benchmarks, we bench or terminal bench or other stuff like those tasks may have absolutely nothing to do with your task.

s204
00:13:55.720 --> 00:13:58.720
Like swe bench is all in Python, we're Ruby on Rails.

s205
00:13:58.720 --> 00:14:00.880
It is not the case that the benchmarks are identical for them.

s206
00:14:00.880 --> 00:14:05.520
There's trends that do compare, but the results can be very, very different.

s207
00:14:05.520 --> 00:14:09.480
And I'm going to swap over to my browser one more time here.

s208
00:14:10.720 --> 00:14:16.400
So, this is These are results on our code base of all these different harnesses.

s209
00:14:16.400 --> 00:14:19.320
This one here is quality versus cost.

s210
00:14:19.320 --> 00:14:21.000
This one here is quality versus time.

s211
00:14:21.000 --> 00:14:23.440
I'll start with this one.

s212
00:14:23.440 --> 00:14:24.880
You can see some trends here.

s213
00:14:24.880 --> 00:14:31.480
Right, you can see that the Anthropic agents have just been consistently getting better, but not really any faster.

s214
00:14:31.480 --> 00:14:35.200
The Codex agents and cursor are actually pretty fast and quite good.

s215
00:14:35.200 --> 00:14:38.680
The open stuff has been getting better and better over time, but they're kind of slow.

s216
00:14:38.680 --> 00:14:39.760
This is for our code base again.

s217
00:14:39.760 --> 00:14:43.080
I'm not trying to make any general claims here.

s218
00:14:43.160 --> 00:14:48.480
By cost, the Anthropic stuff is clearly just so much more expensive for us.

s219
00:14:48.480 --> 00:14:51.160
And the Codex stuff has been cheaper for us.

s220
00:14:51.160 --> 00:14:53.040
And so this causes us to change our behavior.

s221
00:14:53.040 --> 00:14:54.640
We still use the different models.

s222
00:14:54.640 --> 00:14:55.760
There's different use cases for them.

s223
00:14:55.760 --> 00:14:56.520
We like the variety.

s224
00:14:56.520 --> 00:14:57.839
We still use all these things.

s225
00:14:57.839 --> 00:15:01.760
But when we saw these results, they kind of matched our vibe check.

s226
00:15:01.760 --> 00:15:03.920
We wanted to kind of like have hard data, too.

s227
00:15:03.920 --> 00:15:06.720
We switched our default to Codex at that time.

s228
00:15:06.720 --> 00:15:09.320
And Fiable came out, and it was great.

s229
00:15:09.320 --> 00:15:13.800
Kind of switched our default to that for like the few days we had it, and then it went away and switched back to Codex.

s230
00:15:13.800 --> 00:15:18.240
But the most important thing is like because we're agnostic, like none of that had any meaningful disruption on our work.

s231
00:15:18.240 --> 00:15:20.600
Like we're able to just kind of switch back and forth really easily.

s232
00:15:20.600 --> 00:15:26.800
So, the next day something new comes out, see if it's good, and go.

s233
00:15:26.800 --> 00:15:34.200
And the last thing I want to mention around that is like I don't know if this resonates with you all, but I have a lot of friends that like, "Okay, and I heard MiniMax is good.

s234
00:15:34.200 --> 00:15:41.040
I heard, you know, GLM is good, and Kimmy K2 is good, but like haven't had the time to try it out, and everyone keeps telling me I need to cuz it's so much better and faster and cheaper."

s235
00:15:41.040 --> 00:15:47.920
And you kind of have that anxiety for a little while, and then like finally you take the two hours to try it, and it's like, "Oh, actually like didn't really work for us.

s236
00:15:47.920 --> 00:15:50.200
So, like what's, you know, I just wasted those two hours."

s237
00:15:50.200 --> 00:15:51.120
It kind of eliminates that.

s238
00:15:51.120 --> 00:15:56.160
It helps you kind of stay on the cutting edge really like seamlessly.

s239
00:15:57.280 --> 00:15:59.400
Let me go back.

s240
00:15:59.400 --> 00:16:07.800
So, what that kind of turned into us for us is, you know, essentially 100% like 99.9% of our pull requests are like heavily agent generated.

s241
00:16:07.800 --> 00:16:13.240
We know that quality and reliability and security are really important, so we still have humans look at everything.

s242
00:16:13.240 --> 00:16:16.120
We have agents help with it all, but everything's human reviewed.

s243
00:16:16.120 --> 00:16:22.360
Um you know, for our our our our relatively small team, we had 1 and 1/2 billion tokens over the past month.

s244
00:16:22.360 --> 00:16:24.440
And you can kind of see what we're saying about Claude here.

s245
00:16:24.440 --> 00:16:30.800
It's a little small, so I apologize, but we had 3,300 Claude code runs that cost $10,000 in tokens daily.

s246
00:16:30.800 --> 00:16:34.400
We have plans, so we didn't spend $10,000 on it.

s247
00:16:34.400 --> 00:16:39.400
And Codex had four times as many sessions, and it was cheaper overall.

s248
00:16:39.400 --> 00:16:43.480
And so again, the vast majority of our work currently is merged through Codex.

s249
00:16:43.480 --> 00:16:45.680
We still use the other models.

s250
00:16:45.680 --> 00:16:50.200
More and more is happening through GLM 5.2. Going to invest in that.

s251
00:16:50.200 --> 00:17:00.160
And the one thing that we're really excited to do going forward with this benchmarking is automatically like like you've probably heard about people, you know, routing tasks to the right models and all that,

s252
00:17:00.160 --> 00:17:03.640
but how do how does like some third party know what to route for your code base?

s253
00:17:03.640 --> 00:17:12.560
Like this is a way that you can know what's going to work best for which task for your for your project, and we're going to kind of automatically routing that for you.

s254
00:17:12.880 --> 00:17:15.760
So, I'm going to leave you with a few recommendations.

s255
00:17:15.839 --> 00:17:19.400
So, first, um get your code base and agents working in a sandbox.

s256
00:17:19.400 --> 00:17:24.760
It unlocks a lot of different things, a lot of different workflows, everything I've talked about and more.

s257
00:17:24.760 --> 00:17:36.120
Second, integrate agents into the relevant human interfaces so your team and your agents can work together and don't have to like context switch and copy context back and forth.

s258
00:17:36.120 --> 00:17:42.680
Obviously, we think Superagent is the best way to do it, but plenty of people are home rolling things, hacking things together.

s259
00:17:42.680 --> 00:17:46.880
Figure out how to make this happen because if not, the friction is just really high.

s260
00:17:46.880 --> 00:17:56.240
And lastly, find a way to benchmark and become model agnostic so you're not tied to anybody and you can just constantly stay at that right part on the frontier of cost, speed,

s261
00:17:56.240 --> 00:17:58.240
quality.

s262
00:17:58.880 --> 00:17:59.960
So, thank you so much.

s263
00:17:59.960 --> 00:18:01.800
Um, we've got a booth in the expo.

s264
00:18:01.800 --> 00:18:02.800
Please feel free to come by.

s265
00:18:02.800 --> 00:18:08.000
You can sign up at superagent.com or you can email me with any questions at arjun@superagent.com.

s266
00:18:08.120 --> 00:18:10.920
Um, I will be out in the back as well for any questions.

s267
00:18:10.920 --> 00:18:12.552
Thanks so much.

s268
00:18:12.552 --> 00:18:14.552
[applause]

s269
00:18:16.760 --> 00:18:22.240
One one one last thing, um, if, uh, you know, at the booth we're mentioning we're giving away a MacBook Neo.

s270
00:18:22.240 --> 00:18:26.520
If you were here cuz you signed up through for that, just meet us outside and we will announce the winner.

s271
00:18:26.520 --> 00:18:28.680
Thank you.