Multiplayer agentic engineering — Arjun Singh, Superconductor

AI Engineer · 18 min · 271 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12All right.
00:13Hey everyone, I'm Arjun Singh.
00:15Today I'm going to talk to you about multiplayer agentic engineering or how to enable your whole team and your best agents to work together.
00:22If you go to the talks or go around the expo, you're going to see that a lot of people are talking about putting the agents at the center of everything.
00:27Makes sense, they're really powerful, they're really cool.
00:29But you don't see a lot of people talking about the people.
00:32Like this is all for us to make us our lives better, our more our more productive, whatever.
00:36And so we're going to really focus on how the people fit into these agentic workflows.
00:41Just a little bit about us first.
00:44So our team has worked together building software for over a decade.
00:48Um my co-founder Sergey and I, we met in the PhD program at Berkeley.
00:51I worked on robotics, he worked on computer vision.
00:54And during that we co-founded a company called GradeScope.
00:56Some of you may have used it.
00:58It's used by millions of students worldwide at thousands of universities, helping instructors grade their students' work.
01:04And um pretty much the entire team working on Superconductor used to work together on GradeScope.
01:09And so we've had a team that's worked together productively from first user to acquisition, working on something new together again.
01:15And I think it's kind of an interesting experiment because you know, over the past year we've very aggressively integrated agents in our workflows and we've kind of surfaced all the different bottlenecks and friction points that come up and how to do that productively and keep collaborating the way we used to but with the new power of agents.
01:31So today I'm going to talk to you about how, you know, we we the lessons we learned from kind of solving those friction points and solving those bottlenecks.
01:38And in the talk description I mentioned five lessons and I'm going to be an engineer and start from zero and add a sixth one in there.
01:44Um the first one I'm going to start with is just to be model and harness agnostic.
01:49So there's a few reasons for that.
01:50The best model and harness can change weekly.
01:53It could change cuz a new one comes out.
01:55It could change cuz the best one got taken away.
01:58Um things happen and you don't want that to disrupt your entire team's flow.
02:03The other thing is that open weight models are actually pretty good now.
02:06We've been really happy with GLM 5.2. They're much cheaper.
02:10Um and you want to be able to kind of explore with them and integrate them without again having to change your entire workflow.
02:16The last thing I'll mention on this is that the incentives of the people selling you tokens aren't really aligned with yours.
02:23You're here for a reason.
02:24You're you're you're So you're doing things for a reason.
02:25You're trying to make your customers' lives better, make your product better, delight your customers.
02:29They want to sell you more tokens.
02:31And you might be happy to pay for as many tokens as it takes, but you don't want to pay for more than that.
02:36And so again, kind of being able to switch between things lets you stay in control of all of that.
02:43And so, um you know, as I go through the talk, I'll mention a couple of places where our product makes it easy for us, but whether you use us or not, I'm just going to leave things with you that I think are really important
02:53um for you to be able to work collaboratively effectively.
02:57So, the next one is to turn every human interface into an agent and human interface.
03:03So, you know, typically when people are working with coding agents, they're on their laptop, kind of stuck on that laptop.
03:09Nobody else can talk to that agent.
03:12So, the first place people go to kind of expose more interfaces for them is Slack.
03:16Cloud has a Slack bot.
03:18Coda has a Slack bot.
03:18We have a Slack bot.
03:20It's really cool.
03:20You can say, "Hey, Coda bot, do XYZ."
03:22It does it.
03:22Somebody else can talk to it.
03:24But it's not enough.
03:25Cuz now we've taken it from trapped on somebody's laptop to kind of trapped in Slack.
03:29And a lot of work happened in Slack, so that's better than than than nothing, but certainly not all work happens in Slack.
03:36So, what we really wanted was to be able to work with the same session from every relevant interface.
03:41Could be Slack, could be our app, could be GitHub, could be elsewhere.
03:45And so, one possible flow is you start and collaborate on a session in Slack.
03:51And then maybe you continue in a kind of more engineer-focused environment in the desktop app or the mobile app.
03:56And then, you can finish it up in GitHub.
03:58And the important thing here is the exact same agent session.
04:01So, it's like the agent didn't forget what you did in one place in Slack when you go and talk to it from GitHub, it's the same session.
04:07It's got the same context.
04:10And the second lesson builds on top of that, which is to make the agent work visible and collaborative across the team.
04:16And so, obviously Slack makes it more collaborative.
04:18Um But here, we've got that kind of app view.
04:22And you know, Sergey made this ticket.
04:24I've been talking to the same ticket.
04:26Our growth person hopped in as well.
04:28So, you can kind of see at the top here all the different people that interacted with this.
04:31So, I can see who's getting notified about this session, who's seen it.
04:35Um that's especially important when you have work triggered by non-technical people.
04:39Right?
04:39So, it's like, you know, our customer support person created a ticket.
04:42It's working really well.
04:43I want to understand like has this been vetted by an engineer or not?
04:46You can kind of see who's involved really easily.
04:49And then, if I'm reviewing something, I can just pop in and say like, "Hey, why did you do it this way?"
04:52And again, because it's the same agent session, I don't need to wait for Sergey to kind of get my notification on GitHub and respond to me.
04:59The answer to the question is almost certainly in this thread.
05:03I don't also don't want to read the entire thread.
05:05So, I can just ask ask the agent.
05:08Or how we most often kind of make the work visible is with artifacts.
05:14So, it doesn't matter where the work started or where it's finishing, the agent can show you the work it's doing as screenshot or video or other.
05:21And you can see it from everywhere.
05:23So, again, you don't have to worry about like, "Oh, where is that thing?
05:25I got to go to GitHub to see the image or got to go to Slack to see the image."
05:28It's just everywhere.
05:29Work is visible everywhere.
05:30Collaborate from anywhere.
05:35The third lesson
05:36[clears throat]
05:36um I'm going to talk about here is to turn every external signal into code that your team can quickly evaluate.
05:44And I I to show this live, but the Wi-Fi is not quite there.
05:48So, I'm going to show you something from yesterday.
05:51But, what do I mean by external signal?
05:53So, it could be Slack conversation, could be a meeting you have with a customer, an onboarding call, a sales call.
05:59Could be an internal team meeting, could be something from Sentry or bug tracker, a bug report from a customer, an email, feature request.
06:06And right now what's happening is like all that stuff is already exists.
06:08It's in all those different systems.
06:11People hook them together with MCPs.
06:13So, now your coding agent can check the email or check Notion or whatever it might be.
06:16But, like how do you how how does it know what to work on, right?
06:19It's like it's still kind of stuck everywhere.
06:21And so, like some humans are involved in like kind of taking stuff from one place and telling it solve email number 48 or ticket number 6,000.
06:30But, that's still a lot of coordination.
06:32And so, what we do is we we have several different ways to automatically ingest these signals, prioritize what to what to do with it and and kind of act on them.
06:41And my favorite one, the most fun one, is what we call our meeting bot.
06:45And so, I'm going to switch switch over to my browser here for a second.
06:49And uh Okay.
06:52So, we've got a booth at the expo and we had the meeting bot running all day yesterday.
06:57So, this is a 4-hour meeting of a Google Meet.
06:59You just kind of invite the bot to Meet or Zoom or Teams or whatever it might be.
07:04And it listens all day.
07:06And it created all sorts of stuff as it was listening.
07:10If it finds existing work, it'll link to it.
07:12All right?
07:12So, it's not going to just like create new work if it's something already working on.
07:15Some of this is, you know, people testing the meeting bot out and telling it to do some weird things, um or interesting things, or just creative ideas.
07:22But, a lot of it's actually just like really good ideas that come out of people looking at what we're doing, asking questions, having new ideas of what to do with it.
07:29And so, it's kind of nice cuz the last idea that was here was someone saying, "Hey, like when I work with coding agents, I want to make sure that
07:36the agent has clear criteria to evaluate whether it did a good job on the work before it tells me that it's done.
07:43And um so they had that idea, the bot just picked up on it.
07:47None of us did anything manually.
07:49It created this ticket and started working on it.
07:54And then I was able to just say, "Hey, take a screenshot of what what you did."
07:57And here's that screenshot.
07:58And it kind of modified our ticket form to add these two new fields of acceptance criteria.
08:03Now, I'm going to Am I going to ship this one exactly how it is?
08:05Like, no, probably not.
08:06But it's a new idea, it's concrete, I can play with it, I can go and actually like use the live preview and like see if this improves performance.
08:14And so it takes like this you know, you know, your hundreds or thousands of ideas that are everywhere and it helps you kind of move with the speed of what your customers are asking you for and what they're thinking.
08:22And um it's really fun because every time we have an onboarding your customer call or or team meeting, we almost always have dozens of new ideas that are prototyped,
08:33but more importantly, at least a few shippable PRs with a very minimal intervention.
08:38So we talk, stuff comes out, we look at it, we ship it.
08:42It's so much fun.
08:45Put this back.
08:49So the next thing I'm going to mention is that, you know, the these three things that I've talked to you about really rely on having your workflow, your code base, your project
09:00set up to work in an isolated cloud environment.
09:04So that way the agents aren't trapped on an individual's machine.
09:10So there's several reasons why this is important.
09:12So the first one is to eliminate what some people are calling lid anxiety.
09:16You want to be able to close your laptop.
09:17You've probably seen people running around the office with their laptops open while stuff is working or at the airport or you know, there's some posts on Twitter or whatever about um you know, like people having their
09:26laptop tethered to their phone in their cars as they're driving home.
09:29This was actually probably the impetus for for me and for a few people on our team to even start working on this.
09:35You You last year I started working on Cloud code with cloud code a lot.
09:39I had a I think at the time like 6-month-old.
09:41I like didn't want to like be tied to my laptop or have that stress.
09:45I was like, I I don't I I don't ever want to think about whether I can like step away from a laptop or not.
09:50And so, we moved everything to the cloud.
09:52Things are working always.
09:54It eliminated that problem for us that people have been talking about for the past year.
09:59It's really helpful.
10:00It's important to me.
10:02But I don't think that's the most important reason to do this.
10:06I think the most important reason to do this actually was was touched on in the previous talk, if you were here for it.
10:12I think you should only give your access give your agents access to only what they need.
10:17Right?
10:17So, if you think about what's happening, you have a bunch of developers with these agents running on their laptop.
10:23Their laptops, unless you have like impeccable hygiene, probably have a bunch of stuff on it that you don't want the LLMs or agents to have access to.
10:31And yeah, like everybody's working on these sandboxes and approval flows.
10:34And so, you but but but really you're in one of two camps.
10:37You're either approving a bunch of stuff or you're hoping that your auto approval flow or your YOLO mode or whatever is configured properly and your sandbox is configured properly
10:46and doesn't read a bunch of stuff on your laptop that it shouldn't have.
10:50And, you know, as as the previous talk mentioned, like these agents are getting more autonomous.
10:53They're getting really resourceful.
10:54They They're trying to please you and do what you said.
10:57And so, when you say, "Hey, you know, wipe this agent database."
11:00and it finds a token on your laptop that it can use and it thinks it's working with staging, but actually it's production and now it just deleted everything.
11:07I'm not trying to say this is happening constantly, but it still happens.
11:10And for us, the peace of mind of just like letting anybody run with these experiments and ideas and prototypes and and and real code without having to worry about this is is is is really worthwhile.
11:24To go one step further on that, um it's not just, "Hey, make sure they don't have the credentials that they shouldn't have."
11:31It's also make sure they can't exfiltrate your code or your projects or your secrets or your content to somewhere they shouldn't be able to.
11:38And so you have a configurable network sandbox and you say, "Look, these are the places you're allowed to access, these are the ones you can't access."
11:44And anytime it tries to access something that it shouldn't, it just pops up and says, "Hey, tried to access something.
11:49Do you want to give it access?
11:50Maybe you're trying to integrate a new vendor and you need documentation."
11:54And you can do it on a per ticket basis or for the whole project.
11:57And so again, that peace of mind of like people can do things.
12:00If they need new access, it's easy to grant it.
12:03And um we're not going to leak a bunch of important data uh by uh running agents in yellow mode.
12:13And the last thing I'll mention about that is that this is the key for allowing your non-technical team members to trigger real work.
12:21Right?
12:21Your non-technical people don't have development environments set up on their computers.
12:26But we've gotten our support people or growth people to actually be meaningfully impacting the product by just talking to the users, seeing bugs, experiencing themselves, and just go to Slack or the rabbit cell and say, "Hey, fix this."
12:40They fix it.
12:41Screenshots are shown.
12:42Engineer gets it, gets merged.
12:45Without that, they'd have to put it in linear and linear would eventually pick it up and a PM would triage it or whatever.
12:50None of that here.
12:51You just ask for it and it's done.
12:55Now, the reason people didn't do this you know, up until somewhat recently, like this was really painful.
13:00Getting your full thing set up in this like kind of sandbox environment used to be really, really painful.
13:05But agents have gotten better.
13:07We have our own environment setup assistant that kind of takes your project and gets it to work in one of these sandboxes.
13:13But honestly, whether you use this or not, I highly recommend you get your project working this way and you can just get Cloud Coder or CodeX to do this for you.
13:19You don't have to use us, but you know, we think it's the best way.
13:25And the last lesson is to benchmark agents on your code base.
13:29So, um the way we do this, we select pull requests that represent great engineering work.
13:35It could be agent created, could be human created, could be a hybrid, doesn't matter.
13:38You pick the agents you want to use and benchmark.
13:41And then you get a quality versus cost and time breakdown on your code base.
13:46Now, why do you want to do this?
13:47There's There's many reasons, but one is that if you're kind of going off the public benchmarks, we bench or terminal bench or other stuff like those tasks may have absolutely nothing to do with your task.
13:55Like swe bench is all in Python, we're Ruby on Rails.
13:58It is not the case that the benchmarks are identical for them.
14:00There's trends that do compare, but the results can be very, very different.
14:05And I'm going to swap over to my browser one more time here.
14:10So, this is These are results on our code base of all these different harnesses.
14:16This one here is quality versus cost.
14:19This one here is quality versus time.
14:21I'll start with this one.
14:23You can see some trends here.
14:24Right, you can see that the Anthropic agents have just been consistently getting better, but not really any faster.
14:31The Codex agents and cursor are actually pretty fast and quite good.
14:35The open stuff has been getting better and better over time, but they're kind of slow.
14:38This is for our code base again.
14:39I'm not trying to make any general claims here.
14:43By cost, the Anthropic stuff is clearly just so much more expensive for us.
14:48And the Codex stuff has been cheaper for us.
14:51And so this causes us to change our behavior.
14:53We still use the different models.
14:54There's different use cases for them.
14:55We like the variety.
14:56We still use all these things.
14:57But when we saw these results, they kind of matched our vibe check.
15:01We wanted to kind of like have hard data, too.
15:03We switched our default to Codex at that time.
15:06And Fiable came out, and it was great.
15:09Kind of switched our default to that for like the few days we had it, and then it went away and switched back to Codex.
15:13But the most important thing is like because we're agnostic, like none of that had any meaningful disruption on our work.
15:18Like we're able to just kind of switch back and forth really easily.
15:20So, the next day something new comes out, see if it's good, and go.
15:26And the last thing I want to mention around that is like I don't know if this resonates with you all, but I have a lot of friends that like, "Okay, and I heard MiniMax is good.
15:34I heard, you know, GLM is good, and Kimmy K2 is good, but like haven't had the time to try it out, and everyone keeps telling me I need to cuz it's so much better and faster and cheaper."
15:41And you kind of have that anxiety for a little while, and then like finally you take the two hours to try it, and it's like, "Oh, actually like didn't really work for us.
15:47So, like what's, you know, I just wasted those two hours."
15:50It kind of eliminates that.
15:51It helps you kind of stay on the cutting edge really like seamlessly.
15:57Let me go back.
15:59So, what that kind of turned into us for us is, you know, essentially 100% like 99.9% of our pull requests are like heavily agent generated.
16:07We know that quality and reliability and security are really important, so we still have humans look at everything.
16:13We have agents help with it all, but everything's human reviewed.
16:16Um you know, for our our our our relatively small team, we had 1 and 1/2 billion tokens over the past month.
16:22And you can kind of see what we're saying about Claude here.
16:24It's a little small, so I apologize, but we had 3,300 Claude code runs that cost $10,000 in tokens daily.
16:30We have plans, so we didn't spend $10,000 on it.
16:34And Codex had four times as many sessions, and it was cheaper overall.
16:39And so again, the vast majority of our work currently is merged through Codex.
16:43We still use the other models.
16:45More and more is happening through GLM 5.2. Going to invest in that.
16:50And the one thing that we're really excited to do going forward with this benchmarking is automatically like like you've probably heard about people, you know, routing tasks to the right models and all that,
17:00but how do how does like some third party know what to route for your code base?
17:03Like this is a way that you can know what's going to work best for which task for your for your project, and we're going to kind of automatically routing that for you.
17:12So, I'm going to leave you with a few recommendations.
17:15So, first, um get your code base and agents working in a sandbox.
17:19It unlocks a lot of different things, a lot of different workflows, everything I've talked about and more.
17:24Second, integrate agents into the relevant human interfaces so your team and your agents can work together and don't have to like context switch and copy context back and forth.
17:36Obviously, we think Superagent is the best way to do it, but plenty of people are home rolling things, hacking things together.
17:42Figure out how to make this happen because if not, the friction is just really high.
17:46And lastly, find a way to benchmark and become model agnostic so you're not tied to anybody and you can just constantly stay at that right part on the frontier of cost, speed,
17:56quality.
17:58So, thank you so much.
17:59Um, we've got a booth in the expo.
18:01Please feel free to come by.
18:02You can sign up at superagent.com or you can email me with any questions at arjun@superagent.com.
18:08Um, I will be out in the back as well for any questions.
18:10Thanks so much.
18:12[applause]
18:16One one one last thing, um, if, uh, you know, at the booth we're mentioning we're giving away a MacBook Neo.
18:22If you were here cuz you signed up through for that, just meet us outside and we will announce the winner.
18:26Thank you.