Realtime multiplayer, automation, and you! — Idan Gazit, GitHub

AI Engineer · 21 min · 290 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:13going to start 1 minute early, which gives me one extra minute.
00:17And then anybody who came on time is going to miss the super enthralling introduction.
00:22Hi, my name is Eitan.
00:23Nice to meet you all.
00:25Uh I lead GitHub Next, which is the Labs team of GitHub.
00:29I like to call us the Department of Fool Around and Find Out, but I usually don't say the word fool.
00:36We're the team that created Copilot uh and pioneered a ton of areas since then, right?
00:40Uh spec-based programming, natural language to app, lots more.
00:44Not everything uh that we do turns into a finished product.
00:47Our job is to sort of explore the future and scout it out.
00:51Um but our job is to reach for the GitHub that's going to be next year.
00:56Maybe not tomorrow's GitHub, but uh the tools that we're all going to use to make software a year from now, 2 years from now.
01:03That's pretty hard, cuz my crystal ball barely works into uh next week.
01:07Uh and we're really fortunate that we get to do most of our work in the open, so you can check out githubnext.com and our socials, which we occasionally remember to post
01:17stuff to.
01:18And what we do isn't really research, right?
01:20Because the only way to know what's going to be good uh is to make stuff.
01:25So, we make a lot of stuff, and the hard part about being an undirected research team is always the question of what's worth our time.
01:34Even if you're a token billionaire, uh even if you have 10 terminals running Fable night and day, then opportunity cost is is still there.
01:42It's everything.
01:43Uh so, if in a in a world where the uh marginal cost of a line of code is approaching zero, uh and AI can help us to think and to make,
01:54what do we make, right?
01:56How do we even choose what's important uh when the market is super noisy and the tech changes every week?
02:03And this isn't even really a next problem anymore.
02:06This is an all of us problem now.
02:08We're all labs teams now.
02:10And the way that next thinks about this stuff is to look for durable themes.
02:14Things that will be true no matter what the technology that exists tomorrow.
02:20And I think that the theme of this moment is very much an evergreen one, right?
02:26It's AI started with a surge of personal productivity, right?
02:30The LLMs completed what I type and the agents go fetch me the thing that I need.
02:35And now I have many agents helping me to parallelize myself.
02:39But the greatest value doesn't come from multiplying me into more me.
02:45It comes from enabling groups of people to do more.
02:47That's always been true.
02:50And we're thinking about how to accomplish that through two lenses.
02:53Every industrial revolution came about through automation, right?
02:57It's funny to think about our giant software industry as being pre-industrial, but on some level it is because until now the only automations that we had were heuristics like make sure there's a semicolon at the end of every line.
03:11But now AI can help us to automate things that require some amount of basic judgment and intelligence.
03:17And there's no magic trick to making great software, right?
03:19It costs time.
03:22And we can buy that time by automating away the things that we used to need to do manually.
03:27Like the more we automate, the more time we have to spend on craft or on our product or on making it really good or on features, right?
03:35Either you hire more people or you automate away part of what your people are currently doing in order to spend that time.
03:41And at the same time, how are we going to work together, right?
03:44How does collaboration look like in the future?
03:47Whoops.
03:48Oh well, sorry about that.
03:50Yesterday uh, Jeffrey Lit talked about understanding being the bottleneck, and that's very true at a me level.
03:56Uh, but my personal understanding was never sufficient for shipping code inside a team, right?
04:01Our understanding at an us level can only happen at the end of the process.
04:07Um, sorry.
04:08Uh, uh, I it can't only happen at the end of the process, uh, when the process happens so much faster.
04:15So, going faster means that a small misalignment, uh, can snowball into a ton of wasted work, uh, and that work costs tokens, and tokens cost real money now, so,
04:27uh, on top of the time that you're mis spending.
04:30So, today I'll give you a quick tour of two prototypes that we're working on at GitHub Next in each of these themes.
04:35Agentic Workflows is our take Why is that not there?
04:38Oh, I had to click again.
04:40Uh, Agentic Workflows is our take on how automations should work in an agentic world, and Aces a prototype that explores what real-time multiplayer software development looks like.
04:50So, I'll start by showing off Agentic Workflows, and it requires me doing this.
04:54Okay, cool.
04:55Uh, this is my personal website, not that interesting.
04:57I'm showing it to you.
04:59This is like Chekhov's gun, we're going to see it again later.
05:01Um, and my personal website is built with this framework called Astro.
05:04Astro is a great web framework.
05:06The greatest part about it is that they release like 50 things a month, which means that I'm constantly on the upgrade treadmill, and there's a great GitHub product called Dependabot, which notifies me when my stuff is out of date.
05:17Um, but the problem is is that when I do these upgrades, I frequently need to make code changes.
05:23So, what I really want is a kind of super Dependabot that's always there, automatically looking in the background at my dependencies and figuring out how to upgrade me, including the code changes, the breaking changes.
05:33Um, and because I'm lazy, and I like not doing work, um, I used Copilot, uh, to create an agentic workflow, and there's this magic line up top where I supply effectively a skill saying like, "Hey, create an agentic workflow.
05:47Here's a document that tells you everything you need to know about that.
05:50And then what comes below that is something a lot like a Slack message that I'd send to a junior developer on my team.
05:57Like, every day I want you to check if there's a new release, look at the change log, look at the docs, come up with a plan for the upgrade,
06:05and then create a PR with the thing and here's the links to the docs.
06:08Right, this is like a message that I would send to somebody on my team, go write a playbook.
06:12And when I went and created this, it did go and create a playbook.
06:16In fact, that's what agentic workflows kind of look like.
06:18They look like markdown documents.
06:20Like, if GitHub Actions and Copilot had a baby together, and it ran on markdown, this is what it is.
06:26So, what does this agentic workflow look like?
06:31Well, you know, it's an upgrade checker, it's got my tasks, step one, check for new releases.
06:36Again, because it sees my codebase, it was able to infer what it even needs to check and it actually found these specific dependencies.
06:44Review the change log and the upgrade guide, apply the upgrade, and then create a pull request.
06:50Right, I didn't ask for any of this that explicitly, but it turns out that Copilot is pretty good at sussing out my little three-line message into a full playbook.
06:59And then at the top, I've got this special section, this is the what we're calling No, don't collapse it.
07:06Oh, man.
07:08Scrolling is wonky when you zoom in.
07:10This YAML front matter.
07:12This is where we stick the guardrails cuz if we're going to be not supervising agents doing things, then we're going to need much stronger guardrails around what they're allowed to do, what they're allowed to read, what they're allowed to write.
07:24And where are we going to specify that?
07:25And it's not enough to just prompt the agent and be like, "Listen, bro, I don't want you to buy Bitcoin for me ever."
07:32That's not enough cuz somebody else can prompt inject the agent and take it in a direction that you don't expect.
07:37So, any of the guardrails, if you're prompting the guardrails at the agent, you're effectively letting the fox loose in the henhouse.
07:45It's not actually a guard rail.
07:48Um, so here, uh, you can see that I'm specifying deterministically like my permissions are read all, what tools am I allowed to use, uh, what network, uh, requests is it allowed to make?
07:59It's not allowed to just go to bitcoin.com or whatever.
08:02Uh, in fact, it's only allowed to go to some specified set of default websites, the NPM ecosystem cuz it's got to check for like, you know, what's new, GitHub, and of course the Astro docs which I specified in my original prompt.
08:15Uh, and I've got this block called safe outputs which is basically saying these are the only things that the agent is allowed to write.
08:21And so I'm saying in this case the agent will is allowed to create pull request.
08:26Pull request single uh, because I don't want the agent to get prompt injected to create 500 pull requests.
08:32That would be a denial of service.
08:34Um, or and this is the other thing, I explicitly said you're allowed to do nothing, right?
08:38Which sounds silly, but it actually matters because in a world where I have lots of automations, the last thing I want is noise.
08:44I don't want the agents denial of servicing me.
08:48So, okay, I've created this and I've run it and this is actually my actual automation on my actual personal website.
08:56I didn't ask for any of this, but it did a pretty good job of like saying, "Hey, here's the highlights of what you get from going from this version that you're currently on to the version that is the target,
09:05right?
09:05It's read all of the release notes in the middle.
09:07This is normally what I would do as a human.
09:10Uh, and it's built me like, you know, sort of like a tailored description.
09:13It's figured out there's no breaking changes.
09:15It's actually verified this by running and building my project.
09:18And because I happen to have this deployed to Cloudflare, um, or whatever, anything with preview deploys, I can click that open and see that nothing has changed in my website, which is exactly what I want, right?
09:28Like it's done the upgrade and I see that it still works exactly as it did before.
09:32But this was like a minor point release.
09:34That doesn't really count.
09:35Let's look at a major upgrading change.
09:38And actually, I'm lucky that Astro just released Astro 7 because this is actually jumping two major revisions from five to seven.
09:45And so, now it's saying like, "Okay, Astro 7 has brought me all of these things.
09:50And Astro 6 would have brought me all of that stuff, but I neglected to do the upgrade so I could have a cool demo for you all."
09:57And it's found all of the code changes that were broken, and it updated them.
10:02It also verified that the build runs.
10:05And it also highlighted manual steps that things that I would need to do later.
10:11And again, you know, if I go down here and I click on this, I can see, "Hey, still works."
10:17So, cool.
10:18Now, uh it's just markdown.
10:21It's easy to iterate on that markdown, right?
10:23If you don't like the way that the automation works, just edit the English.
10:26It gets recompiled into an Actions workflow.
10:29Like, the markdown is the source code.
10:31The YAML is like a compiled artifact.
10:33You never look at it.
10:35Um but, we've also given you a whole library of Agendic workflows for you to use as a starting point to customize.
10:41So, an issue triager.
10:43Internally, GitHub has actually used this as the basis for like spiking out our own internal issue triager or for like hunting down N+1 queries in our like monolith or all kinds of things.
10:53There's a ton of things that are super helpful there.
10:55Repo assist.
10:56This is actually a swarm of Agendic workflows that work together to help you maintain your project by finding low-hanging fruit, fixing them, identifying tickets that need nudging or feedback that you need from people who have filed issues, whatever.
11:12CI doctor.
11:13How many times have you responded to a busted CI run by just running it again?
11:18All of us.
11:19Anybody who hasn't raised their hand is lying.
11:21Uh a million more.
11:24Like, you know, goals, sure.
11:26Daily team status and repo status.
11:28If I want this to go do like homework on the internet, I can.
11:31So, this is not just for engineers, this is also for product managers whose job it is to look at information over here and summarize those tickets over there, right?
11:39We can start to get everybody involved in automation.
11:42That's how you actually get industrial scale.
11:46Uh So, uh that's Agentic Workflows.
11:50Um the security guardrails, we have sort of four principles that we believe uh everybody should burn into their brains.
11:57Uh defense in depth, one layer is never enough.
12:00Uh that was always true.
12:01Never trust agents with secrets.
12:03If an agent can know a secret, that secret, you need to treat it as if it's already been compromised.
12:09Uh because you have no idea whether or not somebody's injected the agent to reveal that secret somewhere else.
12:14So, if an agent can see the secret, um it's bad.
12:17In Agentic Workflows, the secrets are all kept outside of the agent's jail, and when the agent wants to use the secret to call something, it needs to ask the warden, "Hey, mother may I please go talk to that service?"
12:29Uh stage and vet all rights, just so that it's auditable, and log everything, just so that it's auditable.
12:35Uh and when we give this to existing projects like the Home Assistant project, which is a huge open-source project, um the first uh Agentic Workflow they built was something that looks at every submitted issue, walks the Python stack trace to figure out if the bug is in first-party code or third-party code, closes the issue if it's not
12:53their issue, right?
12:54That's something that was not possible before AI, not possible with heuristics, uh but is possible now.
13:01Agentic Workflows is in public preview today.
13:03You can go and kick the tires.
13:05So, go ahead, go wild.
13:07Uh we actually believe that this is going to be a bigger category than interactive AI because automations that run in the background while you sleep, that's the ballgame.
13:17Okay, so let's talk about the collaboration piece.
13:19So, this is how we've always built software, right?
13:22Because the cost of writing code was so high, uh but that's not true anymore.
13:27We would plan and review together, but the building part was done alone.
13:30Like, you know, illuminated by the light of my monitor, uh, I would build.
13:34But now, none of it is alone, right?
13:35Planning isn't before, and review isn't after.
13:39We iterate on the direction together, and AI takes a step, and then we iterate more in the direction.
13:44So, what's an interface that makes sense for that style of development?
13:48I'm only slightly trolling, right?
13:50Slack was designed to be better, uh, than email for the average office worker.
13:54It was never designed for making software or the needs of everyone involved in that.
13:59But, what this is good for is surfacing all the facts that are not in code.
14:04Anything that's in code, any fact that's in code, the agents can figure out by reading the code.
14:08What's left are the things that are not in code, like political considerations.
14:12Like, "Hey, if we do it that way, that VP over there is going to vibe with that direction."
14:16Or, like, "We should make it purple because that's their favorite color."
14:19Or, "We get a really sweet deal, uh, on infrastructure from Azure.
14:23Therefore, we should be building on Azure, not on, uh, GCP or AWS.
14:28Whatever."
14:30But, the biggest win is the same win that we've already seen over and over, right?
14:33I don't email Word documents around anymore.
14:35I create and collaborate in the same surface, in the same place.
14:40This is coming for code, a trillion percent, right?
14:43So, let me show you what we have here.
14:45Oops.
14:46Here we go.
14:48I got to find the tab.
14:49All right.
14:50Uh, this is Ace.
14:52Let's switch to the repository.
14:54So, Ace looks an awful lot like Slack, right?
14:57And over here on the left, I've got sessions, and I can create new ones.
15:00And, you know, so far this kind of looks like every other conductor-like product out there.
15:05Um, the difference being is that every one of these is not on my machine.
15:09In fact, none of this is running on my machine.
15:11It's all micro VMs in the cloud.
15:12So, every session is just a branch of my repo checked out to a spot in the cloud.
15:17Uh, and I can create them and do stuff in them and talk with my teammates.
15:20So, like, uh hey, um uh what's your favorite color, right?
15:28Uh and meanwhile, I'm going to like install my dependencies.
15:31And then when that's done, I'm going to do like uh bond dev.
15:37I'm going to run the dev server.
15:39Um and here, like, Russ and I are having a discussion, like, are you sure?
15:44Maybe uh maybe green is calmer.
15:49Um Oop, nope.
15:51I sent that as a terminal command.
15:52Good job, me.
15:54I do not want that as a thing.
15:56Great.
15:56I'll do it like this.
15:58Uh and I can open up my preview.
16:01Oops.
16:02Give me a preview.
16:03I'd like a browser preview.
16:04Okay.
16:05So, so far, not that different from developing with any sort of like multiplayer tool.
16:09And here, I've got this sort of calm hacker news thing.
16:11I've just had a whole discussion with my teammate.
16:14I don't want to turn around and now like emit those instructions again.
16:17Instead, I just want to be like, yo, Ace, do it.
16:20Uh and because it sees the entire backscroll of my conversation with my peers, with my team, it's able to act on that uh uh on on that chat history.
16:32And if the Wi-Fi was nice, then it would be doing it faster.
16:36Um uh you're going to have to trust me on this because I don't have enough time to wait for this.
16:39That it's going to just respond to the fact that we had a discussion about colors.
16:43And AI is also really good at fishing out that final state.
16:47Like, very frequently, what do engineering conversations sound like?
16:50They sound like like, hey, we should try it this way.
16:52No, wait, I thought of like an edge case.
16:54We should actually do it that way.
16:55Let's go back to the first idea, right?
16:58But instead of me sort of like figuring teasing out that final state from that long conversation, I can just let AI do it and it'll figure it out.
17:06So, I don't need to work for the robots.
17:08And sometimes we have things that are a lot more um complicated.
17:12Like here, I wanted to add selectable time frames to my app.
17:16And so, I asked it to make a plan, and that plan comes as a uh uh markdown document.
17:21Uh but, this markdown document is not just for me to look at and edit, it's for us to look at and edit together.
17:28So, Russ is somewhere uh here in this document, and like, you know, maybe he thinks that we should add an all time, and I'm going to get rid of the today,
17:37and here I can again do like uh we've updated the plan, do it.
17:43Um Uh uh and it'll just respond to the plan that we've edited together.
17:49And as we see now, we're moving to this future where uh more and more of the work that we're doing with AI results in documents like markdown documents in a docs folder that captures sort of the truth,
18:01and maybe more and more in the future we're going to be editing those documents as the way that we do development.
18:07Like, in order to change something about my application, I'm going to edit a document, and I'm going to tell AI, "Hey, make the document true."
18:15So, this shared document editing is not just like, "Oh, a nice to have."
18:20Maybe this is actually sort of the uh interface that we like to work in.
18:24But, there's also the uh social coding aspect, right?
18:28Like, if I'm working with other people on my team.
18:30Um remember when that was a thing that was a tagline under the GitHub logo?
18:35Um so, uh how can it help me stay up to date with what everybody else on my team is working on?
18:41Like, it's not just enough to have like real-time multiplayer, I also want to be ambiently aware of what everybody's going going on about.
18:48So, Kristoff is working on Vian tooling.
18:50This is actually work that we're doing on Ace, and Maggie wrote this dashboard and hardcoded her name, and so that's why we're looking at Maggie's name.
18:57Um and David worked on whatever.
18:59All this stuff to help me stay aligned with my team.
19:02And when I look to the future, I'm starting to think about how do automations surface themselves in this?
19:08If I want to talk with my automation, uh there's lots of things that I want to do in this kind of interface, like when an agent wants to tap me on the shoulder and ask me a question,
19:18um, that I think are very interesting.
19:21So, that's a short Ace demo.
19:24We're going through this weird inversion of our relationship with the agents.
19:27Like, the better that we get at articulating, uh, our goals to the agents, the less they need us.
19:34Uh, and as the models get better, they're also good at spotting like underspecified behaviors and then asking us to clarify.
19:41Uh, and then whenever they need a pair of hands, they can ask us to be the pair of hands.
19:45But, either way, the interfaces now have the ability to support the ability of agents to listen to everything and invoke us when they need it.
19:55Which is a little funny to think about.
19:56It's maybe like sort of we're coming at it from this side and like open claws coming at it from this side, but like we're landing in sort of a similar spot.
20:08And I'll close with this thought.
20:09Um, for the past few years, AI has helped me to type.
20:13But, if you look at the science of the matter, it's only about 5% of the job.
20:17Like, this was a longitudinal study conducted on like 100 developers over thousands of hours.
20:22Turns out that the hands-on keyboard typing part is 5% of the time.
20:27Now, AI has to help me with the other 95%.
20:29Where is the system that I want to touch?
20:32How does it work today?
20:33What do other people think about like how we could mutate it or should mutate it?
20:38When AI can discover anything in my code base, like, how do we How do we help scale up all those other things, right?
20:46Like, not just the 5%, which is what all the tools have been helping us to do so far.
20:51So, that's Ace and that's a genetic workflows.
20:54Uh, please, uh, come by and talk to us.
20:56Uh, we have, uh, a booth down in the Microsoft booth because we're a Microsoft company.
21:02Uh, and you can find us on the socials and get at next.com.
21:05So, if any of this resonates and you're interested in it and you want to give it a shot, ACE is going to be in technical preview hopefully later this month and genetic workflows is already out there for you to kick the tires and we'd love to hear from you and how you want to use this.
21:17Thank you so much.
21:18[applause]