WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=I2LL_wd89-A

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/I2LL_wd89-A.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.960 --> 00:00:15.519
Hey uh everyone.

s2
00:00:15.519 --> 00:00:16.880
This is a hard tone to follow.

s3
00:00:16.880 --> 00:00:17.760
Why did everybody leave?

s4
00:00:17.760 --> 00:00:22.240
No. Um this is the rerun of a talk that she gave yesterday upstairs.

s5
00:00:22.240 --> 00:00:24.320
So this will be better than the one yesterday.

s6
00:00:24.320 --> 00:00:27.359
So uh kudos to you for catching this one.

s7
00:00:27.359 --> 00:00:35.040
I'm working on VS Code and we move from a monthly release to a weekly release cycle thanks to AI.

s8
00:00:35.040 --> 00:00:43.760
And this is about the things we learned, the systems that made this work, not just to ship faster, but actually to ship more effectively.

s9
00:00:43.760 --> 00:00:48.879
So, it's not about how we build agents into VS Code, which we can try out anytime.

s10
00:00:48.879 --> 00:00:56.559
You can come by the booth and see us showing of agents, but this is about how we use agents to build the product itself.

s11
00:00:56.559 --> 00:01:07.600
And that's VS Code, that's the copci and other areas, but also how as you use agents and more AI, things are breaking and you have to rethink actually how you work.

s12
00:01:07.600 --> 00:01:12.960
And that's a very small team shipping to over 50 million users.

s13
00:01:12.960 --> 00:01:20.240
What happened is over the past year is we started tracking already early on this code survival metrics in VS code.

s14
00:01:20.240 --> 00:01:24.320
So that means the percent of agent written code that actually gets committed.

s15
00:01:24.320 --> 00:01:33.600
how much kind of garbage has the agent produced that you threw out the window again or didn't trust and that you that you read it u and and scrapped.

s16
00:01:33.600 --> 00:01:37.040
So GBD 4.1 started at 55%.

s17
00:01:37.040 --> 00:01:46.320
But then over time and over improving the harness and new models cloud opus 4.6 is now today at 86%.

s18
00:01:46.320 --> 00:01:56.000
And that increase clearly shows developer trust and confidence in shipping AI code faster and more AI code generated.

s19
00:01:56.960 --> 00:01:59.920
And this success created problems on the VS code side.

s20
00:01:59.920 --> 00:02:04.479
We're one of the biggest or the biggest open-source project on GitHub.

s21
00:02:04.479 --> 00:02:09.360
So we saw a massive increase in issues because AI helps to file issues.

s22
00:02:09.360 --> 00:02:17.120
So we actually see an increase in quality issues but also the counterpart of more automated lowquality issues.

s23
00:02:17.120 --> 00:02:24.080
We also see more open PRs by the team by our own velocity and by our community.

s24
00:02:24.080 --> 00:02:37.360
And interesting part we actually see everybody would assume we see a lot of garbage PRs but actually the number of community merged PRs is going up as well which is really amazing that we can unlock more community contributions as well.

s25
00:02:37.360 --> 00:02:44.879
And as part of this new workflow, we actually moved several months ago, we moved from monthly releases, which happened over 10 years.

s26
00:02:44.879 --> 00:02:52.800
Since the inception of VS Code 1.0, we have been shipping a new version of VS Code on a monthly basis.

s27
00:02:52.800 --> 00:02:56.640
Including fun facts like the January release actually ships in February.

s28
00:02:56.640 --> 00:03:03.599
So when you read the release notes in VS Code, you're wondering why it's February and you're reading the January release notes if you're a month late.

s29
00:03:03.599 --> 00:03:07.519
No, that's just how we tracked our iterations.

s30
00:03:08.080 --> 00:03:20.319
So that cadence was to match to velocity, but it also showed us that the it's not about using more AI as you work dayto-day, which a lot of companies are pushing.

s31
00:03:20.319 --> 00:03:22.080
Like you're not token maxing.

s32
00:03:22.080 --> 00:03:25.280
You're not spending every day using AI.

s33
00:03:25.280 --> 00:03:34.879
It's about evolving the whole system, how you're shipping software to evolve to actually make better use of AI along the whole process.

s34
00:03:34.879 --> 00:03:40.879
And that what makes this velocity actually work and doesn't just generate more code.

s35
00:03:40.959 --> 00:03:44.720
I'm breaking this down into this first part which a lot of people have been experiencing.

s36
00:03:44.720 --> 00:03:47.599
I see shipping code faster and that's the easy part.

s37
00:03:47.599 --> 00:03:58.560
You have like this 100x engineers that has built this beautiful skill MCP plug-in system and they're just shipping code like crazy and slowly maybe more in the more people in the team

s38
00:03:58.560 --> 00:04:03.760
uh adopt the same patterns and you see a lot more code shipping faster.

s39
00:04:03.760 --> 00:04:07.120
But the problem is once you ship faster, you get into code review.

s40
00:04:07.120 --> 00:04:12.720
You get into how do you actually ship high quality code that doesn't constantly break.

s41
00:04:12.720 --> 00:04:14.879
And that's important for us as VS Code.

s42
00:04:14.879 --> 00:04:17.199
We're shipping binaries to user machines.

s43
00:04:17.199 --> 00:04:22.959
So if something breaks, that recovery is even more costly.

s44
00:04:22.960 --> 00:04:34.320
But once you hit that velocity with shipping high quality code faster with confidence, how do you actually know you're shipping the right thing or if we should ship this thing?

s45
00:04:34.320 --> 00:04:48.639
And that's where AI kicks in to actually learn faster in the end and shipping better products which a lot of people are missing out as as they write more code is applying that product taste and applying that learning.

s46
00:04:49.520 --> 00:04:51.680
So let's start shipping faster.

s47
00:04:51.680 --> 00:04:54.720
Now that everybody can vibe, we can really all ship faster.

s48
00:04:54.720 --> 00:04:58.880
And it's a big part is getting your codebasation ready.

s49
00:04:58.880 --> 00:05:15.120
So that's kind of the no-brainer slide that everybody should already been doing is thinking about how to get your code bases agent ready is creating agents.mmd that give it that are lightweight enough and give the agents a good kind of map of the codebase where to look at

s50
00:05:15.120 --> 00:05:24.240
mapping out that is every agent now at your uh at your fingertips has a slash in it that gives you a good draft and then you keep itering those are living documents

s51
00:05:24.240 --> 00:05:29.680
that should evolve as the agent makes mistakes and works.

s52
00:05:29.680 --> 00:05:39.440
And that's where I see a lot of teams that have invested in developer experience actually gain a lot because once they have good documentation and onboarding for a codebase

s53
00:05:39.440 --> 00:05:44.000
that is read by the agent as well and that can help the agent as much.

s54
00:05:44.000 --> 00:05:45.600
Next up you have skills.

s55
00:05:45.600 --> 00:05:48.479
VS Code has always messed heavily in accessibility.

s56
00:05:48.479 --> 00:06:00.639
So one skill we added early on was infusing all the accessibility best practices and how we think about accessibility into a skill that everybody will use and that previously was one person

s57
00:06:00.639 --> 00:06:10.560
that had to then got get pulled in or give feedback and now that skill was reviewed and is maintained by the kind of area owner of accessibility.

s58
00:06:10.560 --> 00:06:15.759
So to bring that that accessibility that short work everywhere into everybody's hands.

s59
00:06:15.759 --> 00:06:25.520
So think how you can encode your expert knowledge to unblock this like few users that might be always asked for this input into a skill.

s60
00:06:25.520 --> 00:06:38.639
And lastly once you do it right for us in VS Code the litmus test is can I as a PM effectively VIP code in the VS code repo which I do

s61
00:06:38.639 --> 00:06:42.000
um and how is that accepted by engineering?

s62
00:06:42.000 --> 00:06:43.280
How much work does it cost?

s63
00:06:43.280 --> 00:06:46.000
Is it out of the box working?

s64
00:06:46.000 --> 00:06:48.560
Is it for me working locally?

s65
00:06:48.560 --> 00:06:51.840
Or is it eventually breaking when I ship it?

s66
00:06:51.840 --> 00:07:00.560
So, a lot of these quality improvements will show in the end that you can allow everybody to be more effective in the repo.

s67
00:07:01.599 --> 00:07:08.000
Another improvement, and that's a massive one, is we switched to TypeScript Go.

s68
00:07:08.000 --> 00:07:17.360
And if you've been in any developer codebase that takes some time to build or run linting or run CI/CD, you've noticed it's already annoying.

s69
00:07:17.360 --> 00:07:19.120
But developers find other ways.

s70
00:07:19.120 --> 00:07:20.639
They can review a PR.

s71
00:07:20.639 --> 00:07:24.560
They can context switch into something else while runs and then do something else.

s72
00:07:24.560 --> 00:07:36.800
But once you hit that with agents, you have 10 agents, 20 agents running and hitting the same bottleneck, these any slow part of your CI/CD will compound.

s73
00:07:36.800 --> 00:07:49.520
And for us, Typescript Go was a 10 times improvements in our builds, which at the scale of automatic PRs running with agents and them fixing getting needing that CI/CD

s74
00:07:49.520 --> 00:07:54.720
loop for feedback to improve the code is a massive improvement.

s75
00:07:55.520 --> 00:07:59.360
And that leads to me uh filing big PRs.

s76
00:07:59.360 --> 00:08:09.199
That was beginning of the year to get the ask questions tool in which I believe we really needed and then bringing it to the team to show to let them try it out in the product.

s77
00:08:09.199 --> 00:08:17.520
Within a few weeks we had a perfectly polished and designed experience based on my initially very scrappy PR.

s78
00:08:17.520 --> 00:08:28.560
But it unblocked that this these discussions the participation of everybody else because we had the ground set laid and the idea and the experience was clear.

s79
00:08:30.639 --> 00:08:40.399
Next up, when agents actually work on UI, and we all had this, that the agent gives you a UI and it says it looks perfect, then you open it and it's all misaligned.

s80
00:08:40.399 --> 00:08:44.720
Even with SVG, even with the latest models, you still get it.

s81
00:08:44.720 --> 00:08:48.640
And it's like the number one thing I keep telling everybody now these days.

s82
00:08:48.640 --> 00:09:02.399
If your agent cannot use your application, your product directly to get this feedback loop of is it all working, then that's a really big investment that pays off every time you work on UI.

s83
00:09:02.399 --> 00:09:05.040
And there's two areas we invested in.

s84
00:09:05.040 --> 00:09:13.040
One is the component browser, which is an automated build that runs every time VS Code changes something.

s85
00:09:13.040 --> 00:09:18.240
Let me open this in this issue as an example.

s86
00:09:19.920 --> 00:09:23.360
In this case, we added a back button to our customization screen.

s87
00:09:23.360 --> 00:09:33.120
And because it's changing wide, we actually have this automated process in the back that takes screenshots of every component in VS Code and points out the differences.

s88
00:09:33.120 --> 00:09:42.880
It allows us to detect unexpected changes like we're changing one component as the ripple effect of something else moving or some other icon disappearing.

s89
00:09:42.880 --> 00:09:59.360
and allows you to quickly review a PR as well because you also get directly a view and what the PR is actually doing and that's what we ask every developer before we ask them to at least attach a video or screenshot on what they have been implementing so we don't have to run everything from scratch and we

s90
00:09:59.360 --> 00:10:14.800
can quickly assess and and give feedback and this is now automated with the component explorer next up we have the self-correcting loop because in VS Code we run uh a web browser and open a website.

s91
00:10:14.800 --> 00:10:16.000
Basically, that's what VS Code is.

s92
00:10:16.000 --> 00:10:17.360
It's all HTML.

s93
00:10:17.360 --> 00:10:22.000
We can actually use Playright to automate a lot in the browser.

s94
00:10:22.000 --> 00:10:29.279
So, we have a /launch skill that would just open up VS Code and give it a specific scenario to click through in VS Code.

s95
00:10:29.279 --> 00:10:40.560
So, it's a really powerful way to diagnose issues because you can also get the locks along the way, but also validate any fixes you did before and after without you having to click through it.

s96
00:10:40.560 --> 00:10:55.600
So you can just hit launch move on to the next task and then come back once the agent verified it's worked because it can access your application and to call out this is nice because we have playright and we are a web app uh running in electron

s97
00:10:55.600 --> 00:11:07.839
but there's many other solutions out there one of my favorites is Xcode MCP because on my site I work on iOS apps and once you see Xcode MCP just creating in VS code without Xcode involvement

s98
00:11:07.839 --> 00:11:12.000
just making an iOS app opening it up clicking through it, taking screenshots.

s99
00:11:12.000 --> 00:11:15.600
These moments are magical and give you a really tight feedback loop as well.

s100
00:11:15.600 --> 00:11:20.640
So there's other tools available for Android as well to make this work.

s101
00:11:21.519 --> 00:11:24.320
Let's talk about code review, which I think is the next big investment.

s102
00:11:24.320 --> 00:11:37.120
We know in the past when we enabled code we on GitHub the automated GitHub copiloted one we're not convinced yet but over the next months and weeks they have massively improved and now it's a mandatory

s103
00:11:37.120 --> 00:11:52.800
uh review every time somebody opens a PR and there's more improvements coming actually dial in the uh the effort that a code review does depending on how much risk you have in this repo so you have a low medium high effort for code review so you can dial in kind of the cost benefit

s104
00:11:52.800 --> 00:12:03.120
as well but for us is if a review after review is done, humans will not even look at PR until all the comments are resolved and and addressed.

s105
00:12:03.279 --> 00:12:06.320
Okay, now we're holding we're shipping faster.

s106
00:12:06.320 --> 00:12:08.079
We got to hold quality.

s107
00:12:08.079 --> 00:12:12.800
And a big signal for us because we're on GitHub is of course issues.

s108
00:12:12.800 --> 00:12:15.839
Uh who has filed a GitHub issue?

s109
00:12:15.839 --> 00:12:19.680
Who has had problems in VS Code before?

s110
00:12:19.680 --> 00:12:20.240
Yeah, cool.

s111
00:12:20.240 --> 00:12:21.040
Kind of similar.

s112
00:12:21.040 --> 00:12:21.360
Awesome.

s113
00:12:21.360 --> 00:12:23.120
This the last crowd had less hands up.

s114
00:12:23.120 --> 00:12:24.480
So it's good.

s115
00:12:24.480 --> 00:12:25.040
Um, yeah.

s116
00:12:25.040 --> 00:12:26.720
So it's a massive, it's a really good signal.

s117
00:12:26.720 --> 00:12:36.399
And as I said, we get sometimes even better issues now because of AI and more people with different language backgrounds being able to file very good issues.

s118
00:12:36.399 --> 00:12:37.760
But we also get a lot of issues.

s119
00:12:37.760 --> 00:12:41.040
And previously engineers actually triage them manually.

s120
00:12:41.040 --> 00:12:54.240
And now we have AI doing the filtering to get out spam enriching and translating in the end assigning area owners for all the issues to raise the signal to noise quality of this.

s121
00:12:54.480 --> 00:13:02.480
We also add a human layer to that once you are in issue because the agent will make mistakes and we need that feedback loop when it makes mistakes.

s122
00:13:02.480 --> 00:13:07.680
We create a Chrome extension that allows you to fix duplicated issues as well.

s123
00:13:07.680 --> 00:13:16.079
So we have the agent doing the work and we have a feedback loop where humans can still fix it that we can bring back into the into the agent work up front.

s124
00:13:16.079 --> 00:13:23.120
That's a really important thing to think about as you have agents doing the work that you get the human feedback as well.

s125
00:13:23.600 --> 00:13:27.040
The other input we have is a lot of error stacks.

s126
00:13:27.040 --> 00:13:34.959
Every time there's an exception and most apps will have this you get an error stack reported and we actually don't use any external tool for that.

s127
00:13:34.959 --> 00:13:40.160
It's all in our own data and we built all the tooling around it and it's mostly we already had it.

s128
00:13:40.160 --> 00:13:48.079
We already had an ML doing sorting on top and this is now actually done by a mix of criteria.

s129
00:13:48.079 --> 00:13:50.240
So one is we collect the raw telemetry.

s130
00:13:50.240 --> 00:13:53.760
We filter it down and that's like 51 billion per day.

s131
00:13:53.760 --> 00:13:57.839
We filter it down to just error stacks that actually have the full stack.

s132
00:13:57.839 --> 00:13:59.839
Then we group it and bucket it.

s133
00:13:59.839 --> 00:14:02.160
That's more of the fingerprinting exercise.

s134
00:14:02.160 --> 00:14:14.560
And then in the end after we all of that we have 10 issues filed assigned to specific area owners that a PR is autocreated to actually already try to fix the problem.

s135
00:14:15.120 --> 00:14:23.839
This is our error page where we see all the stacks including how many hits and how many users affected so we can quickly react to anything that's coming.

s136
00:14:23.839 --> 00:14:30.079
And once we have this working we get a issue filed and a PR automatically opened.

s137
00:14:30.079 --> 00:14:31.040
This is the issue.

s138
00:14:31.040 --> 00:14:45.120
is the error with some initial diagnosis and then the PR got opened with and the agent actually goes in and finds out why does this happen who was responsible in this case um

s139
00:14:45.120 --> 00:15:02.079
added a change that was connected to this to find f find symbols and then it figured that the on cancellation request wasn't in the RPC protocol so the agent already opened the PR we could just merge it and that's a really great automated way you don't care about errors you just want to get a more stable code

s140
00:15:02.079 --> 00:15:04.079
base.

s141
00:15:05.040 --> 00:15:18.959
So all of this automated the triage initially mix of agents and deterministic systems then the issue is filed handed off to an agent in the PR multi- aent system and in the end we have fixing fixes landing mostly automatically

s142
00:15:18.959 --> 00:15:23.120
with some human oversight to still approve the PR.

s143
00:15:25.279 --> 00:15:26.880
There you go.

s144
00:15:26.880 --> 00:15:32.880
So previously VS Code did yolo releases in a way that it was release day everything was stable.

s145
00:15:32.880 --> 00:15:35.839
We did our testing and we shipped the release to 100%.

s146
00:15:35.839 --> 00:15:40.639
We just opened the floodgates but because of agents we want to derisk that process.

s147
00:15:40.639 --> 00:15:42.160
We should have probably done it earlier.

s148
00:15:42.160 --> 00:15:48.880
So now actually we do stage rollouts and as we do the roll out we monitor error locks and issues and everything else.

s149
00:15:48.880 --> 00:15:58.399
So just like good citizens in a in an web application, we now apply the same because rollbacks on an install app are so expensive.

s150
00:15:58.880 --> 00:16:02.160
Okay, now we have this shipping faster, shipping at quality.

s151
00:16:02.160 --> 00:16:08.320
How do we actually do this in a in a speed that we actually learn faster, which is the which is a bigger challenge with AI?

s152
00:16:08.320 --> 00:16:12.000
You actually want to build a better product faster.

s153
00:16:12.000 --> 00:16:14.639
So one of the examples is our VSC bench.

s154
00:16:14.639 --> 00:16:16.320
We're shipping an agentic product.

s155
00:16:16.320 --> 00:16:23.759
So it's important that we have our own evals of our product always at hand and being really easy to extend.

s156
00:16:23.759 --> 00:16:25.199
It's all in the GitHub issue.

s157
00:16:25.199 --> 00:16:26.240
It's all in GitHub.

s158
00:16:26.240 --> 00:16:33.199
When we file an issue to add a new scenario, it's actually an agent picking up from a template and bootstrapping the scenario for you.

s159
00:16:33.199 --> 00:16:43.360
So we make it as easy as possible to bring in all the developer scenarios that are coming up in issues through customer conversations through all these channels to bring them back into our emails.

s160
00:16:43.360 --> 00:16:52.399
So we then start hill climbing actually against them and then com also confirm that it's all improving offline and then an online experimentation.

s161
00:16:52.399 --> 00:17:01.040
So classic product life cycle for um for us and we see these interesting things in our stack.

s162
00:17:01.040 --> 00:17:03.360
Understanding your evals is critical.

s163
00:17:03.360 --> 00:17:18.400
One experiment we did that you can read about on our blog is that we gave a very simple eval scenario of just write a file with hello world just testing out our eval harness end to end and how different models react.

s164
00:17:18.400 --> 00:17:26.720
The same five character file took the most expensive model 70x more tokens and that was not the high reasoning model.

s165
00:17:26.720 --> 00:17:27.839
You can read more about unblock.

s166
00:17:27.839 --> 00:17:37.280
We actually don't talk about the models, but just that insight in how your models react with your harness is extremely is a very important exercise.

s167
00:17:38.320 --> 00:17:47.679
Next up, we have prototyping, which I do a lot of because I noticed that I don't actually want to land most of my work into VS Code PRs itself.

s168
00:17:47.679 --> 00:17:57.679
It's a great way to start a conversation, but most often I want to have a quick conversation because now we're meeting on a daily basis where we just want to have very quick feedback loops

s169
00:17:57.679 --> 00:18:01.520
on here's the idea, here's how it could look like, what if we do this.

s170
00:18:01.520 --> 00:18:06.480
Next day you come back with updated uh prototypes and you just keep having this conversation.

s171
00:18:06.480 --> 00:18:20.559
So much quicker feedback from a month release cycle to weekly to daily sprints where you just keep working on problems and prototypes unlock those deeper discussions and how the experience should look like.

s172
00:18:21.840 --> 00:18:34.000
It also means that we have the durable ownership in internal work as I mentioned daily sprints um smaller squads, smaller work streams that really prioritize and drive smaller sets

s173
00:18:34.000 --> 00:18:37.600
um of areas to move faster.

s174
00:18:37.600 --> 00:18:48.160
So try these things, find out where your bottlenecks are to actually ship higher quality, ship faster and learn faster and find fix that next bottleneck.

s175
00:18:48.160 --> 00:18:52.880
And don't just tune how you work with agents, but tune where how how do they get feedback?

s176
00:18:52.880 --> 00:18:55.039
How do you build these loops?

s177
00:18:55.039 --> 00:19:04.160
And then figure out after you have these these success moments, how can you improve that feedback loop for the next moment of what's blocking you to move even faster

s178
00:19:04.160 --> 00:19:09.120
and it's a constant iterative cycle across all these cycle ones.

s179
00:19:09.360 --> 00:19:09.760
That's it.

s180
00:19:09.760 --> 00:19:10.720
Come by code booth.

s181
00:19:10.720 --> 00:19:12.240
Happy to chat now.

s182
00:19:12.240 --> 00:19:15.679
Otherwise, happy coding.

s183
00:19:16.743 --> 00:19:18.743
[applause]

s184
00:19:33.118 --> 00:19:35.118
[music]