WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=uU5Gv2h8-9g

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/uU5Gv2h8-9g.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:12.400 --> 00:00:15.360
Welcome to this uh fireside chat.

s2
00:00:15.360 --> 00:00:20.640
Um we're going to I have with me uh Theik Shihipa and Cat Woo from Anthropic.

s3
00:00:20.640 --> 00:00:29.039
We are going to be diving deep into claude code and we'll probably talk a little about this Fable thing that's been going on that that's that's been out there in the news.

s4
00:00:29.039 --> 00:00:34.880
Actually, on the subject of Fable, literally a minute and a half ago, Fable came back.

s5
00:00:34.880 --> 00:00:37.280
Like Fable is now available to me.

s6
00:00:37.280 --> 00:00:42.640
So, if you all want to run out of the room and start using up your Fable credits, I I wouldn't hold that against you.

s7
00:00:42.640 --> 00:00:43.920
But we're going to have a great conversation.

s8
00:00:43.920 --> 00:00:45.440
So, please please stick around.

s9
00:00:45.440 --> 00:00:50.960
Um but yeah, so um please welcome uh The Cat for me.

s10
00:00:54.000 --> 00:00:54.719
Thanks for having us.

s11
00:00:54.719 --> 00:00:57.360
Yeah, we timed it for the the chat for sure.

s12
00:00:57.360 --> 00:00:57.680
Yeah.

s13
00:00:57.680 --> 00:00:57.920
Yep.

s14
00:00:57.920 --> 00:00:58.079
Yep.

s15
00:00:58.079 --> 00:01:00.079
This is this is why it's all happening.

s16
00:01:00.079 --> 00:01:03.600
Um this year has been somewhat absurd.

s17
00:01:03.600 --> 00:01:05.199
Uh I it's amazing.

s18
00:01:05.199 --> 00:01:07.360
Claude Code came out in February of last year.

s19
00:01:07.360 --> 00:01:12.960
It's under a year and a half old and it was a bullet point on the Claude Sonnet 3.7 launch.

s20
00:01:12.960 --> 00:01:14.479
I'd love to hear from you.

s21
00:01:14.479 --> 00:01:22.799
How has your how has what you do on a day-to-day basis changed in the past year now that we have these coding agents that actually work for us?

s22
00:01:22.799 --> 00:01:33.439
I remember when we first came out with Cloud Code and Sonnet 37, you would give it this task and you would have to closely monitor every single little thing that it tried to do.

s23
00:01:33.439 --> 00:01:36.960
Uh I remember I would read every permission prompt extremely carefully.

s24
00:01:36.960 --> 00:01:41.280
I would frequently say no. I would always say no no no like did you check this file?

s25
00:01:41.280 --> 00:01:42.640
Did you check that file?

s26
00:01:42.640 --> 00:01:45.360
And now it's been incredible with every model generation.

s27
00:01:45.360 --> 00:01:53.600
I feel like we've all gotten a chance to just take a step back, delegate a lot more of the like menial implementation to claude.

s28
00:01:53.600 --> 00:02:04.320
And it just freed up a lot of our time to think about more creative work like what is the right experience that we should be providing to our users now that we know cloud code can implement a lot of it.

s29
00:02:04.320 --> 00:02:08.959
And now with Fable, it's just a totally different step change improvement.

s30
00:02:08.959 --> 00:02:15.040
um we see for a lot of our use cases that you can actually oneshot a ton of features with Fable now.

s31
00:02:15.040 --> 00:02:21.840
So it it's been amazing to see the transition and to go through this with all of you in the community.

s32
00:02:21.840 --> 00:02:25.200
Yeah, I mean I I think I remember the first text I got about cloud code.

s33
00:02:25.200 --> 00:02:28.720
One of my best friends was like, "Oh, you need to go try Claude Code."

s34
00:02:28.720 --> 00:02:32.319
And it was about just when Opus 4 came out and I I tried it and I was like, "Oh, shit."

s35
00:02:32.319 --> 00:02:34.959
Like I I need to work at Anthropic now, you know?

s36
00:02:34.959 --> 00:02:48.080
And that was Opusport which I mean great model but like yeah you were permission prompts and uh yeah I think it's kind of crazy how much amnesia we have I think where I'm like oh like auto mode has always been here right like I I don't even remember

s37
00:02:48.080 --> 00:02:50.400
pressing like yes and allow.

s38
00:02:50.400 --> 00:02:57.360
Um and yeah I think for me the big thing that I'm trying to push myself is like oh we have to do like high higher quality work than we've ever done before.

s39
00:02:57.360 --> 00:03:01.519
You know like the the outputs are like like incredibly high quality.

s40
00:03:01.519 --> 00:03:16.000
like I've been using it to edit videos a bunch and I'm like okay it has to meet the very exacting demands of our brand team and in a couple hours or we just can't do it you know and so um yeah I think that's how I'm sort of like trying to shift with Fable where it's like okay

s41
00:03:16.000 --> 00:03:18.879
the best work we've ever done like faster than we've ever done it before

s42
00:03:18.879 --> 00:03:32.080
I've certainly been finding that myself like software engineering is getting harder because the the level of ambition of the th stuff we can take on has gone up like I I have such higher expectations of myself now that I I have these tools to back me up

s43
00:03:32.080 --> 00:03:34.640
which is fun but it's al it's a lot of work.

s44
00:03:34.640 --> 00:03:34.959
Yeah.

s45
00:03:34.959 --> 00:03:38.879
It's all the thinking is you know it's it's tiring.

s46
00:03:38.879 --> 00:03:47.680
And so what's a piece of conventional software engineering that was true a year ago that you don't think holds anymore in this new world?

s47
00:03:49.280 --> 00:04:05.599
I think one of the biggest shifts that we're seeing in the ENG skill set is I think you know two years ago it was pretty typical for a product manager to go talk to a bunch of customers and over the course of six months align with like cross functional teams on some PRD

s48
00:04:05.599 --> 00:04:19.519
and then write this like thorough spec um and uh dock on how exactly we'll implement this before the first line of code gets written and now things are like completely turned the opposite way.

s49
00:04:19.519 --> 00:04:31.360
I I think for a lot of engineers, the the push I would give to a lot of folks in the room is to develop more of your business sense and product sense on what is it that we should build

s50
00:04:31.360 --> 00:04:36.639
because now that this the timeline between having this idea and building it is so much shorter.

s51
00:04:36.639 --> 00:04:40.240
It's down from six to 12 months to maybe even a week.

s52
00:04:40.240 --> 00:04:45.759
That means all of us need to have better taste on what is it that is worth building.

s53
00:04:45.759 --> 00:04:49.600
what is it that will actually inflect the businesses that we're working on.

s54
00:04:49.600 --> 00:05:02.320
So I think it's like an increase in value on product taste and business sense and a bit lower on execution in in most in most product domains.

s55
00:05:02.320 --> 00:05:09.919
Of course for infra uh there there's still a very heavy emphasis on making sure all the details are right.

s56
00:05:09.919 --> 00:05:10.320
Yeah.

s57
00:05:10.320 --> 00:05:13.759
I I think for me uh it's like rewrites are now good.

s58
00:05:13.759 --> 00:05:15.440
You know what I Like I I think that like

s59
00:05:15.440 --> 00:05:18.080
the worst thing you could do is now actually fine.

s60
00:05:18.080 --> 00:05:18.320
Yeah.

s61
00:05:18.320 --> 00:05:18.720
Exactly.

s62
00:05:18.720 --> 00:05:19.120
Exactly.

s63
00:05:19.120 --> 00:05:20.639
Like all the like especially Yeah.

s64
00:05:20.639 --> 00:05:31.759
mythical man stuff like never rewrite like I I'm a pro rewriting now you know like if you have a good test suite I think actually the rewrite forces you to like make sure you have a good test suite but

s65
00:05:31.759 --> 00:05:42.800
um I think that like what people I think underount is like a codebase is a spec and maybe it's the only copy of the spec that you have right because like no one knows every branching part of the codebase

s66
00:05:42.800 --> 00:05:53.919
and and yeah you can take this as like an artifact and like distill it or create other versions of it obviously like yeah we rewrote bun in rust And uh you know it works great like you know it's it's live for me right now.

s67
00:05:53.919 --> 00:05:54.320
Um

s68
00:05:54.320 --> 00:05:58.560
but you're not shipping clawed code on bun in rust yet right or is that

s69
00:05:58.560 --> 00:05:59.600
internally we have.

s70
00:05:59.600 --> 00:06:00.240
Wow.

s71
00:06:00.240 --> 00:06:01.120
Oh that's exciting.

s72
00:06:01.120 --> 00:06:01.360
Yeah.

s73
00:06:01.360 --> 00:06:01.600
Yeah.

s74
00:06:01.600 --> 00:06:03.600
Yeah.

s75
00:06:04.080 --> 00:06:11.759
I think what you're saying there about um yeah the uh I'm sorry I just lost my train of thought.

s76
00:06:11.759 --> 00:06:20.720
But yeah, the um the rewrites thing has been really interesting for me as well because you can almost come up with a good test suite and then spin up three implementations and pick which of those implementations

s77
00:06:20.720 --> 00:06:21.680
was the most accurate.

s78
00:06:21.680 --> 00:06:23.280
I'm doing a lot more prototyping now.

s79
00:06:23.280 --> 00:06:35.600
I've always been a prototyper and now I pro I prototype things on my phone during the conference just so I've got something that I can pick up later on and that's working now which is kind of extraordinary.

s80
00:06:35.840 --> 00:06:42.880
So the other big launch recently was claude tag which that's what a week old now I think at least for the rest of us.

s81
00:06:42.880 --> 00:06:47.600
Claude tag I understand that's being used in anthropic by non by non-engineers a great deal.

s82
00:06:47.600 --> 00:06:51.199
What kind of things are non-engineers doing with claude tag?

s83
00:06:51.199 --> 00:06:56.160
So claw tag is a claw that lives in your team's collaboration tools.

s84
00:06:56.160 --> 00:06:59.120
We launched it last week within Slack.

s85
00:06:59.120 --> 00:07:03.759
The thing that's different about claw tag is it's multiplayer by default.

s86
00:07:03.759 --> 00:07:13.120
And so once you add cloud tag into a Slack channel, you can chime in, your teammates can chime in, and you can collaborate together on the PR.

s87
00:07:13.120 --> 00:07:16.880
The other big difference is that it's proactive instead of reactive.

s88
00:07:16.880 --> 00:07:31.840
So you can tell claw tag, hey, monitor every bug report in this channel, put up a PR to fix it, and tag the engineer who last touched this part of the codebase, and it'll do it for the lifetime of the channel without you having to manually tag it in.

s89
00:07:31.840 --> 00:07:37.520
And then the third big shift that we've seen is um we've added team memory into this.

s90
00:07:37.520 --> 00:07:43.440
So if you tell quad tag your preferences in the channel, it'll remember this for every future post.

s91
00:07:43.440 --> 00:07:59.120
So if you wanted to um always debug like if you always wanted to debug outages, but you don't want it to debug warnings, um just tell it that in natural language in the channel and it'll remember it for you and everyone else on your team.

s92
00:07:59.120 --> 00:08:02.400
Internally we see quad tag as the evolution of quad code.

s93
00:08:02.400 --> 00:08:06.639
So we see this as a large shift in how we work internally.

s94
00:08:06.639 --> 00:08:13.919
Um claw tag currently lands 65% of our product PRs

s95
00:08:13.919 --> 00:08:18.000
for for all of anthropic and for cloud code or just for cloud code.

s96
00:08:18.000 --> 00:08:20.639
Uh this is just for our product engineering team.

s97
00:08:20.639 --> 00:08:26.639
So our internal version of claw tag lands 65% of our product PRs right now.

s98
00:08:26.639 --> 00:08:27.919
And this is a huge shift.

s99
00:08:27.919 --> 00:08:30.250
This is like this is more than 50%

s100
00:08:30.250 --> 00:08:30.560
[laughter]

s101
00:08:30.560 --> 00:08:31.840
of our PRs.

s102
00:08:31.840 --> 00:08:41.519
Um, and the way that we actually see people split work between cloud code and cloud tag is cloud code is still the best place for your most complex tasks

s103
00:08:41.519 --> 00:08:45.040
when you're interactively iterating with the agent.

s104
00:08:45.040 --> 00:09:00.000
But claw tag is great for having it work proactively on your behalf so that you no longer need to um manually uh kick off quad codes for for all of the bug reports that might come up for features that you're working on.

s105
00:09:00.000 --> 00:09:00.320
Yeah.

s106
00:09:00.320 --> 00:09:10.720
And for like non-coding cases like I I I think we've seen people use claw tag just uh like for example before this before this talk we asked claw tag like hey when is Fable releasing?

s107
00:09:10.720 --> 00:09:14.000
We wanted to make sure that like you know we'd line it up with the announcement.

s108
00:09:14.000 --> 00:09:19.120
Um, and so Cloud Tag would search our Slack and and look at, you know, who who's been saying what.

s109
00:09:19.120 --> 00:09:22.240
Uh, so as a search engine for your company is really valuable.

s110
00:09:22.240 --> 00:09:24.480
Uh, it has all the context for your product.

s111
00:09:24.480 --> 00:09:26.640
So you can ask it like metrics related questions.

s112
00:09:26.640 --> 00:09:31.760
And oftent times when you're making decisions, you want it to be informed by like, you know, what do the metrics say?

s113
00:09:31.760 --> 00:09:34.240
And then so like you hook it up to your event store.

s114
00:09:34.240 --> 00:09:38.320
Um, I've seen like our marketing team do things like, oh, like, hey, tell me about this feature.

s115
00:09:38.320 --> 00:09:40.720
And, you know, they're not programmers, but Claude is a programmer.

s116
00:09:40.720 --> 00:09:44.080
I can clone the code base and be like, "Oh yeah, this is like, you know, the feature.

s117
00:09:44.080 --> 00:09:45.120
This is what it looks like.

s118
00:09:45.120 --> 00:09:48.240
This is a recording of me using the feature, you know.

s119
00:09:48.240 --> 00:09:51.760
So like um yeah, it just enables a whole wide variety of things."

s120
00:09:51.760 --> 00:09:54.560
And I think we're still early on in figuring that out.

s121
00:09:54.560 --> 00:09:54.880
Yeah.

s122
00:09:54.880 --> 00:10:01.839
Well, I feel like this is one of the fascinating things about the Claude Code story is you use Claude Code to build Claude Code and you've been doing this since

s123
00:10:01.839 --> 00:10:05.839
presumably before the public launch of launch of Claude Code a year and a half ago.

s124
00:10:05.839 --> 00:10:12.720
And yeah, one of the problems I've had with coding agents is I get how to use them as an individual, but I'm not really clear on how I use that in a team in a team environment.

s125
00:10:12.720 --> 00:10:18.880
It sounds like Claude Tag is is your current answer to that sort of team collaborative layer for this stuff.

s126
00:10:18.880 --> 00:10:19.440
Exactly.

s127
00:10:19.440 --> 00:10:23.600
And a large percentage of our sessions are actually multiplayer right now.

s128
00:10:23.600 --> 00:10:30.399
So that means maybe I say, "Hey, I think we should like um implement this new feature in co-work."

s129
00:10:30.399 --> 00:10:33.760
And I'll tag in claw tag to do a first pass at it.

s130
00:10:33.760 --> 00:10:39.440
And then I'll tell cloud tag, hey, just like share a recording of of your final implementation.

s131
00:10:39.440 --> 00:10:47.600
And then I'll tag in design to take a look and they'll nudge it and then they'll pass it on to edge to like take it to the finish line and get it out to prod.

s132
00:10:47.600 --> 00:10:51.839
And so it's been this like very fluid experience.

s133
00:10:51.839 --> 00:11:04.320
We're still trying to iron out what the social dynamics are for steering the same session, but we found that people just like observe how others use it and then follow those social norms

s134
00:11:04.320 --> 00:11:10.880
and it's actually been pretty easy for pretty intuitive for us to integrate claw tag into our teams.

s135
00:11:10.880 --> 00:11:13.200
Yeah, I I think it's great for like Yeah.

s136
00:11:13.200 --> 00:11:28.160
teaching people and also like kind of kind of reducing slo because because you know like if you see someone just be like hey at Claude fix this or something you're like uh you know like I I think there's like some societal or not societal like just like uh like the fact that everyone is seeing you use cloud

s137
00:11:28.160 --> 00:11:30.480
together sort of levels up how you use claude as well.

s138
00:11:30.480 --> 00:11:30.640
So

s139
00:11:30.640 --> 00:11:36.800
right you you want to do work that you're proud to do in public that where the quality doesn't doesn't fall off the cliff.

s140
00:11:36.800 --> 00:11:45.360
So since we're talking about using claude to build CL well how the process for building claude how do you deal with the hardest problem in all of engineering it's prioritization

s141
00:11:45.360 --> 00:11:55.268
right how do you decide which features are worth building and shipping when building a feature is so much more inexpensive now this is the hard thing

s142
00:11:55.268 --> 00:11:57.120
[laughter]

s143
00:11:57.120 --> 00:12:10.959
so there's a few ways we approach it one is we dog food our products every single day whenever there's something that we want to be able to do in our products that we're not able to instead of finding a different solution we fix our product

s144
00:12:10.959 --> 00:12:13.200
uh so that it can support this case.

s145
00:12:13.200 --> 00:12:16.320
Uh we have a very heavy dog fooding culture internally.

s146
00:12:16.320 --> 00:12:28.320
So before we are able to share our products with everyone in the world, we share it with everyone within anthropic and we share it with some early customers who give us very honest feedback about it.

s147
00:12:28.320 --> 00:12:31.760
The more brutal the better and we iterate until people love it.

s148
00:12:31.760 --> 00:12:41.600
So we we have a internal bar for the number of active users and the the amount of retention a feature has to have before we share it with the world.

s149
00:12:41.600 --> 00:12:46.399
And because this bar is very clear, every engineer knows what they're trying to hit.

s150
00:12:46.399 --> 00:12:54.399
And I think this also levels up our polish because if the feature isn't polished, people will churn and then we shouldn't ship that feature.

s151
00:12:54.399 --> 00:12:57.200
Do you have an example of a feature which surprised you?

s152
00:12:57.200 --> 00:13:06.560
you you rolled it out and the engagement was off the charts and it it became was something that was unlikely to be shipped that actually turned into a a real product thing.

s153
00:13:06.560 --> 00:13:08.079
I do have one.

s154
00:13:08.079 --> 00:13:12.959
So a lot of folks on our team love remote control.

s155
00:13:12.959 --> 00:13:23.040
So remote control lets you connect use your mobile device or quad in the web browser to connect a local quad code session running in your CLI.

s156
00:13:23.040 --> 00:13:30.800
I never have this need because I just kick off the task directly on mobile and it runs in a cloud session and doesn't use my local environment.

s157
00:13:30.800 --> 00:13:34.639
I think this is because I'm doing very easy coding tasks.

s158
00:13:34.639 --> 00:13:36.880
But this is something where I didn't totally understand it.

s159
00:13:36.880 --> 00:13:40.079
I was like, hey, people should just set up their remote dev environments.

s160
00:13:40.079 --> 00:13:55.040
But in practice, once we rolled out remote control, everyone like so many people who I talk to are like, "Okay, now what I do every night is I um a lot of people tell me that they just plug their lo their laptop into a power charger,

s161
00:13:55.040 --> 00:14:04.079
close the screen or like open a bunch of remote control sessions, lock the screen, and then use their mobile phone from their couch to to control quad code."

s162
00:14:04.079 --> 00:14:09.600
And so this has been this flow that we're now leaning into that I didn't originally get, but now I do.

s163
00:14:09.600 --> 00:14:10.880
I do exactly that.

s164
00:14:10.880 --> 00:14:17.360
I get so much so much work on my laptop done from more comfortable environments because yeah, I can remote control it now.

s165
00:14:17.360 --> 00:14:18.800
That's really fun.

s166
00:14:18.800 --> 00:14:21.839
Um, how does code review work?

s167
00:14:21.839 --> 00:14:27.920
Are you pres are you reviewing does a human being review every line of production code that makes it into clawed code?

s168
00:14:27.920 --> 00:14:29.519
And if not, what are you doing?

s169
00:14:29.519 --> 00:14:32.079
How do you keep the quality up?

s170
00:14:32.079 --> 00:14:32.399
Sure.

s171
00:14:32.399 --> 00:14:32.639
Yeah.

s172
00:14:32.639 --> 00:14:36.079
Um, it it varies on the task a lot.

s173
00:14:36.079 --> 00:14:39.519
So, so we um for important areas we have code owners, right?

s174
00:14:39.519 --> 00:14:47.680
And so, uh the system prompt is kind of an example where we have a code owner uh you really need to like uh submit your uh you know you need to get their approval.

s175
00:14:47.680 --> 00:14:49.120
Um and then uh

s176
00:14:49.120 --> 00:14:53.600
so I guess the code anyway is directly responsible for the quality of that area of the code.

s177
00:14:53.600 --> 00:14:53.920
That's right.

s178
00:14:53.920 --> 00:14:54.079
Yeah.

s179
00:14:54.079 --> 00:14:54.639
Yeah.

s180
00:14:54.639 --> 00:14:56.720
And they need to approve the PR that touches it.

s181
00:14:56.720 --> 00:14:57.199
Right.

s182
00:14:57.199 --> 00:14:58.000
That's right.

s183
00:14:58.000 --> 00:15:11.519
uh we have code review our like uh our uh code review git b you know review everything and and so um that like goes on every PR and often times like that's doing the bulk of the review.

s184
00:15:11.519 --> 00:15:21.519
Um I think that like uh we I something I've seen on the team is like for more complex PRs you might make like an artifact to explain the PR so that other people can then review.

s185
00:15:21.519 --> 00:15:38.480
Um and uh yeah we just invest a lot into verification CI/CD things like that to make sure that like you know uh anytime anything fails like we have a test we have like uh a really robust environment cloud can control cloud code and test it you know what I mean so

s186
00:15:38.480 --> 00:15:46.800
um yeah there is a uh yeah there's just like a multi-pronged approach to like code review I think you have anything to add

s187
00:15:46.800 --> 00:15:56.800
in general we are trying to move to a world where humans don't need to be in the loop And so for the most critical core changes to the core of quad code

s188
00:15:56.800 --> 00:16:04.240
and other the cores of other products, there is always a code owner and they do manually review all the changes.

s189
00:16:04.240 --> 00:16:12.959
But increasingly for the changes that are at the outer layers, we actually have quad code review fully review those.

s190
00:16:12.959 --> 00:16:23.120
That sounds pretty scary but there we've had this six plus monthl long process to get here and I think there are like baby steps that you take to build up trust with code review.

s191
00:16:23.120 --> 00:16:35.680
So in the beginning we would have human review for everything and then increasingly we would say okay for code changes that touch these files code review is catching a 100% of the issues there.

s192
00:16:35.680 --> 00:16:39.759
So we actually don't need a human to be manually reviewing those.

s193
00:16:39.759 --> 00:16:56.000
Um and then also when we have incident review, we look at the PRs that cause the incident and we say okay how do we update code review to catch that and then we also take those PRs and add it to an eval set to make sure that our future changes to code review never regress that metric.

s194
00:16:56.000 --> 00:17:02.399
So it it is a big like removing humans from the code review loop is a big step forward.

s195
00:17:02.399 --> 00:17:12.160
I think it can sound scary and it's not something that you can do overnight, but it is something that you can do through like many months of investment in the infrastructure

s196
00:17:12.160 --> 00:17:16.720
to give you the confidence that code review is catching everything that you care about.

s197
00:17:16.720 --> 00:17:26.319
So, it's interesting you mentioned building trust in the models because that's something I found is like I know that Opus 4.8 if I ask it to build me a JSON endpoint that runs a SQL query and outputs JSON,

s198
00:17:26.319 --> 00:17:27.520
it's just going to get it right.

s199
00:17:27.520 --> 00:17:30.080
Like that that that's not something I have to review closely.

s200
00:17:30.080 --> 00:17:38.559
But then a new model comes along and I still don't know how do I build trust in fable quickly that it's not going to mess things up that Opus didn't.

s201
00:17:38.559 --> 00:17:47.440
Is that something that you you you have to think about much the those new model like how does the new model affect your intuition for for what it can do and what it can't do?

s202
00:17:47.440 --> 00:18:03.840
So the main reason that we're building up this eval base over time is so that new models can be a drop-in replacement because what we do when we have a new model is we run the whole eval set and we make sure that for example Fable is strictly better than Opus 48 and that gives us the confidence

s203
00:18:03.840 --> 00:18:04.960
to drop it in.

s204
00:18:04.960 --> 00:18:11.039
And are those model evals for Anthropic as a whole or are these claud code team specific evals that you're using?

s205
00:18:11.039 --> 00:18:11.760
We have both.

s206
00:18:11.760 --> 00:18:25.039
So we we have evals on our team and we run code review across every repo within Anthropic and so we have evals for that and for things like auto mode we not only have evals across every user within Enthropic

s207
00:18:25.039 --> 00:18:37.280
but we've also commissioned multiple external testers to red team this to create environments with prompt injections and malicious inputs and make sure that automode doesn't let any of those pass.

s208
00:18:37.280 --> 00:18:41.600
So for claude code itself and this is a challenge I've had with stuff I'm building.

s209
00:18:41.600 --> 00:18:45.039
I want to know if the system prompt improvement I made actually improved the product.

s210
00:18:45.039 --> 00:18:45.280
Right?

s211
00:18:45.280 --> 00:18:48.720
That's that's the sort of most basic form of product specific eval.

s212
00:18:48.720 --> 00:18:51.120
And I still don't have a great feel for how to do that.

s213
00:18:51.120 --> 00:19:00.960
Is something is that something that you're doing such that you have complete confidence that this tweak that you've made to the system prompt does result in in better in better output?

s214
00:19:01.520 --> 00:19:07.760
We don't have complete confidence but we do a lot to make sure that we don't regress performance.

s215
00:19:07.760 --> 00:19:17.039
So the starting point that we have is we have a suite of external evals that we trust and we complement that with an even larger suite of internal evals

s216
00:19:17.039 --> 00:19:18.240
that we trust.

s217
00:19:18.240 --> 00:19:21.200
Uh to start we mainly optimize for capability.

s218
00:19:21.200 --> 00:19:32.160
So given given a complete definition of a task and the full codebase does claude make the right decisions and fully fix the the bugs and pass all the tests.

s219
00:19:32.160 --> 00:19:38.559
So that's the starting point and that's the thing that we optimize for because it is like most directly what users want.

s220
00:19:38.559 --> 00:19:44.640
But there's a lot of like behaviors that impact how users feel when they work with quad code.

s221
00:19:44.640 --> 00:19:50.148
For example, people really don't like it when cloud code says it's like time to go to sleep.

s222
00:19:50.148 --> 00:19:51.039
[laughter]

s223
00:19:51.039 --> 00:19:54.640
or people really don't like it when it says like, "Hey, I finished two out of five parts.

s224
00:19:54.640 --> 00:19:55.760
Like, do you want me to continue?"

s225
00:19:55.760 --> 00:19:57.440
Like, "Yes, please continue."

s226
00:19:57.440 --> 00:20:02.080
Um, and so we're building up a set of behavioral emails to catch these.

s227
00:20:02.080 --> 00:20:06.320
And as we get user feedback, please be loud with us about your user feedback.

s228
00:20:06.320 --> 00:20:10.160
As we get user feedback, we just rank, okay, these are the priority issues.

s229
00:20:10.160 --> 00:20:12.960
And we go down one by one and build evals for each of them.

s230
00:20:12.960 --> 00:20:19.200
So, it's not 100% coverage, but we try to it it is a priority for us to increase the coverage.

s231
00:20:19.200 --> 00:20:26.080
And how much overlap is there between the how much interaction is there between the Claude code team and the teams at Anthropic who are training the models in the first place?

s232
00:20:26.080 --> 00:20:28.000
Is is that quite a close collaboration?

s233
00:20:28.000 --> 00:20:30.000
Now

s234
00:20:30.400 --> 00:20:33.520
across Anthropic, we all work quite closely together.

s235
00:20:33.520 --> 00:20:41.520
So um we we meet often to talk about like what do we expect the next generation of models to be able to do.

s236
00:20:41.520 --> 00:20:46.400
I think our research team has also been amazing about just like showing this publicly.

s237
00:20:46.400 --> 00:20:57.679
So we often talk in our blog posts about how we're targeting everinccreasing longer horizon work, how we train quad itself to be honest, harmless, and helpful.

s238
00:20:57.679 --> 00:21:06.240
Um, we also put a lot of effort into making sure that it's aligned with your intent, even if your intent is expressed in a fuzzy way.

s239
00:21:06.240 --> 00:21:09.120
Of course, try your best to be specific about what you want.

s240
00:21:09.120 --> 00:21:22.000
So Quad um has all the context, but even when you're you're not specific, uh we we teach Claude to make good assumptions and yeah, I I think it's been a it's been a productive partnership.

s241
00:21:22.000 --> 00:21:29.120
And so um Derek, you this morning you mentioned that the system prompt for Claude Code has been reduced by 80%.

s242
00:21:29.120 --> 00:21:33.919
Because of Claude Fable, can you go into a little bit more detail about what that looks like?

s243
00:21:33.919 --> 00:21:35.840
What kind of things have you been able to drop?

s244
00:21:35.840 --> 00:21:36.080
Yeah.

s245
00:21:36.080 --> 00:21:39.440
So it wasn't just Fable, it was uh Opus 4.8 as well.

s246
00:21:39.440 --> 00:21:46.080
And um yeah, going forward the future models, but we do sort of um we have different system prompts for different models now.

s247
00:21:46.080 --> 00:21:53.200
Um I think that like uh some of the patterns we saw is that um we were over constraining Claude, right?

s248
00:21:53.200 --> 00:22:05.039
So I think the initial like maybe Opus 4ish kind of models wanted a lot of examples and uh removing examples was extremely helpful because it was just more creative than like

s249
00:22:05.039 --> 00:22:06.799
uh you know the examples we gave it.

s250
00:22:06.799 --> 00:22:12.559
That's really because one of the top prompting tips I give people is give it examples like examples are the easiest ways.

s251
00:22:12.559 --> 00:22:16.320
If that's no longer true that kind of breaks my prompting model a little bit.

s252
00:22:16.320 --> 00:22:17.760
Ex yeah same same here.

s253
00:22:17.760 --> 00:22:19.120
I think I was surprised to hear that.

s254
00:22:19.120 --> 00:22:27.039
I I think that now it's more about like sort of the shape of what you give the tools to Claude and like yeah your system prompt and things like that.

s255
00:22:27.039 --> 00:22:37.200
Um the other thing we did is we um we we try and give it more context and fewer like do not do this you know because like I I think that

s256
00:22:37.200 --> 00:22:46.720
um it's just a very strong impulse to Claude and especially if that uh conflicts with user instructions later on that can be like extremely confusing to Claude, right?

s257
00:22:46.720 --> 00:22:50.960
because you're like, "Oh, like I've got this skill that says this and the system prompt says this."

s258
00:22:50.960 --> 00:22:58.559
Um, and so we try and like uh have fewer hard constraints and more just like sort of context and just like fewer instructions overall.

s259
00:22:58.559 --> 00:23:01.360
Um, yeah, I think it's definitely a science.

s260
00:23:01.360 --> 00:23:03.120
It took a bunch of like evals to build.

s261
00:23:03.120 --> 00:23:06.880
I'm not sure if you had anything else on uh the lean system.

s262
00:23:06.880 --> 00:23:15.280
I think in general when you're prompting these models, you should always think about like are there edge cases to the instruction that I'm giving it?

s263
00:23:15.280 --> 00:23:28.880
And when we went back and we reviewed all the instructions in the cloud code system prompt, we found a few cases where yes, this statement is like 90% true, but there's like a real 10% of cases where this is not true.

s264
00:23:28.880 --> 00:23:34.559
And we didn't want to constrain the model or like confuse it into thinking, hey, it should always do this.

s265
00:23:34.559 --> 00:23:36.640
Like one good example is verification.

s266
00:23:36.640 --> 00:23:39.280
Everyone here wants claw to verify its work.

s267
00:23:39.280 --> 00:23:46.799
Um, and we had some instructions in the prompt that just said um, if you make a front-end change, always verify.

s268
00:23:46.799 --> 00:23:49.760
But you know, there there is a limit to it.

s269
00:23:49.760 --> 00:24:00.720
Like for example, if you if it's changing copy from one string to another string and the user tells says it says like just make a quick fix and update the test, maybe you don't want to verify.

s270
00:24:00.720 --> 00:24:12.799
And so we we've also adjusted our wording from saying always verify verify verify verify to like hey most of the time when you're doing front-end work you can't always understand the full

s271
00:24:12.799 --> 00:24:15.200
experience by hitting the backend endpoints.

s272
00:24:15.200 --> 00:24:26.960
So like when when you make like more changes to the user experience uh please run the app locally and actually in fact that instruction probably isn't even good because

s273
00:24:26.960 --> 00:24:31.279
what is a large change like maybe it wants to change it test it for small changes too.

s274
00:24:31.279 --> 00:24:41.760
Um, in general, whenever you give a prompt to the model, you should always think about the ways in which it could be misinterpreted by like a well-intentioned uh other user or human

s275
00:24:41.760 --> 00:24:54.080
in order to better understand how the model might interpret it and in order to make sure that you can soften the prompt such that it is actually 100% accurate because you are giving this prompt to the model 100% of the time.

s276
00:24:54.080 --> 00:24:57.679
But what's fascinating about that is you're relying on the model's judgment.

s277
00:24:57.679 --> 00:25:00.799
And that's got to be a opus fable level thing.

s278
00:25:00.799 --> 00:25:08.159
Like models a year ago did not have the levels of judgment necessary to decide if they were going to like test a change or not.

s279
00:25:08.159 --> 00:25:09.360
That's absolutely fascinating.

s280
00:25:09.360 --> 00:25:16.159
But that does break down if you're building for a wide range of models and trying to run the cheaper models for cheaper tasks and stuff.

s281
00:25:16.159 --> 00:25:20.880
We actually have a different system per model now because of this very reason.

s282
00:25:20.880 --> 00:25:28.720
So, it's only our uh most frontier models that have this 80% token decrease and the older models actually still have the full system prompt.

s283
00:25:28.720 --> 00:25:39.919
Do you think Fable and Opus are smart enough to be able to prompt Haiku with more details because they understand that Haiku has less judgment, has less taste?

s284
00:25:41.279 --> 00:25:50.080
We haven't been able to evalu, but we don't have any hard data to show it.

s285
00:25:50.080 --> 00:26:02.880
I think there's a tough thing with um smaller models sometimes because like uh you know we saw this with um just like sometimes the the larger models can be more token efficient on a hard problem than the smaller models.

s286
00:26:02.880 --> 00:26:14.320
And so uh you know there's like a little bit of that uh intuition to build about like you know sometimes you really just want frontier intelligence almost all the time you know um but it's uh

s287
00:26:14.320 --> 00:26:17.600
yeah like the paralle curve shifts you know and so it's hard to find.

s288
00:26:17.600 --> 00:26:17.840
Yeah.

s289
00:26:17.840 --> 00:26:19.120
I mean that's something I found fascinating.

s290
00:26:19.120 --> 00:26:22.799
I feel like a year ago I did not trust a model to write a prompt.

s291
00:26:22.799 --> 00:26:26.080
Like today the good models are very good at prompting.

s292
00:26:26.080 --> 00:26:31.360
Like a lot of my prompts are written by models which feels absurd but it actually works really well.

s293
00:26:31.360 --> 00:26:39.679
And something that helped me come to terms with that was thinking about sub agents which is entirely about a clawed model setting up a prompt for another claw model so that it knows what to go and do.

s294
00:26:39.679 --> 00:26:47.039
Yeah, I think workflows are actually a really good example of this because it's like cloud not just prompting a single sub agent, but it's like pro prompting like the orchestration

s295
00:26:47.039 --> 00:26:51.120
of many sub aents and each one of them gets like, you know, a very detailed prompt.

s296
00:26:51.120 --> 00:26:56.559
So, it's like almost like a level above like, you know, just spawning a sub agent.

s297
00:26:56.559 --> 00:26:58.080
Um, so yeah, it's quite good at that.

s298
00:26:58.080 --> 00:27:04.880
Yeah, I've also been using on my personal machine like giving it the Gemini API and being like, oh, like here, generate images.

s299
00:27:04.880 --> 00:27:09.600
And it's so good at it's way less lazy than I am at prompting an image model, you know.

s300
00:27:09.600 --> 00:27:13.840
So, um yeah, it's just Claude prompting Claude all the way down.

s301
00:27:13.840 --> 00:27:14.240
Yeah,

s302
00:27:14.240 --> 00:27:18.400
I think Claude also wrote the prompt for the workflow tool.

s303
00:27:18.400 --> 00:27:20.480
For the workflow tool, I've read that prompt.

s304
00:27:20.480 --> 00:27:22.559
It's a good prompt.

s305
00:27:22.559 --> 00:27:28.080
I mean, that's actually um a frustration I have with Anthropic generally is you publish the prompts for Claude.

s306
00:27:28.080 --> 00:27:32.159
There's there's a web page with them on, but you don't include the tool prompts and the Claude code prompts.

s307
00:27:32.159 --> 00:27:34.799
I still have to run a proxy to intercept them.

s308
00:27:34.799 --> 00:27:39.279
I would love it if the Claude code prompts were deliberately published because they're the documentation.

s309
00:27:39.279 --> 00:27:42.000
They're how you know what the tool can do and how it works.

s310
00:27:42.000 --> 00:27:44.000
I'll write down that feature request.

s311
00:27:44.000 --> 00:27:45.200
Please do.

s312
00:27:45.200 --> 00:27:47.440
I'll have Claude tag do it.

s313
00:27:47.440 --> 00:27:55.440
And also the diffs like every now and then I'll do I'll diff the older and the newer prompt and that's how I learn the capabilities of the new model.

s314
00:27:55.440 --> 00:27:59.039
I'm really looking forward to seeing what this 80% reduction actually looks like.

s315
00:27:59.039 --> 00:27:59.919
Yeah, this is on me.

s316
00:27:59.919 --> 00:28:02.559
I have to make a post about this in detail.

s317
00:28:02.559 --> 00:28:03.360
Yeah.

s318
00:28:03.360 --> 00:28:04.559
So, what's your bar?

s319
00:28:04.559 --> 00:28:06.159
Let's talk about tools a little bit.

s320
00:28:06.159 --> 00:28:09.840
Claude code is basically a just bag of to a big bag of tools.

s321
00:28:09.840 --> 00:28:12.480
What's your bar for introducing a new tool?

s322
00:28:12.480 --> 00:28:16.720
How do you decide when it's worth doing that additional engineering at that level?

s323
00:28:16.720 --> 00:28:17.600
Do you want to take it?

s324
00:28:17.600 --> 00:28:20.640
Because you introduced one of the best tools we have.

s325
00:28:20.640 --> 00:28:21.039
Yeah.

s326
00:28:21.039 --> 00:28:25.360
I mean, yeah, it's like my career peaked when I introduced the ask user question tool.

s327
00:28:25.360 --> 00:28:26.399
I I think so.

s328
00:28:26.399 --> 00:28:28.640
Um, it it's really hard.

s329
00:28:28.640 --> 00:28:38.240
I I I think is the the especially for some tools like ask user question is claude's tool to ask you and so it's hard to eval and and sometimes it's more of a user preference thing.

s330
00:28:38.240 --> 00:28:41.600
So like um especially back then we had fewer evals.

s331
00:28:41.600 --> 00:28:48.720
It was very ant fooding based um or sorry dog fooding is yeah ant fooding is you know our ant version of that.

s332
00:28:48.720 --> 00:28:54.480
Um but yeah I mean I think overall we've been trying to trend towards fewer tools.

s333
00:28:54.480 --> 00:29:02.159
I think the last set of tools we introduced were like the task tool I think um and try and give cloud more general versions to do this.

s334
00:29:02.159 --> 00:29:02.799
Um

s335
00:29:02.799 --> 00:29:12.480
right one of the most interesting tools is the file editing tool which but you can have file editing as a tool or you can tell teach it tell it to use said and GP and and do things that way.

s336
00:29:12.480 --> 00:29:15.360
What's the latest evolution of your file editing tool?

s337
00:29:15.360 --> 00:29:20.799
Uh I think we still have one but like for example we removed our GP and uh other search tools.

s338
00:29:20.799 --> 00:29:34.480
glob tools for just like native uh like bash and so uh yeah we still have one I I think this is kind of like I said in my uh talk earlier that the models are kind of like more of a biology than a physics you know and so like

s339
00:29:34.480 --> 00:29:51.600
you know it's hard to like especially tool design I think is quite hard and I'm not sure if actually if cat disagrees and is like oh like we should no there's like a science to the ebal of it but I'm sort of like yeah tool design is more of an art maybe or like a biology Yeah,

s340
00:29:51.600 --> 00:30:08.640
I think I largely agree, but that I think in general as we introduce more tools, it like in general we try to keep the cardality pretty low and make sure that every tool we add has a distinct function from every other tool so that Claude can very easily distinguish

s341
00:30:08.640 --> 00:30:10.000
when to call each.

s342
00:30:10.000 --> 00:30:11.520
for file edit.

s343
00:30:11.520 --> 00:30:19.760
Actually, the reason that we have file edit is because we can render it because back back in the day we used to show or I guess we still do.

s344
00:30:19.760 --> 00:30:28.640
Um, so we show people when quad makes a file change and there's this nice like dedicated UI that just says uh do you approve this edit to this file?

s345
00:30:28.640 --> 00:30:36.960
And the reason that we had a dedicated file edit tool was so that we could deterministically know that quad was making a file so we could show people this nice UI.

s346
00:30:36.960 --> 00:30:41.600
And for a lot of the new users who are onboarding, I think they still really like this experience.

s347
00:30:41.600 --> 00:30:43.039
So, we've kept it around.

s348
00:30:43.039 --> 00:30:48.159
But for a lot of us who are on auto mode right now, or hopefully you're not on YOLO mode.

s349
00:30:48.159 --> 00:30:57.039
But anyway, for a lot of us right now, I don't think actually it matters and we probably could just remove file edit and we'll be totally fine.

s350
00:30:57.039 --> 00:31:01.120
So, let's talk about auto mode or let's talk about safety and security in general.

s351
00:31:01.120 --> 00:31:10.559
Like I am deeply aware of the risks of prompt injection and there there are so much bad things can happen if somebody else tells my clawed code what to do.

s352
00:31:10.559 --> 00:31:15.200
I still mostly run clawed code in yellow mode and feel incredibly guilty about it.

s353
00:31:15.200 --> 00:31:20.000
What's the advice within anthropic for safely running clawed code?

s354
00:31:20.000 --> 00:31:21.520
Like what do you tell people to do?

s355
00:31:21.520 --> 00:31:23.200
Why not auto mode?

s356
00:31:23.200 --> 00:31:27.840
I am starting to use auto mode and I don't understand it enough to get how safe it is.

s357
00:31:27.840 --> 00:31:32.399
But yeah, that's as of maybe three weeks ago, I'm defaulting to auto mode.

s358
00:31:32.399 --> 00:31:38.000
Okay, so broadly within Enthropic, almost every single person uses auto mode.

s359
00:31:38.000 --> 00:31:44.240
It is the best way to do longunning work in quad code while being safe.

s360
00:31:44.240 --> 00:31:46.080
We've done extensive bashing.

s361
00:31:46.080 --> 00:31:47.519
We have thousands of eval.

s362
00:31:47.519 --> 00:31:56.559
We've commissioned many red team teamers to create adversarial environments in order to trick cloud code into doing bad actions.

s363
00:31:56.559 --> 00:31:59.919
and we've mitigated every single issue that they found.

s364
00:31:59.919 --> 00:32:08.080
And so we're going to publish some evals in the in the coming weeks, but we we've pretty much mitigated every attack.

s365
00:32:08.080 --> 00:32:09.679
That is a big claim.

s366
00:32:09.679 --> 00:32:10.720
That's very exciting.

s367
00:32:10.720 --> 00:32:12.000
If that holds up,

s368
00:32:12.000 --> 00:32:26.480
we we will we'll share the evals for it so uh f folks can assess, but we've been extremely diligent about identifying all the ways in which quad might mess up and then updating auto mode to counter it.

s369
00:32:26.480 --> 00:32:29.120
It doesn't catch 100% of things.

s370
00:32:29.120 --> 00:32:33.200
Um I don't think any Yeah, that would be way too strong of a claim.

s371
00:32:33.200 --> 00:32:44.640
But for the main categories of risks that we're concerned about like prompt injection, data exfiltration, um the risks are far lower than the average human reviewer.

s372
00:32:44.640 --> 00:32:47.679
So, and oh yeah, a little bit on how auto mode works.

s373
00:32:47.679 --> 00:32:49.519
I I think it's useful to build this mental model.

s374
00:32:49.519 --> 00:33:01.440
So, whenever Claude is doing a turn, uh there's a or a bash call, uh there's a sonet classifier that is judging the tool and also the context of the conversation, your instruction,

s375
00:33:01.440 --> 00:33:01.840
right?

s376
00:33:01.840 --> 00:33:18.559
And so there are some things around like uh permissions which are dependent on your request right so you don't want to give git push like permissions all the time but if you say hey push this to github you want it to do it right and so auto mode will or if you say don't push you want it

s377
00:33:18.559 --> 00:33:28.159
to deny it right and so auto mode will do that particular thing happens to me a lot where it's like oh like auto mode like I like claw tried to do this because it's very helpful and proactive

s378
00:33:28.159 --> 00:33:33.760
and auto mode saw like oh like you know uh you know don't do this and it like surfaced it.

s379
00:33:33.760 --> 00:33:41.120
So it's good at like the dynamic permissions that you yourself give inside of the prompt which I think is really important.

s380
00:33:41.120 --> 00:33:48.720
Um it also works well with our sandboxing infrastructure because like sandboxing is one of those things where there are so many different edge cases.

s381
00:33:48.720 --> 00:33:52.480
Uh and it's hard for like us to deterministically follow them.

s382
00:33:52.480 --> 00:34:06.240
But if you we have a sandbox and something needs to escape the sandbox like a you know a network request auto mode can then look at that request and be like oh hey does this like you know does this make sense right and just allow that in.

s383
00:34:06.240 --> 00:34:10.480
So I hadn't realized auto mode is interacting with the networking sandbox as well.

s384
00:34:10.480 --> 00:34:11.040
Yeah exactly.

s385
00:34:11.040 --> 00:34:12.480
So it's also part of sandbox.

s386
00:34:12.480 --> 00:34:13.440
Um yeah

s387
00:34:13.440 --> 00:34:17.280
it it interacts with any permission prompt that the user would otherwise see.

s388
00:34:17.280 --> 00:34:19.119
And how old is auto mode?

s389
00:34:19.119 --> 00:34:23.679
like I feel like as a feature that I had access to, it's it's only a couple of months old, right?

s390
00:34:23.679 --> 00:34:26.800
We've been using it within Anthropic since January.

s391
00:34:26.800 --> 00:34:27.280
Okay.

s392
00:34:27.280 --> 00:34:30.320
So, we've been hardening it for for quite a while.

s393
00:34:30.320 --> 00:34:34.960
And it's obviously Anthropic is extremely focused on safety and security.

s394
00:34:34.960 --> 00:34:47.200
And so, we've been working broadly across our um alignment and safeguards teams in order to enable the rollout internally, build out these eval even more robust before sharing it out with the world.

s395
00:34:47.200 --> 00:34:51.200
I think my only my main problem with automotive is I don't understand it deeply enough.

s396
00:34:51.200 --> 00:35:00.800
Like for anything that's looking after my security, I want to know as much as I can about how it works and what it protects me against and what it doesn't so I can decide like how much I can trust it.

s397
00:35:00.800 --> 00:35:03.359
I I think yeah Dell was working on a post about this.

s398
00:35:03.359 --> 00:35:06.320
So um a little bit just a little bit more about auto mode.

s399
00:35:06.320 --> 00:35:08.960
This is also the reason cloud tag is so so good, right?

s400
00:35:08.960 --> 00:35:16.880
because cloud tag uses auto mode and like you can imagine that like one of like I've heard a lot of like build versus buy questions on on a Slackbot.

s401
00:35:16.880 --> 00:35:23.839
I'm like please you probably shouldn't build your own AI Slackbot, you know, like there's so many attack vectors, you know what I mean?

s402
00:35:23.839 --> 00:35:31.359
And like um like you have a a feedback channel that like you know users can post feedback into and now your bot is reading it, right?

s403
00:35:31.359 --> 00:35:38.320
And and so I think that like this the work we've put in with auto mode and you know we have a general Swiss cheese defense of like security, right?

s404
00:35:38.320 --> 00:35:42.480
We also like yeah you know like RL against this stuff and things like that.

s405
00:35:42.480 --> 00:35:45.440
Um I think this is really what makes cloud tag work.

s406
00:35:45.440 --> 00:35:53.599
It like just works seamlessly with your permissions and yeah like you know we you don't want to be prompt injected in your Slack.

s407
00:35:53.599 --> 00:35:53.760
Yeah.

s408
00:35:53.760 --> 00:35:54.160
Yeah.

s409
00:35:54.160 --> 00:36:00.320
Do you have any are there any more security things in the pipeline beyond that go beyond auto mode?

s410
00:36:00.480 --> 00:36:14.880
Um I I I think I mean I I think we're very secure like like so we with claude tag you can uh peri can provision your own like sort of credentials for claude so it doesn't need to act on your uh on your behalf.

s411
00:36:14.880 --> 00:36:21.040
You can have like claude as an identity and and that also makes it easier to audit and inspect what claude is doing.

s412
00:36:21.040 --> 00:36:24.160
Well I guess cuz claude tag is influenced by anyone who can talk to it.

s413
00:36:24.160 --> 00:36:28.160
So it's it's got a much wider pool of people who are telling it what to do.

s414
00:36:28.160 --> 00:36:28.560
That's right.

s415
00:36:28.560 --> 00:36:28.720
Yeah.

s416
00:36:28.720 --> 00:36:28.800
Yeah.

s417
00:36:28.800 --> 00:36:37.760
And of course we have probes as well like with like mythos and uh sorry with fable um and that's also like a downstream effect of our safety and research work.

s418
00:36:37.760 --> 00:36:53.119
And I think this is the moment where you sort of see AI like Anthropic being an AI safety company really paying off when you like you know we really want Claude to be able to run in an aligned way over long periods of time and like yeah automode has to be basically flawless for this to work right

s419
00:36:53.119 --> 00:36:58.640
and it's sort of like all downstream of our like you know our being an AI safety company.

s420
00:36:58.640 --> 00:37:06.079
We also launch trusted devices for the remote control users out here who want to be safer.

s421
00:37:06.079 --> 00:37:12.160
Um, and for all of our remote environments, we support uh credential injection.

s422
00:37:12.160 --> 00:37:21.200
So if you want like quad code to be able to access data dog but you don't want quad code itself to hold the data dog credential you can set up

s423
00:37:21.200 --> 00:37:31.040
um our identity uh credential management system so that the data dog credentials are only usable by the agent but not accessible by the agent.

s424
00:37:31.040 --> 00:37:34.880
So we insert it on the fly when the agent tries to make a data dog request.

s425
00:37:34.880 --> 00:37:37.760
This is that that token the proxying trick, right?

s426
00:37:37.760 --> 00:37:42.400
Where the proxy knows anytime somebody calls this an API.

s427
00:37:42.400 --> 00:37:46.720
Dog.com address with a token, replace the token with the real thing.

s428
00:37:46.720 --> 00:37:47.839
I love that pattern.

s429
00:37:47.839 --> 00:37:49.280
I'm seeing that in a whole bunch of places.

s430
00:37:49.280 --> 00:37:53.040
It feels so obviously right to me.

s431
00:37:53.200 --> 00:37:55.920
Let's talk a little bit about the human element.

s432
00:37:55.920 --> 00:38:10.640
And you touched on this in the keynote this morning, but um a lot of people are feeling a sense of loss now that so much of what they considered to be their role in in building software is is is being subsumed by the models.

s433
00:38:10.640 --> 00:38:12.000
Um how do you think about that?

s434
00:38:12.000 --> 00:38:19.520
Like um how firstly how has the past year and a half thought changed the way you think about your own craft and the value that you add?

s435
00:38:19.520 --> 00:38:23.520
Yeah, I I think for me and I think Kat is always such a good reminder.

s436
00:38:23.520 --> 00:38:26.480
Ken and Boris are such good reminders of like you have to be more ambitious.

s437
00:38:26.480 --> 00:38:33.440
They're always like, you know, you have to like like, you know, we're growing so fast, we have to be on the edge, we have to do like the best work we can.

s438
00:38:33.440 --> 00:38:40.800
Um, I think that that's kind of like a constant reminder for me where I'm like anytime I'm like kind of like slow on something.

s439
00:38:40.800 --> 00:38:42.320
I'm like, okay, can I do it faster?

s440
00:38:42.320 --> 00:38:43.839
Can I be more ambitious here?

s441
00:38:43.839 --> 00:38:49.359
Um, I think the like point on and like oftentimes the answer is claude because claude is getting better as you go.

s442
00:38:49.359 --> 00:38:53.680
So, I'm like, "Oh, the last time I tried this, I it was with the previous model or something."

s443
00:38:53.680 --> 00:38:56.560
Um, I think with your point on loss, I think this is real.

s444
00:38:56.560 --> 00:39:07.520
And I I I do feel that like if you're only trying to do the same work you were doing before LLMs and now it's like a prompt, it it is like I think kind of a sad feeling.

s445
00:39:07.520 --> 00:39:10.400
And I think the way you offset that is by being more ambitious, right?

s446
00:39:10.400 --> 00:39:21.760
And you're like, you know, like I love like I think Jared is such a good example where he's like hand wrote all of the Zig code in his Oakland apartment in like a year barely left his house, you know, and then

s447
00:39:21.760 --> 00:39:28.320
uh and he had so much fun doing that and now I see him like rewrite all of Bon into Rust and he's having so much fun doing that, right?

s448
00:39:28.320 --> 00:39:41.200
and it's like so much more ambitious and that's how how like he sort of like offsets that and I think just generally being like okay like how do I do the bigger thing and and and do more and I think success is fun you know and that's how I kind of like

s449
00:39:41.200 --> 00:39:51.680
the it's changing your ambition it's changing what you do because what you did before is a lot easier in quotes but now we can take on these bigger challenges

s450
00:39:51.680 --> 00:40:00.720
yeah I think there's just like on average everyone has things they wish they did you know what I mean and that they were better at and I think now it's Let's let's do it, you know, like let's Yeah.

s451
00:40:00.720 --> 00:40:04.560
And Cap, what does that look like from a sort of product management perspective?

s452
00:40:04.560 --> 00:40:16.160
I feel like the product role just changes every single month and it's very much just identifying, okay, what are um like all the PMs on our team are like this like mix of engineer,

s453
00:40:16.160 --> 00:40:17.520
designer, PM.

s454
00:40:17.520 --> 00:40:22.800
Um most of the engineers on our team actually used to be uh full-time engineers in the past.

s455
00:40:22.800 --> 00:40:28.240
And so for us it really means like plugging in whenever there's any kind of gap.

s456
00:40:28.240 --> 00:40:39.200
So if it's like okay we have this idea and we didn't inspire any engineer to go build it then like we should just build it and then put it into a notebook and inspire people to take this to production.

s457
00:40:39.200 --> 00:40:49.280
Or if the designs look a little off, well let's let's take a page that's similar and do a first pass design and uh tag in tag in someone who's

s458
00:40:49.280 --> 00:40:51.839
very detail oriented to like fill in the gaps.

s459
00:40:51.839 --> 00:41:06.000
or if we notice that like hey now now our team uh and our product adoption is a bit bigger within the company and more people need to know what's coming down the pipe for quad code quad tag and co-work

s460
00:41:06.000 --> 00:41:20.400
uh what we do then is like okay let let us like automate figuring out our whole launch calendar let's automate getting those status updates asynchronously so we're not bugging people and then let's figure out okay these are our three internal announce channels

s461
00:41:20.400 --> 00:41:25.680
and make sure that our updates there are fully detailed and to the point.

s462
00:41:25.680 --> 00:41:36.480
And so for us, it's very much just understanding what is the gap right now between a great idea and getting something to our customers and then how do we automate it as much as possible.

s463
00:41:36.480 --> 00:41:39.359
It sounds to me like with product management, there's always more to do, right?

s464
00:41:39.359 --> 00:41:49.200
It's I feel like one of the things that makes me feel good is I've never worked at a company that didn't have a backlog of a thousand things they wanted to do and didn't have the resources to take on.

s465
00:41:49.200 --> 00:41:52.880
Um, so what's a moment when Claude has surprised you?

s466
00:41:52.880 --> 00:41:59.440
Like when when when the model has done something that genuinely surprised you that you you didn't think it would be able to do that.

s467
00:41:59.440 --> 00:42:09.520
Yeah, I mean I I've posted a lot about uh cloud video editing, but like like most recently I I gave a talk at the ACM Agentic conference and uh I was like, "Hey guys, do you have the edited video?

s468
00:42:09.520 --> 00:42:11.440
I'd love to post it and share with my coms team."

s469
00:42:11.440 --> 00:42:13.119
And they're like, "Oh, it's taking so long."

s470
00:42:13.119 --> 00:42:15.760
And I'm like, "Okay, could you send me the raw files?"

s471
00:42:15.760 --> 00:42:23.119
So they send me the video of me talking on stage, the video of the deck and the audio file, and they're like, "Good luck."

s472
00:42:23.119 --> 00:42:25.200
And and so I like give this to Claude.

s473
00:42:25.200 --> 00:42:30.880
I give it my HTML deck as well, and I'm like, "Hey, can you just like edit this together, you know?"

s474
00:42:30.880 --> 00:42:33.440
And um what it does is like honestly incredible.

s475
00:42:33.440 --> 00:42:34.800
Like I I'm ready to ship it.

s476
00:42:34.800 --> 00:42:37.280
Like so it transcribes the entire video.

s477
00:42:37.280 --> 00:42:41.359
Um it notices that sometimes the video of my deck is a little bit weird.

s478
00:42:41.359 --> 00:42:47.680
there's like a popup of like an auto update in the middle and it's like, "Oh, I probably shouldn't use the video of your deck.

s479
00:42:47.680 --> 00:42:55.760
Actually, what I'm going to do is I'm going to slice up and figure out which um slide you're on and instead use your h the HTML source, right?"

s480
00:42:55.760 --> 00:42:57.680
And so, it's displaying the HTML source.

s481
00:42:57.680 --> 00:43:02.480
Then, it's got a video of me, but you know, like I'm only taking up a small part of the stage.

s482
00:43:02.480 --> 00:43:06.400
And so, it's cropping dynamically where I am on the stage.

s483
00:43:06.400 --> 00:43:08.240
Um, and like I'm I'm pacing.

s484
00:43:08.240 --> 00:43:10.640
So, it's like tracking me as I'm pacing.

s485
00:43:10.640 --> 00:43:16.000
And I've got like this crop of me, the the deck, and then like it's transcribing what I'm saying.

s486
00:43:16.000 --> 00:43:16.960
This was Fable, right?

s487
00:43:16.960 --> 00:43:17.440
This Fable.

s488
00:43:17.440 --> 00:43:17.680
Yeah.

s489
00:43:17.680 --> 00:43:17.920
Yeah.

s490
00:43:17.920 --> 00:43:18.319
Absolutely.

s491
00:43:18.319 --> 00:43:19.119
Yeah.

s492
00:43:19.119 --> 00:43:24.240
Um and it's just like like, you know, it was a good prompt, but it was a oneshot prompt.

s493
00:43:24.240 --> 00:43:32.160
And and then I asked it to like add some like interesting animations and graphics, and it I was just like blown away kind of, you know, and it just like does all this stuff.

s494
00:43:32.160 --> 00:43:35.280
It does ffmpeg, it does remotion, it does.

s495
00:43:35.280 --> 00:43:36.480
So I have to ask the follow-up.

s496
00:43:36.480 --> 00:43:37.839
What can't it do?

s497
00:43:37.839 --> 00:43:39.599
What are the things where you're still disappointed?

s498
00:43:39.599 --> 00:43:44.240
You're waiting for Claude Fable 6 to to figure out for you.

s499
00:43:44.240 --> 00:43:47.119
I wanted to have better design and UX taste.

s500
00:43:47.119 --> 00:43:47.235
Uhhuh.

s501
00:43:47.235 --> 00:43:48.880
[laughter]

s502
00:43:48.880 --> 00:43:56.960
Like I feel like it's now at the point where if I give it if I write out a prompt with a detailed spec of how I want a feature to behave,

s503
00:43:56.960 --> 00:43:59.280
it will usually behave that way.

s504
00:43:59.280 --> 00:44:08.240
But, you know, the the paddings might be off or like the the interface is it's just not delightful yet.

s505
00:44:08.240 --> 00:44:18.480
Um, I think it kind of leans on like existing best practices for apps for how apps are designed, but I feel like for Frontier AI products, there's so many new

s506
00:44:18.480 --> 00:44:22.720
interaction experiences that we still have to have yet to design.

s507
00:44:22.720 --> 00:44:24.400
There's an Opus aesthetic.

s508
00:44:24.400 --> 00:44:26.960
You can look at something and go, "Yeah, that was designed by Opus."

s509
00:44:26.960 --> 00:44:28.880
It' be good if we could move beyond that.

s510
00:44:28.880 --> 00:44:29.119
Yeah.

s511
00:44:29.119 --> 00:44:29.440
Yeah.

s512
00:44:29.440 --> 00:44:36.079
It like I I'm very excited for future models to hopefully be like interaction design thought partners.

s513
00:44:36.079 --> 00:44:36.720
Hm.

s514
00:44:36.720 --> 00:44:38.079
Uh what can't it do?

s515
00:44:38.079 --> 00:44:44.800
Um I I think I would love to see it, you know, interact more with the real world like okay like can it do this like you know can it solve science right?

s516
00:44:44.800 --> 00:44:54.560
Like can it like orchestrate you know the experiments and there's some amount of coding that goes into that but there's also this like other taste of uh you know like the broader world that it needs.

s517
00:44:54.560 --> 00:44:54.960
So

s518
00:44:54.960 --> 00:44:58.319
Claude Science is a new product that just came out a few days ago right?

s519
00:44:58.319 --> 00:44:59.839
Yeah but I have no context on it.

s520
00:44:59.839 --> 00:45:03.680
I was going to ask is that part of Claude code or is that a separate separate sex section?

s521
00:45:03.680 --> 00:45:04.800
It's our partner team.

s522
00:45:04.800 --> 00:45:05.200
Gotcha.

s523
00:45:05.200 --> 00:45:07.280
Try it out, though.

s524
00:45:07.280 --> 00:45:10.319
Um, so we've got I've got a couple of closing questions.

s525
00:45:10.319 --> 00:45:18.560
Um, which parts of Anthropic's company culture do you think uniquely help Anthropic be productive with these tools that other companies should steal?

s526
00:45:18.560 --> 00:45:23.359
What are the cultural hacks that people should be should be adopting from you?

s527
00:45:23.359 --> 00:45:25.200
I'll share one and then you go.

s528
00:45:25.200 --> 00:45:27.680
Um, I'll share one for quag.

s529
00:45:27.680 --> 00:45:35.920
So claw tag works best when you have it in a public channel and when most of your channels are public.

s530
00:45:35.920 --> 00:45:43.359
Claw tag is able to search across all public channels to get as much context as possible to give you the highest accuracy answer.

s531
00:45:43.359 --> 00:45:47.920
And it's only able to do this if it has access to everything.

s532
00:45:48.000 --> 00:45:51.520
Yeah, I mentioned this in my keynote, but I I think I it's so important to me.

s533
00:45:51.520 --> 00:46:03.839
I want to reemphasize like I I think the co-founders say like we don't negotiate against ourselves, you know, and I think this is really important where you're like you can imagine trade-offs in your head and talk yourself out of doing something ambitious,

s534
00:46:03.839 --> 00:46:04.400
you know?

s535
00:46:04.400 --> 00:46:07.280
Um or you can just try and do the ambitious thing.

s536
00:46:07.280 --> 00:46:11.520
And I think that like we're just so often being like, okay, what if we just did it?

s537
00:46:11.520 --> 00:46:15.680
Like what if like, you know, like is this a real trade-off or not, right?

s538
00:46:15.680 --> 00:46:17.920
Or like and if so, like why?

s539
00:46:17.920 --> 00:46:22.400
like where's the proof that it's a real trade-off and not just like it sounds reasonable, right?

s540
00:46:22.400 --> 00:46:27.839
And so I think just yeah like you know make the trade-offs show themselves to you.

s541
00:46:27.839 --> 00:46:29.599
Be as ambitious as you can.

s542
00:46:29.599 --> 00:46:36.240
That's so because that goes against I've got 25 years of software experience that says the default answer should be no. Everything is a trade-off.

s543
00:46:36.240 --> 00:46:37.040
Everything has a cost.

s544
00:46:37.040 --> 00:46:40.400
And now we're having to reimagine all of those intuitions.

s545
00:46:40.400 --> 00:46:42.160
It's it's kind of fascinating.

s546
00:46:42.160 --> 00:46:42.400
Okay.

s547
00:46:42.400 --> 00:46:44.720
And so final question for both of you.

s548
00:46:44.720 --> 00:46:53.200
What is something what what's one of your favorite absurd things that you've built with Claude just because you could build it?

s549
00:46:54.720 --> 00:46:55.200
I can go.

s550
00:46:55.200 --> 00:47:04.079
Well, you think um I'm working on a 2D Street Fighter fighting game uh with me as a character and like my friends as well.

s551
00:47:04.079 --> 00:47:12.560
Um and it uses cloud code to prompt you know Gemini and honestly the sea dance model is pretty good like uh to make like video animations.

s552
00:47:12.560 --> 00:47:17.119
Um, and uh, it works great like like it's so good at prompting.

s553
00:47:17.119 --> 00:47:21.119
It's like, you know, it can verify like the frames to check if this was a good animation.

s554
00:47:21.119 --> 00:47:24.400
Is this Street Fighter 2 level 2D sprites that you're generating?

s555
00:47:24.400 --> 00:47:24.800
Yeah, exactly.

s556
00:47:24.800 --> 00:47:24.960
Yeah.

s557
00:47:24.960 --> 00:47:25.200
Yeah.

s558
00:47:25.200 --> 00:47:25.359
Yeah.

s559
00:47:25.359 --> 00:47:26.560
Like like 2D sprites.

s560
00:47:26.560 --> 00:47:27.599
The animation looks amazing.

s561
00:47:27.599 --> 00:47:29.119
And it can also figure out hitboxes.

s562
00:47:29.119 --> 00:47:33.280
It can be like, oh, you know, your fist is like here, I'll draw the JSON hitbox.

s563
00:47:33.280 --> 00:47:33.440
Yeah.

s564
00:47:33.440 --> 00:47:33.920
Yeah.

s565
00:47:33.920 --> 00:47:35.599
It's like incredible.

s566
00:47:35.599 --> 00:47:35.839
Yeah.

s567
00:47:35.839 --> 00:47:36.079
Yeah.

s568
00:47:36.079 --> 00:47:39.119
So, um, I don't know if I'll put this out, but it's uh

s569
00:47:39.119 --> 00:47:40.880
I feel like we need a screenshot at least.

s570
00:47:40.880 --> 00:47:43.200
This sounds I can make a screenshot happen.

s571
00:47:43.200 --> 00:47:43.440
Yeah.

s572
00:47:43.440 --> 00:47:44.560
Yeah.

s573
00:47:44.560 --> 00:47:46.720
Mine is much more simple.

s574
00:47:46.720 --> 00:47:58.319
Um I'm a big rock climber and a lot of my friends climb and so we have this little app that we built with quad code that where we just log all the projects that we're working on and we also uh go outdoors together a lot.

s575
00:47:58.319 --> 00:48:02.160
So we have uh quad do all this research with workflows.

s576
00:48:02.160 --> 00:48:03.920
Workflows is amazing.

s577
00:48:03.920 --> 00:48:09.119
Like we brand it as a coding tool but it's amazing for doing deep research for travel.

s578
00:48:09.119 --> 00:48:14.480
Um, I also plan our team off sites and it's good at finding venues that can fit all of us.

s579
00:48:14.480 --> 00:48:18.880
Um, it yeah there it has a lot of uh side benefits.

s580
00:48:18.880 --> 00:48:28.319
But anyway, um I also use workflows to just research all the climbing destinations that we might want to go to, what has direct flights from where all of us are located.

s581
00:48:28.319 --> 00:48:32.480
It goes to mountain project and finds all the climbs that are in our grade level.

s582
00:48:32.480 --> 00:48:40.000
It finds the Airbnb and it actually maps out like I I don't like hiking and so I care a lot about it having a very short approach.

s583
00:48:40.000 --> 00:48:44.400
So very short walking distance from where the car parks to where the rock actually is.

s584
00:48:44.400 --> 00:48:56.480
And so it it filters for this and so I like with existing apps I have to like manually click through mountain project but with this I just put in all of our preferences and it's just a custom app for us.

s585
00:48:56.480 --> 00:49:00.160
So you're you're basically vibe coding Jira for mountain climbing.

s586
00:49:00.160 --> 00:49:00.880
Exactly.

s587
00:49:00.880 --> 00:49:02.559
That's that's pretty fantastic.

s588
00:49:02.559 --> 00:49:03.359
Um, you know what?

s589
00:49:03.359 --> 00:49:05.920
We have time for a couple of audience questions.

s590
00:49:05.920 --> 00:49:13.280
Um, if you want to sh if you want to come forward and say them to me and I will repeat them so everyone can hear them.

s591
00:49:13.280 --> 00:49:17.680
But yeah, um, tell you what, anyone who gets here first gets to ask a question.

s592
00:49:17.680 --> 00:49:19.520
Sorry for people at the back.

s593
00:49:19.520 --> 00:49:21.520
Hey um actually

s594
00:49:21.520 --> 00:49:33.760
yeah uh my question is that um do you have any near plan to build more uh eval uh tools for us to build eval data set or anything like that and um more observability

s595
00:49:33.760 --> 00:49:39.200
tools to monitor the performance of agents and uh workflows.

s596
00:49:40.480 --> 00:49:50.559
We've considered building eval tools, but I think the limiting factor actually tends to be that it takes a long time for customers to build really high quality evals.

s597
00:49:50.559 --> 00:49:56.079
And so I think the tooling is less of the constraint and more of the skill set of how do you build a great eval?

s598
00:49:56.079 --> 00:50:03.920
And that's an area where we're excited to both invest internally and also hopefully we can share some of the best practices externally.

s599
00:50:04.000 --> 00:50:06.000
Hey Katar, uh my name is Sai.

s600
00:50:06.000 --> 00:50:11.119
Uh so my question was because I'm more interested in the memory and the multiplayer.

s601
00:50:11.119 --> 00:50:13.760
How does how is memory being designed?

s602
00:50:13.760 --> 00:50:19.839
So two questions right uh so how is memory being designed today I I assume it's around files.

s603
00:50:19.839 --> 00:50:29.599
So and second part of it is have you thought about thinking in an orthogonal direction where you would actually need a data store to store these memory instead of files to scale it better.

s604
00:50:29.599 --> 00:50:31.599
So I think that's my question.

s605
00:50:31.599 --> 00:50:34.640
Yeah right now for cloud tag the memory is channel specific.

s606
00:50:34.640 --> 00:50:43.680
So every claude in that channel has a shared memory and then you know the instances have a session um but like the session can contribute back to main memory.

s607
00:50:43.680 --> 00:50:48.960
We do a lot of memory research and it's you know can be kind of unintuitive like what what is the right way to do memory.

s608
00:50:48.960 --> 00:50:50.640
Uh but yeah we're always working on this.

s609
00:50:50.640 --> 00:50:52.880
So yeah

s610
00:50:53.760 --> 00:50:56.079
uh yeah I mean we're always running me memory experiments.

s611
00:50:56.079 --> 00:51:00.960
I don't have you know anything to yeah like how it works right now in cloud tag is a markdown file per channel.

s612
00:51:00.960 --> 00:51:01.599
Yeah.

s613
00:51:01.599 --> 00:51:02.400
Thank you.

s614
00:51:02.400 --> 00:51:04.079
So I'm afraid we're out of time.

s615
00:51:04.079 --> 00:51:10.160
Please join me in thanking Cat and Tariq and we will be around for more questions um in the hallway.

s616
00:51:10.160 --> 00:51:10.640
Thanks guys.

s617
00:51:10.640 --> 00:51:12.800
Thank you.

s618
00:51:27.050 --> 00:51:29.050
[music]