Field Guide to Fable — Thariq Shihipar, Anthropic

AI Engineer · 19 min · 233 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Please welcome to the stage member of technical staff at Anthropic, Tarik Shaupar.
00:20[music]
00:33Hey everyone, I'm Tarik.
00:34I work at Anthropic on Claude Code.
00:37Before we get started, we have a tradition on Claude Code where we take a selfie before a talk.
00:41So, if you don't mind, if you strike a pose with me, I'll take a quick selfie at AI engineer.
00:50Okay, incredible.
00:51Well, uh yeah, to kick things off, like we said, Fable is back.
00:57Um we're rolling it out later today.
01:03Uh keep stay tuned for exact timeline.
01:05Me and Kat Woo and Simon Wilson will be doing a fireside chat at 12:30.
01:10We might have some updates for you then.
01:12Um but Fable is a model I'm just so so excited about.
01:18It's one of those Anthropic models where you just like you're just going to remember it, like Sonnet 3.5 new, Opus 4, Opus 4.5. It's a model that I just have a lot of like affection and excitement for.
01:31And the best way to describe Fable to me is like the the map is opening up, you know, like you were playing like an RPG and you've been on the tutorial.
01:41And now you get to the point where the like, you know, the open world starts, right?
01:46And there's so much that you can do and explore.
01:50Uh but there's also it's also a little bit intimidating and confusing, right?
01:54Cuz there's so much you can do.
01:56And so, what I wanted to do in this talk is give you guys a field guide to Fable, right?
02:03How do you work with this new class of models?
02:07So, I've got four parts to it.
02:10I've been working on this as a series of articles and blog post.
02:14Uh but, you know, when we announced Fable is coming out, I was like, "Okay, let me do uh all of this at once at the talk, uh you know,
02:23uh speed run."
02:24So, the four parts: unhobbling Claude, finding your unknowns, dealing with the grief, and being unreasonable.
02:32So, first, unhobbling Claude.
02:37Um I think something we say really often is that the models are grown, not designed, right?
02:44We don't wake up and be like, "We need 99% on SWE-bench," right?
02:49Like, the models are, you know, something we we grow carefully.
02:52We give it data and feedback and compute.
02:55Um but, ultimately, it's, you know, something that we it's a little bit organic, and we sort of figure out and learn with the model as we use it.
03:06And so, um that what that also means is that what contains them is us, right?
03:11The harness we put them in and the way we prompt them is basically like a function of our understanding of Claude, right?
03:19And by unhobbling it, I mean, how can we understand Claude better to unleash it?
03:25And we need to understand Fable more.
03:28So, I think one of my points is that, you know, uh we're still so early, and I think there's a lot more it understanding in Fable uh to unlock.
03:39And uh I I think I'll give you a quick example about how models get smarter, cuz it's a little bit unintuitive, right?
03:46Like, there I saw this viral tweet a couple weeks ago being like, you know, "Why can't LLMs say which Pokémon end in AW?
03:54There are a thousand Pokémon, right?
03:56And turns out there are two who whose names end in AW, Croconaw and Drednaw, right?
04:01And it turns out if you ask like a normal chat model, it can't answer it.
04:04Which is kind of confusing because like, you know, it definitely knows all the names of the Pokémon, right?
04:10But if you ask Claude Code, it can, right?
04:13Cuz what it does is that it fetches every Pokémon and writes a script to filter for AW, right?
04:20And so, this is what I mean by like unhobbling Claude.
04:26We call this capability overhang, right?
04:28Claude gets smarter in spiky ways.
04:31So, it doesn't just remember every Pokémon and reason through it, but if you give it the code execution tool, it can find the two Pokémon that end with AW, right?
04:41And so, this is I think part of the challenge with Fable is figuring out this capability overhang.
04:45What is now possible?
04:47And I think this is like a discovery that I'm excited to go on with you.
04:50Uh to make this a little bit clearer, I'm going to talk about a few different examples of how models have progressed in the past.
04:57Um one of the big examples obviously is like chat, you know, the chat models were had to be given context, right?
05:04Like maybe you paste in your code base and maybe naively you might have thought like, you know, the way we solve coding is by the context just gets really large and I can just paste in my entire code base, you know, it'll be a 100 million context window.
05:16But it turns out that instead, if you give it arms, like you give it the bash tool and ways to work with the environment, it can build and search its own context.
05:25And that's sort of like the insight that led to Claude Code, right?
05:29And so, again, spiky, like a new like innovation kind of, right?
05:34In how we think about and work with the model.
05:37And then recently we we rolled out Claude Tag.
05:39Uh and what sort of unlocked Claude Tag is its ability to work proactively in multiplayer.
05:45Uh Claude Code, you know, is something that you have to prompt for it to do work, right?
05:49And uh, this ability for Claude to wake itself up and do work is something that we think is unlocking the new wave of agents.
05:58But there's there's more here.
05:59So, for example, uh, we recently removed 80% of the system prompt for Claude code, right?
06:06And this is one of the ways in which models, you know, and what they need uh, changes over time.
06:12So, originally, like, you know, maybe back in 3 sign of 3.5 new, the best practices for a system prompt was a small system prompt, few tools, and lots of examples,
06:22right?
06:22And then as the models get smarter, you can give it more information and more instructions, and they start following them.
06:27And so, it's a larger system prompt with lots of examples and many tools, right?
06:32But most recently, we found this new class of models want fewer what want a smaller system prompt.
06:40The examples tend to constrain it cuz it's actually more imaginative than the examples we give it.
06:45And so, uh, and we tried to give it context and not just constraints.
06:50We're really trying to avoid being like, "Do not do this."
06:53Um, which was really necessary for the previous models.
06:56Um, and so, this is like, uh, a way that the system prompt is changing and and probably will continue to change.
07:02Uh, another feature I really like is the ask you a question tool.
07:06This is something I worked on when I first got to Claude code.
07:09And and it's uh, when Claude, you know, is is planning or wants to ask you a question, it can show you a multiple choice dialogue.
07:17Uh, for Opus 4, it could barely call it.
07:20I had to like really tweak the tool to make sure that it was uh, that it would work, right?
07:26And then, sometime at Opus 4.5, I was like, well, what if I asked it to like, you know, ask me 40 questions about the spec.
07:33It could start interviewing me, right?
07:34And so, its ability to ask questions jumped, right?
07:38And then, most recently with Opus 4.8 and Fable, I can now build a whole HTML report with the questions embedded inside of them.
07:47And uh it's just like a whole new way of interacting with uh with Claude, right?
07:52And and so this progression of like how Claude can get information from you has also changed.
07:58Um speaking of which, uh Markdown and HTML is something I've also talked a lot about.
08:03Um you know, it turned Initially, Markdown was a a good output for the model.
08:07Um you know, it could show a little bit of rich information.
08:11And then, you know, with plan mode, it started to be for you.
08:14Like you could understand what Claude was about to do.
08:17Um and now, you know, Claude can build you these in-depth HTML reports, right?
08:22So again, a way of this the model is getting smarter in a spiky way.
08:27I really like to emphasize that this is closer to a biology than a physics, right?
08:33It's still very empirical, very organic.
08:36Um we don't know all the rules, but there is some sort of science behind it, right?
08:40Like there is an intuition to build as well.
08:42And so I really, you know, encourage you to treat Fable like that.
08:46Uh one of my favorite papers uh that at Anthropic that we've written is on the biology of a large language model.
08:52Um all of our research papers are meant to be read by, you know, people with various degrees of technical expertise, but this is one of my favorite.
08:59So uh if you're looking to learn a little bit more, suggest you check it out.
09:04But so uh yeah, we talked about unhobbling Claude.
09:08But it turns out when you're working with Fable, you also need to unhobble yourself, right?
09:14And so one of the things that I think a lot about is that the map is not the territory, right?
09:19When I'm working on a coding problem, the plan and prompt and spec that I have in my mind is the map, right?
09:27But the territory is the actual code base, the real world, the constraints that Claude needs to navigate, right?
09:34And whenever Claude runs into something in the territory that's not in the map, I call that an unknown, right?
09:39Claude has to figure out what to do about it.
09:41It's a decision point that I haven't specified.
09:44And Fable is one of the first models where I felt that like I really have to figure out my unknowns because if not, it's going to traverse such a large area that like it's going to run into a lot of them.
09:57So, how do you figure out your unknowns?
10:00Um Yeah, it I Fable's bottleneck my ability by my ability to match the map and the territory to find my unknowns.
10:09So, a few um few ways to think about this.
10:13I like to think of it in a a matrix.
10:15So, like for any problem, I have a bunch of known knowns.
10:19This is usually like what I write in my prompt.
10:21What do I want, right?
10:23Then I have known unknowns.
10:24Things that like I know I haven't don't really know yet, but I just haven't figured out yet.
10:29I can um Uh yeah.
10:31Then I've got unknown knowns.
10:32Like what's so obvious that I just wouldn't write it down, you know, but I I'd know it when I see it, right?
10:38And then finally unknowns unknowns.
10:40What haven't I considered at all?
10:41What do I not know, right?
10:43Like what is something that if I knew could change how I prompt Claude?
10:48And and luckily, you can use Claude, you can use Fable to find your unknowns.
10:53So, I'm going to go over a few examples of how I do that with Fable.
10:57Um The first is I like to do what I call a blind spot pass.
11:02So, I like to say something like, "Hey, I'm working on a new auth provider that I know nothing about uh like in this code base.
11:09Can you do a blind spot pass to help me figure out my relevant unknown unknowns and help me prompt better, right?"
11:14And so, this like might have Claude go through the the auth module and figure out like, "Oh, you know, this is kind of like a hairy little uh dead end that comes up a lot."
11:23Maybe search in my Git diff or Slack.
11:25I might tell it where there's context, right?
11:28So, that I can learn about, you know, all the gotchas.
11:31And and you can use this very broadly, right?
11:33You can use it to teach you about new fields.
11:35I I recently did this for color grading when doing video editing.
11:39Um I think this is really powerful and and Sable is incredible at it.
11:42Um in many ways, the model knows more about you know, almost everything than I do.
11:49I just need to get it out of it.
11:50Um then I like to do some brainstorms and prototypes.
11:55Uh this helps me figure out my unknown knowns, right?
11:58Things like especially for design, it for me it's like know it when you see it, right?
12:02So, I might ask it to uh create a dashboard.
12:05Um and I tell it I have no visual taste.
12:08Uh make me an HTML page with four widely different design decisions so I can react to them, right?
12:13And then you know, you tweak this as you want, but like the idea is to sort of get an idea of like what are the things that you uh
12:21you know, you can't describe in words, right?
12:23And uh like work with the model to help figure that out.
12:28Uh then then interviews.
12:30So, once I have an idea of like, you know, this is what I want to do, uh there's probably still a lot of like uh unknowns here, right?
12:39Where I might not have considered something, I might not have specified it.
12:43And so, I'll ask Claude to interview me, right?
12:45And I'll give it a little bit more context.
12:47In any of these questions, like giving it a little bit more context about you and the work and the stage you're at, like, "Hey, yeah, prioritize questions that would change the architecture."
12:55is extremely helpful.
12:58Uh then references.
13:00One of the best ways to give Claude a map is to give it another map, right?
13:05So, instead of me writing out the spec, uh I can just say, "Hey, here's some code that represents what I want to be done, right?
13:13It could be in a different uh system or language.
13:17Uh but just read this code, understand it, and then use that to start your work, right?
13:22And uh again, this can be in a lot of different ways.
13:25If I'm making a a React component, I might have an HTML mock-up that is my map, right?
13:29That I pass in as a reference.
13:31I think this is a really, really powerful.
13:32Fable is really incredible at it.
13:35Uh something else I've like really appreciated is implementation notes.
13:39So, if uh while you're running Fable uh and it runs into an unknown, ask it to log it, right?
13:46So that um you uh you can see where the deviations happened and then you can sort of figure out why as well, you know?
13:54It will usually give you some context about what happened.
13:57And then finally, I'd like to get a Fable to quiz me about what happened uh just to make sure I understand what I'm doing and I can represent this work, you know, when I'm creating a PR or merging it.
14:08Um this is a really great way of like making sure that you're like really in the loop with Fable.
14:14And I think that's like one of the most important parts of Fable is like staying in the loop and making sure that you uh you get what you want.
14:23So, um those are some of my tips for working with Fable.
14:28Uh I also want to say that the first time I used a Mithril class model, uh used Fable, I felt both a huge sense of like gain, but also a sense of loss.
14:40And I I wanted to talk a little bit about that, you know?
14:44Um when I think about coding before LLMs, it feels like a foreign country, you know?
14:49Like I used to run a YC startup about 30 people and we were just constantly forced into trade-offs because of how hard code was, right?
14:57Like we could make the the app fast or we could try prototyping new feature and and this might take a month or this would take 2 months.
15:04And so we had to choose.
15:05It was just really, really hard.
15:07Um and now I went back to that code base a couple weeks ago and I thought about some of the things that I wanted to do.
15:15And uh it was just way easier.
15:17It was like the things that would have taken me weeks, I could do in hours, you know?
15:23And uh at some point it's like, yeah, like how can you not laugh?
15:25Also, how can you not cry, all honestly?
15:28Like, it's like one of these things where I really, really loved programming and writing code by hand.
15:34I love the feeling of like seeing the code base in my mind and like rotating it.
15:39But I also remember just, you know, like staying up late nights trying to debug, working on things for weeks without working, right?
15:48I just remember swimming in failure.
15:50I just remember that like the most of the projects I've ever worked on have failed.
15:54Most startups go bankrupt, you know?
15:56I think just overall programming and coding is extremely hard.
16:00And like as much as I enjoyed those highs, I I can cannot go back, right?
16:07And the way my reflection here is like the only way out is through, right?
16:11There's still a lot to learn with the gentle coding.
16:14There's a lot to learn with Fable.
16:16But I think if we try really hard and if we like stay in the loop, we on hobble it, we can get there, you know?
16:22And we can come out on the other side with just so much more.
16:28And so, the last bit I wanted to talk about is is the so much more part, right?
16:33I call this being unreasonable.
16:36One of my favorite parts of Anthropic is that we believe that tradeoffs are not real.
16:44Um, like I think that very often, I like in my previous company, I was very used to being reasonable.
16:50So I'd like write down this list of priorities and I'd be like, well, I guess we can prioritize this against this, right?
16:56And like, you know, that makes sense.
16:58So we'll this will be our priority this quarter.
17:01But what if you just did all of it, you know?
17:04What if you forced reality to show you the tradeoff, right?
17:09Um, this is one thing I really valued out of our culture in Anthropic.
17:11And that my reflection going forward is that I'm going to be a lot less reasonable.
17:16Um, I think one of this like the math of Claude and Fable really changes how you think about tradeoffs.
17:23And there are so many tradeoffs that you make implicitly in your head, right?
17:26Like good, fast, cheap.
17:28Now it's pick three, right?
17:30Um I think that like the best way to like do more ambitious work is to uh like reframe and make big make ourselves more ambitious.
17:40Because I think the only way to prove that agents work is to do the best work of our lives faster than ever before.
17:48Um you know, for example, I made this deck last night in about 4 hours with Fable.
17:54I feel like it's a it's a deck I really like and I I really enjoyed it, but I also um you know, I did it really fast.
18:01Uh and I think that if you're here, you know, at AI engineer, the world is kind of looking at you to prove that AI works, right?
18:08That it's not just like a fad or something, but that it can make us more productive and also save us time.
18:14And and that's my resolution for this year is to work be more productive, but work less and spend more time with people I really care about.
18:22Uh I think it's also worth calling out that building is easier, but generating value is still hard.
18:29And I think this is something that we run into, you know, as AI engineers sometimes where we think so much about the process of building and our our setups.
18:38Um but the the point is to generate value, right?
18:42And uh there it takes a lot of swings, it takes a lot of tries to find the valuable stuff.
18:49Uh but that really is the goal and that's like, you know, again, what the world is looking to us to prove that AI can really transform it.
18:58So, to to end, I just want to say like go explore, make it real, and uh yeah, be less reasonable.
19:06Thank you.
19:25[music]
19:27Woo!