Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley

AI Engineer · 20 min · 238 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Okay, I'm getting rolling and uh welcome aboard.
00:15We just had a little technical issues, but uh we resolved them.
00:18So, my name is Frank Coyle.
00:20Uh I am a computer science guy.
00:23I've been teaching computer science for over 30 years, and I'm now teaching at Berkeley.
00:28And one of the problems that uh all my students, past and present, are having is AI, because computer science is no longer the magic pathway to a job.
00:40So, I've been trying to figure out ways to uh help them come up with schemes to help them get ready for this world of agentic AI.
00:49And one of the things that sort of uh dropped into my uh plate was the something called the Claude Certified Architect exam, which I will be talking about today, and it has
01:02um a number of aspects to it.
01:03And I think if you're interested in a career in agentic AI, then certainly take a look at least what the exam is about, because I feel that um Anthropic
01:14knows how people are using their system and what the issues are going to be.
01:19So, before we jump into that, I want to give a little bit of my uh my philosophy.
01:26bop bop bop bop May have to do this manually, getting stuck.
01:36So, this is a quote from uh a woman named Sister Corita Kent.
01:42Nothing is a mistake.
01:44There's no win and no fail.
01:46There's only make.
01:48Bottom line here is experiment, experiment, experiment.
01:52Not only should you read, but you should do.
01:54You should make stuff.
01:56Now, what happens when you make stuff?
01:59A lot of times things don't work.
02:03Thomas Edison said, "I have not failed.
02:07I've only found 10,000 ways that don't work."
02:11And what I want to emphasize here is that what this shows us are something that in the design patterns movement, which came around in the early 1990s with object-oriented programming,
02:24we had patterns for objects.
02:27We now have patterns for agents, but there's also anti-patterns.
02:31And I think anti-patterns are a key to understanding what you should not do because understanding what you should not do is the key to leading you to what you should do.
02:44So, a little bit about the Claude Certified Exam, released in March, so it's brand new.
02:50It is uh it is based on scenarios.
02:55It is timed.
02:55It is proctored.
02:57It is available to companies in the Claude ecosystem, the Anthropic ecosystem, but individuals can pay $99 and take the exam once every once every 6 months.
03:11And it's not just multiple-choice questions.
03:15It is multiple-choice, but they're they are based on uh realistic constraints and realistic scenarios.
03:24The five domains.
03:26There are five domains that are covered and they give you the percentages of each.
03:30So, agentic architecture, 27%.
03:33Claude code, how to configure the Claude code system and workflow, 20%.
03:39How to doing prompt engineering, structuring your output, using JSON all over the place.
03:47Tool design.
03:49Model context protocol integration.
03:51These are topics that you should understand and know whether you're going to take the exam or not.
03:56This is going to help you get ready for whatever the agentic world is going to throw at you.
04:02And then there's going to be contact management and reliability.
04:06So these are the areas of of the kind of questions you're going to run into.
04:13Then there are and they they provide you with six production scenarios and your the exam will randomly choose four and all the questions will be centered around the four that they choose.
04:29And what I'm going to do is walk you through um the production scenarios and give you some anti-patterns to be aware of because there's a number of ways you can solve the problem but one of the big things is what not to do and that often can be the key
04:45to getting these questions right.
04:47So, number one customer support resolution agent.
04:50So we have agentic loops, control, something called stop reason which is uh what Cloud Code has.
04:58Every time something happens, there's a stop reason and you need to take a look at that because that can give you a lot of information about what's going on.
05:05Uh scenario two, code generation.
05:08Three, multi-agent research system which we'll look at.
05:12How do you How do you distribute your agents?
05:15Hub and spoke.
05:17Who's the orchestrator?
05:18How much information should they know?
05:20All these are important factors.
05:22Um scenario four, developer productivity with code.
05:27So how do you do subtask isolation?
05:30Keep your tasks in their little universes.
05:33And this hearkens back to what we learn in computer science from doing multi-threaded programming.
05:39When you have multiple threads operating and sharing memory, then you get into issues with synchronization.
05:45You You to put locks Keep the little threads independent.
05:50Keep your agents independent.
05:52Um and then some cloud code for continuous integration.
05:58And then we'll look at some patterns for structured data extraction.
06:03Okay, that's kind of where we're going to go.
06:06Now, here's something that I I I like to point out.
06:10Everybody's talking about loops, right?
06:12Every The loop is the new thing.
06:14Um uh Boris Cherney says he doesn't write code, but his job is to write loops.
06:22And Peter Steinberger master of Open Claw says, "I don't I don't uh I don't code anymore.
06:28I just design loops that prompt your agents."
06:32So, loops are the new big thing, right?
06:34Well, no, they're not.
06:36Okay?
06:37Um back in the day uh early days of computing, we had programming languages were exploding.
06:45We had Fortran, we had COBOL, and there were big fights.
06:49My program My programming language is better than yours.
06:53It can do more.
06:54No, it can't.
06:55We can do this.
06:56Böhm and Jacopini, 1966 proved that if you want a language to be Turing complete, which means can compute anything that computers are possibly able to compute, then you need only three things.
07:13The ability to to to write statements sequentially, okay?
07:18To have if-then conditionals, and the third piece is the loop.
07:24If you add the loop, you have Turing computability.
07:28And now we are seeing this being resurrected in the agentic world with the focus on loops, cuz up to now we've had sort of sequences.
07:38You have prompts, you have maybe if-then, but now we have a loop.
07:42And now this is what's giving us the power.
07:44This is where the agentic stuff is getting very exciting.
07:48Okay.
07:50I'm start with uh with scenario one, customer support resolution.
07:56So here we have a loop operating and the I'm going to jump to the anti-pattern.
08:04What you don't want is just to let the agent go and do something and get the response back and use it, okay?
08:12What you want to do is you want to loop with something called the stop reason.
08:16So I'm going to show you a little code here.
08:19So here we have while loop.
08:20It's a while true, it's a loop.
08:22We're looping right here, okay?
08:24So the first little block is where we call uh we call the model, okay?
08:29And we pass it the messages.
08:31The messages are essentially the sequence of prompts that exist in the context window, okay?
08:38And we are asking the and we have a we have a prompt and we have we have the context and we have a tool.
08:47And we're asking the LLM to do something with this tool and help us out.
08:53The problem is the LLM can't do anything.
08:56It is just a probabilistic next word predictor.
09:01It can't execute tools.
09:03So what it does though is it can figure out if you point it to a tool, it can figure out how to set things up so that you or your code can execute it.
09:16So it's important to understand that the LLM is not executing these tools.
09:20It can't do anything except talk back to you, very intelligently sometimes, but all it can do is talk back to you.
09:26So when it finishes this task and has a result which is basically here is I've I know what you want.
09:39I know what the tool can do.
09:41Here's how I It sets up the parameters that can then be or that then used to actually execute the tool.
09:48So, the second block you see why did the LLM come back to us?
09:55That's our stop reason.
09:57Tool use.
09:58Oh, okay.
09:59We've stopped because the LLM it wants to use the tool.
10:03So, let's just run the tool.
10:05So, that's what the second block is.
10:07Run tool, the response is what the LLM said, and it's basically the parameters that it has extracted from the data that you provided it.
10:17Okay?
10:17Then it executes that.
10:19Then it goes back.
10:20That then it continues.
10:22Continues means the LLM sees it and says, "Oh, successful run.
10:26So, okay."
10:28Come back down.
10:31We're not running a tool anymore.
10:32We're end the end of our loop.
10:34Bingo.
10:35Now, then we take the answer, and this is an opportunity for you to have a human in the loop potentially.
10:43You check the confidence.
10:44If it looks good, you keep it.
10:46If you don't, then you escalate to a human.
10:49So, now there's another reason why you need to make sure you check your stop reason.
10:55One of the stop reasons may be you have run out of tokens, and this response is based on partial when the LLM had to stop.
11:06And it's going to give you a response, but if you have run out of tokens, then you need to take action.
11:12Okay.
11:13Um Next scenario.
11:17Uh code generation with Claude.
11:19So, Claude code has this has this concept of the Claude MD file, a markdown file, where you put all the things you wanted to know.
11:27What Anthropic recommends is you have three levels of Claude.
11:34One that you have at the top level of your project, the other that you have in inside your sort of the project folder, and then within directories you can also specify.
11:48So, the idea is to have a hierarchical set of rules that that can then control how the system is going to respond.
11:58Okay.
12:00Moving right along, uh we have a multi-agent research system.
12:05So, here we're going to have uh the problem is how do I how do I get my agents to to go off and do stuff and bring the answers back in a reasonable way?
12:16The anti-pattern you have one agent and you load it up with tools, all right?
12:22So, I like to think about you you know, you hire somebody to come to your house, you hire a carpenter to come to the house, and the guy shows up with uh
12:31plumbing tools, carpenter tools, electrical tools.
12:34He says, "I can do anything."
12:35Well, maybe you don't want this guy, maybe you want a a professional carpenter.
12:39So, that's the kind of idea.
12:41And this kind of back takes us back to some of the the functional programming uh ideas that functions should be do one thing.
12:52And if you can get your agents to do one thing, you with maybe one or two tools available to it, then that's going to be a win, and that's going to help you with this exam.
13:03So, specialize, don't overload.
13:07The other part of this is don't let your agents context spill over into the main context because context means tokens, tokens mean money, and the more context you have, the more confused
13:24the LLM is going to be in giving you an answer.
13:27So, even though oh, a million token context window, I can put everything in there.
13:32No, no, don't put everything in there.
13:34Limit what's going to go in there because then you're going to get a much more accurate system.
13:41So, here's a Here's an example of a specialized sub agents.
13:48You're giving it So, this would be the critic.
13:52So, let's say you've run some stuff.
13:54Now, you want to get an agent to look at what's happened.
13:57What you want to do is just give it what it needs to solve that critic problem.
14:03I'm only giving it here the we're passing it the claim and the evidence.
14:10So, this is your claim is sort of how we're going to solve the problem.
14:14Here's Here's the evidence, but we're not giving it the the thought processes that went in to creating this claim.
14:24Why?
14:25When you When you get a bunch of agents together collaborating and talking to each other, there's a tendency to have group think.
14:35And all the agents seem to kind of devolve into one idea.
14:41I mean, it's it's like, you know, you're in a group, you know, you're at a party, and everybody wants pizza except you, but then people talk you into you you know, you don't want to be uh
14:53you don't want to spoil the party, so you'll go along.
14:55And it seems that agents kind of work in the same way.
14:58So, you're going to return Basically, you're going to give each agent only a slice.
15:04I didn't think about the pizza analogy, but yes.
15:07Every agent gets its own slice, and and it it should come through.
15:12Okay.
15:17Fourth scenario, developer productivity.
15:21So, the anti-pattern.
15:25Let every subtask dump its full output into the primary thread, crowding out the context.
15:30Again, this is what we're I was just talking about.
15:33This is bad.
15:34Let the context grow unbounded.
15:37Bad, right?
15:38For the reasons we just talked about.
15:40You want to isolate your subtask output, and you want to compact long sessions.
15:47I'm going to take a second to talk about that.
15:49So, here's here's a an example of a pattern.
15:54Uh you want to have your agent uh look at the logs and create a summary of where the problems are in the log.
16:06So, here's your task, scan all the logs for error.
16:10Context fork.
16:12So, you're forking the agent into a like a separate thread where whatever the agent does and thinks and adds tokens to does not come back and pollute the main
16:25uh the main context.
16:28Now, you see here what happens, then you take this summation, and then you add that summation without all the other stuff into the overriding context.
16:40Now, this last little block is kind of interesting, I think.
16:45Because you can check your token count, and you can determine how big the token count is.
16:53And if you can set some limit and you know, if if you have more than 150,000 tokens, then what you want to do is you can run a compact.
17:03So, Anthropic and Claude have these compaction algorithms that take this giant context and and compact it in some way, shape, or form.
17:12Not quite sure how the implementation is of that, but there is compaction.
17:17Now, a little side effect a little side channel I've been walking around when you walk outside, you see see these guys handing out these books.
17:26Okay?
17:27Anybody see these guys handing out these but take them.
17:30This is this is actually a pretty good little book.
17:32In fact, I was looking at it last night and one of the things it had in it was this is by this guy Sam Sam Bagwell.
17:40I have no connection I didn't even know Sam, but it there's a online page 32.
17:46It says uh his company provides custom logic for compression of context.
17:53So, he's got an and you can write your own.
17:55He's got a he's got he you can extend his base class and have your own compression of your data, whatever you think is important.
18:02So, I think that's kind of an interesting spin on this whole thing.
18:07Okay.
18:09Cloud code for uh uh continuous integration uh anti-pattern Always have interactive modes in a pipeline.
18:20Well, no no no cuz interactive modes mean uh Cloud will stop and ask you, "You want to do this?
18:27You want to do that?
18:28Can I have permission for that?"
18:29So, there are ways to set it up so that it'll just run straight through, okay?
18:34The other uh the other tip that I'll give you here is there's something called the uh the batch.
18:46So, you can take your prompts, you can take your work, and you can put them in a batch and for 50% fewer token cost you will get the result they promise
18:58in at at least 24 hours.
19:00So, if you're going to go take a nap, you're going to go on vacation, you're going to go out, take a a day off, run your stuff in batch mode, and you're going to have a a
19:09less to pay.
19:13Where am I here?
19:15All right, I've only got a few few minutes left, few seconds left, but I want to conclude with this.
19:22Remember, nothing is a mistake.
19:24There's no win, there's no fail, there's no exam, only make.
19:29You do it and you make it and you're going to succeed.
19:32If you want to reach out to me, reach out to me uh coil at Berkeley, look at my websites.
19:38I got a website co-supreme AI.
19:40I'm a big jazz fan and I named this website after John Coltrane, Love Supreme, if you know that song, great.
19:45Anyway, that's my story and I'm sticking to it and I'm about to zero time.
19:50Okay,
19:50[applause]
19:51thank you.