Design Patterns for AI Trust: Juries, Libraries, and Agent Tiers — Alex Bauer, Upside.tech

AI Engineer · 17 min · 193 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Well, hopefully this is the one talk that's different from all the others today.
00:15So, let's start with audience participation.
00:18Who here has kids?
00:22Oh.
00:23Okay, there's a couple.
00:24Um Who here kind of has a parent-child relationship with Claude?
00:32Okay, there we go.
00:33I have a kid.
00:34This is Auto.
00:36Uh he learns to drool this week, which surprisingly is something that these things don't come out of the box with.
00:42He also discovered his fingers, but he doesn't know about his hands.
00:45So, they're fascinating and they're in front of his face, but he doesn't know how they got there and he can't bring them back.
00:51But anyway, I finished these slides this morning at 2:00 a.m. and he decided that playtime started at 5:00.
00:56So, we're going to see how the rest of this talk goes, but clearly I need practice telling bedtime stories.
01:01So, we're going to do one of those first.
01:04Once upon a time, a go-to-market team set out on a simple quest.
01:10They wanted to know one thing.
01:13How their business was doing.
01:16Between them and the answer lay one small, tidy pile of data.
01:24And a dragon, because of course the best quests always have dragons.
01:29And every quest also has guardians.
01:31The first was large, worried, and had a great many questions about their security posture.
01:39They defeated it in the way that important bosses are usually defeated, not with a sword, but with paperwork.
01:49The next monster was worse.
01:51It had many heads, and no two of them could agree on what a single word meant.
02:04To beat it, our heroes had to accept a hard truth.
02:09The systems they trusted had never actually promised to tell them the whole story.
02:16So, they stopped believing any single one of them and started building something underneath all of them, a place where the data finally agreed with itself.
02:31And then came the final boss.
02:33It didn't attack.
02:35It did something even worse.
02:37It asked one question.
02:39The hardest one of all.
02:44Now, our heroes had learned a few things by then.
02:48They'd seen what happens when you send a brand new adventurer in with no guidance at all.
02:56And they realized the secret.
02:58You don't need a genius adventurer.
03:00You need a well-prepared one.
03:03First, you write down how your world actually works.
03:07Then you give every adventurer a knowledgeable guide.
03:11They can ask for directions.
03:13And then the adventurers start teaching each other.
03:16So that no one ever has to start from anything and nothing again.
03:23So, they gathered up everything they'd learned.
03:25The permissions, the definitions, the knowledgeable guide, the habit of asking for a second opinion before they believed anything.
03:34And they forged all of it into a single tool.
03:39And when they had finally reached that tidy data pile, they found out that the dragon had never been the villain.
03:52And so, the kingdom entered a golden age of growth and creativity.
03:57The end.
04:00Just kidding.
04:01That's only where the bedtime story ends.
04:02Uh the actual talk starts here.
04:04Because every part of that story is a real problem.
04:08And I want to show you how three of them actually get solved.
04:12So, our agenda for today is I'm going to talk a bit about the age of agentic GTM.
04:16And then we're going to walk through three actual examples that we either have worked through as a company recently or that we've built into the product that we sell to our customers.
04:26So, briefly, to establish credibility here, I'm going to give you 1 minute about our company, Upside.
04:33We are the data layer for the age of agentic go-to-market, which means that we start by helping you get go-to-market data ready for AI.
04:42And that means knowing what actually happened with a data foundation and a dashboard you can use.
04:49And then helping you answer things that used to be impossible at scale with things like deep research or multi-touch attribution model.
04:57We do a lot to help you make AI correct about your business so that it's not hallucinating and giving you trust issues.
05:05And then we make sure that this data is in all the places that you want to use it.
05:08So, a lot of things you used to spend money to buy are actually things that you, the person closest to the problem, can now build.
05:15So, with that established, this is the age of agentic go-to-market.
05:21We have this interesting situation right now where go-to-market teams have not historically been the place where you'd find a high density of builders.
05:30At least compared to product and engineering.
05:32Obviously, there are exceptions to this, but in general, most marketers and sales people were never writing a ton of code.
05:38And that's very much not to say they weren't aware of their problems and didn't have ideas of how they would solve them, but the historical toolbox here was basically
05:46spreadsheets and PowerPoint slides.
05:50And AI has changed that.
05:52Cloud does for building basically what the bicycle did for mobility.
05:56It makes it accessible.
05:57And so spreadsheets and slides are no longer the default business tool that everyone has available, which means that a lot of people who historically had ideas and didn't consider themselves builders
06:09now can be.
06:11Before AI, I would have described myself as technical enough to be dangerous.
06:17I can read code.
06:18I can write code.
06:19You probably wouldn't want my stuff in production generally if you could help it, but I'd be interested who in this room today considers yourselves an actual engineer.
06:30Okay, awesome.
06:31Thank you for being brave.
06:33Who would tell themselves that they are also technical enough to be dangerous?
06:40All right.
06:41And who doesn't read code at all?
06:44Okay, thank you for being brave in the back.
06:47Uh those categories should have been mutually exclusive collectively exhaustive.
06:51So if anybody didn't raise your hand, I recognize that there's no free coffee here and I am the first to talk after lunch, but the thing about AI is it makes all of us technical enough to be dangerous.
07:03What AI has done for those second two groups is basically provided you with an infinite supply of valedictorian interns with computer science degrees who can help you implement all of the solutions
07:15that you probably know better than any engineer that you'd work with on a different team.
07:19And that incredibly speeds up iteration, and it also kind of makes me feel like I'm Harry Potter walking around with a magic wand for the first time.
07:26But essentially, when you set interns loose on problems led by non-technical people to build things, you're going to get some interesting results.
07:35And a couple of years ago, I remember that everyone was talking about the AI hallucination problem.
07:39Somehow that big word seems to not come up as often anymore, but I think we have its older sibling now.
07:44It's a trust problem.
07:46And if you ask Claude to do something like report on revenue, it doesn't say, "I'm not sure."
07:52It says, "Here you go."
07:54And it gives you a wrong answer that looks exactly like being right.
07:58So, a lot of the talks this week I think have been around the bleeding edge of technical optimizations, and a lot of the influencer fluff you see on LinkedIn and Twitter tells you about the magic improvements if you, you know, do your prompt with exactly the right incantation.
08:12I think both of those things might well be true, but I don't generally have time to do either of them.
08:17And in practice, I find that the most practical thing for working with AI is something that we all know pretty intuitively anyway, because we have a lot of experience
08:27working with other people.
08:30And in that framing, establishing trust actually isn't new either.
08:33We already know a lot about how to do this for people.
08:36So, the main thesis of the talk today is actually, when in doubt, manage your agents like other humans.
08:41If you were helping a team of humans figure out how to do something, great.
08:46Do that with your AI agents, too.
08:48And if you take nothing else from this talk, this is my one practical tip.
08:52Use commander's intent when you prompt.
08:55This is something that comes out of the armed forces doctrine, but basically tell your agents why you want them to do something, and they will do it a lot better.
09:05And by the way, this works for humans as well.
09:07People don't generally like being micromanaged, so give them commander's intent and you'll get great outcomes.
09:12But, beware because the agents have been trained on material from other humans, and so they like to micromanage themselves, and that's not usually what you want.
09:21Don't tell Claude to improve itself, you'll get micromanagement.
09:24You have to pull it back and say, "Remember, we're talking about the why."
09:28Anyway, onto the examples.
09:29So, the first one here is I needed to redo our entire website recently.
09:34And at my previous company, which was 600 people and I led the product marketing team, this would have meant several product marketers, a designer, a couple of consulting PMs, a web team, and it would have taken 2 months.
09:46And so, in general, with AI, sometimes you just kind of have to give it a go to see if it's improved enough from the last time you tried.
09:53And so, I did that, you know, YOLO mode, here are the sources, go read everything, please give me my website.
09:59And it was obviously a fail, even with using a nice plan mode in Claude first.
10:04I'm sure this will work eventually, but it doesn't today.
10:08So, you have to do some scaffolding.
10:12And by scaffolding, what I mean is you have to tell it what to know about your business.
10:19So, I maintain this list of anchor assets for our entire company.
10:23I think documentation is something that humans generally should have used anyway, but that was often something that was also generally delegated and forgotten.
10:31Anyway, I maintain these anchor assets, and one of the really important ones here is the product capabilities reference, because on the website, you want to talk about your product.
10:41And this is something that I actually did use AI to compile.
10:46So, this wasn't me sitting here and typing all of these out, but if you pull up one of these cards, you'll see that it tells you what does the product capability
10:54do, why it matters for a bunch of the personas, which are also defined in these anchor assets.
11:00And then, it also shows the track record of how Claude found all of this.
11:04So, these are citations from across every system that it's connected to.
11:07I can follow them back if I need more details, but that helps me know that it didn't hallucinate the important parts.
11:13And if you look back at what we had on this website, that actually ladders pretty much directly up to what was on the homepage, and then if I were to go into these product pages, it also ladders up there.
11:24So, the step here was define the structure first, and then turn Claude loose.
11:29Don't try and YOLO it from the beginning.
11:32Uh if this kind of thinking is new to you on product capability references, by the way, this I think is probably the best book on how to do it.
11:39There's also a nice summary here that somebody wrote up online, particularly steps five to six.
11:45I think are the important ones for this, but the whole book is worth reading, honestly.
11:51Um this ticker is staring me in the corner of the space, so I'm going to give you five more seconds to scan that and then we're moving on.
11:59Two other important things are general D Sloop scale, there are a ton of these floating around, and also consult Persona Bench.
12:05I have a set of agents that inhabits each of the important personas and I can have them review something on demand and give their opinions.
12:12But the second one is our radiant librarian.
12:15So this is something that actually exists inside our product, and I'm going to show you how it works with a diagram.
12:22So this is the librarian.
12:23If you, the user, have a question, "How much pipeline did we create in Q1?"
12:28Then your agent is happily going to go off and say, "Oh, okay.
12:31Well, quarter means, you know, January to March.
12:33I'll just look at created date."
12:35And that is probably not correct.
12:38So what we actually have it do is consult the librarian first.
12:42And the librarian has access to documentation and the library of knowledge items about your company and the schema of prior failed queries.
12:50And so it basically gives your agent a just-in-time memory of all the important things.
12:55So we know that the fiscal year here is actually February through April.
12:59And you pipeline means things that only meet me uh stage two or later.
13:03And then you get a nice trustworthy answer with citations back rather than something that you discovered for the first time.
13:10I was going to do a live demo of this, but I think we're going to skip that for the purposes of time and move on to example number three,
13:17which is what we call our jury and judge workflow.
13:21So we started this company to solve multi-touch attribution, which is kind of like the holy grail of anything in go-to-market.
13:30And it took us 2 years to get anywhere close.
13:32But, along the way, we built this AI-native data layer and then discovered that that was actually what everybody needed for everything else.
13:39And when we came back to multi-touch attribution, enabled by Opus, we actually get really interesting results that are actually pretty impressive.
13:48So, we're going to look at how this workflow works because there's this classi- classification of challenges in go-to-market where there is no empirically correct answer.
13:58And we have a model for how we deal with that in the real world.
14:01It usually involves a trial by a jury of your peers.
14:04So, here we're doing a very similar thing where you might say, "I'd like attribution on the Acme deal."
14:11And your agent is not going to go off and immediately answer this on their own.
14:15They are going to spin up a team of independent analysts who all look at the data independently and come up with an evidence-cited opinion for what they think the attribution credit of that deal should be.
14:28And none of these are necessarily correct, but they are independent research that then goes back to the consensus judge who says, "Oh, I'm not treating these as fact.
14:38I'm treating them as input, and I'm going to weigh the reasoning quality of each of these analysts, and then I'm going to help you come up with the final version here.
14:46And if there's not enough consensus, then I'll escalate and expand the jury.
14:51But, my job is not to do research on my own.
14:53It's to say, "Here's the final result based on a whole team of independent analysts doing a really good job of research."
15:01And this is something that we see in the real world.
15:04We would do it with humans, too.
15:06Turns out multiple researchers with somebody who helps at the end is better than a single person kind of perseverating on that forever.
15:15So, thus ends the three examples.
15:17But, I have a bonus.
15:20So, bonus is right here, agent tears.
15:27In my experience, you can't fix stupid.
15:31So, this basically means friends don't let friends use really bad harnesses or low intelligent models for important work.
15:37An example of this is Slackbot released its MCP client functionality a couple weeks ago.
15:43So, you can now hook Slackbot up and it will talk to MCPs and I thought, "Oh, that's awesome."
15:47Now, everyone who has Slack can suddenly consult the Upside Librarian and I was so excited to try it and it turns out that this is horrifically stupid.
15:57So, basically, the general case here is any AI product where it's been crowbarred into a pre- subscription model is probably not something that you should be using for anything important because the margin on those plans just doesn't leave enough space for an intelligent reasoning model to work.
16:13So, find something that is at least tier two here and tier two means it's got to be a powerful model.
16:20You need it to have attributes like sub agents, plan mode, full MCP support.
16:24It should be able to use file editing and there are a lot of these out here.
16:28So, do not let your team just use the chat GPT web interface and expect that it will be a great result.
16:36And that ends the talk.
16:38So, I hope you stick around for the rest of the track.
16:40I know Jeff from XAI is coming up later and he's got a really bang of a talk about how they basically created Upside for all the rest of the world's data
16:47and I shall be outside for a few minutes if you have any questions.
16:50Thank you.