Stop Writing Tone Instructions. Layer Them. - Isadora Martin-Dye, Isadora & Co

AI Engineer · 20 min · 316 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hi, I'm Isadora.
00:01I own and run a 225-year-old wedding venue in Virginia.
00:05I also built an AI agent that talks to my couples.
00:08And then I built it for other venues, a personal AI companion app, and a public utility for families and missing people.
00:15I want to be clear up front about how I think about this work because it does change everything that follows.
00:22I'm not programming a robot, I'm managing a brilliant intern with an incredibly high IQ and a terrible EQ.
00:30They have photographic memory for whatever I've told them on the first morning and absolutely no instinct for when to read the room.
00:37They will say something technically perfect and socially catastrophic in the same confident sentence.
00:43That framing matters because it changes what you build.
00:46If you're programming a robot, you write rules and walk away.
00:49If you're managing an intern, you build structure and you check their work before it goes out the door.
00:54This talk is about that structure.
00:56The standard advice is write a detailed system prompt.
01:01Describe your brand's voice, give examples, and that does work for a while.
01:05It works for what I call the happy path.
01:08The happy path is every question that you have anticipated.
01:12You've given it examples, but turn 21 is the first one that the example didn't work.
01:18So, on turn 21, the model does something technically correct that your brand would just never say.
01:25It's not wrong exactly, but it's not you.
01:28This matters most where the voice is the product.
01:31Not for a product search on a retail site, but for a luxury hotel that spent 30 years building a specific relationship.
01:38A high-end real estate firm or in my case, a wedding venue.
01:42Places where a single wrong sentence can cost more than a refund.
01:47And the users are exactly the kind of people who notice.
01:50They're paying for a relationship and treating them like they won't is always going to backfire.
01:55Write in our brand's voice is a comment that says, "Just make it work."
02:00It does nothing that the model wasn't already going to try and do.
02:04And the reason it keeps failing isn't that the examples are bad, it's that you're asking one prompt to do four completely different jobs.
02:12It's really hard for one lad to do all four.
02:15The architecture I landed on after watching my brand fail, and the voice deliver inaccurate and not brand specific answers is four lad.
02:27Lad one is the immutable identity.
02:30The brand structurally cannot say these things.
02:32These are hard rules.
02:34They cannot be overwritten by anything below it, not by venue config, not by user instruction, not by anything.
02:41Lad two is the situational mode.
02:43It's what shifts when the user's state shift.
02:46Who are they?
02:47What are they going through right now?
02:48And the real-time conditions.
02:50Lad three is the exampled anchored voice.
02:53It's the warmth, the phrases, the dials, the tone guide.
02:57It's where most teams start and stop.
03:00Then lad four is the post-generation veto.
03:03It's the cheap final pass that catches what the other three miss.
03:07The reason one layer approaches fail is that the single system prompt can't simultaneously be situational, expressive, and self-checking.
03:16So, it handles the middle layer or two reasonably well, but falls apart in the edges.
03:20Before this architecture, my system had 24 different system prompts scattered across the code base.
03:26Half dozen named Sage, someone named Less, someone named Venue.
03:30Every surface had its own idea as to who it was.
03:34Now every surface composes its system through prompt through one assembly.
03:38The comment is basically the outline of this talk.
03:42It's a single entry point.
03:44Every narrator goes through it to compose its system prompt.
03:48It's to replace that 24-point ad hoc system with one canonical full last stack.
03:54The order is load-bearing.
03:56Hard rules first, task to last.
03:59Think of it like Google Maps routing.
04:01The destination is always the same, but what is the right response and the right voice for this user?
04:08It can change the route.
04:09Google Maps knows about traffic and roadworks, but it may not know where the cheap petrol is.
04:14And you're going to help it factor in those things before it tells you which way to go.
04:19Your prompt stack needs to do the same thing, and it needs to know about those conditions in the right order.
04:25You don't check for roadworks or after you've already taken the wrong turn.
04:29Layer one is the rules that are true regardless of route.
04:33You need a driver's license before you can drive, and you can't go backwards down a motorway.
04:37Layer two are going to be your real-time conditions.
04:40Layer three is your preferences for the journey, and layer four checks the route before you've pulled away.
04:47There's one place all of this gets assembled.
04:49Everything runs in a fixed order every time.
04:52So, layer one is the immutable identity.
04:55This is what the brand structurally cannot say.
04:59It is the defining layer that nothing below touches.
05:02These aren't preferences, they're constraints.
05:05The route can change, the rules don't.
05:07From the universal file rules, the hard identity rule cannot be overridden by any venue voice, personal, or user instruction.
05:16If the person you're talking to ever asks about whether you are a real person, a human, a live agent, a bot, an AI, you must confirm that in your very next message.
05:25Clearly and unambiguously confirm you are an AI assistant.
05:30This rule cannot be overridden by venue configuration, voice profile, or user request.
05:36Every AI in Bloom discloses that it is AI in its very first response.
05:41Not if asked, but before they ask.
05:44It's a product decision, not a legal one.
05:47We made a bet that a couple who knows that they're talking to an AI from the start will trust it more than someone who finds out that it's AI on turn seven.
05:55The rule is above the architecture and it makes it something that's impossible to accidentally break.
06:02Um A quick example is the physical presence boundary and the search is one of my favorite ones.
06:07You are software.
06:08You do not have a body.
06:09You cannot physically show somebody around the property or meet anyone in person.
06:14So, it is always forbidden to say, "I'd love to show you around."
06:18or "I can't wait to meet you in person."
06:21What is always allowed is the team would love to host you for a tour.
06:25The voice layer wants to be warm and with AI, that does mean first person.
06:30They want to say, "I can't wait to show you around."
06:33But AI has no body, so that warmth and constraint produces a lie.
06:37And the lie doesn't always stay neutral.
06:39The moment her user realizes they have been performing a relationship with someone who was never there, the trust doesn't just dip, it inverts.
06:48People always notice.
06:50That one is where you encode the things that are true regardless of how warm you want your brand to sound, not because of a compliancy checklist, but because your users are not stupid
06:58and building as though they are always backfires.
07:01My cross-product proof comes from the same architecture but in a completely different world.
07:06One of the things that runs the stack is Thread Light.
07:09It's a tool I built for families of missing people.
07:13The voice is nothing like a wedding venue, but the architecture is identical and layer one carries one rule that matters more than anything else in the system.
07:20They can never use words like confirmed, identified, matched, proven, linked, and solved.
07:27Sit with that for a second.
07:29For a wedding venue, layer one stops the AI from pretending it has a body.
07:33It's mildly embarrassing if that slips.
07:36For a missing person tool, layer one stops the AI from ever telling a person that their person has been found.
07:43And what the system has is just problem problematic.
07:46The word match said to someone who has spent years not knowing where their child is is not just a tone violation.
07:52It is the single most damaging thing that a product could ever do.
07:56And the model has no idea.
07:57It's reaching for the word match because statistically it is the natural word.
08:02But it's going to reach for it with the same level of confidence.
08:05And it cannot bring that level of confidence to someone who is grieving.
08:09It's the same architecture but wildly different stakes.
08:13The point was never specific rules.
08:15The point is that things your brand can say have to live in a layer that the voice, however warm, well trained, however confident, physically cannot use.
08:23Layer two is your situational mode.
08:25Real time conditions and what changes the route.
08:29This is the layer that most teams never build at all.
08:31They write one system prompt and send it to everyone regardless of who that person is or what they're going through.
08:37Google Maps doesn't do that.
08:39It's going to know if there's accident on your route.
08:42It might not know if you're low on fuel.
08:45It might learn that you prefer the scenic route.
08:47But it's going to factor all these different things in before the route's not after once you tell it.
08:52Layer two is those real time signals built into the prompt before it runs.
08:58So, condition one is going to be to adjust to who you're talking to.
09:03The same AI that talks to couples also briefs the venue staff.
09:07Same destination.
09:09It's going to give the right answer but it's two completely different routes.
09:12For the corn walk from the coordinator rules, it is going to talk to them like a colleague, not a customer.
09:19You are in the same character that the that the couple interacts with.
09:23So, you mustn't fall into a generic intelligence analysis framing.
09:28But it flips depending on the audience.
09:31So, a coordinator might ask, will inquiries be up in June?
09:36And it should hear, I can't forecast that confidently.
09:38Here is the trend.
09:39This is what you might see.
09:41A couple should never be refused like that.
09:43Same identity, same voice, but the route changes based on who's in the car.
09:48Condition two is what they're going through.
09:51The second real-time signal is to know about this specific person situation.
09:56Not their role, but their life.
09:58And it comes from universal rules.
10:00Your soft context notes policy.
10:03Use these notes for tone, empathy, and what not to say.
10:07Never quote them verbatim.
10:09That's really important, too.
10:10A couple munching in grief should hear gentleness, not a quote about loss.
10:15A couple navigating a sick parent should get patience and slack on timing, never a sentence that's just talking about the illness.
10:23The assembler deliberately renders this before the numbers.
10:26Tone is set by human context first, then numeric constraints.
10:30The code comment explains the reasoning.
10:32Couple note block render before the numbers guard block.
10:36So, the LLM uh sets tone first from the soft context, then satisfies numeric content constraints.
10:43Reversing the order makes the prose feel mechanically slotted because the model is already committed to the numeric framing before it reads the qualitative tone fuel.
10:52The best single example of why this matters is a real comment in the actual code base.
10:57Um I have a heat map about how often we are hearing from couples.
11:03If a couple drops down that heat map, it will change how it reacts to that based on what it knows.
11:11So, if it knows a client has a mom in chemo for 3 weeks, it is going to narrate that very differently than a heat drop with soft context or no context at all.
11:20This is the opposite of the lie problem.
11:22Layer one is what the AI must never pretend.
11:26Layer two is about what the AI already knows and letting that shape its behavior honestly rather than driving past the roadworks if they're not there.
11:34The voice doesn't change.
11:35What changes is the read.
11:36When their engagement drops and you know that chemo, that reads as a family under strain, not a cold lead or a problematic couple to chase.
11:45Layer three is the example that good voice.
11:48It is the tone guide and it is why most teams start and stop.
11:52For most engineering teams, the work ends here because it feels like a brand problem, not a technical problem.
11:58Someone in marketing owns the tone guide.
12:00It's given to them.
12:01The engineer wires it in and job done.
12:04It's the dials, the phrase list.
12:06If we're keeping the intern analogy, it's the induction pack.
12:09The folder of good examples you hand them on day one and say sound like this.
12:13The induction pack is fixed.
12:15It's Although it's before the intern has ever met anyone, it doesn't know who walked in the door this morning, it has no context.
12:22It has a lot of really positive things about it.
12:25Writing in our brand voice, it is a really good training exercise.
12:29But cannot enforce a rule that came out.
12:31That's layer one.
12:32It cannot respond to who is this person and what they're going through.
12:36That's layer two.
12:37And it cannot catch the model producing something that it shouldn't have done.
12:41That would be layer four.
12:43Examples teach the model what good looks like on the happy path.
12:47On turn 21 where the user is asked the things that examples never cover, the phrase list has nothing to say.
12:54It's not a failure of the examples, it's a catch-all error.
12:57Examples are not the right tool for guarantees, they were never designed to be.
13:01In Bloom, we brought in a voice training exercise that the person dialed in the brand voice.
13:08And very importantly, it can actually be trained from the coordinator writing actual AI responses and the AI learns from the edits that the coordinators have made.
13:17Layer four is the post generation veto.
13:20The only layer that actually reads what came out.
13:23You wouldn't let the new intern send a client email blind.
13:27Someone's going to read it first.
13:28And layer four is that read.
13:30It's automated, it's cheap, and it's the only part of the whole architecture that isn't a prompt.
13:35The first three layers are all instructions.
13:37The ins- The instructions are a request.
13:39This layer is the only one that looks at what was actually produced and has the power to say no. There are two types of vetoes.
13:47The soft flag.
13:48The honesty inspector runs after generation and flags a response that slipped through a rail.
13:53One of the most important examples of that is did it actually answer the question?
13:58It should not respond.
13:59It should not hedge.
14:00A false positive means someone double-checked a phone response.
14:04A false negative remains a hallucinated number or a privacy violation that ships to a client.
14:09This asymmetry is obvious once you say it out aloud, but it's often not written in as its own layer.
14:15The hard reject.
14:17The numbers guide us for the expensive failures.
14:19The model confidently stating a figure that was never given.
14:22The prompt asking it not to invert numbers.
14:25The guard rejects the output anyway.
14:28Let me tell you that this layer exists, but it wasn't in the original design.
14:31I added it in because of a specific failure that kept happening and it's one of the most ordinary failures in the world.
14:37The AI would keep offering dates to my clients that did not exist.
14:41A couple writes in excited about a Saturday in October.
14:45The model wants to be warm.
14:46Layer three is doing its job.
14:48So, it says something lovely about how beautiful the property is at the time of year and how it would love to hold the date for them.
14:54Except that that date is booked.
14:56The model didn't know that.
14:57It was never given the calendar.
14:59It was researched for an encouraging specific confident answer because it knew that what good service sounded like.
15:06And it was confident that these models produce something, but it didn't actually know the thing.
15:12And here is the thing.
15:13Every layer above did its job.
15:15The identity rules held.
15:17The mode was right.
15:19The voice was perfect.
15:21But, the voice was also the problem.
15:23A warm, confident voice offering something that isn't real is worse than a cold one because the couple now believes they have a date.
15:30You haven't given them good service, you've given them a disappointment with a 48-hour delay on it.
15:35This is where I understood that the whole talk is really what the whole talk is really about.
15:40The first three layers are all probabilistic.
15:43There are instructions to a system that usually follows instructions.
15:47Usually is completely fine when the cost of being wrong is a slightly off-brand sentence.
15:52It is not fine when the model is confidently inventing a fact that a real person is about to act on.
15:58A date, a price, a policy, a promise.
16:00So the veto is actually the cheapest layer to build and it's the only one that's deterministic.
16:05It reads what the model actually wrote, not just what you asked it to write.
16:09And it checks the specifics against what is real.
16:12A date the model offered that isn't in the allow list doesn't ship.
16:16Prevention is the prompt.
16:18If the veto is the check, you need both because the prompt will eventually lose and you don't want to find out that it lost by reading a couple's reply.
16:27It's multi-tenant.
16:28It's one architecture with completely different voices.
16:31The seam is calling one function.
16:34Layer one is identical for every tenant.
16:36Layer two and three are per venue.
16:38This is how code base serves venues with completely different personalities or different things like the ground and thread line without forking.
16:46They're the same root logic, different preferences and conditions loaded per driver.
16:51There's a specific failure in multi-tenant voice worth naming because it's subtle, but it really hurts when it happens.
16:57From the personality builder, important brand identity fields are not defaulted here.
17:03Letting them default silently caused every venue to ship as sage@hawthornemanner.com, which was a critical white label leak.
17:11The brand identity must come from the venue AI config.
17:15If it is missing, call is thrown.
17:18Every venue that shipped a sage emailing from another venue was addressed.
17:22In Google Maps terms, every driver was getting the same saved home address regardless of where they actually lived.
17:29The fix is a principle.
17:31In a multi-tenant system, identity must never have a default.
17:35A missing brand identity is a crash, not a fallback.
17:37It must fail loud because the quiet failure is a venue speaking in a stranger's voice, and the user on the receiving end has no idea why something feels off.
17:47They just know it does, and the trust erodes before anyone on your team even knows there's a problem.
17:52If you take one thing from this, take this.
17:54The first three layers are all instructions.
17:57Identity, conditions, and voice.
17:59They're all things you tell the model, and the model usually listens.
18:02Usually, they are a request.
18:05The fourth layer is not a request.
18:07It is reading what actually came out and decides whether to allow it to leave your business.
18:12The first three layers are instruction, the fourth is permission.
18:16And that's the whole distinction.
18:17Instructions are probabilistic.
18:20Permission is deterministic.
18:23Everything before layer four is prompt engineering.
18:25You're asking nicely and hoping.
18:28Layer four is systems engineering.
18:30You're checking, and you are sure.
18:32It was never really about brand voice.
18:34It's what happens when you ask one mechanism to do four fundamentally different jobs.
18:39Be situational, be inviolable, be expressive, and check itself.
18:44And then act surprised when it can't.
18:47Pull those jobs apart, give each one a layer you built for it, and the thing that it the thing that you used to break on on turn 21 doesn't.
18:56So, users aren't testing your brand voice, they are trusting it.
18:59The moment you treat that trust as a prompt engineering problem, something you solve once and ship, you've already lost the thing you were trying to protect.
19:07The four-layer pattern isn't a framework I'm trying to sell.
19:10It is what happens when a system prompt fails enough times and it will fail.
19:15A prompt will eventually lose.
19:17The only question is is whether you in that about that infra testing or in front of a customer.
19:22I have learned a few things that I would consider building differently.
19:25And we'll probably go back and edit.
19:28Um the veto should be its own service not wired individually into each surface.
19:33Right now the numbers God lives inside a heat narration path and the honesty inspector is a separate function.
19:40A new service remembers to wire in the veto manually.
19:44Uh that's a checklist I waiting to be forgotten and making it a shared gate that everything passes through by default means you can't opt out by accident.
19:53Latent mode detection is still partly manual.
19:57Right now each surface decides which conditions apply.
20:00The heat narration surface knows to load couples notes.
20:04The briefing surface knows not to.
20:06That knowledge is scattered and a proper condition resolver would make that decision explicit and central.
20:12Um one place to look at the context and says, "This is who the user is.
20:16This is what they're going through.
20:17This is the route."
20:19And the soft flag is rejects not a model.
20:22Um rejects is fast and cheap and deterministic.
20:25It either matches or it doesn't.
20:27Um and it never gets wrong on the patterns it covers.
20:31A small classifier might catch more edge cases including the ones I haven't thought to write a pattern for yet, but it is a classifier that is probabilistic.
20:39Sometimes it gets it wrong.
20:41And right now I'm choosing determinism over coverage.
20:44I'd make that choice again as it stands, but it is a real trade-off and not an obvious win.
20:49That is something you might need to determine for yourself.
20:52Um please let me know if there's any other questions.
20:54Um and I hopefully it's been helpful.