Build for the Memo, Not the Demo — Shawn Chan, China Resources Holdings

AI Engineer · 24 min · 231 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Good afternoon.
00:13Thank you for being here, day four of our conference, a room with no windows.
00:19Right just after lunch.
00:21You are the strongest people in this building.
00:24Let me start with a confession.
00:26For 15 years, my job has been one thing.
00:30I sitting in a room, a very smart, confident person hands me a piece of paper, and before my company spends a hundred million dollars, I have to decide, do I believe this paper?
00:43This year, my job still exactly that.
00:49Except now, some of time, the very confident person handing me the paper is a chatbot.
00:58And honestly, the chatbot is often better written than humans, better grammar, nicer formatting, never get defensive when you ask a follow-up question, very very sure of itself.
01:13The problem is being sure of yourself and being right are two different skills.
01:19Some of most the confident people I have ever met in finance were also the most wrong.
01:26AI just learned that trick faster than rest of us.
01:31So, here my whole war whole talk in two sentence.
01:35First, almost every AI finance product is built to impress people for five minutes, and almost none are built to survive a room whose entire job is to not be impressed.
01:52Second, and this is the part I promised the organizers added the exact same skills that fix your product is the skills that gets investors like me to write you a check.
02:09Same muscle.
02:10I've proved both.
02:11And I'm telling this now, this week, because a lot of you are about to get pulled into exactly this.
02:21Your CEO saw a demo somewhere.
02:24Your biggest comes a customer suddenly has a compliance department, or you are 3 months from raising your next round.
02:34When any of those days arrive, I'd rather you to hear the hard part from someone who sit on the other side of the table, than discover them live in a room in front of the people
02:50who bill by hour.
02:55A quick quick a bit about me.
02:58I promise this is the boring part, and I will keep it fast.
03:0215 years of cross-border deals, Hong Kong, mainland China, the UK, the US.
03:10Mergers, IPOs, big strategic investment on names you'd actually recognize, the kind of companies that go public, and your LinkedIn feed won't shut up and about it for a week.
03:26Along the way, I've sit in about 200 investment committee meetings.
03:32I have the grey hairs to prove it.
03:36I checked.
03:37It's not genetics.
03:40And here the part that matters for second half of this talk.
03:47I've also read hundreds and hundreds of pitch decks, founders' decks, bankers' decks.
03:5547 slide seed decks.
03:59I have seen fonts that should be illegal.
04:04[sighs and gasps]
04:05200 committee meetings taught me one thing.
04:10No textbook says outlawed.
04:12The number on the page is not what gets a deal approved.
04:16Trust is a number.
04:18What gets a deal approved?
04:20Money doesn't follow intelligence.
04:24Money follows trust, and the trust is fragile, especially when the thing that wrote the page has never once in its entire life said the words, "I'm not sure."
04:39Let me give you one small taste of what those rooms feel like.
04:45Early in my career, a very polished, very expensive banker present a beautiful slides full of confident wrong numbers.
04:56One senior person in room asked a one quiet question, "Where does this number come from?"
05:03The banker posed uh what felt like an entire physical quarter.
05:08The pose taught me more about finance than 3 years of exams did.
05:13So, I'm not sure as a So, I'm not here as a builder.
05:20I don't build these systems.
05:22I sit across the table from them and from the founders selling them, deciding whether to trust them.
05:32Today, I'm going to give you both halves of that.
05:37What breaks trust in your product and uh what builds trust in your pitch.
05:46Let's define two machines that moves everyone keep confusing.
05:51Machine one is a demo.
05:53One clean document in, one fluent answer out is the whole job is to make a room go oh for 5 minutes.
06:03Machine two is a is a memo.
06:06The real document is a real committee race before real money moves.
06:11Hundreds of pages, filings, transcripts, broken nose, scratches, someone's rushed nose from a call last Tuesday.
06:22Half the sources disagree with each other and the memo's job is not to make you go oh.
06:31Its job's to survive an argument.
06:34Basically, a family dinner except somebody's uncle brought a spreadsheet.
06:39Here's everyday worship of the gap.
06:42Ask your phone to summarize a long email.
06:46That's a demo.
06:47It's not just sound plausible for 10 seconds.
06:52Now, imagine that some same phone has to stand in front of a bank and defend out loud why you deserve a mortgage.
07:04Suddenly, plausible isn't enough.
07:06Now, it has to be right and it has to prove it.
07:12That second situation is what a memo actually is.
07:19Now, you might think the demo world and the memo world never touch.
07:24Let me tell you about the most expensive typo in history.
07:29February 2023, one of the biggest tech company on the earth launches its shiny new AI assistant with a promotional demo.
07:39In that demo, the assistant answers a simple question about a space telescope and it gets a wrong the wrong answer.
07:49One sentence one wrong fact about a telescope in um marketing demo, the market noticed the company's stock dropped around 8% in a day.
08:03That's roughly $100 billion of value gone because of unchecked sentence.
08:10$100 billion for one sentence.
08:13Nobody in that company asked the one question.
08:19Every junior analyst on my team is trained to ask before anything leaves the building, wait.
08:27Where does this claim come from?
08:30Did everyone check it?
08:32So, here's the punchline.
08:34Even the demo failed the memo test.
08:38The moment real money is watching and the real money is always watching.
08:44Every sentence become a memo sentence.
08:48There is no safe demo anymore.
08:51Keep that story in your head because of the same trust breaking moment happens in six smaller, quieter, very predictable ways inside of your product every day.
09:05Let's go through them fast.
09:10Six ways trust quietly breaks.
09:15Which source do you believe?
09:17Do the numbers agree?
09:19Do you hide contradictions or show them?
09:23Is that the a fact or a guess?
09:27Can you prove it in 30 seconds?
09:30And whose name is actually on the decision?
09:34Six.
09:35Keep count with me.
09:37I will keep each one short and I will bring receipts.
09:45Model one.
09:47Not every source deserves the same trust, but most AI system treat them like they do.
09:55Think of like this, a number from an audit filing is your accountant speaking under oath.
10:04A number from a analyst note is a friend at a party confident probably from someone's internal email is a thing you overheard in in a elevator.
10:19Most of retrieval systems can't tell them these apart.
10:25They grab whichever text is close to your question and hand it over like a gospel.
10:32True story.
10:34Anonymized.
10:35I know I once watched a um very expensive AI tool confidently called a number from a group chat someone's rough guess texted six months earlier.
10:48The model loved the confident freezing.
10:51The real audit number was the three rows away in the actual filings.
10:58The AI just liked the group chat version better.
11:02It sounded more enthusiastic.
11:05If your system can can't tell an accountant under oath from a rumor in a group chat, it it is not ready for real money.
11:22Model two.
11:23The numbers have to agree each other everywhere, every time.
11:30Here's a memo that already died.
11:33Page one says revenue growth 18% 18% page 11 little table nobody reads for fun says 17.4 nobody in the room cares about the missing 0.6. They care about what it what it means if this person didn't check the easy
12:01mathematics what did they not check on the hard stuff.
12:08That memo didn't pass not because of the number because of what number you implied.
12:16And if you want the industrial strength version of this failure remember the giant American real estate company that let algorithm to buy houses at scale.
12:31The algorithm was extremely confident about the house prices the house disagreed.
12:41The company ended up writing off around half a billion dollars shut the whole unit down and let a quarter of the staff go.
12:52The model wasn't stupid the model was unsupervised nobody built a boring mechanic machinery that forces the numbers to keep agreeing with reality after lunch day.
13:09Fluent and a confident remember is not the same as right.
13:19Model three surprises people.
13:24A contradiction is not a bug a contradiction is a gift.
13:28If the CEO says one gross number on the earning call and the official filing says the different a different one, the gap is the single most interesting thing in the real
13:42story highs.
13:44Real diligence leaves for that gap.
13:48AI does the opposite.
13:53It is a trained to sound smooth and helpful.
13:57So, when it hears a conflict, it quietly picks whichever version reads nicer and moves on.
14:07You never even learn there was a disagreement.
14:12I've sit through exactly these CEO's number and the filing numbers meaningfully different.
14:23Neither one uh Nobody Nobody flagged it.
14:27Everyone just used the nicer one.
14:31We caught it because one person happened to have both documents open at once.
14:37Pure luck.
14:39Luck is not a control.
14:42Your job as a builder isn't resolve the argument.
14:47It's to make sure that the argument happens in front of a human instead of quietly along inside box.
14:59Model four.
15:01Facts and the guesses have to live in separate box.
15:08And the fluent AI loves melting them into one smooth sentence.
15:12Example, the company will likely receive approval next quarter.
15:18Reads like a fact.
15:21Sounds like a fact.
15:23it is a guess.
15:24Somebody buys the estimator wearing fact shaped the clothing.
15:29A committee's entire job is to agree with the guesses while trusting the facts.
15:36If your system melts them together, the committee can't find them seems, and then all all they can do is to is is approve or reject the genuine map of the document.
15:52You should not spend a hundred million dollars on maps.
15:56I watched approval accept soon.
16:01Turn across three drafts of a demo into approval received.
16:06Nobody lied.
16:07The guess just rose its fact costume a little longer each rewrite until nobody remembered it then it started as a guess.
16:21The approval did didn't arrive on schedule.
16:25That was an uncomfortable phone call.
16:28This fix almost embarrassingly cheap.
16:31Label your guesses, a tag, a color, anything that survives being copy pasted into someone else's slides three weeks later.
16:47Model five, if nobody can find where a claim come from it doesn't matter how right it is.
16:58You've all heard about the New York's lawyer.
17:03Uh he filed a legal brief written with a chat bot's help.
17:08The brief cited six court cases, beautiful citations proper proper formatting, very convincing.
17:19One small issue, the cases didn't exist.
17:22The AI invented invented all six.
17:27And here, my favorite detail, the part that should be taught in school.
17:33Before filing, the lawyer got suspicious, so he asked the chatbot, "Are these cases real?"
17:41And the chatbot said, "Yes."
17:44That is like asking the guy who sold you the watch whether the watch is real.
17:51The judge fined him.
17:54The story went around the world, and my second favorite detail, the fake cases even had a real real estate surrounding names and the page numbers.
18:07The AI didn't just lie, it just the formatting.
18:11It cited itself beautifully, wrong but beautifully.
18:16The license is a 30-second test.
18:19When someone points at a sentence and says, "Show me where this come from."
18:27You either click once or land on exact source paragraph, or you open seven browser tabs and start swiping.
18:40I have personally been the guy with seven tabs in a real meeting while the room full of people watched my scroll.
18:4910 out of 10.
18:51Wouldn't would not recommend.
18:54If you remember only one sentence from this whole talk.
18:59The click-through is a product.
19:02Everything else is well-written packaging.
19:11Model six, my favorite, because it's the most human someone has to sign.
19:21Here's the story you probably know.
19:24An airline website chatbot told a grieving customer he could book a full-price ticket now and claim a bereavement discount afterwards, that the policy didn't exist.
19:39The chatbot made it up politely, fluently, confidently.
19:46The customer took the airlines to a tribunal.
19:52And the airlines' defense, this is real, was was that the chatbot is called a separate legal entity responsible for its own action.
20:07That is the corporate version of my dog ate my homework.
20:12The tribunal didn't buy it.
20:14The airline paid, and every one of us in our boardroom quietly took a note that day.
20:22You can't outsource accountability to your own software.
20:27At the bottom of every real decision, a human signs.
20:32If your architecture doesn't have a fundable human at the end of it, you you have not built a product.
20:43You have have built an excuse generator.
20:47So, build your AI around that accountable person, not instead of them.
20:56Okay, the fix.
20:58Five things.
21:00Each one is a direct cure for a story you have just heard.
21:06This is almost a word for word.
21:10What I demand from our vendor before lighting their system near a live deal.
21:16One, every claim comes with a receipt.
21:21Each sentence linked us straight to its source paragraph with the source trust level attached.
21:31Not a citation tab at the end.
21:34Two, fact and the guesses stay visibility separate.
21:41At glance at the page, I insistently see what's proven and what's somebody's best estimate.
21:50Three, numbers agree with each other automatically.
21:55The system refuse to ship a memo where the figures don't match.
22:00No human checking at 2:00 in the morning.
22:04Four, contradictions get surfaced, never smoothed over.
22:09When sources disagree, the system raises its hand instead of picking the front layer answer.
22:19Five, a real human approval gate and is logged.
22:24Who's Who reviewed what change when they signed.
22:29That log is audit trail.
22:32Notice what's not on the list.
22:34A smart promoter not One of these is a bigger brain problems.
22:44All five plumbing and honesty problems.
22:48The winners in the in this category won't win on benchmark points.
22:55They will win because a tired, skeptical finance person finance person can trust that their output at 11:00 at night without opening seven tabs.
23:13Now, the part I promised, the money.
23:17Many of you are not just the building AI products.
23:21You are raising for them or you will be.
23:26So, let me tell you what actually happens after you leave the pitch meeting.
23:33Your deck becomes a memo.
23:35Literally, literally, literally, someone like me sits down and writes an internal memo about you.
23:43Every number you said out loud gets checked get get checked against your data room, what which means everything I just told you about the documents apply to you personally.
23:59Four lessons.
24:00Okay, my time's up.
24:01Thank you.
24:04[applause]