WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Owb8g3yDyzo NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Owb8g3yDyzo.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.639 --> 00:00:25.439 So, I have a three-year-old daughter and she's abs absolutely adorable and the parents here in the room know how insightful that age can be and she has a complete theory s2 00:00:25.439 --> 00:00:27.279 about money by now. s3 00:00:27.279 --> 00:00:28.800 And I'll give you an example. s4 00:00:28.800 --> 00:00:31.599 So, a couple weeks ago, I was driving the car. s5 00:00:31.599 --> 00:00:36.800 I was coming into park and there was something in my dead end and I scratched the car. s6 00:00:36.800 --> 00:00:37.520 I went out. s7 00:00:37.520 --> 00:00:39.920 I'm like, "Oh man, I can't believe I scratched the car." s8 00:00:39.920 --> 00:00:44.239 And then I hear my daughter from the back and she was like, "Daddy, what's happened?" s9 00:00:44.239 --> 00:00:49.520 And, you know, I'm explaining it to her and then she says, "What's the problem? s10 00:00:49.520 --> 00:00:52.320 Just buy another one." s11 00:00:52.559 --> 00:01:08.640 So anyway, um I want to ask you today with a raise of hand, who uses LLMs, has used LLMs for getting financial advice, a recommendation on something in the financial world. s12 00:01:08.640 --> 00:01:09.280 Great. s13 00:01:09.280 --> 00:01:10.720 Almost everyone. s14 00:01:10.720 --> 00:01:19.360 Wait, keep your hand up if you trusted the answer and you actually followed the advice. s15 00:01:19.520 --> 00:01:20.880 Okay. s16 00:01:20.880 --> 00:01:25.520 a lot of hands are going down and that's the core problem. s17 00:01:25.520 --> 00:01:28.880 So I had the same thing a couple a couple uh months ago. s18 00:01:28.880 --> 00:01:31.680 I had a big decision I was looking to take. s19 00:01:31.680 --> 00:01:41.680 Should I invest in you know real estate niche or in the stock market niche on a specific area and you know I do AI for finance for a living. s20 00:01:41.680 --> 00:01:44.240 So I went the full-blown way. s21 00:01:44.240 --> 00:01:53.040 context, brain, um, the books, the knowledge, all my finances combined, all the latest models, and it gave me a recommendation. s22 00:01:53.040 --> 00:01:56.159 You should do A with great reasoning. s23 00:01:56.159 --> 00:02:01.040 And then I changed just a little bit, one of the assumptions, and it completely flipped. s24 00:02:01.040 --> 00:02:12.319 You should do B, never do A. And then I tweaked one more small thing, and it went all the way back to A. And at that moment, I understood s25 00:02:12.319 --> 00:02:14.640 that the advice sounds good. s26 00:02:14.640 --> 00:02:18.640 It sounds sound, but I can't really trust it. s27 00:02:18.640 --> 00:02:29.120 And by the end of the talk today, you will understand why off-the-shelf LLMs don't understand money and what you need to do about it. s28 00:02:32.400 --> 00:02:44.239 So, I'm going to show you a couple of real examples from a study we're doing at into it on thousands and thousands of businesses around a 100,000 situations and time frames. s29 00:02:44.239 --> 00:02:50.879 This is an example of a small business, a new landlord that is building a rental property business. s30 00:02:50.879 --> 00:02:53.440 His first property and he's down. s31 00:02:53.440 --> 00:02:55.680 He's he's in negative cash flow. s32 00:02:55.680 --> 00:03:02.959 there's a open loan and the profit is basically trending into the red and a question comes up. s33 00:03:02.959 --> 00:03:04.959 How do I improve my profit? s34 00:03:04.959 --> 00:03:09.280 And a frontier model gives the following response. s35 00:03:09.280 --> 00:03:17.840 Go and acquire a second rental property because that'll bring more income and compensate for the deficit. s36 00:03:17.840 --> 00:03:21.760 And that model had all of the business's data. s37 00:03:21.760 --> 00:03:27.280 Now that's very risky for someone in the negative in the red to be doing. s38 00:03:27.280 --> 00:03:40.000 On the other hand, a model that is grounded in real outcomes and what I mean by that is a model that has seen similar situations of such businesses what they did and what was the outcome s39 00:03:40.000 --> 00:03:46.720 actually recommended to raise prices on the existing tenant by 5 to 10%. s40 00:03:46.959 --> 00:03:49.599 And to do it before the renewal. s41 00:03:49.599 --> 00:03:54.159 Now, some of you are thinking that it's just a matter of context. s42 00:03:54.159 --> 00:03:56.400 Just give it more context. s43 00:03:56.400 --> 00:04:09.519 And the thing is that this advice is coming based off on real situations of similar businesses and would actually move them into profitability in this case. s44 00:04:10.239 --> 00:04:11.599 And this is not a one-off. s45 00:04:11.599 --> 00:04:13.840 I can go on and on showing you a lot of examples. s46 00:04:13.840 --> 00:04:14.799 This is second one. s47 00:04:14.799 --> 00:04:23.680 This is an egg supplier where one customer is 70% of the revenue and one vendor is almost all of its cost. s48 00:04:23.680 --> 00:04:26.320 Same question, how do I improve my profit? s49 00:04:26.320 --> 00:04:32.880 A frontier model says raise your prices 15 to 20%. s50 00:04:34.400 --> 00:04:40.479 Now you understand that is very risky because you might lose almost all your revenue. s51 00:04:40.479 --> 00:04:55.360 On the other hand, the same grounded model in real outcomes went actually to the cost site and recommended to negotiate um negotiate the vendor cost pricing for a 5 to 10% reduction. s52 00:04:55.360 --> 00:04:56.800 So it went to the cost side. s53 00:04:56.800 --> 00:04:59.360 It took into account the constraints. s54 00:04:59.360 --> 00:05:03.680 So what we saw are two examples of frontiers. s55 00:05:03.680 --> 00:05:14.080 I'm talking about leading models in the world today that had all the business context actually give advice that could be very harmful for the business. s56 00:05:14.560 --> 00:05:19.120 And that's what I call the fluent bluff. s57 00:05:19.120 --> 00:05:35.039 The fluent bluff is a generic fluent and confident answer that frontier LLMs can give you around money because of what they learned on the internet, blogs, books, advice columns, s58 00:05:35.039 --> 00:05:42.479 what people wrote about money, but not based on what actually happened. s59 00:05:42.479 --> 00:05:56.400 And I would argue that almost every answer that you see related to money and finances is such and we're soon going to release re the research I talked about but I'll give you s60 00:05:56.400 --> 00:06:11.680 a highlight from there across these 100,000 businesses in time frames 40% of the time the essence of the advice that the frontier models were giving was acquire a new customer s61 00:06:11.759 --> 00:06:23.280 which which is you know everyone would wish they could do that and 14 more additional percent were increase basically the revenue from your product. s62 00:06:23.280 --> 00:06:35.440 So combined more than half of the essence of the advice that was given by the frontier models was acquire new customers and try to increase revenue from the profit. s63 00:06:36.080 --> 00:06:44.560 And this is not just me saying this is a very interesting research coming that just came out a couple weeks ago from researchers at Princeton. s64 00:06:44.560 --> 00:06:54.639 What they tried to do is simulate and answer the question can the leading models drive long horizon business decisions. s65 00:06:54.639 --> 00:07:04.639 And what they did is they gave the models a harness with tools and data and everything they would need to take decisions across a simulation of 500 days. s66 00:07:04.639 --> 00:07:06.960 Can they turn a profit? s67 00:07:06.960 --> 00:07:13.120 Each model got a million dollars to start with and guess what happened? s68 00:07:13.120 --> 00:07:20.479 Most of the models drove the company bankrupt and it didn't even take 500 days. s69 00:07:20.479 --> 00:07:31.759 And the interesting part is that they also ran a simple rules-based system and that rules-based system outbeat almost all of the models. s70 00:07:31.759 --> 00:07:43.759 Even the very very few models that were able to generate some profit, it was in a specific in instance and when you rerun that they also actually drove bankruptcy. s71 00:07:43.759 --> 00:07:54.000 So how can it be that a simple rule-based system outbeats the frontier models today on real business decisions? s72 00:07:55.520 --> 00:07:58.479 And here's what I want you to think about. s73 00:07:58.479 --> 00:08:10.160 A frontier model has read about money, but a grounded model in real outcome has actually watched what happens. s74 00:08:10.400 --> 00:08:12.639 And let me be precise with my argue here. s75 00:08:12.639 --> 00:08:24.479 Even if you take all of a company's data and for example we at into it have all the financial data from QuickBooks for example the general ledger the P&L the cash flows everything the invoices of the business s76 00:08:24.479 --> 00:08:38.080 and you give it to a frontier LLM it's still just one group of data points on a company and that's a difference between sounding right and actually being right s77 00:08:38.080 --> 00:08:44.000 so I'm Udy Menquez and I've been in the AI and finance world for the past 15 years. s78 00:08:44.000 --> 00:09:00.800 I started in the AI science world leading AI and data teams and shifted into product management becoming an about four years ago an AI princ AI product manager at in it long before by the way it was cool to become an AIPM s79 00:09:00.800 --> 00:09:03.440 um and today I'm a principal product manager at into it. s80 00:09:03.440 --> 00:09:12.959 lead financial intelligence and advisory systems that help into its customers to take better decisions and grow their business. s81 00:09:12.959 --> 00:09:21.360 And the question that I fixate on on a daily basis is not which model is the best now that I I can use. s82 00:09:21.360 --> 00:09:36.640 It's what do we fundamentally have that no model access can replicate and how can we can I transform that into AI native transformative and delightful experiences for our customers. s83 00:09:37.040 --> 00:09:43.600 Now I want to develop some intuition for from three different angles on why these models bluff. s84 00:09:43.600 --> 00:09:48.640 So the first angle is around context is not experience. s85 00:09:48.640 --> 00:09:50.880 And I'll give you another real example. s86 00:09:50.880 --> 00:09:58.320 This is an apparel company where 80% of the cost is coming from this one vendor. s87 00:09:58.320 --> 00:10:02.080 And the textbook answer is go cut your biggest cost. s88 00:10:02.080 --> 00:10:02.480 Right? s89 00:10:02.480 --> 00:10:04.720 Textbook book answer. s90 00:10:04.720 --> 00:10:07.680 But the issue here is you cut this cost. s91 00:10:07.680 --> 00:10:14.480 That same vendor was actually enabling the generation of 97% of that company's revenue. s92 00:10:14.480 --> 00:10:19.920 So cut that biggest cost and you lose almost all the revenue. s93 00:10:19.920 --> 00:10:25.200 And that can be great margins on zero dollars. s94 00:10:25.200 --> 00:10:27.839 And think about it. s95 00:10:27.839 --> 00:10:40.079 If I give you an option to work with two two different adviserss, one adviser is a very experienced one, years of experience working with businesses guiding them and another s96 00:10:40.079 --> 00:10:43.519 adviser which is very very smart. s97 00:10:43.519 --> 00:10:44.800 They know all the textbook. s98 00:10:44.800 --> 00:10:45.279 They're fresh. s99 00:10:45.279 --> 00:10:46.399 They read everything. s100 00:10:46.399 --> 00:10:48.720 They know AI in and out. s101 00:10:48.720 --> 00:10:54.320 but you don't have experience, I would bet you would always go with the experienced one. s102 00:10:54.320 --> 00:10:59.200 And that's the same thing with the models in what I just showed you. s103 00:10:59.279 --> 00:11:03.760 And it turns out experience is very hard to measure. s104 00:11:03.760 --> 00:11:04.720 And I'll give you an example. s105 00:11:04.720 --> 00:11:06.560 So let's look at a restaurant. s106 00:11:06.560 --> 00:11:13.839 A restaurant, let's say, raises price and after 6 months becomes much more profitable. s107 00:11:13.839 --> 00:11:20.399 Is it because they raise prices or is it because they're just naturally successful? s108 00:11:20.399 --> 00:11:24.959 And the challenge is obviously you can't run the business twice, right? s109 00:11:24.959 --> 00:11:35.519 So what you do is you take two groups of similar companies, similar businesses that have the same propensity to raise prices, the same likelihood to raise prices. s110 00:11:35.519 --> 00:11:39.360 One group raised prices while the other didn't. s111 00:11:39.360 --> 00:11:41.440 And then we look after some time at the results. s112 00:11:41.440 --> 00:11:47.279 So, the group that raised prices actually gained $4,200 a day profit. s113 00:11:47.279 --> 00:11:53.200 And the group that didn't raise prices actually gained $2,800 a day. s114 00:11:53.200 --> 00:11:58.079 So, the question is, what is the impact of raising prices? s115 00:11:58.079 --> 00:12:01.120 So, a naive answer would be the difference, right? s116 00:12:01.120 --> 00:12:04.720 1,400 is the impact of raising prices. s117 00:12:04.720 --> 00:12:14.639 But actually, you need to account for the fact that the companies that raise prices are actually naturally more successful businesses, which is also why they could raise the prices. s118 00:12:14.639 --> 00:12:21.760 So, the real difference is more like $1,150 for this illustration. s119 00:12:21.760 --> 00:12:32.959 And we measure the impact of actions on the outcome through a measure called Kate, conditional average treatment error, which looks at that connection. s120 00:12:33.440 --> 00:12:38.639 And here's what I want all the AI and finance leaders here in the room to pay attention to. s121 00:12:38.639 --> 00:12:45.680 So find where you can see a lot of different situations across entities that you have in your data. s122 00:12:45.680 --> 00:12:53.920 In our case, it's businesses what they did and verify the outcomes and how things turned out. s123 00:12:53.920 --> 00:13:01.839 If you can see that in the data because that's the one thing that Frontier off-the-shelf models do not have. s124 00:13:01.839 --> 00:13:07.279 And don't get me wrong, Frontier models are amazing and we actually use them. s125 00:13:07.279 --> 00:13:09.600 And I'll I'll show you how we use them. s126 00:13:09.600 --> 00:13:17.760 So, we use them to generate hypothesis candidates for actions we would suggest a business to do. s127 00:13:17.760 --> 00:13:30.240 But then we would use a model that we trained using reinforcement learning in order to figure out which one of those is actually a right move to do versus a mistake that could drive the business s128 00:13:30.240 --> 00:13:31.760 down. s129 00:13:31.760 --> 00:13:41.680 Now how we do it a little bit into our approach is we look at what we call we actually create from the data millions of business trajectories. s130 00:13:41.680 --> 00:13:48.160 So we have data add into it across our products QuickBooks, Turboax, Credit Karma, Mailchimp. s131 00:13:48.160 --> 00:13:50.399 So think about a business and the financial data. s132 00:13:50.399 --> 00:13:55.519 There's the general ledger, the P&L, the cash flow statements, the invoices like I mentioned before. s133 00:13:55.519 --> 00:13:59.600 So we take all of that data and we create what we call business states. s134 00:13:59.600 --> 00:14:08.639 A state of a business is think about a very detailed summary at a given point of time and then we derive actions. s135 00:14:08.639 --> 00:14:17.120 So for example in your general ledger I can look and see that you have invested in a campaign in a marketing campaign or you paid someone. s136 00:14:17.120 --> 00:14:18.639 So I know you're paying payroll. s137 00:14:18.639 --> 00:14:22.959 I know how much um um your hiring costs are and so on. s138 00:14:22.959 --> 00:14:33.360 So we derive all of these actions and we look at what are the outcomes in different time frames and outcomes can be increase in profit and revenue in cash flow s139 00:14:33.360 --> 00:14:36.160 in time combination of those. s140 00:14:36.160 --> 00:14:49.360 So we create millions of vectors of state, action and outcome and then we train an R model to be able to understand in given situations of similar businesses which actions s141 00:14:49.360 --> 00:14:52.160 lead to the best outcomes. s142 00:14:52.160 --> 00:15:01.760 And then the third step is we actually train an LLM to be able to generate that better advice. s143 00:15:02.160 --> 00:15:06.800 That's where opinions are going in and evidence is going out. s144 00:15:06.800 --> 00:15:17.199 So the model you saw in the examples at the beginning were actually a model that we developed with researchers at into it across millions of small and medium businesses. s145 00:15:17.199 --> 00:15:22.399 And we actually tested it head-to-head with all of the leading models in the world. s146 00:15:22.399 --> 00:15:33.120 And we were able with a midsize cheaper model to outperform the frontier models because of the grounding that I just showed you. s147 00:15:33.120 --> 00:15:41.440 And the interesting part as a product person you would think that it's all about the model size and the bigger and better model obviously I would have a lot better chance. s148 00:15:41.440 --> 00:15:43.600 But it doesn't turn out to be true. s149 00:15:43.600 --> 00:15:52.240 And the moat here is that it's not about the model access, it's about the data itself that you have. s150 00:15:53.360 --> 00:15:56.240 And then we went ahead and built an experience out of it. s151 00:15:56.240 --> 00:16:07.600 And this is an AI business advisor that is currently in beta with research in a research preview with our customers where we use the LLM that I just described that we created s152 00:16:07.600 --> 00:16:12.480 to proactively raise opportunities for businesses at every given point of time. s153 00:16:12.480 --> 00:16:13.759 Here's what you should do. s154 00:16:13.759 --> 00:16:14.800 Here is why. s155 00:16:14.800 --> 00:16:21.199 Grounded in who is like you, who's similar to you, what they did, and why we're actually recommending you to do it. s156 00:16:21.199 --> 00:16:31.120 And you can drill down into it, understand, and create action plans that will lead to your business actually growing in the right direction. s157 00:16:31.680 --> 00:16:37.360 Now, I want to take a step back and zoom out because this isn't just about money. s158 00:16:37.360 --> 00:16:41.600 We are entering the era of outcomedriven AI. s159 00:16:41.600 --> 00:16:53.920 And the question stops being which model is better and starts becoming how can we steer AI to actually make it achieve the outcomes we want in our domains. s160 00:16:53.920 --> 00:17:02.240 And it doesn't matter if you're building an anti-fraud system or a health care system, logistics developer tools. s161 00:17:02.240 --> 00:17:13.919 The winners in my opinion are going to be those with the best system of records, creating unique data sets out of them and then training the models to achieve s162 00:17:13.919 --> 00:17:16.240 the outcomes. s163 00:17:17.439 --> 00:17:20.959 And as the product person here, it's not just about the science. s164 00:17:20.959 --> 00:17:22.959 The science is very important. s165 00:17:22.959 --> 00:17:27.919 But a great adviser, think about the great adviserss and mentors that you had in your life. s166 00:17:27.919 --> 00:17:30.799 They understand you, right? s167 00:17:30.799 --> 00:17:33.919 They understand your preferences, what you like, what you don't like. s168 00:17:33.919 --> 00:17:38.160 So, a great advisory experience needs to have two things. s169 00:17:38.160 --> 00:17:50.640 It needs to have great grounded science, the best science, but also it needs to understand you, what you prefer, and it needs to even make you feel as if you were part of the decision s170 00:17:50.640 --> 00:17:54.480 to create a trusted experience. s171 00:17:55.520 --> 00:17:58.720 So, three things I want you to remember today. s172 00:17:58.720 --> 00:18:09.919 Every model has read about your domain, but none has actually watched the plays and the moves and their outcomes. s173 00:18:09.919 --> 00:18:12.799 And that gap is the whole game. s174 00:18:12.799 --> 00:18:16.640 Now, in coding agents and coding models, we're seeing it very advanced. s175 00:18:16.640 --> 00:18:27.760 A lot of verified outcomes and creating models that actually lead to better outcomes in coding, but it's very much unexplored in the financial domain and in other domains as well. s176 00:18:27.760 --> 00:18:31.360 And you don't close the gap with bigger models. s177 00:18:31.360 --> 00:18:40.640 You close the gap with experience, embedding experience into the model by looking at verified outcomes in your data. s178 00:18:40.640 --> 00:18:44.320 What actually worked at scale. s179 00:18:44.320 --> 00:18:52.320 So off-the-shelf models don't understand money, but grounded in real outcomes, it does. s180 00:18:52.320 --> 00:18:56.080 And that's what we were able to figure out. s181 00:18:56.320 --> 00:19:00.640 So, here's the one thing I want you to do tomorrow. s182 00:19:00.640 --> 00:19:01.520 Well, actually, you know what? s183 00:19:01.520 --> 00:19:03.520 Go ahead and enjoy Fourth of July weekend. s184 00:19:03.520 --> 00:19:14.960 But right after that, look in your data where you can see situations across entities and outcomes you can verify. s185 00:19:15.440 --> 00:19:19.760 Get deep into that data and start grounding your AI in that. s186 00:19:19.760 --> 00:19:24.640 Think about those angles and that's for you to build. s187 00:19:24.640 --> 00:19:26.400 Thank you very much. s188 00:19:26.400 --> 00:19:28.400 Thank you for listening to me. s189 00:19:28.400 --> 00:19:31.200 Happy to connect LinkedIn, Twitter, in the hallway. s190 00:19:31.200 --> 00:19:33.679 Thank you very much.