WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=REascnFlq_8 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/REascnFlq_8.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.560 --> 00:00:13.560 Can you all hear me? s3 00:00:13.560 --> 00:00:14.560 All right. s4 00:00:14.560 --> 00:00:16.920 Um Well, first things for coming. s5 00:00:16.920 --> 00:00:20.000 I can't believe there's anybody in the room, but that's very nice. s6 00:00:20.000 --> 00:00:22.720 Uh Uh my name is Jean-Denis. s7 00:00:22.720 --> 00:00:25.080 Um I'm CTO at a company called Town. s8 00:00:25.080 --> 00:00:31.400 We're not going to really talk about Town, so you can go to town.com and check that out if you want, but that's not the point of the talk today. s9 00:00:31.400 --> 00:00:36.760 I was CTO at Plaid for 7 years, and then I was at Dropbox before. s10 00:00:36.760 --> 00:00:38.920 And then before that I built software for hedge funds. s11 00:00:38.920 --> 00:00:50.000 I've done lots of stuff in my career, and right now I'm working on uh assistance agents for for normal people, not not for engineers, but for like basically everyone in America and the world. s12 00:00:50.000 --> 00:00:54.960 And one of the things we've been working on are systems where agents work with other agents. s13 00:00:54.960 --> 00:00:56.600 So, agent-to-agent. s14 00:00:56.600 --> 00:01:07.680 And the main idea is that we think there's huge network effects if agents can work together to get things done for people because in the real world the way most of us do work s15 00:01:07.680 --> 00:01:09.880 is with other people, right? s16 00:01:09.880 --> 00:01:11.440 Uh more is better. s17 00:01:11.440 --> 00:01:16.640 But, actually I don't think agent-to-agent makes much sense as a concept. s18 00:01:16.640 --> 00:01:20.360 So, I want to reframe the entire talk in terms of search. s19 00:01:20.360 --> 00:01:25.760 So, I think most LLM systems are just a search problem. s20 00:01:25.760 --> 00:01:37.520 And what you're trying to do is you're trying to make sure the context window right before you either return results to the user or before a tool call, you're trying to make sure the context window has the right information for the user. s21 00:01:37.520 --> 00:01:46.000 If you put the right information in the context window then based on the intelligence, so to speak, of the LLM, you will get the best result possible. s22 00:01:46.000 --> 00:01:52.760 Um so, you know, 4 years ago the way we did that is humans would populate the context window manually. s23 00:01:52.760 --> 00:02:00.400 Then a couple years ago, most people are ragging, so they were like, let's have a tool, like a search tool, that can look across systems and bring the data in there. s24 00:02:00.400 --> 00:02:04.000 And then people were like, well, that doesn't scale super well, has issues. s25 00:02:04.000 --> 00:02:15.520 And now we're all about agentic search, which is the idea that you give the agent a lot of tools, and it'll search through the space of all content, and then hopefully before it makes a tool call, it has exactly the right content s26 00:02:15.520 --> 00:02:19.080 to make the right tool call to return the right information to the user. s27 00:02:19.080 --> 00:02:21.920 Um and in this, by the way, there's no there's no people. s28 00:02:21.920 --> 00:02:25.800 It's just a one LLM call at the like the one that matters having the right context. s29 00:02:25.800 --> 00:02:26.840 That's That's what you're trying to do. s30 00:02:26.840 --> 00:02:30.200 You're trying to engineer um that system. s31 00:02:30.200 --> 00:02:30.440 Cool. s32 00:02:30.440 --> 00:02:32.520 So, what what does that have to do with agent agent? s33 00:02:32.520 --> 00:02:35.640 So, I want you to imagine the following world. s34 00:02:35.640 --> 00:02:39.360 There's not many agents that can do things. s35 00:02:39.360 --> 00:02:41.680 There's just one agent, right? s36 00:02:41.680 --> 00:02:47.200 And it has one context window, and it has access to all the information in the universe. s37 00:02:47.200 --> 00:02:56.880 It can look at any one person's email, can look at any company's information, can look at any government's information, and it has it right there in the context window, s38 00:02:56.880 --> 00:02:58.600 and then you ask it to do something. s39 00:02:58.600 --> 00:03:05.120 You You have your little system prompt with all that data, and what's going to happen is it'll give you the best possible outcome. s40 00:03:05.120 --> 00:03:09.640 And that actually That is a multi-agent world. s41 00:03:09.640 --> 00:03:11.760 It's just an agent that has access to all the world's information. s42 00:03:11.760 --> 00:03:13.200 That's the natural state of things. s43 00:03:13.200 --> 00:03:14.640 That's the ideal state of things. s44 00:03:14.640 --> 00:03:30.080 There's a problem with this state of things, and the problem comes from a few So, you're learning economics something called the Coase theorem, and it says that basically even humans, if they all have access to all the right information and there's no transaction costs, we get the economically s45 00:03:30.080 --> 00:03:33.959 ideal outcome out of a out of a contract or negotiation. s46 00:03:33.959 --> 00:03:34.880 Well, it's the same thing. s47 00:03:34.880 --> 00:03:38.880 We can't put all of the world's contacts We can't make it available to the LLM. s48 00:03:38.880 --> 00:03:44.400 Like theoretically even with infinite context window, because of privacy and security. s49 00:03:44.400 --> 00:03:45.320 We're humans. s50 00:03:45.320 --> 00:03:52.200 I don't let you look at my email, so there cannot be an agent that I'm willing to just let it look at my email all the time. s51 00:03:52.200 --> 00:03:55.760 But, if it existed, it would be very, very powerful. s52 00:03:55.760 --> 00:04:02.080 So, I I think this is like this is the test for a multi-agent system, which is how well does it approximate this? s53 00:04:02.080 --> 00:04:10.560 If it approximates this, that means if you can get the same data in your window that a perfect system that has access to all the world's data could, then you get the optimal outcome. s54 00:04:10.560 --> 00:04:11.920 That's what you need to try to do. s55 00:04:11.920 --> 00:04:22.000 So, we're going to talk about five strategies that people use at various companies to try to get the right data into that that LLM call with an externality. s56 00:04:22.000 --> 00:04:28.360 So, the first one is approximate access to everything within a trust boundary. s57 00:04:28.360 --> 00:04:39.160 So, my wife and I, we have an agent together, and that agent has access to my email and her email, uh including emails before we were married. s58 00:04:39.160 --> 00:04:48.400 Uh and it's okay, she doesn't ask my agent questions about that, but she does ask about, you know, whether I like schedule something for our kids or, you know, uh if I followed up on some third-party thing. s59 00:04:48.400 --> 00:04:52.120 And so, the fact that our agent has access to both of our systems is wonderful. s60 00:04:52.120 --> 00:05:03.640 Um and the work context, this might be there's a an HR team agent that has access to all the HR systems, just like an employee of the HR team would, or maybe as much access as the lowest employee in the HR team. s61 00:05:03.640 --> 00:05:10.680 All the employees in the HR team have the ability to ask this agent questions, and boom, it gets pretty good results. s62 00:05:10.680 --> 00:05:12.760 And this is very popular right now. s63 00:05:12.760 --> 00:05:20.560 Uh it's very popular with IT teams and security teams cuz it's the same model as SaaS for security, so it works really well. s64 00:05:20.560 --> 00:05:27.640 I think it has a problem, which is a fundamental problem that if I wake up in the morning, it's like basically the only thing I think about, which is s65 00:05:27.640 --> 00:05:32.480 does it get Over time, does this system naturally require fewer humans? s66 00:05:32.480 --> 00:05:36.280 And then, as the models get better, does this approach get better? s67 00:05:36.280 --> 00:05:39.160 And the problem with this approach is the answer is no to both. s68 00:05:39.160 --> 00:05:45.800 Uh you still need humans to think about all the data, and you don't get magical de-siloification of your data. s69 00:05:45.800 --> 00:05:49.120 You've just created a new silo cuz a human thought about it. s70 00:05:49.120 --> 00:05:56.760 So, the problem with this is I do think if this is your approach to building better AI, you're going to be in the next couple years. s71 00:05:56.760 --> 00:05:58.120 Um but that's okay. s72 00:05:58.120 --> 00:06:00.880 Your is my opportunity. s73 00:06:00.880 --> 00:06:03.240 Uh I'm just I'm just not an I'm sorry. s74 00:06:03.240 --> 00:06:03.680 That was mean. s75 00:06:03.680 --> 00:06:06.760 But like I think it's not I think it's a good now way to think about it. s76 00:06:06.760 --> 00:06:09.080 It's not the good end game way to think about it. s77 00:06:09.080 --> 00:06:18.440 The other approach which is I think is a little more clever and then I'm going to try to explain it is basically you try to have tools that make a different trade-off between power and privacy. s78 00:06:18.440 --> 00:06:20.800 So, I'm going to give you an example here. s79 00:06:20.800 --> 00:06:29.400 Um the use case is I want to ask my agent, does anyone in my company is anyone in my company connected to someone on the finance team at Acme Corp? s80 00:06:29.400 --> 00:06:36.120 And so, what the the no silo way to do that is just give me access to everyone's Gmail in my company. s81 00:06:36.120 --> 00:06:38.600 I'll see who has emails with people from Acme Corp. s82 00:06:38.600 --> 00:06:44.440 Then I'll look at their profile on Google or LinkedIn and then I'll be like, oh, you seem to email a lot with the CFO. s83 00:06:44.440 --> 00:06:45.880 Can you do the intro for me? s84 00:06:45.880 --> 00:06:48.160 But right, obviously silos, we don't want that. s85 00:06:48.160 --> 00:06:53.400 So, what if you build a tool and what the tool did is it looked at everyone's Gmail. s86 00:06:53.400 --> 00:06:58.960 So, that tool had access to everyone's Gmail and it just returned a relationship strength score. s87 00:06:58.960 --> 00:07:04.160 So, the tool you would give it like a domain and you would say I'm looking for someone who's a CFO. s88 00:07:04.160 --> 00:07:11.000 It would look at everyone at the company who sent emails to that company and then it would like rank their score and they would give you back the score s89 00:07:11.000 --> 00:07:13.120 and then the agent would get the score and it would be like cool. s90 00:07:13.120 --> 00:07:15.919 Then they would use a Slack tool to text that person the company. s91 00:07:15.919 --> 00:07:20.000 It's like, hey Bob, I see that you're connected to the Jane who's the CFO at Acme Corp. s92 00:07:20.000 --> 00:07:21.919 And then Bob would be like, yes, I am. s93 00:07:21.919 --> 00:07:26.840 And then your AI would be like, oh, can you draft an e- can I draft an email or can you draft an email introducing me? s94 00:07:26.840 --> 00:07:31.360 And then Bob would say, yes, and he would do that and you'd be connected and everything would be wonderful. s95 00:07:31.360 --> 00:07:33.360 So, this is actually very cool approach. s96 00:07:33.360 --> 00:07:34.680 I don't know how many of you do it. s97 00:07:34.680 --> 00:07:38.200 Like we we do this at Town for a few things that we see a lot of our users do. s98 00:07:38.200 --> 00:07:42.680 We ask ourself, what is a privacy preserving tool that all of our users would be okay existing? s99 00:07:42.680 --> 00:07:49.080 They can opt out if they don't want it, but it has a natural network effect because it breaks through silos in interesting way. s100 00:07:49.080 --> 00:07:54.040 Um, like another one that's interesting here is letting other people put draft emails in your inbox. s101 00:07:54.040 --> 00:08:02.040 You let other people at your company draft emails on your behalf cuz they're going to ask you to anyway to get intros if they're on the sales team, so might as well save yourself a few clicks. s102 00:08:02.040 --> 00:08:08.000 So, the question here is like are people going to be okay with a privacy trade-off that you make within a corporation? s103 00:08:08.000 --> 00:08:09.440 Uh, bad. s104 00:08:09.440 --> 00:08:11.200 Bad. s105 00:08:11.200 --> 00:08:13.040 Oh, boy. s106 00:08:13.040 --> 00:08:16.360 Within a corporation that will, you know, mostly it works. s107 00:08:16.360 --> 00:08:21.240 Um, so the problem here again is it's again manual and not dynamic. s108 00:08:21.240 --> 00:08:23.720 It's manual cuz humans need to think about the tools. s109 00:08:23.720 --> 00:08:25.880 Maybe I could build the tools. s110 00:08:25.880 --> 00:08:29.840 Uh, and it's also manual cuz you need to explain it to everyone that it's happening. s111 00:08:29.840 --> 00:08:36.479 Humans may not like it if this is happening if they're not okay with the privacy security the privacy kind of power trade-off that you've made. s112 00:08:36.479 --> 00:08:37.280 Cool. s113 00:08:37.280 --> 00:08:40.039 And again, this doesn't really get better as they I guess better. s114 00:08:40.039 --> 00:08:41.680 That's the problem. s115 00:08:41.680 --> 00:08:41.840 Cool. s116 00:08:41.840 --> 00:08:42.800 So, now the third category. s117 00:08:42.800 --> 00:08:46.200 This one's super popular, but only mostly in the single user context. s118 00:08:46.200 --> 00:08:49.000 So, this is, you know, like personal wikis in claw land. s119 00:08:49.000 --> 00:08:50.480 That's what we would call it. s120 00:08:50.480 --> 00:08:51.640 But it's across teams. s121 00:08:51.640 --> 00:08:54.240 So, it's a shared silo. s122 00:08:54.240 --> 00:08:59.839 Create a new place where data accumulates within your company within subgroups at your company. s123 00:08:59.839 --> 00:09:03.520 Um, and you start to put more and more stuff there over time. s124 00:09:03.520 --> 00:09:05.600 And all the agents have access to that stuff. s125 00:09:05.600 --> 00:09:11.120 Because they have access to it, you no longer have information that would be okay to be shared that's stuck in a silo. s126 00:09:11.120 --> 00:09:13.720 It now automatically filters out into this public space. s127 00:09:13.720 --> 00:09:16.280 So, examples, shared skills. s128 00:09:16.280 --> 00:09:20.560 If you code in an organization, probably in your repo you have shared skills. s129 00:09:20.560 --> 00:09:21.960 Anyone can make them better. s130 00:09:21.960 --> 00:09:25.120 Someone has a better way to, you know, profile your database or whatever. s131 00:09:25.120 --> 00:09:25.920 They can write the skill. s132 00:09:25.920 --> 00:09:29.839 Next time someone's sitting there is like, "Oh my god, the database query is slow." s133 00:09:29.839 --> 00:09:33.440 It uses the profiling skill and everyone's a better engineer. s134 00:09:33.440 --> 00:09:34.240 So, that's one version. s135 00:09:34.240 --> 00:09:44.240 The other one that's pretty popular is people decide they have some shared mediums, like a wiki, airtable, etc. And they uh they have a scale that says, "Hey, put more data in there over time." s136 00:09:44.240 --> 00:09:45.880 So, these are cool. s137 00:09:45.880 --> 00:09:56.920 Um and they work as long as your your agents have those tools and also some trajectory incentives to really like get data out in and out of of these shared silos. s138 00:09:56.920 --> 00:10:02.880 Um I think the next version of this that a few people are working on is like you have a sweeper AI. s139 00:10:02.880 --> 00:10:08.320 So, this actually if there's one good idea in this talk that I think works really well is this. s140 00:10:08.320 --> 00:10:09.160 It's a sweeper AI. s141 00:10:09.160 --> 00:10:12.280 So, you have an AI inside each private silo. s142 00:10:12.280 --> 00:10:17.560 An AI has a policy about what has to stay in the silo. s143 00:10:17.560 --> 00:10:21.960 And then it also has a description of all the shared spaces that you have. s144 00:10:21.960 --> 00:10:26.920 And at the end of the day, it looks at new information in the silo and it puts it in the public spaces. s145 00:10:26.920 --> 00:10:28.680 Well, public, public to your company. s146 00:10:28.680 --> 00:10:37.000 So, this is the same as the personal wiki that you all have AI building for you at the end of the day so that I can know your goals and your friends and all that stuff, but it's at the company level. s147 00:10:37.000 --> 00:10:44.400 Um the hard part is how do we pick what private information is okay to to share and to put it in shared silos. s148 00:10:44.400 --> 00:10:45.800 And I think there's two approaches. s149 00:10:45.800 --> 00:10:48.240 There's the ask a human approach. s150 00:10:48.240 --> 00:10:56.640 So, this is like the LLM comes up with a list of things to contribute and then it asks the user, "Hey, are you okay with me putting this in the shared space?" s151 00:10:56.640 --> 00:10:57.200 And you read it. s152 00:10:57.200 --> 00:10:58.240 You're like, "Yep." s153 00:10:58.240 --> 00:10:59.720 Saved you a bunch of time. s154 00:10:59.720 --> 00:11:00.000 Right? s155 00:11:00.000 --> 00:11:02.200 I mean, you weren't going to do it otherwise. s156 00:11:02.200 --> 00:11:07.600 Uh I think the other version is you actually ask the LLM to enforce a policy. s157 00:11:07.600 --> 00:11:11.200 And I think that actually is where things are going to go very, very quickly. s158 00:11:11.200 --> 00:11:22.839 Um and I think in the next 6 months we'll have a bunch of systems where companies have trusted an LLM with a policy to automatically surface more and more information that otherwise would have been private into a public space. s159 00:11:22.839 --> 00:11:32.360 If you're like at a Fortune 500 enterprise company, unfortunately, I don't think that's going to happen for a while, but I think if you look at smaller companies, like 10, 50 person employees, high trust, s160 00:11:32.360 --> 00:11:41.840 like low likelihood of something one one doing bad with the data, which really clear to know what data couldn't be shared, basically finance and HR data, you're going to see a ton of this. s161 00:11:41.840 --> 00:11:47.760 And the cool thing here is this really improves trajectories of systems on on common work. s162 00:11:47.760 --> 00:11:49.000 All right, that was third approach. s163 00:11:49.000 --> 00:11:52.440 Fourth approach pretty obvious, use humans as the conduit for information. s164 00:11:52.440 --> 00:11:55.320 So this is like traditional agent to agent. s165 00:11:55.320 --> 00:12:03.840 My agent ask your agent, "Hey, who is connected to someone on the finance team at Acme Corp?" s166 00:12:03.840 --> 00:12:07.280 You as a human see the request and you're like, "Yeah, I'm okay with that. s167 00:12:07.280 --> 00:12:10.520 Go and find the information inside of my email." s168 00:12:10.520 --> 00:12:12.240 And then it shows you the result. s169 00:12:12.240 --> 00:12:17.000 And then you're like, "Yes, I'm okay with that result going to the person who asked." s170 00:12:17.000 --> 00:12:25.480 The big problem with it is for for any request that has low where it's like only a few people will have the information, you're kind of spamming everyone the request. s171 00:12:25.480 --> 00:12:38.160 So if I ask this question to 100-person company, 100 people are being pinged on Slack, being like approves on these requests to like farm your personal network for this for this, you know, for this like the answer to this question. s172 00:12:38.160 --> 00:12:39.760 That's not very efficient. s173 00:12:39.760 --> 00:12:43.120 Um And so that's why there's a better version of it. s174 00:12:43.120 --> 00:12:48.560 Um which I this is this is very powerful, but uh I haven't seen it in practice much. s175 00:12:48.560 --> 00:12:51.040 It's it's a black box approach. s176 00:12:51.040 --> 00:12:52.400 I wish I had a diagram for this. s177 00:12:52.400 --> 00:12:54.480 Unfortunately for you all, I do not. s178 00:12:54.480 --> 00:12:56.040 So here's what this means. s179 00:12:56.040 --> 00:13:04.520 The black box approach is where when you ask a question that can only be answered by looking at information in other people's silos. s180 00:13:04.520 --> 00:13:13.839 You have an LLM, the trace of which no one has access to, that gets access to all of the data, and it gets to the answer. s181 00:13:13.839 --> 00:13:14.280 Right? s182 00:13:14.280 --> 00:13:20.640 Like by say get to the answer, either gets the answer or it's about to do the the the any tool call that's all right. s183 00:13:20.640 --> 00:13:30.480 And then it looks at what information did it need to make that tool call, and it only ask the people who own that information for their approval to do the tool call. s184 00:13:30.480 --> 00:13:36.920 So, in the example before that I gave, when I ask 100 people at my company, "Hey, do you know the CFO at Acme Corp?" s185 00:13:36.920 --> 00:13:41.040 The request goes to everyone's agents in my company. s186 00:13:41.040 --> 00:13:47.240 All of their agents look in their Gmail and their private silos to see if they're connected to the CFO. s187 00:13:47.240 --> 00:13:48.600 That happens automatically. s188 00:13:48.600 --> 00:13:51.920 No No human is being asked for approval for that to happen. s189 00:13:51.920 --> 00:13:59.280 Then it the agent in the black box ha gets the list of the 20 people who are connected. s190 00:13:59.280 --> 00:14:03.480 It looks at contacts from the emails to determine who has the strongest connection. s191 00:14:03.480 --> 00:14:06.200 It determines that it's Bob. s192 00:14:06.200 --> 00:14:14.680 And then, it just asks Bob, "Hey, Jean-Denis wants you to introduce him to Jane, the CFO at Acme Corp. s193 00:14:14.680 --> 00:14:16.000 I know you're well connected to her. s194 00:14:16.000 --> 00:14:19.160 Am I okay sharing that bit of information with Jean-Denis?" s195 00:14:19.160 --> 00:14:20.120 And you're like, "Yeah, sure." s196 00:14:20.120 --> 00:14:20.800 You click yes. s197 00:14:20.800 --> 00:14:21.880 No big deal. s198 00:14:21.880 --> 00:14:23.800 The important thing is you have to trust the black box. s199 00:14:23.800 --> 00:14:37.000 So, you have to trust that you can break down all the silos for an LLM that has full access and that doesn't ask for permission until there's this sharing moment or this right step. s200 00:14:37.000 --> 00:14:40.800 So, actually, within a company, this is not impossible to do. s201 00:14:40.800 --> 00:14:45.839 Uh and actually, your security and compliance team can get okay with it. s202 00:14:45.839 --> 00:14:56.240 Um you just have to have You have to be sure that the human in the loop step is correct, and you have to be sure that you're not letting other information go through without last answer. s203 00:14:56.240 --> 00:15:03.400 So, you know, like the the nightmare scenarios and things like this are things like uh um Sorry. s204 00:15:03.400 --> 00:15:06.040 I'm like We have plenty of time. s205 00:15:06.040 --> 00:15:06.640 I'm almost done. s206 00:15:06.640 --> 00:15:07.720 So, it's great. s207 00:15:07.720 --> 00:15:19.720 Um the nightmare scenarios with things like this is someone asks a question like, "Are you connected to a recruiter at the other company that you have no business being recruited to as a way for them to find out that you're interviewing somewhere else, right?" s208 00:15:19.720 --> 00:15:26.480 So, you know, there are you you still sometimes with a black box inadvertently get an information out that you shouldn't be able to. s209 00:15:26.480 --> 00:15:30.120 You have to really think about how you would build a great system. s210 00:15:30.120 --> 00:15:32.440 So, those are the those are the approaches. s211 00:15:32.440 --> 00:15:44.880 I think if I were to bet on one that has immediate ROI that we're going to all see in both like open source claw-ish worlds and then like small companies, it's going to be the wiki that's automatically created by AI s212 00:15:44.880 --> 00:15:46.800 like the information base that's kept up to date. s213 00:15:46.800 --> 00:15:55.160 I think there will be database versions of it, wiki versions of it, and I think more and more we're going to trust LLMs to make the decision about what's okay to share and what's not. s214 00:15:55.160 --> 00:15:56.400 There are problems. s215 00:15:56.400 --> 00:16:01.120 So, prompt injection in in the silos can be a real problem. s216 00:16:01.120 --> 00:16:02.080 Obviously. s217 00:16:02.080 --> 00:16:12.520 So, if you have a silo that has like that's more open and someone can put something bad in there and then that as part of the energetic search you pull it out, you know, bad things can can happen. s218 00:16:12.520 --> 00:16:23.000 Uh you can have it's very easy to have a shared wiki that just goes like totally off the rails, you know, like the information there one piece of information there is incorrect cuz LLM made a mistake s219 00:16:23.000 --> 00:16:24.800 and then it like poisons it forever. s220 00:16:24.800 --> 00:16:30.640 I have a personal wiki that thinks my agent's name is Apex right now, but I renamed my agent a month ago to Ivy. s221 00:16:30.640 --> 00:16:38.920 And like somewhere in memory bank uh uh of my like setup uh Apex lives and so I can't get rid of it. s222 00:16:38.920 --> 00:16:39.839 Uh that's fine for Apex. s223 00:16:39.839 --> 00:16:45.360 That's a funny one, but it's like much more difficult if it's a really wrong piece of information about your business. s224 00:16:45.360 --> 00:16:53.240 Um if you don't have human in the loop for any of the steps, obviously there'll be false and wrong disclosures. s225 00:16:53.240 --> 00:16:57.040 You know, sometimes when there's a wrong disclosure of information, someone gets fired. s226 00:16:57.040 --> 00:17:00.440 Someone there's a wrong disclosure, it doesn't matter at all. s227 00:17:00.440 --> 00:17:02.120 Sometimes a customer sues you. s228 00:17:02.120 --> 00:17:05.360 So, you know, uh you got to be careful. s229 00:17:05.360 --> 00:17:12.120 Um and then I think this all sounds nice, but like who approves what, what's logged, what's reversible? s230 00:17:12.120 --> 00:17:16.360 The black box idea is really great, but it can't truly be a black box. s231 00:17:16.360 --> 00:17:19.079 Someone at your company will want to audit it at some point. s232 00:17:19.079 --> 00:17:20.839 They want to understand what's going in there, right? s233 00:17:20.839 --> 00:17:26.600 So, at some level there must be some person in the CISO suite or somewhere that has access to all the data. s234 00:17:26.600 --> 00:17:29.560 Um Yeah. s235 00:17:29.560 --> 00:17:32.240 So, um what do I think? s236 00:17:32.240 --> 00:17:33.840 Well, I do think the frontier is auto. s237 00:17:33.840 --> 00:17:34.600 So, I've said that. s238 00:17:34.600 --> 00:17:36.880 I think in coding we used to approve everything. s239 00:17:36.880 --> 00:17:39.040 Then we were like, "YOLO, live dangerously." s240 00:17:39.040 --> 00:17:42.240 And now the gods at Anthropic have granted us auto mode. s241 00:17:42.240 --> 00:17:46.160 And auto mode tries to figure out when we're maybe being a little silly and it tells us. s242 00:17:46.160 --> 00:17:50.160 Well, I think A to A across information silos will be the same way. s243 00:17:50.160 --> 00:17:56.520 I think what's going to happen is we're going to get comfortable with low sensitivity information being pulled out and put into common spaces. s244 00:17:56.520 --> 00:18:00.920 And then we will have a place that's like human review or always human approved. s245 00:18:00.920 --> 00:18:04.800 And then over time what's going to happen is the LLMs will get more powerful. s246 00:18:04.800 --> 00:18:07.360 We will be better at encoding safe policies within them. s247 00:18:07.360 --> 00:18:13.640 We'll be better at designing for the really hard areas tools that get the privacy trade-off correct. s248 00:18:13.640 --> 00:18:19.840 And it'll just be more and more auto for building shared silos and even sometimes for deciding whether involve a human. s249 00:18:19.840 --> 00:18:32.040 So, the example that I have is like if I ask for notes from a weekly recurring call with, you know, a supplier of ours and their finance team, maybe the the LLM's like, "Oh, well, in giving your role, s250 00:18:32.040 --> 00:18:34.440 I don't need to ask anyone on those teams for permission. s251 00:18:34.440 --> 00:18:36.000 I can just share the notes with you. s252 00:18:36.000 --> 00:18:37.200 It's like it's fine. s253 00:18:37.200 --> 00:18:38.640 I they can it can look at the content. s254 00:18:38.640 --> 00:18:39.840 It can see what my role is. s255 00:18:39.840 --> 00:18:43.920 It can decide from a risk perspective I'm okay with that disclosure." s256 00:18:43.920 --> 00:18:49.320 Um and the cool thing about auto, by the way, is if you design your systems that way, it'll scale with model capacity. s257 00:18:49.320 --> 00:18:59.520 So, my encouragement would be like, you need to start if you have agent to agent or or work across silos, which is I think the better way to think about it, definitely define a low sensitivity zone s258 00:18:59.520 --> 00:19:02.480 where you're okay with the LLM making a call. s259 00:19:02.480 --> 00:19:04.560 And get okay with that. s260 00:19:04.560 --> 00:19:13.240 And then magically, as time goes on, it'll get bigger and your system will naturally get more powerful, which is what you you So, you want to be on a beach. s261 00:19:13.640 --> 00:19:14.320 That's what you want to do. s262 00:19:14.320 --> 00:19:15.480 That's where I want to be. s263 00:19:15.480 --> 00:19:17.400 My kids in Hawaii. s264 00:19:17.400 --> 00:19:18.280 Okay, network effects. s265 00:19:18.280 --> 00:19:19.480 I've 1 minute. s266 00:19:19.480 --> 00:19:20.840 Uh this is the conclusion. s267 00:19:20.840 --> 00:19:30.040 So, we talked about five approaches, blah blah blah, trust boundaries, custom tools, shared silos, humans in the loop, and this human in the black box version. s268 00:19:30.040 --> 00:19:32.360 Uh I think this stuff is very powerful. s269 00:19:32.360 --> 00:19:37.960 I think the interesting questions a little bit are in within companies, I think this will all work very soon. s270 00:19:37.960 --> 00:19:43.600 The big question is where how how how comfortable are we are we with something like an auto mode around privacy? s271 00:19:43.600 --> 00:19:52.280 And I think the really interesting question that I don't have an answer for, but I think whoever does this will be wealthier than I am, is if you can think of s272 00:19:52.280 --> 00:20:01.000 scenarios where you can get multiple companies to agree to their information silos having a common agent working across them. s273 00:20:01.000 --> 00:20:08.800 So, there's like a company that I won't name in in in in somewhere in the world, uh working on like finance stuff where they have a lot of investment banks. s274 00:20:08.800 --> 00:20:15.080 And actually the investment banks there's a benefit to them sharing private data about private companies for purposes of things like lending. s275 00:20:15.080 --> 00:20:17.600 And they're starting to look in this direction. s276 00:20:17.600 --> 00:20:25.360 Where they're trusting each other's agents to be able to work across what before would have been private information, with agents deciding what can be accessed or not. s277 00:20:25.360 --> 00:20:26.680 And it's it's cool. s278 00:20:26.680 --> 00:20:27.200 It's very cool. s279 00:20:27.200 --> 00:20:36.440 And I think once you find some use cases across across companies, uh I think that'll be a really good beachhead to to move more in this direction. s280 00:20:36.440 --> 00:20:38.840 So, yeah. s281 00:20:38.840 --> 00:20:44.880 You know, as a human though, I ask myself, do I trust the future where agents make all the decisions about privacy? s282 00:20:44.880 --> 00:20:45.680 I don't know about that. s283 00:20:45.680 --> 00:20:50.240 I just think it's a it is for better or worse the direction things are going. s284 00:20:50.240 --> 00:20:52.280 And I just I'm like so good on time. s285 00:20:52.280 --> 00:20:52.960 So, I'm on time. s286 00:20:52.960 --> 00:20:53.840 Thank you for coming. s287 00:20:53.840 --> 00:20:56.304 I again, uh yeah. s288 00:20:56.304 --> 00:20:56.400 [applause] s289 00:20:56.400 --> 00:20:59.280 Thanks for being here, and s290 00:21:13.989 --> 00:21:15.989 [music] s291 00:21:16.440 --> 00:21:18.440 Mhm.