WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=xn8EsW0wxyc NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/xn8EsW0wxyc.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.480 --> 00:00:13.280 Hi everyone. s2 00:00:13.280 --> 00:00:14.880 Uh, welcome to my talk. s3 00:00:14.880 --> 00:00:26.080 I'm Joshua Mo and today we're going to be talking about some of the challenges of building a privacy respecting AI system. s4 00:00:26.240 --> 00:00:34.719 So just for an introduction as to who I am, I'm currently the lead developer relations engineer at Venice AI. s5 00:00:34.719 --> 00:00:47.520 So currently I handle partnerships, documentation, talking to developers, um some engineering and other things other things as well. s6 00:00:48.079 --> 00:00:58.320 Uh so prior to this I was actually the lead maintainer for rig which is the leading AI framework in Rust. s7 00:00:58.640 --> 00:01:02.079 Um so what is Venice? s8 00:01:02.079 --> 00:01:06.960 So Venice is a private and unrestricted AI platform. s9 00:01:06.960 --> 00:01:11.840 Uh so we host all the latest open-source models. s10 00:01:11.840 --> 00:01:15.760 We're 100% open AI compatible. s11 00:01:15.760 --> 00:01:26.240 We don't store any prompts or conversations and we also have a web app that you can use to talk to our models as well. s12 00:01:26.240 --> 00:01:41.040 We also have a studio offering that is um a offering for our content creators that includes video, audio, image generation and transcription. s13 00:01:42.479 --> 00:01:45.200 So why does privacy matter? s14 00:01:45.200 --> 00:01:48.640 So let's d deep dive into this. s15 00:01:48.640 --> 00:02:00.479 So as we know data breaches are increasingly becoming an eventuality which means that essentially reputational risk matters. s16 00:02:00.479 --> 00:02:07.759 You only need one data breach before your users stop trusting you and they move on to another provider. s17 00:02:07.759 --> 00:02:18.160 Or if you're enterprise, maybe you lose the enterprise customer who's been fueling almost 90% of your annual revenue. s18 00:02:18.239 --> 00:02:25.360 So in this case, sensitive data can severely increase the impact of any security incident. s19 00:02:25.360 --> 00:02:37.599 As well as that, storing less use uh user data directly reduces the blast radius of a security security incident that happens. s20 00:02:37.599 --> 00:02:40.319 So, how does Venice do privacy today? s21 00:02:40.319 --> 00:02:42.400 Well, there's a few ways. s22 00:02:42.400 --> 00:02:54.160 The first way is that we don't store any prompt conversations, of course, as we mentioned previously, which means that any digital footprint incurred in your side is is essentially reduced to zero. s23 00:02:54.160 --> 00:03:03.040 So, when you use models like GLM 5.2 and Kimmy 2.7, you can guarantee that none of your conversations will be saved on our side. s24 00:03:03.040 --> 00:03:05.040 It will be all stored locally. s25 00:03:05.040 --> 00:03:08.319 We also have predefined tiers of privacy. s26 00:03:08.319 --> 00:03:20.159 So private by default means that the affforementioned uh measures apply for providers like open AI and claude. s27 00:03:20.159 --> 00:03:32.080 Uh we anonymize your user footprint so that none of well they can't identify you from your simply from your prompt or your request. s28 00:03:32.080 --> 00:03:46.799 We also have models hosted in trusted execution environments, meaning that you can verifiably prove that your inference was run in a totally private process. s29 00:03:46.799 --> 00:03:52.000 As well as that, we also use a privacy first software architecture. s30 00:03:52.000 --> 00:04:11.920 So for endpoints like video uh generation where you have to pull the endpoints, we store the results in a totally encrypted payload using split key custody, meaning that attackers can't access your data and neither can the team should their accounts be compromised. s31 00:04:12.080 --> 00:04:18.560 So I just want to introduce you to the guys the uh the idea that not all systems are made equal. s32 00:04:18.560 --> 00:04:22.079 So, not all systems are made private. s33 00:04:22.079 --> 00:04:26.560 Some companies may claim their systems are private, but they're not really. s34 00:04:26.560 --> 00:04:29.759 Uh, you guys know how it goes. s35 00:04:29.759 --> 00:04:38.080 So, while models are a very important part of an aentic system, in fact, they're a vital part of the system. s36 00:04:38.080 --> 00:04:41.199 Um, they are still only one part. s37 00:04:41.199 --> 00:04:51.199 And in fact to create a complete system to accompany an LLM in doing a task you have to build an entire process around it. s38 00:04:51.199 --> 00:04:56.479 However, these systems are also much more prone to data leakage and breaches. s39 00:04:56.479 --> 00:05:02.160 So the real question is how can we optimize for this kind of problem. s40 00:05:02.560 --> 00:05:06.479 So one example I just wanted to bring up was observability logging. s41 00:05:06.479 --> 00:05:13.280 So observability as many of you know is a core part of any serious production system today. s42 00:05:13.280 --> 00:05:20.000 So just to break it down quickly, observability can be broken down into metrics, logs and traces. s43 00:05:20.000 --> 00:05:25.120 So traces are simply a span of work uh done over a unit of time. s44 00:05:25.120 --> 00:05:32.880 Metrics as are simply numbers like counters, gauges for instance and logs are the actual messages. s45 00:05:32.880 --> 00:05:40.560 So logging, as we all know, is actually pretty great for debugging, especially in production or dem dev environments. s46 00:05:40.560 --> 00:05:50.880 But you can also accidentally store prompts and secrets by default if you're not careful, which can also hold personal information. s47 00:05:50.880 --> 00:05:56.160 However, when you use Venice, you don't need to worry about any of that. s48 00:05:56.160 --> 00:06:00.560 So let's talk about how secrets can flow through a regular agent. s49 00:06:00.560 --> 00:06:04.319 So for example, let's say you're you're using open claw. s50 00:06:04.319 --> 00:06:08.240 You send a user request that gets sent to an agent runtime. s51 00:06:08.240 --> 00:06:16.960 The runtime might need a secret that it will fetch from a one password vault for instance or hashc vault. s52 00:06:16.960 --> 00:06:22.960 The secret will then get sent to the LLM reasoning layer. s53 00:06:24.080 --> 00:06:28.400 Um, oh my god. s54 00:06:31.280 --> 00:06:42.000 So the reasoning layer will then make an API request through a tool call which will then return the answer back to the user. s55 00:06:42.000 --> 00:06:52.880 However, due to this the way that it's architected, the secrets get fed directly to the LLM, which means it then may get used in training data. s56 00:06:52.960 --> 00:07:06.080 So in terms of privacy preserving patterns that work, one thing that we've used at Venice is storing hashes that rep represent ids that we need as opposed to direct data. s57 00:07:06.080 --> 00:07:15.440 So for example, if you need to identify a user for rate limiting purposes, you might not want to use their direct user footprint directly. s58 00:07:15.440 --> 00:07:24.319 You might want to, you know, store it as a hash for instance, just so you can identify them without directly using any of their personal data. s59 00:07:24.319 --> 00:07:31.440 As well as that, we encrypt data and store it in an inaccessible enclave using split key custody. s60 00:07:31.440 --> 00:07:38.400 So for context, encryption keys are stored in two halves, a read half and a write half. s61 00:07:38.400 --> 00:07:49.280 By splitting these, the attacker cannot access the data due to needing both halves for the decryption to complete. s62 00:07:49.280 --> 00:07:54.720 Additionally, you can use design patterns like handles that can be easily revoked. s63 00:07:54.720 --> 00:08:00.160 So a handle is simply an object that grants you access to something. s64 00:08:00.160 --> 00:08:05.360 So for example, let's say you have a database pool using a handle. s65 00:08:05.360 --> 00:08:09.520 So when you have the handle, you can access the database pool. s66 00:08:09.520 --> 00:08:13.199 When you don't, then you can't access it. s67 00:08:13.199 --> 00:08:26.960 This is very important especially when you're using authorization like JSON web tokens or session keys that may be uh a core part of your authorization system. s68 00:08:26.960 --> 00:08:40.080 As well as that, of course, engineering your prompts with privacy in mind can be uh very helpful for engineering teams who are using coding harnesses like cursor and clawed code s69 00:08:40.240 --> 00:08:42.800 in terms of protecting agent handle data. s70 00:08:42.800 --> 00:08:48.959 So essentially ensuring narrow agent scope to to decrease risk. s71 00:08:48.959 --> 00:08:55.279 So for example, let's say you pull tools from an NCP server that may have 15 20 different tools. s72 00:08:55.279 --> 00:08:56.880 you don't really want to use all of them. s73 00:08:56.880 --> 00:09:05.519 You actually only really need one, but you also need an API key to handle that uh API call. s74 00:09:05.519 --> 00:09:18.480 So if you pull that API key in from you know your your password vault and then you send it as part of the agent um what can happen is that it gets exfiltrated s75 00:09:18.480 --> 00:09:27.040 through through the agent uh on a tool hall to whatever you know web server is being hosted by a bad actor for instance. s76 00:09:27.040 --> 00:09:37.519 As well as that um we also have a classification layer in front of an agent to check for prompt injection for example. s77 00:09:37.519 --> 00:09:40.000 So there's a couple of ways you can do this. s78 00:09:40.000 --> 00:09:57.839 So you can either use an LLM based uh layer to do this or you can alternatively use a semantic um search layer for example using vector search uh via something like quadrant s79 00:09:57.839 --> 00:10:04.640 lance db or additionally you can use an in-memory process. s80 00:10:05.279 --> 00:10:14.399 It should also be noted that tight context loops caused by narrow agent scope can also improve context window performance. s81 00:10:14.399 --> 00:10:21.839 So for example, because you don't have as many tools, uh your agents may be able to perform much better. s82 00:10:21.839 --> 00:10:35.680 It's well known that when you fill your context window, by doing so, when you reach near the limit of your context window, performance can actually degrade uh considerably. s83 00:10:35.680 --> 00:10:46.320 By keeping an empty or near empty context window, we can make sure that the agent is performing at its best. s84 00:10:46.720 --> 00:10:51.200 So let's look at keeping secrets out of the model. s85 00:10:51.200 --> 00:10:56.000 So here we have a user request that gets sent to the reasoning layer. s86 00:10:56.000 --> 00:11:06.000 This reasoning layer is actually then uh formulating a series of non-deterministic inputs for an agentic runtime. s87 00:11:06.000 --> 00:11:18.959 So by tying the password vault to the agentic runtime instead of the reasoning layer, what happens then is that the run the agent runtime is essentially turned into an execution s88 00:11:18.959 --> 00:11:23.600 layer which then we can you know we can consider it deterministic. s89 00:11:23.600 --> 00:11:33.200 We can do all kinds of things that you normally wouldn't be able to do for privacy and security reasons at the LLN layer. s90 00:11:33.200 --> 00:11:44.399 Uh so for example, let's say you want to use a uh browser user tool and you need to log into a into a website that requires your credentials. s91 00:11:45.200 --> 00:12:06.320 With this uh software architecture, it's much more reasonable to say that it's private and secure rather than feeding your password see uh directly into the reasoning layer as well as you know calling external APIs. s92 00:12:06.320 --> 00:12:20.160 So in terms of sandboxing your agents um it it's a very well-known fact that using sandboxes and VMs for example firecracker um and you know AWS EC2 in a VPC s93 00:12:20.160 --> 00:12:26.160 can significantly upgrade the security of your agent. s94 00:12:26.160 --> 00:12:32.639 However this doesn't act as a cure all for your agentic systems. s95 00:12:32.639 --> 00:12:46.720 Uh so for example, if you have internet access enabled for your agent, it can still accidentally exfiltrate data through external API access if you're not careful and therefore monitoring permissions s96 00:12:46.720 --> 00:12:49.120 is essential. s97 00:12:49.120 --> 00:12:57.360 So my personal suggestion is building your own agentic runtime. s98 00:12:57.360 --> 00:13:07.120 So essentially building your own coding harnesses where you can add your own steps, your own observability, uh your own tools, things of that nature. s99 00:13:07.120 --> 00:13:12.800 Essentially owning the execution layer and reasoning layer. s100 00:13:13.120 --> 00:13:16.399 So let's talk about privacy at the hardware level very quickly. s101 00:13:16.399 --> 00:13:30.480 So in terms of this um there's something I like to call confidential compute where a trusted execution environment essentially uh at the hardware level enables the inference process to be run privately. s102 00:13:30.480 --> 00:13:43.040 So with Venice, how this works is that when you send a request, uh you get a response and then we actually return an attestation that you can then um s103 00:13:43.040 --> 00:13:46.800 verify with our atestation verifier. s104 00:13:47.040 --> 00:14:03.440 So in terms this what this actually means is that you can fully isolate an agent at the reasoning layer cryptographically and prove it in terms of oh confidential tool calls there is still some work to be done um tees s105 00:14:03.440 --> 00:14:16.160 are still great for tool calls however we also offer end to end encryption um but there's also you know some work to be done there because of uh the need to provide an ergonomic s106 00:14:16.160 --> 00:14:23.120 API I experience while still allowing users to return entirely encrypted tool calls. s107 00:14:23.120 --> 00:14:26.000 This is still kind of an ongoing challenge. s108 00:14:26.000 --> 00:14:35.519 Um in terms of this, a very reliable solution would be to allow providers to own the computes layer at which your tool calls being run. s109 00:14:35.519 --> 00:14:37.710 However, um s110 00:14:37.710 --> 00:14:39.710 [laughter] s111 00:14:40.079 --> 00:14:43.040 sorry, just give me a sec. s112 00:14:51.760 --> 00:14:58.399 So in terms of confidential tool calls um yeah there's definitely some work to be done. s113 00:15:02.320 --> 00:15:03.680 Um yeah that's it. s114 00:15:03.680 --> 00:15:06.160 Thank you for listening.