WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Bz1HFxQA1o8 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Bz1HFxQA1o8.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.760 --> 00:00:20.640 Uh when I was in the 10th grade, I had a math teacher, Mr. Huff, who was uh notoriously brutal on our math tests. s3 00:00:20.640 --> 00:00:28.720 He would uh make us write our answers down to the uh fourth decimal place in case you happen to get to the answer by luck or by chance. s4 00:00:28.720 --> 00:00:37.280 And most of the credit for the question was actually from the proof that you wrote to show how you got to the answer. s5 00:00:37.280 --> 00:00:51.840 And when I first started playing AI observability software, uh I realized that most of these compliance and risk teams that I work with every day, they're kind of like Mr. Huff, but with a a multi-million s6 00:00:51.840 --> 00:00:54.080 dollar compliance budget. s7 00:00:54.080 --> 00:01:01.240 And I would let you know that playing agents AI within banks takes a long long time. s8 00:01:01.240 --> 00:01:04.160 And the reason is because that keyword production. s9 00:01:04.160 --> 00:01:14.000 I'm not talking about prototypes in dev or staging or even like internal rag agents to increase, you know, internal employee productivity. s10 00:01:14.000 --> 00:01:18.160 I'm talking about the applications that these banks millions of customers use every single day. s11 00:01:18.160 --> 00:01:22.560 The same ones you use when you log into your bank's website or mobile app. s12 00:01:22.560 --> 00:01:24.760 And those are the use cases that matter. s13 00:01:24.760 --> 00:01:35.000 In my team, we're on the front lines of deploying not only our software, but also engineering processes into the largest banks in the world. s14 00:01:35.000 --> 00:01:46.080 And I want to share some of the issues that our customers bring to us and the strategies we use to help reduce the time to production from a year down to s15 00:01:46.080 --> 00:01:48.640 a couple weeks. s16 00:01:49.160 --> 00:01:55.040 So, let's take first take a look to see how standard SaaS AI gets deployed at a bank. s17 00:01:55.040 --> 00:01:57.080 Uh spoiler alert, it doesn't work. s18 00:01:57.080 --> 00:02:04.280 There's outbound egress to the internet and sensitive information can potentially be leaked out. s19 00:02:04.280 --> 00:02:12.840 Within when you deploy on prem AI at an air-gapped bank, the flow is a little more complicated. s20 00:02:13.240 --> 00:02:15.760 I'm going to walk through three pillars today. s21 00:02:15.760 --> 00:02:18.600 Number one is the actual build pipeline. s22 00:02:18.600 --> 00:02:23.600 How our banks pull images from our software into their cluster. s23 00:02:23.680 --> 00:02:28.840 If a region fails mid-request, what is the data resiliency plan? s24 00:02:28.840 --> 00:02:40.160 And then finally, one of the most important parts is how do you ensure no sensitive information leaks out to internal bank employees and outside of the the bank's VPC. s25 00:02:42.680 --> 00:02:46.480 Compliance velocity uh and CVEs go hand in hand. s26 00:02:46.480 --> 00:02:58.640 And no platform worth their salt does not have a plan for how they address this first compliance hurdle, which is common vulnerabilities and exposures. s27 00:02:58.640 --> 00:03:00.480 I'm going to walk you through why this process is complicated. s28 00:03:00.480 --> 00:03:06.640 So, number one, you can take one CVE and you can research on the NVD, CISA, or MITRE websites. s29 00:03:06.640 --> 00:03:13.200 But, in reality, our platform, most platforms, you have multiple images and each image has multiple CVEs. s30 00:03:13.200 --> 00:03:18.000 And I'm not trying to remediate CVEs for one bank, but rather for multiple banks. s31 00:03:18.000 --> 00:03:20.640 So, we have to create a system and process around this. s32 00:03:20.640 --> 00:03:29.480 And the complication there is that our banks are on different versions of our images, each with their own unique CVEs. s33 00:03:29.480 --> 00:03:36.680 On top of that, banks utilize different scanning tools, which output a report on what CVEs are present. s34 00:03:36.680 --> 00:03:49.040 Furthermore, they have specific policies around what are the actual CVEs in the images, what the severity level is, and then what the SLA is for time to remediation. s35 00:03:49.760 --> 00:04:00.840 Our images rely on many open-source projects and those projects contain many transient dependencies which have CVEs that our engineering team cannot remediate. s36 00:04:00.840 --> 00:04:10.640 So, our team is we're constantly looking into the release notes for a lot of these open-source projects cuz we have to provide a detailed explanation back to the security teams at these banks s37 00:04:10.640 --> 00:04:17.200 on why we cannot remediate and why we're dependent on these open-source project maintainers. s38 00:04:17.200 --> 00:04:24.440 So, once you've identified the CVEs that your team your engineering team can actually remediate there's multiple ways of remediation. s39 00:04:24.440 --> 00:04:32.080 You can bump the package version up to a higher version or rebuild your image against the base Docker image. s40 00:04:32.080 --> 00:04:37.680 But, then we also have this complication of certain banks saying we want to be on version 3.8 of the package. s41 00:04:37.680 --> 00:04:39.960 But, another bank might say hold wait, hold on. s42 00:04:39.960 --> 00:04:41.520 That version is too too high. s43 00:04:41.520 --> 00:04:42.840 There's too many CVEs. s44 00:04:42.840 --> 00:04:46.160 I want to be on an older version of that package. s45 00:04:46.160 --> 00:04:50.400 So, given all of this your software still has to work and function. s46 00:04:50.400 --> 00:04:55.880 And so, one solution that you can use is creating a scale. s47 00:04:55.880 --> 00:05:05.040 So, we have a scale that does a lot of the heavy heavy lifting and reporting and the internal research as well. s48 00:05:05.040 --> 00:05:15.840 And the output is a nice kind of table format that has tells our engineering team what CVEs we have to fix and also provides explanations back to our security s49 00:05:15.840 --> 00:05:22.200 teams from the bank on what can we cannot fix due to these transient dependencies. s50 00:05:23.080 --> 00:05:33.560 Most of these scanning tools Sysdig, Wiz, Prisma they just check to see if the package has been bumped to a specific version. s51 00:05:33.560 --> 00:05:40.880 But, it says absolutely nothing about whether or not that exploit is actually still in your application code. s52 00:05:40.880 --> 00:05:43.400 And so, how do you actually test for that? s53 00:05:43.400 --> 00:05:53.240 So, one solution is creating a harbor task and this is kind of like creating a harness or a world for your for an agent to try things out. s54 00:05:53.240 --> 00:06:02.560 And so within this harbor task you have instructions for the agent, you have an environment which contains the actual docker images for your application, and then tests that you can run against the agent s55 00:06:02.560 --> 00:06:05.240 to see if they passed. s56 00:06:05.240 --> 00:06:11.520 The instruction here is a CVE for uh this service, this package. s57 00:06:11.520 --> 00:06:19.720 And the instruction I tell the agent is 2.6.2 of this package supposedly remediates the CVE. s58 00:06:19.720 --> 00:06:27.440 I want you to go and write an exploit based on the older version to see if this new version still has that exploit. s59 00:06:27.440 --> 00:06:32.560 Then write a test to see if that exploit actually passed or not. s60 00:06:32.560 --> 00:06:44.360 And then with harbor you can run a bunch of docker containers or um harbor uh modal or Daytona sandboxes, run these all in parallel, and you have a bunch of trials to see how the agent performed over various trials. s61 00:06:44.360 --> 00:06:53.000 And the What I'm trying to say here is that it's not enough just to say we bumped the package version up to a specific number. s62 00:06:53.000 --> 00:07:00.640 We have to provide the evidence to show that the exploit is no longer actually in the application code. s63 00:07:02.040 --> 00:07:08.600 Uh the second pillar, disaster recovery, is not like optional infra hygiene for a bank. s64 00:07:08.600 --> 00:07:25.080 The complication here is that the main database platform most AI observability tools use, including some of the companies that are sponsoring this conference, there's no managed service for failover across regions and data replication. s65 00:07:25.360 --> 00:07:28.080 Anyone happen to know what database platform I'm talking about? s66 00:07:28.080 --> 00:07:37.320 It isn't this So, with Redis, Postgres, there's managed failover across regions, but this one database does not have that story. s67 00:07:37.560 --> 00:07:46.720 ClickHouse is an amazing database platform, but even in their data resiliency docs they call out that there's no managed service managed failover across regions. s68 00:07:46.720 --> 00:07:53.640 So this means the banks are kind of up to themselves to to figure out that disaster recovery situation. s69 00:07:53.640 --> 00:08:05.640 So a simple pattern that you can employ is you have ClickHouse in a primary region, a job that backs up to a bucket, and then another job that restores that bucket to a DR region. s70 00:08:05.640 --> 00:08:10.240 Within that region, you might have multiple pods that have replicas of that data. s71 00:08:10.240 --> 00:08:25.040 This whole procedure relies on ClickHouse backup, which is an open-source project maintained by Altinity, who they themselves are a competitor to ClickHouse in terms of managing this data failover s72 00:08:25.040 --> 00:08:27.440 across regions. s73 00:08:29.640 --> 00:08:37.760 It's uh so with this procedure, they ClickHouse backup shares logs on whether or not the backup and restore happens, so you can get this nice table. s74 00:08:37.760 --> 00:08:45.080 But this status says nothing about your data at this point in time. s75 00:08:45.320 --> 00:08:55.480 A bad disk, a bad node, um bad migration, replication lag, these are all issues that can cause your data to become stale, that is different from when this job actually ran. s76 00:08:55.480 --> 00:09:05.400 So you can go the extra step in checking the row counts between tables, uh running a check to see if the replicas within a region have converged to each other. s77 00:09:05.400 --> 00:09:14.360 And the takeaway here is that it's about data correctness, and not necessarily just checking a box and seeing if the backup and restore job ran. s78 00:09:14.360 --> 00:09:17.960 Similar to the CVs, you can't just bump the version and assume the exploit's gone. s79 00:09:17.960 --> 00:09:20.320 You have to do these extra checks. s80 00:09:20.320 --> 00:09:33.120 And uh one call out for any of the uh architects who are creating this kind of backup and restore within your cluster is that you're also making a decision around data residency, whether you like it or not. s81 00:09:33.120 --> 00:09:38.720 When you're replicating to a bucket, the bucket doesn't care if it's in Oregon or Virginia. s82 00:09:38.720 --> 00:09:44.120 Data residency, however, cares about where those individual records are stored. s83 00:09:44.120 --> 00:09:54.920 So, one solution is you can replicate just the IDs and the metadata around your traces and your data into different regions. s84 00:09:54.920 --> 00:10:05.600 But, then the actual core trace data, that's stored within one region and doesn't leave that region because of certain jurisdiction laws. s85 00:10:07.040 --> 00:10:20.920 Uh the final pillar is most banks think that they are not leaking any sensitive information to their internal employees, their users outside of their VPC, but there may be side doors that are open. s86 00:10:20.920 --> 00:10:26.800 And this comes in the form of PII MNPI being redacted. s87 00:10:26.800 --> 00:10:28.480 And there's many different strategies for this. s88 00:10:28.480 --> 00:10:34.040 You can redact information at the SDK level, at the gateway level, using various middleware. s89 00:10:34.040 --> 00:10:42.520 And what this looks like in practice is on the UI, you might have some customer information, sensitive information is redacted. s90 00:10:42.520 --> 00:10:45.680 But, what about the judges that you're using? s91 00:10:45.680 --> 00:10:58.920 I challenge all of you to look at the system prompts for your LLM's judges and check the traces to see if those traces with the reasoning from those judges contain any PII s92 00:10:58.920 --> 00:11:01.120 or MNPI. s93 00:11:01.760 --> 00:11:03.480 Here's a system prompt for a simple judge. s94 00:11:03.480 --> 00:11:09.480 It outputs a score and explanation on how well the agent performed for a given task. s95 00:11:09.480 --> 00:11:19.800 But, there's one line here in the prompt, which is seemingly innocuous because you want the human reviewer to be able to verify if the agent did a good job and was accurate. s96 00:11:19.800 --> 00:11:29.440 This just potentially may have allowed the judge to leak and unredact PII or MNPI. s97 00:11:29.440 --> 00:11:41.520 So, your UI still looks good, but then on the back-end your LLM as LLM as judges reasoning may un-redact customer information and sensitive information. s98 00:11:41.520 --> 00:11:49.640 So, one strategy for this is you can still have your judge read the raw trace data, but the reasoning for that judge is not persisted. s99 00:11:49.640 --> 00:12:01.800 The score, however, you want that score it's for good good for evals, that always persists through the UI, but then on this other branch the reasoning, as we saw, can still leak PII. s100 00:12:01.800 --> 00:12:11.200 But in your application code, you can have a check that tells the agent this record does not contain sensitive information and therefore can persist to the UI. s101 00:12:11.200 --> 00:12:15.680 So, now you have a score plus the reasoning for the for this record. s102 00:12:15.680 --> 00:12:22.320 And then if that check comes back negative, it does have sensitive information, then you just suppress that reasoning from the UI. s103 00:12:22.320 --> 00:12:29.080 Most of the banks we work with, they want to see the LLM as judges reasoning on this sensitive information. s104 00:12:29.080 --> 00:12:42.520 And the extra upgrade you can add to this flow is you have a second LLM as a judge where the input has is redacted of sensitive information and therefore you can have confidence s105 00:12:42.520 --> 00:12:50.040 that the output from the judge is clean and then you can have that show up in the UI. s106 00:12:50.160 --> 00:12:57.400 The uh takeaway here is that having zero leakage and data sovereignty is not free. s107 00:12:57.400 --> 00:13:03.560 In this case, it's the cost of running another LLM call on the same request. s108 00:13:06.440 --> 00:13:13.480 So, one way we're able to build trust with these security and compliance teams is that we're not just saying trust us, here's the answer. s109 00:13:13.480 --> 00:13:19.280 We have to show the evidence and the work on how we arrived at the at the answer. s110 00:13:19.280 --> 00:13:30.520 And many vendors will tell you, you know, we're our deployment time to production takes a few weeks, but what they're not saying is they didn't check that time against s111 00:13:30.520 --> 00:13:37.960 your infrastructure, your security posture, and the nuances about your specific cluster. s112 00:13:37.960 --> 00:13:50.040 And so certain signals that we look for in terms of shortening that time to production is really just having a lot of processes set up in terms of, you know, how you pull images down from an from a registry, s113 00:13:50.040 --> 00:13:53.960 having GPUs sourced and ready to go for running inference. s114 00:13:53.960 --> 00:14:02.720 That timeline gets pushed out a couple months when you have formal security reviews, or you're building a custom LM gateway or RBA configuration. s115 00:14:02.720 --> 00:14:17.880 And then that timeline gets pushed even further out when you don't even have an approval process for pulling images down to your your registry, and not having a not having a story around sourcing and procuring GPUs. s116 00:14:17.880 --> 00:14:30.320 Now, I found that most of these signals that push the timeline out are less technical and more about internal human and organizational challenges and alignments at the banks. s117 00:14:30.320 --> 00:14:35.720 Uh but still important factors to figure out. s118 00:14:37.680 --> 00:14:41.680 So, the theme for today is showing your work. s119 00:14:41.680 --> 00:14:48.880 That is the real lever that you have on compliance velocity when working with a bank. s120 00:14:48.880 --> 00:14:53.600 And when our banks, when one of our customers come to us and tell us, "Hey, our customers are having problems." s121 00:14:53.600 --> 00:14:55.720 Those problems now become my team's problems. s122 00:14:55.720 --> 00:15:04.000 And we break those problems down into uh engineering, workflow, and human and organizational work streams. s123 00:15:04.000 --> 00:15:10.640 The goal is to automate as much as possible, but those problems are the ones that keep me up at night. s124 00:15:10.640 --> 00:15:19.920 And I want to ensure that our competitors don't outwork us and don't outsmart us on any of those vectors. s125 00:15:20.280 --> 00:15:28.240 And I would say that you don't want to just be another vendor, you want to actually be solving your customers' problems. s126 00:15:28.240 --> 00:15:38.800 And I would say the biggest lesson I've learned from working with these banks and security teams is that there's so much at stake. s127 00:15:38.800 --> 00:15:42.760 Checking accounts, mortgages, credit cards. s128 00:15:42.760 --> 00:15:48.640 Getting your answers down to the fourth decimal place matters in these situations. s129 00:15:48.640 --> 00:15:54.920 And there's a reason why a lot of the software within these banks is old and outdated. s130 00:15:54.920 --> 00:16:07.080 But the the line between legacy software and old banking processes and innovation is moving a little bit every day and it's because of all of you. s131 00:16:07.080 --> 00:16:14.720 So I think all of you're, you know, exploring different agent frameworks and building with, you know, new harnesses, playing with Jev. s132 00:16:14.720 --> 00:16:28.760 And I would say for the few of you who care about the the slow, the boring, the bureaucratic, the methodical, the messy human and organizational challenges within a bank, s133 00:16:28.760 --> 00:16:36.400 there's nothing like deploying AI within the financial services sector because there's so much so much at stake. s134 00:16:36.400 --> 00:16:44.560 And yeah, if you all want to score your own deployments against um these pillars, just just reach out. s135 00:16:44.560 --> 00:16:55.320 I only discussed three pillars today, but um you know, there's a whole production checklist that we use to uh that underpins these pillars. s136 00:16:56.360 --> 00:17:08.280 Going back to my 10th grade math test, one of the best parts about those tests were afterwards we me and my friends we, you know, go to the lunch table and we'd try to scribble down all the problems we remembered from the test. s137 00:17:08.280 --> 00:17:14.120 And then we use that time to talk about, "Hey, here's how we approached solving this problem and that problem." s138 00:17:14.120 --> 00:17:20.720 And it allowed me to kind of think about different approaches and different ways to solving these problems. s139 00:17:20.720 --> 00:17:29.000 And conferences like this are kind of like that uh lunch table back from when I was in 10th grade because I want to learn from all of you on how you're s140 00:17:29.000 --> 00:17:32.880 solving your your problems for for your customers. s141 00:17:32.880 --> 00:17:34.562 Thank you very much. s142 00:17:34.562 --> 00:17:36.562 [applause]