Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco

AI Engineer · 17 min · 142 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Uh when I was in the 10th grade, I had a math teacher, Mr. Huff, who was uh notoriously brutal on our math tests.
00:20He would uh make us write our answers down to the uh fourth decimal place in case you happen to get to the answer by luck or by chance.
00:28And most of the credit for the question was actually from the proof that you wrote to show how you got to the answer.
00:37And when I first started playing AI observability software, uh I realized that most of these compliance and risk teams that I work with every day, they're kind of like Mr. Huff, but with a a multi-million
00:51dollar compliance budget.
00:54And I would let you know that playing agents AI within banks takes a long long time.
01:01And the reason is because that keyword production.
01:04I'm not talking about prototypes in dev or staging or even like internal rag agents to increase, you know, internal employee productivity.
01:14I'm talking about the applications that these banks millions of customers use every single day.
01:18The same ones you use when you log into your bank's website or mobile app.
01:22And those are the use cases that matter.
01:24In my team, we're on the front lines of deploying not only our software, but also engineering processes into the largest banks in the world.
01:35And I want to share some of the issues that our customers bring to us and the strategies we use to help reduce the time to production from a year down to
01:46a couple weeks.
01:49So, let's take first take a look to see how standard SaaS AI gets deployed at a bank.
01:55Uh spoiler alert, it doesn't work.
01:57There's outbound egress to the internet and sensitive information can potentially be leaked out.
02:04Within when you deploy on prem AI at an air-gapped bank, the flow is a little more complicated.
02:13I'm going to walk through three pillars today.
02:15Number one is the actual build pipeline.
02:18How our banks pull images from our software into their cluster.
02:23If a region fails mid-request, what is the data resiliency plan?
02:28And then finally, one of the most important parts is how do you ensure no sensitive information leaks out to internal bank employees and outside of the the bank's VPC.
02:42Compliance velocity uh and CVEs go hand in hand.
02:46And no platform worth their salt does not have a plan for how they address this first compliance hurdle, which is common vulnerabilities and exposures.
02:58I'm going to walk you through why this process is complicated.
03:00So, number one, you can take one CVE and you can research on the NVD, CISA, or MITRE websites.
03:06But, in reality, our platform, most platforms, you have multiple images and each image has multiple CVEs.
03:13And I'm not trying to remediate CVEs for one bank, but rather for multiple banks.
03:18So, we have to create a system and process around this.
03:20And the complication there is that our banks are on different versions of our images, each with their own unique CVEs.
03:29On top of that, banks utilize different scanning tools, which output a report on what CVEs are present.
03:36Furthermore, they have specific policies around what are the actual CVEs in the images, what the severity level is, and then what the SLA is for time to remediation.
03:49Our images rely on many open-source projects and those projects contain many transient dependencies which have CVEs that our engineering team cannot remediate.
04:00So, our team is we're constantly looking into the release notes for a lot of these open-source projects cuz we have to provide a detailed explanation back to the security teams at these banks
04:10on why we cannot remediate and why we're dependent on these open-source project maintainers.
04:17So, once you've identified the CVEs that your team your engineering team can actually remediate there's multiple ways of remediation.
04:24You can bump the package version up to a higher version or rebuild your image against the base Docker image.
04:32But, then we also have this complication of certain banks saying we want to be on version 3.8 of the package.
04:37But, another bank might say hold wait, hold on.
04:39That version is too too high.
04:41There's too many CVEs.
04:42I want to be on an older version of that package.
04:46So, given all of this your software still has to work and function.
04:50And so, one solution that you can use is creating a scale.
04:55So, we have a scale that does a lot of the heavy heavy lifting and reporting and the internal research as well.
05:05And the output is a nice kind of table format that has tells our engineering team what CVEs we have to fix and also provides explanations back to our security
05:15teams from the bank on what can we cannot fix due to these transient dependencies.
05:23Most of these scanning tools Sysdig, Wiz, Prisma they just check to see if the package has been bumped to a specific version.
05:33But, it says absolutely nothing about whether or not that exploit is actually still in your application code.
05:40And so, how do you actually test for that?
05:43So, one solution is creating a harbor task and this is kind of like creating a harness or a world for your for an agent to try things out.
05:53And so within this harbor task you have instructions for the agent, you have an environment which contains the actual docker images for your application, and then tests that you can run against the agent
06:02to see if they passed.
06:05The instruction here is a CVE for uh this service, this package.
06:11And the instruction I tell the agent is 2.6.2 of this package supposedly remediates the CVE.
06:19I want you to go and write an exploit based on the older version to see if this new version still has that exploit.
06:27Then write a test to see if that exploit actually passed or not.
06:32And then with harbor you can run a bunch of docker containers or um harbor uh modal or Daytona sandboxes, run these all in parallel, and you have a bunch of trials to see how the agent performed over various trials.
06:44And the What I'm trying to say here is that it's not enough just to say we bumped the package version up to a specific number.
06:53We have to provide the evidence to show that the exploit is no longer actually in the application code.
07:02Uh the second pillar, disaster recovery, is not like optional infra hygiene for a bank.
07:08The complication here is that the main database platform most AI observability tools use, including some of the companies that are sponsoring this conference, there's no managed service for failover across regions and data replication.
07:25Anyone happen to know what database platform I'm talking about?
07:28It isn't this So, with Redis, Postgres, there's managed failover across regions, but this one database does not have that story.
07:37ClickHouse is an amazing database platform, but even in their data resiliency docs they call out that there's no managed service managed failover across regions.
07:46So this means the banks are kind of up to themselves to to figure out that disaster recovery situation.
07:53So a simple pattern that you can employ is you have ClickHouse in a primary region, a job that backs up to a bucket, and then another job that restores that bucket to a DR region.
08:05Within that region, you might have multiple pods that have replicas of that data.
08:10This whole procedure relies on ClickHouse backup, which is an open-source project maintained by Altinity, who they themselves are a competitor to ClickHouse in terms of managing this data failover
08:25across regions.
08:29It's uh so with this procedure, they ClickHouse backup shares logs on whether or not the backup and restore happens, so you can get this nice table.
08:37But this status says nothing about your data at this point in time.
08:45A bad disk, a bad node, um bad migration, replication lag, these are all issues that can cause your data to become stale, that is different from when this job actually ran.
08:55So you can go the extra step in checking the row counts between tables, uh running a check to see if the replicas within a region have converged to each other.
09:05And the takeaway here is that it's about data correctness, and not necessarily just checking a box and seeing if the backup and restore job ran.
09:14Similar to the CVs, you can't just bump the version and assume the exploit's gone.
09:17You have to do these extra checks.
09:20And uh one call out for any of the uh architects who are creating this kind of backup and restore within your cluster is that you're also making a decision around data residency, whether you like it or not.
09:33When you're replicating to a bucket, the bucket doesn't care if it's in Oregon or Virginia.
09:38Data residency, however, cares about where those individual records are stored.
09:44So, one solution is you can replicate just the IDs and the metadata around your traces and your data into different regions.
09:54But, then the actual core trace data, that's stored within one region and doesn't leave that region because of certain jurisdiction laws.
10:07Uh the final pillar is most banks think that they are not leaking any sensitive information to their internal employees, their users outside of their VPC, but there may be side doors that are open.
10:20And this comes in the form of PII MNPI being redacted.
10:26And there's many different strategies for this.
10:28You can redact information at the SDK level, at the gateway level, using various middleware.
10:34And what this looks like in practice is on the UI, you might have some customer information, sensitive information is redacted.
10:42But, what about the judges that you're using?
10:45I challenge all of you to look at the system prompts for your LLM's judges and check the traces to see if those traces with the reasoning from those judges contain any PII
10:58or MNPI.
11:01Here's a system prompt for a simple judge.
11:03It outputs a score and explanation on how well the agent performed for a given task.
11:09But, there's one line here in the prompt, which is seemingly innocuous because you want the human reviewer to be able to verify if the agent did a good job and was accurate.
11:19This just potentially may have allowed the judge to leak and unredact PII or MNPI.
11:29So, your UI still looks good, but then on the back-end your LLM as LLM as judges reasoning may un-redact customer information and sensitive information.
11:41So, one strategy for this is you can still have your judge read the raw trace data, but the reasoning for that judge is not persisted.
11:49The score, however, you want that score it's for good good for evals, that always persists through the UI, but then on this other branch the reasoning, as we saw, can still leak PII.
12:01But in your application code, you can have a check that tells the agent this record does not contain sensitive information and therefore can persist to the UI.
12:11So, now you have a score plus the reasoning for the for this record.
12:15And then if that check comes back negative, it does have sensitive information, then you just suppress that reasoning from the UI.
12:22Most of the banks we work with, they want to see the LLM as judges reasoning on this sensitive information.
12:29And the extra upgrade you can add to this flow is you have a second LLM as a judge where the input has is redacted of sensitive information and therefore you can have confidence
12:42that the output from the judge is clean and then you can have that show up in the UI.
12:50The uh takeaway here is that having zero leakage and data sovereignty is not free.
12:57In this case, it's the cost of running another LLM call on the same request.
13:06So, one way we're able to build trust with these security and compliance teams is that we're not just saying trust us, here's the answer.
13:13We have to show the evidence and the work on how we arrived at the at the answer.
13:19And many vendors will tell you, you know, we're our deployment time to production takes a few weeks, but what they're not saying is they didn't check that time against
13:30your infrastructure, your security posture, and the nuances about your specific cluster.
13:37And so certain signals that we look for in terms of shortening that time to production is really just having a lot of processes set up in terms of, you know, how you pull images down from an from a registry,
13:50having GPUs sourced and ready to go for running inference.
13:53That timeline gets pushed out a couple months when you have formal security reviews, or you're building a custom LM gateway or RBA configuration.
14:02And then that timeline gets pushed even further out when you don't even have an approval process for pulling images down to your your registry, and not having a not having a story around sourcing and procuring GPUs.
14:17Now, I found that most of these signals that push the timeline out are less technical and more about internal human and organizational challenges and alignments at the banks.
14:30Uh but still important factors to figure out.
14:37So, the theme for today is showing your work.
14:41That is the real lever that you have on compliance velocity when working with a bank.
14:48And when our banks, when one of our customers come to us and tell us, "Hey, our customers are having problems."
14:53Those problems now become my team's problems.
14:55And we break those problems down into uh engineering, workflow, and human and organizational work streams.
15:04The goal is to automate as much as possible, but those problems are the ones that keep me up at night.
15:10And I want to ensure that our competitors don't outwork us and don't outsmart us on any of those vectors.
15:20And I would say that you don't want to just be another vendor, you want to actually be solving your customers' problems.
15:28And I would say the biggest lesson I've learned from working with these banks and security teams is that there's so much at stake.
15:38Checking accounts, mortgages, credit cards.
15:42Getting your answers down to the fourth decimal place matters in these situations.
15:48And there's a reason why a lot of the software within these banks is old and outdated.
15:54But the the line between legacy software and old banking processes and innovation is moving a little bit every day and it's because of all of you.
16:07So I think all of you're, you know, exploring different agent frameworks and building with, you know, new harnesses, playing with Jev.
16:14And I would say for the few of you who care about the the slow, the boring, the bureaucratic, the methodical, the messy human and organizational challenges within a bank,
16:28there's nothing like deploying AI within the financial services sector because there's so much so much at stake.
16:36And yeah, if you all want to score your own deployments against um these pillars, just just reach out.
16:44I only discussed three pillars today, but um you know, there's a whole production checklist that we use to uh that underpins these pillars.
16:56Going back to my 10th grade math test, one of the best parts about those tests were afterwards we me and my friends we, you know, go to the lunch table and we'd try to scribble down all the problems we remembered from the test.
17:08And then we use that time to talk about, "Hey, here's how we approached solving this problem and that problem."
17:14And it allowed me to kind of think about different approaches and different ways to solving these problems.
17:20And conferences like this are kind of like that uh lunch table back from when I was in 10th grade because I want to learn from all of you on how you're
17:29solving your your problems for for your customers.
17:32Thank you very much.
17:34[applause]