Deploying AI Inside a Bank Is Not an Engineering Problem — Paden Gayle, Applied AI Engineers
https://www.youtube.com/watch?v=3ZlQk4-knWM
AI Engineer · 15 min · 250 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:00] Hello and welcome.
[00:00] Today we will be talking about deploying AI inside of a bank and how it is not an engineering problem.
[00:06] Before we get into that, I want to introduce myself.
[00:08] My name is Payeyton Gail.
[00:09] I worked across many different fields.
[00:11] So I started off at consulting at Accenture.
[00:14] Before moving over to Bloomberg as a software engineer where I worked on a trade infrastructure team and a data platform team.
[00:20] Then afterwards I would be at Google working on security and deployment.
[00:23] And right now I am a founding FTE at a startup and writing the book on for deploy engineering with O'Reilly.
[00:30] In this example, the system reviews client documents and drafts exceptions for an analyst.
[00:35] The demo shows the intended workflow and everybody can see why it would be useful.
[00:40] Then security asks, okay, which environment will host it?
[00:43] The data owners ask who agreed to this use of documents and someone from risk asked who will review a wrong exception before it reaches the next system.
[00:51] Those are reasonable questions, right?
[00:53] Each question can change what we are allowed to build.
[00:56] The team has already invested a design while several people responsible for that design have never seen it.
[01:01] So in this case, the demo worked.
[01:03] The model was fine.
[01:03] However, it still did not ship.
[01:05] I want to now show you how to uncover those decisions in week one so you can design a system with a credible route into production.
[01:14] These four labels represent decisions your deployments depend on.
[01:17] For each one, I want a person who can explain the decision, the evidence they need, and the next step.
[01:22] So, let's start with data.
[01:23] Data, who owns the source, who can grant access, who can approve using this in the workflow.
[01:29] Those responsibilities may belong to different people, but being able to query a data set does not settle every question about how the application may use it.
[01:36] For the model risk, find a person who can tell you which review applies to the intended use.
[01:41] Ask what a previous submission looked like.
[01:43] Learn who will own the system after launch and what evidence that owner will need to maintain it.
[01:48] For third party risk, draw the actual dependency chain.
[01:51] Include the model provider, the host arrangements, and your role in the operating service.
[01:56] Find out which relationships have already been reviewed and which proposed changes require another decision.
[02:02] For audit, ask someone, right?
[02:06] For audit, ask what someone would need to reconstruct a consequential action.
[02:10] Which input was used?
[02:11] Which configuration produced the output?
[02:13] What did the reviewer see?
[02:15] And who approved the action?
[02:17] Those are the questions that affect the records you need to capture from from the beginning.
[02:21] Now imagine getting those answers after you built the whole entire system, a deployment boundary, a data path or action that may have to change.
[02:29] The useful move is to bring these people in and bring them into the design early.
[02:34] Their responsibilities help us identify the system we can actually operate.
[02:39] And here discovery is the engineering.
[02:42] So now we want to talk about what discovery is and who we need to really find out to understand what exactly we need to build.
[02:51] So you want to think every data set has an owner.
[02:53] The work I did at Bloomberg on the data platform team made this concrete for me.
[02:57] So building infrastructure for for an LLM meant understanding which data sets could be used and who could approve that use.
[03:04] The idea we want to get across is that every data set has an owner.
[03:07] My work at Bloomberg on the data platform team made this concrete for me.
[03:10] Building an infrastructure for an LLM meant understanding which data sets can be used and who could approve that use.
[03:17] Carry that into each bank deployment.
[03:19] Honestly, any deployment.
[03:20] So before designing retrieval, find the person accountable for each source.
[03:24] Confirm the allowed use and the environment where the processing can actually start to happen.
[03:30] If those answers change, your design changes with them.
[03:33] Imagine the key on this slide stands for a real permission with a real owner.
[03:36] We should be able to name that owner to know where exactly this key needs to come from.
[03:43] When starting to scope out an architecture, a lot of the questions you may have can be answered by looking at the or chart.
[03:50] Read the top row first.
[03:51] Documents go into the model and an analyst reviews the output.
[03:54] You can probably sketch that system in a few minutes.
[03:57] Now read the row underneath it.
[03:58] The documents have a data owner.
[04:00] The model service needs a platform owner.
[04:01] The analyst workflow needs someone who can confirm the relevant risk requirements.
[04:05] And the last name still has a question mark.
[04:08] That question mark is a dependency.
[04:10] We have designed a workflow without finding the person who can tell us whether its intended use is acceptable and what evidence will be required.
[04:18] So I would walk through this diagram with a sponsor or someone you trust and figure out, okay, who owns the documents?
[04:23] Have they heard about this use?
[04:25] Who can help us reach the appropriate reviewer?
[04:27] Then I would confirm the answers directly with those people.
[04:30] The real map will have more relationships than this.
[04:33] A technical component may depend on several approvals.
[04:36] Some decisions span the entire system.
[04:38] The point is to make those dependencies visible while the design is still easy to change.
[04:42] An unnamed owner becomes a specific task with a person responsible for resolving it.
[04:47] That is how discovery starts turning uncertainty into an implementation plan.
[04:54] And given that you may start to ask yourself, okay, Peyton, what sort of question should I be asking?
[04:58] So here are some of the questions I would bring to a discovery call.
[05:02] Where does the data live and who can approve reading it?
[05:05] Who has stopped it in a similar project before and why?
[05:09] Which existing controls could cover this workflow and who owns them?
[05:13] Where will this production run and who are the administrators for that environment?
[05:17] Who signs the relevant risk review and what have they signed before?
[05:21] What obligation is the sponsor personally worried about and who can explain it?
[05:25] Notice what I want to leave with.
[05:27] It's either names, decisions, or next conversations.
[05:30] If someone says security will be fine, ask will actually review the design and when you can meet them.
[05:35] Then close the loop.
[05:36] Send each owner a short description of the proposed use and confirm what they need from you.
[05:40] By Friday, the people whose decisions the deployment depends on should know the project exists.
[05:45] That gives scoping a much firmer starting point.
[05:48] And once all that is done and we know who owns the decisions, I can scope the work.
[05:52] When the customer cannot write or give out the spec, I rely on these three things.
[05:56] Shadow, draft, and shrink.
[06:00] For example, at Bloomberg, I worked on a system that handled cash flows.
[06:03] That work had to reconcile.
[06:04] That gave me a concrete place to start.
[06:06] So, it started with following.
[06:09] For example, at Bloomberg, I worked on a system that handled cash flows.
[06:11] The work had to reconcile and that gave me a concrete place to start.
[06:15] Follow the flow and the people responsible for it.
[06:18] I use that approach when a sponsor wants AI for onboarding.
[06:20] Sit with an analyst, watch which systems they open, where they make a judgment, and when they ask for permission, ask what happens when a document is missing.
[06:29] Write down the actual sequence.
[06:30] That is the beginning of your specification.
[06:32] Now, imagine we are building the document exception assistant.
[06:36] This is a hypothetical example, but I would still put these three lines in front of the sponsor.
[06:40] One, review the documents.
[06:42] Two, draft the exceptions.
[06:43] And three, analyst submits.
[06:46] Then I would make each line specific.
[06:48] Which documents can the analyst access?
[06:50] Is it access approved for this processing?
[06:52] What should the model produce?
[06:54] And that'll be a highlighted passage, a draft form or both.
[06:57] And who checks the draft?
[06:59] And also who submits it?
[07:00] For first version, suppose the output is a highlighted exception and a draft form.
[07:04] The analyst checks the source, edits the form and submits.
[07:08] That is the boundary someone can correct.
[07:10] I would also propose a success criteria.
[07:13] So either less review time with no increases and misconceptions.
[07:17] We still need to agree on how to measure that against the current workflow because that again right is the whole point.
[07:22] It is a proposal until we have an agreement.
[07:25] Take the draft to the sponsor and the control owners.
[07:28] They can cross out a document type.
[07:30] They can change the reviewer or they could just, you know, flat out reject the output.
[07:33] Each correction gives a decision we can build around.
[07:37] So once the workflow is complete, scope becomes a series of approval questions.
[07:42] So now let's grow our hypothetical system one step at a time.
[07:46] First, it summarizes a document that the analyst already has.
[07:49] We will still confirm whether that existing access covers its use.
[07:53] Next, it reads from a second system.
[07:56] Now, there's another data path and somebody needs to approve it.
[07:59] Next, after that, it drafts a response that a customer will see.
[08:02] Who checks the content and which will review the applies that we can use.
[08:07] Finally, it sends a response automatically.
[08:09] Who owns that action?
[08:10] What happens if it sends something wrong?
[08:12] And how do we stop it and recover it?
[08:14] Those are the four different engineering scopes.
[08:16] The applicable reviews depend on the institution and the use case.
[08:20] So I would confirm them with people who own those decisions.
[08:24] Then I ask which increment creates the useful value today.
[08:27] Could I draft inside the analyst's current workflow to solve enough of the problem?
[08:31] If so, start there.
[08:33] Confirm that the existing control really fits and build the smallest useful version or the MVP with a credible path of approval.
[08:42] That brings me to autonomy.
[08:43] I start with the least authority that the problem solved then ask what task can be proven as evidence to give the system more.
[08:49] So when working on cash flows at Bloomberg I thought about automation a lot of the time.
[08:54] So every automated step needs a control and somebody owns a number.
[08:57] I carry that discipline into AI.
[08:59] So find the consequential action name its owner decide what happens when the system is wrong.
[09:04] So for example in a hypothetical exception workflow the submission is that action.
[09:08] An analyst reviewing the draft needs the source evidence and visible uncertainty to make a real judgment.
[09:14] A review button only helps if the person can actually check the work.
[09:19] So when thinking about autonomy, I want you to read the ladder from bottom up.
[09:22] Suggest means the analyst receives a recommendation and chooses what to do with it.
[09:26] Draft means the system prepares something the analyst checks and submits.
[09:30] That is a highlighted step.
[09:32] Acts with approval means a person authorizes each action before it happens.
[09:35] Acts means the system operates within limits that have already been approved.
[09:39] Each step changes the consequences of an error.
[09:42] However, it also changes which controls must work and which reviewer needs to be convinced.
[09:47] I would start at the lowest step that creates value.
[09:49] Then collect evidence from there.
[09:51] Are the reviews overriding the drafts?
[09:53] Which failures keep recurring?
[09:54] Can people recover when something breaks?
[09:56] Use those observations to make the next authority decision.
[09:58] A model does not become more capable by itself.
[10:02] Answer how much authority this particular workflow should give it.
[10:06] So now let's make that concrete with the same hypothetical exception assistant.
[10:10] Suppose the initial request is an agent that reads client documents, decides which ones fall outside of policy, and files exceptions automatically.
[10:19] In that design, the model's judgment can become an operational action without an analyst checking each case.
[10:25] Now keep the model and change the action boundary.
[10:27] It highlights a likely exception, points to the relevant source, and prepares a draft form.
[10:32] The analyst checks the evidence, corrects the form, and chooses whether to submit.
[10:36] Follow that sequence on the slide.
[10:37] Model, human, submit.
[10:39] I'll take that design to the people we found during discovery.
[10:42] Can this use the analyst existing workflow?
[10:45] Does the approved data access cover the processing?
[10:47] Which controls apply?
[10:49] And what evidence do the reviewers need?
[10:51] Human review does not guarantee approval and it does not determine a regulatory category on its own.
[10:56] We still have to confirm the intended use and the required reviews.
[11:00] The useful question is whether this drafting version solves enough of the problems to ship first.
[11:04] If it does, we can collect evidence about errors, overrides, and recovery while operating within that boundary.
[11:10] The next decision about authority can draw on what happened in real use.
[11:14] That makes the evaluation part of the operating plan.
[11:17] And that brings us to our fourth promise.
[11:19] An evaluation the risk team can use for evidence, which really just means the ones they're responsible for managing.
[11:26] At Google, the question I needed answered was what happens when the system breaks, who notices, what stops, who takes responsibility, how long will it take to bring back up.
[11:35] For an AI deployment, I want the same concrete account, right?
[11:39] The intended use, a threat model, the failure modes, and a response plan.
[11:43] Show me the controls working when the model gets something wrong.
[11:47] Name the person who receives the alert and can stop the workflow.
[11:50] That gives a practical way to evaluate our example.
[11:54] So going back to our lovely example, the hypothetical exception assistant.
[11:58] If it invents an exception, the analyst checks the source passage and rejects the draft.
[12:04] If extraction produces unreadable text, stop and request a clean file.
[12:08] If it misses exceptions, targeted offline cases help expose that weakness.
[12:13] They do not detect every live miss.
[12:15] Keep affected work under manual review.
[12:17] For each failure, name the detection path, the fallback, and its owner.
[12:20] Put those decisions in an evidence pack.
[12:23] These five files make the review complete.
[12:25] Intended use, record users, boundaries and prohibited actions.
[12:29] Data lineage, records, each example source and use approval, failure modes, records detection and fallback, results by segment, record coverage and errors controls, record reviews, thresholds, stopping rules and owners.
[12:44] So with all of this, we generate the evidence with each evaluation run.
[12:48] alongside it preserve the input prompts model version configuration output reviewer action that lets us reconstruct the historical decision.
[12:56] Calling the model again later may produce a different answer.
[12:59] So we preserve what actually happened.
[13:01] A reviewer can then expect that supporting record.
[13:04] So now we can take a look to see like what exactly is being used to organize that result which is in the fourth file.
[13:12] For a hypothetical workflow, I would separate clean documents, scans, rare exceptions, and new document types.
[13:18] Measure the errors and control behaviors in each group, then lead with the weakest result.
[13:22] An average can hide a serious gap in one segment.
[13:25] If a group has a little evidence, say so and restrict that use until you have enough.
[13:30] This tells reviewers what the results support.
[13:33] And it also brings us back to our week one decisions.
[13:36] Now, just rounding everything up by Friday, I want five answers.
[13:41] Is the data owner engaged and aware of this project?
[13:44] Are the production environment and its administrators named?
[13:47] Can the sponsor describe the workflow without saying AI?
[13:50] Can the smallest useful scope fit existing controls confirmed with their owners?
[13:55] Do we know the risk signer and what happened to the person that signed last time?
[14:00] Use those answers as a planning heristic.
[14:02] Four or five yeses suggest a credible path to ship.
[14:04] Three means unresolved decisions need owners and dates.
[14:07] Zero to two means resolve this decisions before committing to delivery, including possibly shrinking scope or even redefining scope.
[14:14] This score does not predict approval, but it does tell us where to spend the next week.
[14:19] And here is what I would carry into these next conversations.
[14:24] So to wrap this up, what I want people to walk away with is that discovery is the engineering.
[14:29] Draft the spec, then shrink it.
[14:31] Give the system only the authority it needs.
[14:33] Build the evidence pack in sprint one.
[14:36] The demo worked.
[14:37] The model was fine.
[14:38] It still did not ship.
[14:40] So what do we do?
[14:41] We go find the people.
[14:43] Thank you for going through this presentation with me.
[14:46] My name is Peyton Gail.
[14:47] I am a founding for deployed engineer writing the book on FTE for O'Reilly.
[14:52] And yeah, love to talk about FTE and everything related to it.
[14:55] If that's of interest, you can find me on LinkedIn or my website painting.com has all of my information.
[15:02] Thank you again for watching today and yeah looking forward to