Deploying AI Inside a Bank Is Not an Engineering Problem — Paden Gayle, Applied AI Engineers

AI Engineer · 15 min · 250 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:00Hello and welcome.
2. 00:00Today we will be talking about deploying AI inside of a bank and how it is not an engineering problem.
3. 00:06Before we get into that, I want to introduce myself.
4. 00:08My name is Payeyton Gail.
5. 00:09I worked across many different fields.
6. 00:11So I started off at consulting at Accenture.
7. 00:14Before moving over to Bloomberg as a software engineer where I worked on a trade infrastructure team and a data platform team.
8. 00:20Then afterwards I would be at Google working on security and deployment.
9. 00:23And right now I am a founding FTE at a startup and writing the book on for deploy engineering with O'Reilly.
10. 00:30In this example, the system reviews client documents and drafts exceptions for an analyst.
11. 00:35The demo shows the intended workflow and everybody can see why it would be useful.
12. 00:40Then security asks, okay, which environment will host it?
13. 00:43The data owners ask who agreed to this use of documents and someone from risk asked who will review a wrong exception before it reaches the next system.
14. 00:51Those are reasonable questions, right?
15. 00:53Each question can change what we are allowed to build.
16. 00:56The team has already invested a design while several people responsible for that design have never seen it.
17. 01:01So in this case, the demo worked.
18. 01:03The model was fine.
19. 01:03However, it still did not ship.
20. 01:05I want to now show you how to uncover those decisions in week one so you can design a system with a credible route into production.
21. 01:14These four labels represent decisions your deployments depend on.
22. 01:17For each one, I want a person who can explain the decision, the evidence they need, and the next step.
23. 01:22So, let's start with data.
24. 01:23Data, who owns the source, who can grant access, who can approve using this in the workflow.
25. 01:29Those responsibilities may belong to different people, but being able to query a data set does not settle every question about how the application may use it.
26. 01:36For the model risk, find a person who can tell you which review applies to the intended use.
27. 01:41Ask what a previous submission looked like.
28. 01:43Learn who will own the system after launch and what evidence that owner will need to maintain it.
29. 01:48For third party risk, draw the actual dependency chain.
30. 01:51Include the model provider, the host arrangements, and your role in the operating service.
31. 01:56Find out which relationships have already been reviewed and which proposed changes require another decision.
32. 02:02For audit, ask someone, right?
33. 02:06For audit, ask what someone would need to reconstruct a consequential action.
34. 02:10Which input was used?
35. 02:11Which configuration produced the output?
36. 02:13What did the reviewer see?
37. 02:15And who approved the action?
38. 02:17Those are the questions that affect the records you need to capture from from the beginning.
39. 02:21Now imagine getting those answers after you built the whole entire system, a deployment boundary, a data path or action that may have to change.
40. 02:29The useful move is to bring these people in and bring them into the design early.
41. 02:34Their responsibilities help us identify the system we can actually operate.
42. 02:39And here discovery is the engineering.
43. 02:42So now we want to talk about what discovery is and who we need to really find out to understand what exactly we need to build.
44. 02:51So you want to think every data set has an owner.
45. 02:53The work I did at Bloomberg on the data platform team made this concrete for me.
46. 02:57So building infrastructure for for an LLM meant understanding which data sets could be used and who could approve that use.
47. 03:04The idea we want to get across is that every data set has an owner.
48. 03:07My work at Bloomberg on the data platform team made this concrete for me.
49. 03:10Building an infrastructure for an LLM meant understanding which data sets can be used and who could approve that use.
50. 03:17Carry that into each bank deployment.
51. 03:19Honestly, any deployment.
52. 03:20So before designing retrieval, find the person accountable for each source.
53. 03:24Confirm the allowed use and the environment where the processing can actually start to happen.
54. 03:30If those answers change, your design changes with them.
55. 03:33Imagine the key on this slide stands for a real permission with a real owner.
56. 03:36We should be able to name that owner to know where exactly this key needs to come from.
57. 03:43When starting to scope out an architecture, a lot of the questions you may have can be answered by looking at the or chart.
58. 03:50Read the top row first.
59. 03:51Documents go into the model and an analyst reviews the output.
60. 03:54You can probably sketch that system in a few minutes.
61. 03:57Now read the row underneath it.
62. 03:58The documents have a data owner.
63. 04:00The model service needs a platform owner.
64. 04:01The analyst workflow needs someone who can confirm the relevant risk requirements.
65. 04:05And the last name still has a question mark.
66. 04:08That question mark is a dependency.
67. 04:10We have designed a workflow without finding the person who can tell us whether its intended use is acceptable and what evidence will be required.
68. 04:18So I would walk through this diagram with a sponsor or someone you trust and figure out, okay, who owns the documents?
69. 04:23Have they heard about this use?
70. 04:25Who can help us reach the appropriate reviewer?
71. 04:27Then I would confirm the answers directly with those people.
72. 04:30The real map will have more relationships than this.
73. 04:33A technical component may depend on several approvals.
74. 04:36Some decisions span the entire system.
75. 04:38The point is to make those dependencies visible while the design is still easy to change.
76. 04:42An unnamed owner becomes a specific task with a person responsible for resolving it.
77. 04:47That is how discovery starts turning uncertainty into an implementation plan.
78. 04:54And given that you may start to ask yourself, okay, Peyton, what sort of question should I be asking?
79. 04:58So here are some of the questions I would bring to a discovery call.
80. 05:02Where does the data live and who can approve reading it?
81. 05:05Who has stopped it in a similar project before and why?
82. 05:09Which existing controls could cover this workflow and who owns them?
83. 05:13Where will this production run and who are the administrators for that environment?
84. 05:17Who signs the relevant risk review and what have they signed before?
85. 05:21What obligation is the sponsor personally worried about and who can explain it?
86. 05:25Notice what I want to leave with.
87. 05:27It's either names, decisions, or next conversations.
88. 05:30If someone says security will be fine, ask will actually review the design and when you can meet them.
89. 05:35Then close the loop.
90. 05:36Send each owner a short description of the proposed use and confirm what they need from you.
91. 05:40By Friday, the people whose decisions the deployment depends on should know the project exists.
92. 05:45That gives scoping a much firmer starting point.
93. 05:48And once all that is done and we know who owns the decisions, I can scope the work.
94. 05:52When the customer cannot write or give out the spec, I rely on these three things.
95. 05:56Shadow, draft, and shrink.
96. 06:00For example, at Bloomberg, I worked on a system that handled cash flows.
97. 06:03That work had to reconcile.
98. 06:04That gave me a concrete place to start.
99. 06:06So, it started with following.
100. 06:09For example, at Bloomberg, I worked on a system that handled cash flows.
101. 06:11The work had to reconcile and that gave me a concrete place to start.
102. 06:15Follow the flow and the people responsible for it.
103. 06:18I use that approach when a sponsor wants AI for onboarding.
104. 06:20Sit with an analyst, watch which systems they open, where they make a judgment, and when they ask for permission, ask what happens when a document is missing.
105. 06:29Write down the actual sequence.
106. 06:30That is the beginning of your specification.
107. 06:32Now, imagine we are building the document exception assistant.
108. 06:36This is a hypothetical example, but I would still put these three lines in front of the sponsor.
109. 06:40One, review the documents.
110. 06:42Two, draft the exceptions.
111. 06:43And three, analyst submits.
112. 06:46Then I would make each line specific.
113. 06:48Which documents can the analyst access?
114. 06:50Is it access approved for this processing?
115. 06:52What should the model produce?
116. 06:54And that'll be a highlighted passage, a draft form or both.
117. 06:57And who checks the draft?
118. 06:59And also who submits it?
119. 07:00For first version, suppose the output is a highlighted exception and a draft form.
120. 07:04The analyst checks the source, edits the form and submits.
121. 07:08That is the boundary someone can correct.
122. 07:10I would also propose a success criteria.
123. 07:13So either less review time with no increases and misconceptions.
124. 07:17We still need to agree on how to measure that against the current workflow because that again right is the whole point.
125. 07:22It is a proposal until we have an agreement.
126. 07:25Take the draft to the sponsor and the control owners.
127. 07:28They can cross out a document type.
128. 07:30They can change the reviewer or they could just, you know, flat out reject the output.
129. 07:33Each correction gives a decision we can build around.
130. 07:37So once the workflow is complete, scope becomes a series of approval questions.
131. 07:42So now let's grow our hypothetical system one step at a time.
132. 07:46First, it summarizes a document that the analyst already has.
133. 07:49We will still confirm whether that existing access covers its use.
134. 07:53Next, it reads from a second system.
135. 07:56Now, there's another data path and somebody needs to approve it.
136. 07:59Next, after that, it drafts a response that a customer will see.
137. 08:02Who checks the content and which will review the applies that we can use.
138. 08:07Finally, it sends a response automatically.
139. 08:09Who owns that action?
140. 08:10What happens if it sends something wrong?
141. 08:12And how do we stop it and recover it?
142. 08:14Those are the four different engineering scopes.
143. 08:16The applicable reviews depend on the institution and the use case.
144. 08:20So I would confirm them with people who own those decisions.
145. 08:24Then I ask which increment creates the useful value today.
146. 08:27Could I draft inside the analyst's current workflow to solve enough of the problem?
147. 08:31If so, start there.
148. 08:33Confirm that the existing control really fits and build the smallest useful version or the MVP with a credible path of approval.
149. 08:42That brings me to autonomy.
150. 08:43I start with the least authority that the problem solved then ask what task can be proven as evidence to give the system more.
151. 08:49So when working on cash flows at Bloomberg I thought about automation a lot of the time.
152. 08:54So every automated step needs a control and somebody owns a number.
153. 08:57I carry that discipline into AI.
154. 08:59So find the consequential action name its owner decide what happens when the system is wrong.
155. 09:04So for example in a hypothetical exception workflow the submission is that action.
156. 09:08An analyst reviewing the draft needs the source evidence and visible uncertainty to make a real judgment.
157. 09:14A review button only helps if the person can actually check the work.
158. 09:19So when thinking about autonomy, I want you to read the ladder from bottom up.
159. 09:22Suggest means the analyst receives a recommendation and chooses what to do with it.
160. 09:26Draft means the system prepares something the analyst checks and submits.
161. 09:30That is a highlighted step.
162. 09:32Acts with approval means a person authorizes each action before it happens.
163. 09:35Acts means the system operates within limits that have already been approved.
164. 09:39Each step changes the consequences of an error.
165. 09:42However, it also changes which controls must work and which reviewer needs to be convinced.
166. 09:47I would start at the lowest step that creates value.
167. 09:49Then collect evidence from there.
168. 09:51Are the reviews overriding the drafts?
169. 09:53Which failures keep recurring?
170. 09:54Can people recover when something breaks?
171. 09:56Use those observations to make the next authority decision.
172. 09:58A model does not become more capable by itself.
173. 10:02Answer how much authority this particular workflow should give it.
174. 10:06So now let's make that concrete with the same hypothetical exception assistant.
175. 10:10Suppose the initial request is an agent that reads client documents, decides which ones fall outside of policy, and files exceptions automatically.
176. 10:19In that design, the model's judgment can become an operational action without an analyst checking each case.
177. 10:25Now keep the model and change the action boundary.
178. 10:27It highlights a likely exception, points to the relevant source, and prepares a draft form.
179. 10:32The analyst checks the evidence, corrects the form, and chooses whether to submit.
180. 10:36Follow that sequence on the slide.
181. 10:37Model, human, submit.
182. 10:39I'll take that design to the people we found during discovery.
183. 10:42Can this use the analyst existing workflow?
184. 10:45Does the approved data access cover the processing?
185. 10:47Which controls apply?
186. 10:49And what evidence do the reviewers need?
187. 10:51Human review does not guarantee approval and it does not determine a regulatory category on its own.
188. 10:56We still have to confirm the intended use and the required reviews.
189. 11:00The useful question is whether this drafting version solves enough of the problems to ship first.
190. 11:04If it does, we can collect evidence about errors, overrides, and recovery while operating within that boundary.
191. 11:10The next decision about authority can draw on what happened in real use.
192. 11:14That makes the evaluation part of the operating plan.
193. 11:17And that brings us to our fourth promise.
194. 11:19An evaluation the risk team can use for evidence, which really just means the ones they're responsible for managing.
195. 11:26At Google, the question I needed answered was what happens when the system breaks, who notices, what stops, who takes responsibility, how long will it take to bring back up.
196. 11:35For an AI deployment, I want the same concrete account, right?
197. 11:39The intended use, a threat model, the failure modes, and a response plan.
198. 11:43Show me the controls working when the model gets something wrong.
199. 11:47Name the person who receives the alert and can stop the workflow.
200. 11:50That gives a practical way to evaluate our example.
201. 11:54So going back to our lovely example, the hypothetical exception assistant.
202. 11:58If it invents an exception, the analyst checks the source passage and rejects the draft.
203. 12:04If extraction produces unreadable text, stop and request a clean file.
204. 12:08If it misses exceptions, targeted offline cases help expose that weakness.
205. 12:13They do not detect every live miss.
206. 12:15Keep affected work under manual review.
207. 12:17For each failure, name the detection path, the fallback, and its owner.
208. 12:20Put those decisions in an evidence pack.
209. 12:23These five files make the review complete.
210. 12:25Intended use, record users, boundaries and prohibited actions.
211. 12:29Data lineage, records, each example source and use approval, failure modes, records detection and fallback, results by segment, record coverage and errors controls, record reviews, thresholds, stopping rules and owners.
212. 12:44So with all of this, we generate the evidence with each evaluation run.
213. 12:48alongside it preserve the input prompts model version configuration output reviewer action that lets us reconstruct the historical decision.
214. 12:56Calling the model again later may produce a different answer.
215. 12:59So we preserve what actually happened.
216. 13:01A reviewer can then expect that supporting record.
217. 13:04So now we can take a look to see like what exactly is being used to organize that result which is in the fourth file.
218. 13:12For a hypothetical workflow, I would separate clean documents, scans, rare exceptions, and new document types.
219. 13:18Measure the errors and control behaviors in each group, then lead with the weakest result.
220. 13:22An average can hide a serious gap in one segment.
221. 13:25If a group has a little evidence, say so and restrict that use until you have enough.
222. 13:30This tells reviewers what the results support.
223. 13:33And it also brings us back to our week one decisions.
224. 13:36Now, just rounding everything up by Friday, I want five answers.
225. 13:41Is the data owner engaged and aware of this project?
226. 13:44Are the production environment and its administrators named?
227. 13:47Can the sponsor describe the workflow without saying AI?
228. 13:50Can the smallest useful scope fit existing controls confirmed with their owners?
229. 13:55Do we know the risk signer and what happened to the person that signed last time?
230. 14:00Use those answers as a planning heristic.
231. 14:02Four or five yeses suggest a credible path to ship.
232. 14:04Three means unresolved decisions need owners and dates.
233. 14:07Zero to two means resolve this decisions before committing to delivery, including possibly shrinking scope or even redefining scope.
234. 14:14This score does not predict approval, but it does tell us where to spend the next week.
235. 14:19And here is what I would carry into these next conversations.
236. 14:24So to wrap this up, what I want people to walk away with is that discovery is the engineering.
237. 14:29Draft the spec, then shrink it.
238. 14:31Give the system only the authority it needs.
239. 14:33Build the evidence pack in sprint one.
240. 14:36The demo worked.
241. 14:37The model was fine.
242. 14:38It still did not ship.
243. 14:40So what do we do?
244. 14:41We go find the people.
245. 14:43Thank you for going through this presentation with me.
246. 14:46My name is Peyton Gail.
247. 14:47I am a founding for deployed engineer writing the book on FTE for O'Reilly.
248. 14:52And yeah, love to talk about FTE and everything related to it.
249. 14:55If that's of interest, you can find me on LinkedIn or my website painting.com has all of my information.
250. 15:02Thank you again for watching today and yeah looking forward to