AI Cut Our Research Time From 7 Days to 15 Minutes — Gareth Yoder, Vanguard
https://www.youtube.com/watch?v=uA4ah-dVHvU
AI Engineer · 15 min · 132 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:01] [music]
[00:12] My name is Gareth Yoder.
[00:13] I'm the cash program technical lead here at Vanguard.
[00:17] I help oversee four to six teams in the cash program.
[00:22] This is no cash left behind, a Vanguard modernization journey.
[00:27] This journey really starts after we delivered our cash plus offering to our Vanguard brokerage account clients, and we launched that in 2022.
[00:37] And after we did the launch, we started to get the whispers that modernization is now coming down the pipeline, and that will be our next major project.
[00:47] How many here have legacy code bases?
[00:51] Right?
[00:52] That's a lot of you.
[00:52] How many of you wish that legacy code base would just modernize?
[00:57] Right?
[00:57] Many of you would say the same.
[00:59] So, looking here at the journey, we're going to go through this timeline in today's presentation, but I want to first talk to you what a cash ledger actually is.
[01:10] A cash ledger involves all money movement transactions into our Vanguard brokerage accounts from the outside world.
[01:17] It also involves cash being moved between accounts.
[01:22] And so, the payment rails, think ACH, wires, and more.
[01:26] We support the web channels, mobile channels, and batch channels.
[01:31] Our system runs in an active-active multi-region fault-tolerant ecosystem.
[01:38] And our services will receive upwards to 30 million calls per day.
[01:44] Our clients will generate a half million ledgers every day as well.
[01:52] And so, as we start to go down this modernization approach, we broke it down into three pillars: a data migration, a producer migration, and a consumer migration.
[02:03] But, because this was an enterprise priority, we had to do all three in parallel.
[02:09] So, now, as we look towards the research analysis, remember we're at the beginning of our timeline, the AI tooling wasn't there.
[02:17] And so, oftentimes we would work with mainframe engineers, and we would work with architects to try to determine what that data schema model would look like.
[02:28] And there was a turning point in this modernization journey, and that's when we had our legacy codebase being available to our AI tooling via an MCP server.
[02:40] And now we saw faster feedback loops.
[02:43] What was taking 7 days for a mainframe engineer to go into the code, do their analysis, and then kind of and come back to us for an answer, now could all be done via an AI tool prompt.
[02:57] And what we saw was that the feedback loop was reduced from 7 days down to 15 minutes.
[03:06] It was a huge win for our product owners, our technical leads, and developers alike.
[03:13] They all contribute this tool to helping our modernization succeed.
[03:20] So, let's talk about data modernization.
[03:23] We utilize the standard CDC change data capture approach that's common in industry, and we had to then scan both databases.
[03:33] A team did daily audits.
[03:36] We first made sure we had strong consistency by verifying the row level data counts, and then we ensured that we had strong integrity by checking the individual columns, making sure that the amounts matched from the legacy
[03:50] to the modern, making sure that the account information would match or the state withholding, the federal withholding.
[03:57] All of the attributes in the legacy database had to match to the to the cloud.
[04:02] A major tip here is that you don't want to over-engineer the solution because as modernization would complete, this code would then get deprecated.
[04:11] Now, let's talk about producer migration.
[04:15] Our producers called our infrastructure in three ways from the legacy state.
[04:21] One was through a service request.
[04:23] The second was an API that wrapped that service request.
[04:27] And the last and probably the most daunting was direct table inserts because we would see data appear in our database and we weren't sure how it got there or who produced that data.
[04:38] So, we took that common data schema and we put it onto a gateway that wrapped our APIs that would do our cloud data inserts.
[04:47] Now, take your modernization journey as an opportunity to reimagine your data ecosystem.
[04:53] We took concepts from FinTechs such as item potency.
[04:57] We also ensured that we had strong consistency across multiple writers.
[05:03] And so, we used utilized a similar paradigm to what a Snowflake ID would provide us.
[05:08] We are reimagined and we didn't do a lift and shift.
[05:13] Now, we have a modernized producer offering available.
[05:18] How do we get our producers?
[05:20] We had 35 producers that had to then modernize to our ecosystem.
[05:26] Early on, we had to produce manual documentation to facilitate this transition.
[05:33] And as the AI tooling became available, we provided that documentation in our company's wiki, but then exposed it via the MCP server.
[05:42] And that was really great because what it did was we used to have many teams having chats with us, and we had to try to keep all of those organized.
[05:51] Now that communication and that back and forth could all be done via the prompt on the producer side, and we saw that the number of chats and and producer support that our team was doing was greatly reduced when we had the MCP available.
[06:08] The AI tools also did a really good job of transitioning state from the legacy ecosystem to the cloud.
[06:16] We made our cloud ecosystem, the ledger enums, to be human readable versus just ledger status codes in our legacy ecosystem.
[06:27] And so, our AI tooling did a really good job at bridging the gap of the intent from the legacy ecosystem to the cloud.
[06:38] So now let's look at the consumer migration path.
[06:42] We had three ways to access consumer level data.
[06:48] We had real-time use cases, batch use cases, and streaming use cases.
[06:53] And the commonality here is that we provided a schema registry, no matter which channel that your uh consumer would access that data.
[07:04] And so, for a funds available, that's our most that's our most uh heavy traffic uh real-time API, and that goes into determining what is the balance that's displayed on your account.
[07:16] They're going to be calling us via GraphQL, and they're going to subscribe to only the attributes that meet their use cases.
[07:25] Now, for batch use cases, think of reconciliation or data analytics.
[07:30] We have our data warehouse that will expose data share views to each individual consumer.
[07:36] Each of the consumers tells us their use case, and we provide a a view that's specific to their data needs.
[07:43] This This was supported by our data governance team that strongly advocated that only the consumers that need the certain data attributes, they should only see uh those pieces of data.
[07:56] Our last piece was around streaming.
[07:58] We have a Kafka event topic that we would stream all of the inserts and updates to that database.
[08:05] Our fraud teams were able to take take advantage of this technology to provide real-time data ingestion into their fraud tooling.
[08:15] We provided at least once delivery.
[08:18] And so, our concepts of item potency that we did through the producer side, we carried forward through the event streaming.
[08:27] And so, as they subscribe to at least once delivery, we made sure that they had item potent safe uh operations.
[08:36] So, that if a batch job would would fail halfway through the process, those teams could retry, and we can ensure that we don't have uh duplicate ledgers being created.
[08:46] But, the at least once delivery from the eventing side ensured that they had all of the ledger events that were created.
[08:57] Now, when we onboarded consumers, we had 50 different consumers across our organization that needed our data.
[09:05] At Vanguard, when you onboard an application, you need to have a three-character application code to register to onboard.
[09:15] This application code describes what your application is and and what it performs.
[09:21] But, we took it a step further.
[09:23] We requested that our consumers would provide us a description and a business case justification for their need for the data.
[09:34] And so, think about if I am a trade order that needs a cash ledger to fill fulfill the cash obligation of the trade, that could be a business use case.
[09:46] Or for a retirement account that needs to do a a required mandatory distribution.
[09:51] A 529 account could do a contribution to their college tuition account.
[10:00] By grouping together all of these business descriptions, now the AI tooling can compile that together and provide an accurate description of what use cases have been modernized in our ecosystem,
[10:13] as well as we now have what is the top business functionality that we support.
[10:18] We can also take it a step further by asking what is the impact if our SLAs are breached.
[10:27] And this is critical because if you're in a major incident and if we see an app code that starts to go go below the five nines of availability, we can go in and we can see, okay,
[10:42] a batch application we may have hours to respond.
[10:46] A real-time API where there's a client on the other side that's receiving errors, we have minutes and time is of the essence to self-heal.
[10:56] We're continuing to evaluate how we can get faster response times by combining the business use case justification into our observability and and log traces.
[11:12] So what we've learned, we've learned that although we started this journey before the AI tooling was even available, we adapted and continued to adopt the newer technologies.
[11:26] This greatly aided in our research, in our development, and our testing.
[11:32] Today our cash program, 90% of our engineers utilize these AI tools on a daily basis.
[11:41] Our teams use multi-agent workflows, and we have shared knowledge bases.
[11:46] We're continuing to evolve this AI harness to meet our modernization demand.
[11:51] We have the concept of a team brain that as engineers are are building out product features, that knowledge goes into a common storage where it's now accessible by all members on the team.
[12:06] And if there's an engineer that has just joined, they can tap into this common knowledge base, and now they have the same context in their prompt as an engineer
[12:16] that has been here for the last 5 years.
[12:20] Now, when you're in a modernization process, maintaining two code bases is challenging.
[12:28] And what you'll want to do is you'll want to continue to fix forward.
[12:33] Don't fix the past.
[12:36] As you encounter bugs that have may have listed have existed for decades, continue to fix the modernized state and allow that to be the incentive for your producers and your
[12:51] consumers to modernize to the cloud.
[12:55] Now, this will throw off your audit logs, so you'll want to have white listing available, and that's really critical.
[13:01] And the last piece is ensure that you have the right metrics, alarms, and traceability, because you're going to review those often as your product continues to grow.
[13:12] Now, because this is a leadership track, I want to call out five main points.
[13:16] Number one was make your knowledge make your legacy code base AI queryable.
[13:24] That exposing that via an MCP server, our legacy code base it cut that research cycle time and gave quicker feedback loops.
[13:32] That's 7 days down to 15 minutes.
[13:34] That was a huge win.
[13:37] Treat your migration docs as part of your product.
[13:40] It needs to your migration docs are just as in critical as your business functionality because that's what all of your producers are going to utilize to onboard to your application.
[13:52] Eventually, yes, those will get deprecated out but for that interim process those migration docs are critical.
[14:01] Reimagine and don't lift and shift is is the third point.
[14:05] The consumers and producers that we had the most trouble onboarding to our modernized state were the ones that wanted to do a direct lift and shift transfer of old to new
[14:19] versus those that took a moment to step back and say, "How could we better evolve our product by taking advantage of the modern capabilities that the cash ledger ecosystem
[14:32] provides?"
[14:35] The fourth component is adopt AI incrementally.
[14:41] As the AI tools arrived, you continue to enhance your product.
[14:46] There's never going to be a perfect starting point in the world that we live in today.
[14:51] The important thing is that you get started and you continue to enhance and adapt as the technology changes.
[14:59] Last piece is observability is non-negotiable.
[15:03] You need to know who's producing your data and who's consuming your data.
[15:07] And this is ever more important as your product goes to BAU and once the modernization is complete, your teams will now have to support this as their uh daily operations.
[15:21] Thank you very much, all.
[15:24] If we have time, we'll take questions.
[15:26] Otherwise, I'll be outside of the room.
[15:27] You can connect with me later.
[15:29] And thank you again for the opportunity.
[15:32] [applause]