AI Cut Our Research Time From 7 Days to 15 Minutes — Gareth Yoder, Vanguard

AI Engineer · 15 min · 132 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

  1. 00:01[music]
  2. 00:12My name is Gareth Yoder.
  3. 00:13I'm the cash program technical lead here at Vanguard.
  4. 00:17I help oversee four to six teams in the cash program.
  5. 00:22This is no cash left behind, a Vanguard modernization journey.
  6. 00:27This journey really starts after we delivered our cash plus offering to our Vanguard brokerage account clients, and we launched that in 2022.
  7. 00:37And after we did the launch, we started to get the whispers that modernization is now coming down the pipeline, and that will be our next major project.
  8. 00:47How many here have legacy code bases?
  9. 00:51Right?
  10. 00:52That's a lot of you.
  11. 00:52How many of you wish that legacy code base would just modernize?
  12. 00:57Right?
  13. 00:57Many of you would say the same.
  14. 00:59So, looking here at the journey, we're going to go through this timeline in today's presentation, but I want to first talk to you what a cash ledger actually is.
  15. 01:10A cash ledger involves all money movement transactions into our Vanguard brokerage accounts from the outside world.
  16. 01:17It also involves cash being moved between accounts.
  17. 01:22And so, the payment rails, think ACH, wires, and more.
  18. 01:26We support the web channels, mobile channels, and batch channels.
  19. 01:31Our system runs in an active-active multi-region fault-tolerant ecosystem.
  20. 01:38And our services will receive upwards to 30 million calls per day.
  21. 01:44Our clients will generate a half million ledgers every day as well.
  22. 01:52And so, as we start to go down this modernization approach, we broke it down into three pillars: a data migration, a producer migration, and a consumer migration.
  23. 02:03But, because this was an enterprise priority, we had to do all three in parallel.
  24. 02:09So, now, as we look towards the research analysis, remember we're at the beginning of our timeline, the AI tooling wasn't there.
  25. 02:17And so, oftentimes we would work with mainframe engineers, and we would work with architects to try to determine what that data schema model would look like.
  26. 02:28And there was a turning point in this modernization journey, and that's when we had our legacy codebase being available to our AI tooling via an MCP server.
  27. 02:40And now we saw faster feedback loops.
  28. 02:43What was taking 7 days for a mainframe engineer to go into the code, do their analysis, and then kind of and come back to us for an answer, now could all be done via an AI tool prompt.
  29. 02:57And what we saw was that the feedback loop was reduced from 7 days down to 15 minutes.
  30. 03:06It was a huge win for our product owners, our technical leads, and developers alike.
  31. 03:13They all contribute this tool to helping our modernization succeed.
  32. 03:20So, let's talk about data modernization.
  33. 03:23We utilize the standard CDC change data capture approach that's common in industry, and we had to then scan both databases.
  34. 03:33A team did daily audits.
  35. 03:36We first made sure we had strong consistency by verifying the row level data counts, and then we ensured that we had strong integrity by checking the individual columns, making sure that the amounts matched from the legacy
  36. 03:50to the modern, making sure that the account information would match or the state withholding, the federal withholding.
  37. 03:57All of the attributes in the legacy database had to match to the to the cloud.
  38. 04:02A major tip here is that you don't want to over-engineer the solution because as modernization would complete, this code would then get deprecated.
  39. 04:11Now, let's talk about producer migration.
  40. 04:15Our producers called our infrastructure in three ways from the legacy state.
  41. 04:21One was through a service request.
  42. 04:23The second was an API that wrapped that service request.
  43. 04:27And the last and probably the most daunting was direct table inserts because we would see data appear in our database and we weren't sure how it got there or who produced that data.
  44. 04:38So, we took that common data schema and we put it onto a gateway that wrapped our APIs that would do our cloud data inserts.
  45. 04:47Now, take your modernization journey as an opportunity to reimagine your data ecosystem.
  46. 04:53We took concepts from FinTechs such as item potency.
  47. 04:57We also ensured that we had strong consistency across multiple writers.
  48. 05:03And so, we used utilized a similar paradigm to what a Snowflake ID would provide us.
  49. 05:08We are reimagined and we didn't do a lift and shift.
  50. 05:13Now, we have a modernized producer offering available.
  51. 05:18How do we get our producers?
  52. 05:20We had 35 producers that had to then modernize to our ecosystem.
  53. 05:26Early on, we had to produce manual documentation to facilitate this transition.
  54. 05:33And as the AI tooling became available, we provided that documentation in our company's wiki, but then exposed it via the MCP server.
  55. 05:42And that was really great because what it did was we used to have many teams having chats with us, and we had to try to keep all of those organized.
  56. 05:51Now that communication and that back and forth could all be done via the prompt on the producer side, and we saw that the number of chats and and producer support that our team was doing was greatly reduced when we had the MCP available.
  57. 06:08The AI tools also did a really good job of transitioning state from the legacy ecosystem to the cloud.
  58. 06:16We made our cloud ecosystem, the ledger enums, to be human readable versus just ledger status codes in our legacy ecosystem.
  59. 06:27And so, our AI tooling did a really good job at bridging the gap of the intent from the legacy ecosystem to the cloud.
  60. 06:38So now let's look at the consumer migration path.
  61. 06:42We had three ways to access consumer level data.
  62. 06:48We had real-time use cases, batch use cases, and streaming use cases.
  63. 06:53And the commonality here is that we provided a schema registry, no matter which channel that your uh consumer would access that data.
  64. 07:04And so, for a funds available, that's our most that's our most uh heavy traffic uh real-time API, and that goes into determining what is the balance that's displayed on your account.
  65. 07:16They're going to be calling us via GraphQL, and they're going to subscribe to only the attributes that meet their use cases.
  66. 07:25Now, for batch use cases, think of reconciliation or data analytics.
  67. 07:30We have our data warehouse that will expose data share views to each individual consumer.
  68. 07:36Each of the consumers tells us their use case, and we provide a a view that's specific to their data needs.
  69. 07:43This This was supported by our data governance team that strongly advocated that only the consumers that need the certain data attributes, they should only see uh those pieces of data.
  70. 07:56Our last piece was around streaming.
  71. 07:58We have a Kafka event topic that we would stream all of the inserts and updates to that database.
  72. 08:05Our fraud teams were able to take take advantage of this technology to provide real-time data ingestion into their fraud tooling.
  73. 08:15We provided at least once delivery.
  74. 08:18And so, our concepts of item potency that we did through the producer side, we carried forward through the event streaming.
  75. 08:27And so, as they subscribe to at least once delivery, we made sure that they had item potent safe uh operations.
  76. 08:36So, that if a batch job would would fail halfway through the process, those teams could retry, and we can ensure that we don't have uh duplicate ledgers being created.
  77. 08:46But, the at least once delivery from the eventing side ensured that they had all of the ledger events that were created.
  78. 08:57Now, when we onboarded consumers, we had 50 different consumers across our organization that needed our data.
  79. 09:05At Vanguard, when you onboard an application, you need to have a three-character application code to register to onboard.
  80. 09:15This application code describes what your application is and and what it performs.
  81. 09:21But, we took it a step further.
  82. 09:23We requested that our consumers would provide us a description and a business case justification for their need for the data.
  83. 09:34And so, think about if I am a trade order that needs a cash ledger to fill fulfill the cash obligation of the trade, that could be a business use case.
  84. 09:46Or for a retirement account that needs to do a a required mandatory distribution.
  85. 09:51A 529 account could do a contribution to their college tuition account.
  86. 10:00By grouping together all of these business descriptions, now the AI tooling can compile that together and provide an accurate description of what use cases have been modernized in our ecosystem,
  87. 10:13as well as we now have what is the top business functionality that we support.
  88. 10:18We can also take it a step further by asking what is the impact if our SLAs are breached.
  89. 10:27And this is critical because if you're in a major incident and if we see an app code that starts to go go below the five nines of availability, we can go in and we can see, okay,
  90. 10:42a batch application we may have hours to respond.
  91. 10:46A real-time API where there's a client on the other side that's receiving errors, we have minutes and time is of the essence to self-heal.
  92. 10:56We're continuing to evaluate how we can get faster response times by combining the business use case justification into our observability and and log traces.
  93. 11:12So what we've learned, we've learned that although we started this journey before the AI tooling was even available, we adapted and continued to adopt the newer technologies.
  94. 11:26This greatly aided in our research, in our development, and our testing.
  95. 11:32Today our cash program, 90% of our engineers utilize these AI tools on a daily basis.
  96. 11:41Our teams use multi-agent workflows, and we have shared knowledge bases.
  97. 11:46We're continuing to evolve this AI harness to meet our modernization demand.
  98. 11:51We have the concept of a team brain that as engineers are are building out product features, that knowledge goes into a common storage where it's now accessible by all members on the team.
  99. 12:06And if there's an engineer that has just joined, they can tap into this common knowledge base, and now they have the same context in their prompt as an engineer
  100. 12:16that has been here for the last 5 years.
  101. 12:20Now, when you're in a modernization process, maintaining two code bases is challenging.
  102. 12:28And what you'll want to do is you'll want to continue to fix forward.
  103. 12:33Don't fix the past.
  104. 12:36As you encounter bugs that have may have listed have existed for decades, continue to fix the modernized state and allow that to be the incentive for your producers and your
  105. 12:51consumers to modernize to the cloud.
  106. 12:55Now, this will throw off your audit logs, so you'll want to have white listing available, and that's really critical.
  107. 13:01And the last piece is ensure that you have the right metrics, alarms, and traceability, because you're going to review those often as your product continues to grow.
  108. 13:12Now, because this is a leadership track, I want to call out five main points.
  109. 13:16Number one was make your knowledge make your legacy code base AI queryable.
  110. 13:24That exposing that via an MCP server, our legacy code base it cut that research cycle time and gave quicker feedback loops.
  111. 13:32That's 7 days down to 15 minutes.
  112. 13:34That was a huge win.
  113. 13:37Treat your migration docs as part of your product.
  114. 13:40It needs to your migration docs are just as in critical as your business functionality because that's what all of your producers are going to utilize to onboard to your application.
  115. 13:52Eventually, yes, those will get deprecated out but for that interim process those migration docs are critical.
  116. 14:01Reimagine and don't lift and shift is is the third point.
  117. 14:05The consumers and producers that we had the most trouble onboarding to our modernized state were the ones that wanted to do a direct lift and shift transfer of old to new
  118. 14:19versus those that took a moment to step back and say, "How could we better evolve our product by taking advantage of the modern capabilities that the cash ledger ecosystem
  119. 14:32provides?"
  120. 14:35The fourth component is adopt AI incrementally.
  121. 14:41As the AI tools arrived, you continue to enhance your product.
  122. 14:46There's never going to be a perfect starting point in the world that we live in today.
  123. 14:51The important thing is that you get started and you continue to enhance and adapt as the technology changes.
  124. 14:59Last piece is observability is non-negotiable.
  125. 15:03You need to know who's producing your data and who's consuming your data.
  126. 15:07And this is ever more important as your product goes to BAU and once the modernization is complete, your teams will now have to support this as their uh daily operations.
  127. 15:21Thank you very much, all.
  128. 15:24If we have time, we'll take questions.
  129. 15:26Otherwise, I'll be outside of the room.
  130. 15:27You can connect with me later.
  131. 15:29And thank you again for the opportunity.
  132. 15:32[applause]