WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=rAUyRDOFyHU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/rAUyRDOFyHU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.400 --> 00:00:02.159 Hello everyone and welcome. s2 00:00:02.159 --> 00:00:07.919 My name is Edu and alongside Javi, we represent the AI platform experience team at Flywire. s3 00:00:07.919 --> 00:00:17.680 Our core mission is to build practical agentic infrastructure that empowers our software engineers while following our established engineering standards and practices. s4 00:00:17.680 --> 00:00:28.800 Today we'll share how structure workflow automation delivers real enterprise value, ensuring control, safety, and reliability across our development process. s5 00:00:28.800 --> 00:00:36.399 Before we dive into the deep technical details of our AI platform, let us give you some essential context about Flywire. s6 00:00:36.399 --> 00:00:46.719 Flywware is built to process the world most complex payments across global industries like education, healthcare, travel, and business services. s7 00:00:46.719 --> 00:00:53.039 Today, we serve over 5,000 clients across more than 240 countries around the world. s8 00:00:53.039 --> 00:01:01.520 Because we process highv value international transactions, regulatory compliance, privacy and security are not optional add-ons. s9 00:01:01.520 --> 00:01:04.239 They are core requirements of our business. s10 00:01:04.239 --> 00:01:17.439 We operate under strict compliance frameworks like PCI, DSS and so 2. This means that customer data, personal identification and payment details must remain completely safe. s11 00:01:17.439 --> 00:01:25.200 We cannot simply take sensitive context and pass it to third party services without strict governance and architectural boundaries. s12 00:01:25.200 --> 00:01:34.479 To give you an idea of our engineering organization, we have around 400 software engineers distributed globally across our teams. s13 00:01:34.479 --> 00:01:43.040 We maintain diverse technology stacks including Java, Ruby, Go, Python, .NET, and PHP. s14 00:01:43.040 --> 00:01:51.840 Almost everything we build and run is deployed on Victoria, our internal platform as a service that operates on top of GitLab and AWS infrastructure. s15 00:01:51.840 --> 00:01:57.360 Victoria provides standardized build pipelines and consistent deployment practices. s16 00:01:57.360 --> 00:02:00.719 So how did our AI journey begin? s17 00:02:00.719 --> 00:02:08.080 About 6 months ago, back in April, we formally established the dedicated AI platform experience team. s18 00:02:08.080 --> 00:02:20.560 Even before that, passionate engineers across Flywire were already experimenting with AI tools, creating skills, building MCP servers, and assembling an early code review bot. s19 00:02:20.560 --> 00:02:26.239 That organic activity proved that our engineers wanted AI automation. s20 00:02:26.239 --> 00:02:33.840 It also showed that we needed a dedicated team to provide infrastructure for safe and reliable agendic solutions. s21 00:02:33.840 --> 00:02:40.080 Since then, we have moved fast with close to a thousand commits to our internal tool. s22 00:02:40.080 --> 00:02:42.480 But the industry does not stop. s23 00:02:42.480 --> 00:02:54.640 In the same six months, Cloud Code alone published more than 160 releases, almost one per day, and the major AI labs release more than 20 new models. s24 00:02:54.640 --> 00:02:59.920 Every week brings a new model and a new tool or a new way of working. s25 00:02:59.920 --> 00:03:04.720 Our challenge is not only to build fast but to keep up. s26 00:03:04.720 --> 00:03:07.599 This bring us to Orbit, our own toil killer. s27 00:03:07.599 --> 00:03:17.280 Orbit is an internal platform that eliminates daily engineering repetitive tasks by including LMS directly into structural workflows. s28 00:03:17.280 --> 00:03:26.239 Instead of letting AI agents run freely in open-ended loops, Orbit pairs deterministic code with a small LLM steps. s29 00:03:26.239 --> 00:03:35.599 We crystallize repetitive procedural logic into code and we call a model only when reasoning is actually needed. s30 00:03:35.599 --> 00:03:42.879 When we evaluated whether to build or buy, building inhouse made complete sense for our environment. s31 00:03:42.879 --> 00:03:50.159 Frontier models made feature development really fast which drastically lowers the cost of a custom platform. s32 00:03:50.159 --> 00:04:05.680 More importantly, as a financial enterprise handling PCI, PII and PHI data, keeping execution inside our existing Victoria infrastructure ensures that sensitive data never leaves our control environment. s33 00:04:05.680 --> 00:04:11.680 We also skip long vendor security reviews and extra compliance work. s34 00:04:11.680 --> 00:04:14.959 So what does a workflow actually look like? s35 00:04:14.959 --> 00:04:26.240 Every agent in orbit is defined in a single JML file that lives in a repository and goes through the same merge request review as any other code. s36 00:04:26.240 --> 00:04:29.120 Take our code review agent as an example. s37 00:04:29.120 --> 00:04:37.120 It starts with a trigger, a Gilda web hook that fires when an engineer mentions the review bot on a merge request. s38 00:04:37.120 --> 00:04:49.040 From there, a sequence of deterministic activities, fetches the merge request metadata, classifies its risk tier, loads prior findings and clones the repository only. s39 00:04:49.040 --> 00:04:58.960 Then we call the model in an LLM step that can read the code but has no permission to modify it and that must answer in a strict output schema. s40 00:04:58.960 --> 00:05:04.080 Finally, another deterministic step post the findings back to the mer request. s41 00:05:04.080 --> 00:05:06.960 This is what we mean by workflows, not autopilot. s42 00:05:06.960 --> 00:05:14.160 Code handles the predictable parts and the model is called only where reasoning is actually needed. s43 00:05:14.160 --> 00:05:18.960 We choose to start with JAML files and a request review process on purpose. s44 00:05:18.960 --> 00:05:25.360 YAML is a format our engineers already know and the review works the way they already work every day. s45 00:05:25.360 --> 00:05:28.000 It also keeps the codebase AI friendly. s46 00:05:28.000 --> 00:05:33.840 Coding agents can read, write, and review workflow definitions as easily as people can. s47 00:05:33.840 --> 00:05:38.479 However, not every use case comes from an engineering team. s48 00:05:38.479 --> 00:05:49.520 For corporate use cases, we are considering an authoring tool in the orbit UI where people can build a workflow by dragging and dropping from a set of predefined activities s49 00:05:49.520 --> 00:05:51.600 without writing a jam. s50 00:05:51.600 --> 00:05:54.400 A workflow can start in three ways. s51 00:05:54.400 --> 00:06:01.280 Engineers and other system can call Orbit's internal API directly or using our internal CLI. s52 00:06:01.280 --> 00:06:12.160 Web hooks from GitLab, Jira, Sentry and AWS start a workflow from the moment something happens such as a new merge request or a new production error. s53 00:06:12.160 --> 00:06:20.960 Schedule jobs run workflows on a fixed cadence like our daily translation syncs, feature flag cleanups, and knowledgebased refreshes. s54 00:06:20.960 --> 00:06:31.840 Beyond these entry points, we are currently building an MCP server for Orbit so that the other AI services and tools we use across FlyWire can start workflows directly. s55 00:06:31.840 --> 00:06:41.360 For example, an engineer will be able to ask cloth in Slack through CL tag to run an orbit workflow without ever leaving the conversation. s56 00:06:41.440 --> 00:06:45.440 We started small with clear high impact use cases. s57 00:06:45.440 --> 00:06:56.080 First the daily engineering work automated code reviews on mer request and our Victoria upgrades agent which fixes Victoria interf findings automatically. s58 00:06:56.080 --> 00:07:00.479 Second translations one workflow now handles the full cycle. s59 00:07:00.479 --> 00:07:07.280 It fetches the new strings, translates them, updates the files and notifies the teams with the updated docs. s60 00:07:07.280 --> 00:07:08.960 Third production errors. s61 00:07:08.960 --> 00:07:12.639 A sentry web hook starts the sentry investigator. s62 00:07:12.639 --> 00:07:20.479 It assesses the issue, post the assessment in Slack and shares the mer request with a fix in the same place. s63 00:07:20.479 --> 00:07:27.280 On the support side, ticket triage classifies new tickets and routes them to the right engineering area. s64 00:07:27.280 --> 00:07:37.039 AI reading is a scorecards measures how ready a repository is for AI assisted development and a fixer acts on the findings. s65 00:07:37.039 --> 00:07:42.000 Finally, the J implement takes a ticket and turns it into a mer. s66 00:07:42.000 --> 00:07:45.599 It works like a small toward factory. s67 00:07:45.599 --> 00:07:56.879 One of the biggest benefits that we found is that we can make changes in bulk across many repositories that lets us do software engineering at scale. s68 00:07:56.879 --> 00:08:02.160 It is very convenient for migrations, library updates and deprecations. s69 00:08:02.160 --> 00:08:09.039 And we also use it to run cloud doctor and to launch our own AR readiness scorecard. s70 00:08:09.039 --> 00:08:18.400 Even with all this automation, the mer request is still our unit of work and it is where a human stays in the loop. s71 00:08:18.400 --> 00:08:25.199 We operate in a regulated environment, so people still review and approve those merge requests. s72 00:08:25.199 --> 00:08:29.360 Let's take a deeper look at the technical architecture of Orbit. s73 00:08:29.360 --> 00:08:40.800 Originally we began with a simple container setup on AWS ECS Fargate where the orchestration layer harness and the runtime environment lived inside the same container. s74 00:08:40.800 --> 00:08:49.680 As our needs grew, we evolved towards separating the orchestration layer, the agent runtime environment and the session state into distinct components. s75 00:08:49.680 --> 00:08:54.240 This improves isolation, maintainability and resource allocation. s76 00:08:54.240 --> 00:09:05.920 It also makes the system more resilient because an error in the effort does not kill the sandbox and it lets us scale each piece independently and adapt to changing requirements. s77 00:09:05.920 --> 00:09:12.959 Fargate lets us scale task elastically so we can run a very large number of agents in parallel. s78 00:09:12.959 --> 00:09:21.040 It was also really easy to deploy orbit because we already had Victoria our past that scales applications without any effort. s79 00:09:21.040 --> 00:09:23.360 But that scale needs guard rails. s80 00:09:23.360 --> 00:09:25.680 so the platform is not misused. s81 00:09:25.680 --> 00:09:27.360 First, timeouts. s82 00:09:27.360 --> 00:09:30.800 Every agent stops after a configurable amount of time. s83 00:09:30.800 --> 00:09:34.399 Second, red limits, which are aggressive by default. s84 00:09:34.399 --> 00:09:40.399 A broken API integration could start spawning thousands of agents and make our build much bigger. s85 00:09:40.399 --> 00:09:42.880 The limits keep that under control. s86 00:09:42.880 --> 00:09:44.959 Third, concurrency control. s87 00:09:44.959 --> 00:09:50.320 Our control plane uses locks to prevent duplicate executions of the same task. s88 00:09:50.320 --> 00:09:56.240 For example, if a code review is already running, we drop the duplicated request. s89 00:09:56.560 --> 00:09:59.920 Running agents in a Victoria comes with container limits. s90 00:09:59.920 --> 00:10:02.640 Victoria uses standard Fargate task definitions. s91 00:10:02.640 --> 00:10:05.760 So the containers are lightweight and tightly restricted. s92 00:10:05.760 --> 00:10:09.680 Our baseline agent container includes the common language runtimes. s93 00:10:09.680 --> 00:10:14.880 But it intentionally leaves out package managers like Bundler, Maven, and Composer. s94 00:10:14.880 --> 00:10:22.079 So heavy agents like our Jer implement cannot easily run native builds or unit tests inside the container. s95 00:10:22.079 --> 00:10:28.480 When we started orbit in April, AWS Lambda microVMs did not exist yet. s96 00:10:28.480 --> 00:10:30.320 AWS released them in June. s97 00:10:30.320 --> 00:10:33.760 So our first version run only on Fargate. s98 00:10:33.760 --> 00:10:39.200 These are managed microVMs that work like Lambda functions resumable processes. s99 00:10:39.200 --> 00:10:42.399 So they can pause and continue where they stop. s100 00:10:42.399 --> 00:10:45.600 that fits the nature of agentic execution really well. s101 00:10:45.600 --> 00:10:49.519 So we now use them with a runtime chosen by project type. s102 00:10:49.519 --> 00:10:54.399 Each tech stack gets its own tool chain quickly in an isolated environment. s103 00:10:54.399 --> 00:10:58.160 This is not full sandboxing yet and we are actively moving there. s104 00:10:58.160 --> 00:11:07.519 The goal is to support setups like dockering docker or multiple microvms so agents can run test or more complex verifications on any repository. s105 00:11:07.519 --> 00:11:11.519 Inside each LLM step, we did not build our own harness. s106 00:11:11.519 --> 00:11:17.920 Instead, we reuse the clot agents SDK, the same harness that powers cloud code. s107 00:11:17.920 --> 00:11:24.560 This gives us a mature, well- tested harness with tool usage, sub aents, and skills out of the box. s108 00:11:24.560 --> 00:11:28.320 So, we can focus on the workflow rather than on the plumbing. s109 00:11:28.320 --> 00:11:30.399 What we add on top is control. s110 00:11:30.399 --> 00:11:33.839 We restrict four different things for every step. s111 00:11:33.839 --> 00:11:43.600 the network the agent can reach, the tools it can call, the skills it can load, and the MCP servers it can use down to the specific tools allowed from each server. s112 00:11:43.600 --> 00:11:46.000 All of this is declared in the YAML file itself. s113 00:11:46.000 --> 00:11:51.680 A code review step, for example, can read and search the code, but cannot edit files or push changes. s114 00:11:51.680 --> 00:11:58.720 And our central investigator can query traces through our MCP server, but only with a short list of readonly tools. s115 00:11:58.720 --> 00:12:07.680 In practice, each LM step runs with only the axis it needs and the axis is visible and reviewable like any other code. s116 00:12:07.680 --> 00:12:10.480 Skills themselves come from a single place. s117 00:12:10.480 --> 00:12:22.639 At Flywire, we maintain a central repository of skills that is published internally as a cloud code plug-in marketplace and orbit connects to that marketplace at build time. s118 00:12:22.639 --> 00:12:26.079 This means the same skills works in two places. s119 00:12:26.079 --> 00:12:32.959 An engineer can run it in cloud code on their laptop and orbit can run it inside an automated workflow. s120 00:12:32.959 --> 00:12:37.120 That share foundation also shapes our recommendation to teams. s121 00:12:37.120 --> 00:12:38.560 Start with the skill. s122 00:12:38.560 --> 00:12:43.440 Skill is cheap to write, easy to iterate on, and useful from day one. s123 00:12:43.440 --> 00:12:50.480 Once a team is sure that a task is worth automating, we move it into orbit and turn it into a workflow. s124 00:12:50.480 --> 00:12:53.600 The predictable parts become deterministic code. s125 00:12:53.600 --> 00:13:01.839 The model is called only where it adds value, and each run becomes more reliable and less expensive than an open-ended session. s126 00:13:01.839 --> 00:13:09.040 Workflows also run in the background, so engineers do not have to sit with an open session and guided step by step. s127 00:13:09.040 --> 00:13:14.639 That means less context switching and more time for the work that needs them. s128 00:13:14.639 --> 00:13:22.000 Beyond runtime isolation, we also rely heavily on structure output to guarantee systemic reliability. s129 00:13:22.000 --> 00:13:28.160 every LM step in an orbit workflow enforces strict JSON schemas for model responses. s130 00:13:28.160 --> 00:13:40.560 By constraining output formats, we ensure that subsequent execution branches, error handling, and state transitions are driven by terministic control flow rather than unpredictable string parsing. s131 00:13:40.560 --> 00:13:51.760 If an NLM response fails validation against the expected JSON schema, Orbit automatically triggers a targeted retry or fallback pass before proceeding. s132 00:13:51.760 --> 00:13:56.480 Another crucial pattern we rely on is isolated component evaluation. s133 00:13:56.480 --> 00:14:07.040 Evaluating complex end to end agentic workflows in synthetic benchmark environments is often noisy, expensive, and difficult to reproduce consistently. s134 00:14:07.040 --> 00:14:14.639 Instead of trying to evaluate entire workflows from start to finish, our evaluation harness isolates individual LLM steps. s135 00:14:14.639 --> 00:14:27.600 By injecting exact historical state and context directly into specific workflow nodes, we can evaluate model output quality, measure PROM adjustments, and catch performance regressions quickly and cost effectively. s136 00:14:27.600 --> 00:14:30.399 So, how do we know that an agent is good enough? s137 00:14:30.399 --> 00:14:34.079 Today, a new agent usually reaches production after a b check. s138 00:14:34.079 --> 00:14:39.279 The team tries it on a few real cases and confirms that it behaves as expected. s139 00:14:39.279 --> 00:14:48.720 Once it runs in production, we trace every execution end to end with open telemetry, including each activity, model call, and tool call. s140 00:14:48.720 --> 00:14:54.000 We then extract traces from real executions and add them to a golden data set. s141 00:14:54.000 --> 00:15:00.959 For now, each golden data set is version in the repository next to the code and we run it on demand. s142 00:15:00.959 --> 00:15:07.440 Based on those results, we improve the prompt or change the model and we run the data set again to compare. s143 00:15:07.440 --> 00:15:12.000 We do not have online evaluations yet, but we are working on them internally. s144 00:15:12.000 --> 00:15:18.720 Our goal is to run the whole evaluation loop inside orbit from the UI or through the MCP server. s145 00:15:18.720 --> 00:15:20.880 First, you deploy your agent. s146 00:15:20.880 --> 00:15:25.040 Then, you monitor and evaluate its execution in production. s147 00:15:25.040 --> 00:15:30.880 Finally, you improve the agent based on those evaluations without the need to deploy it again. s148 00:15:30.880 --> 00:15:33.120 That is our Roma for evaluations. s149 00:15:33.120 --> 00:15:38.240 One place to ship, measure and improve every workflow. s150 00:15:38.480 --> 00:15:44.000 The operational impact of this approach has been significant across our engineering organization. s151 00:15:44.000 --> 00:15:53.519 In the past 30 days alone, our automated code review agent completed over 9,700 runs, averaging under three minutes per merge request. s152 00:15:53.519 --> 00:16:05.120 Beyond co- review, our audit fixer open 159 merge requests to fix vulnerable dependencies and the AI readiness fixer improve around 80 repositories. s153 00:16:05.120 --> 00:16:13.360 The Victoria llinter fixer fixed 413 projects and the merch conflict resolver run 921 times. s154 00:16:13.360 --> 00:16:21.120 Across these workflows, we run over 11,000 executions with a 99% success rate. s155 00:16:21.120 --> 00:16:26.720 Reflecting on our journey, adopting a self-service model was one of the best decisions we've made. s156 00:16:26.720 --> 00:16:35.839 A standardized guard rays allow engineers across SR, security, IT, and support to build custom workflow steps easily and safely. s157 00:16:35.839 --> 00:16:40.160 We also learned that cost control starts with model selection. s158 00:16:40.160 --> 00:16:42.800 Not every step needs the most capable model. s159 00:16:42.800 --> 00:16:46.959 So each LLM step in a workflow declares its own model. s160 00:16:46.959 --> 00:16:51.519 For example, a planning step and an implementation step can use different models. s161 00:16:51.519 --> 00:16:58.000 Matching the model to the task keeps cost under control and it is why a cost aware model router is in our road map. s162 00:16:58.000 --> 00:17:02.880 On the other hand, managing data separation remains an ongoing responsibility. s163 00:17:02.880 --> 00:17:11.839 We maintain a strict distillation between engineering SDLC data, corporate operational data and sensitive payment production data. s164 00:17:11.839 --> 00:17:20.720 Additionally, because demand for automation is high, maintaining focus and prioritizing high impact work is essential. s165 00:17:21.280 --> 00:17:27.600 Looking ahead, our road map focuses on expanding our platform features and increasing overall architecture flexibility. s166 00:17:27.600 --> 00:17:37.360 We are actively moving towards a complete sandboxing so agents can handle complex builds, local environments, and test suits seamlessly across all project types. s167 00:17:37.360 --> 00:17:42.960 Moving forward, we are also introducing resumable sessions to handle complex multi-step task. s168 00:17:42.960 --> 00:17:50.720 We are also building more evaluation tools directly into orbit so teams can measure the quality of their workflows on the platform itself. s169 00:17:50.720 --> 00:18:04.240 In parallel, we are refactoring orchestration layer to be model agnostic, enabling flexible routting across different model vendors, including an automated model routter that selects the most cost-ffective model for its task. s170 00:18:04.240 --> 00:18:08.720 that will help us control cost as we keep scaling the platform. s171 00:18:08.720 --> 00:18:16.240 We also uh want to make the platform harness agnostic so that we are not tied to a single harness. s172 00:18:16.400 --> 00:18:19.200 To wrap up, here are three things we learn. s173 00:18:19.200 --> 00:18:20.640 First, start small. s174 00:18:20.640 --> 00:18:25.200 Pick a repetitive well-defined task and prove it works before you scale it. s175 00:18:25.200 --> 00:18:28.720 Second, use the model only where it adds value. s176 00:18:28.720 --> 00:18:33.520 Everything predictable stays as code and that is what keeps the workflows reliable. s177 00:18:33.520 --> 00:18:37.039 And third, keep guard rails and humans in the loop. s178 00:18:37.039 --> 00:18:40.000 Owners limits and may request abuse. s179 00:18:40.000 --> 00:18:46.320 That is how we move from AI experiments to workflows our engineerings use every day. s180 00:18:46.320 --> 00:18:48.720 Thank you all so much for your time and attention today. s181 00:18:48.720 --> 00:18:54.080 Feel free to reach out to us in case you have further questions or comments.