Agents in Orbit: Workflows, Not Autopilot — Eduardo Simon & Javier Casal, Flywire

AI Engineer · 18 min · 181 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:00Hello everyone and welcome.
00:02My name is Edu and alongside Javi, we represent the AI platform experience team at Flywire.
00:07Our core mission is to build practical agentic infrastructure that empowers our software engineers while following our established engineering standards and practices.
00:17Today we'll share how structure workflow automation delivers real enterprise value, ensuring control, safety, and reliability across our development process.
00:28Before we dive into the deep technical details of our AI platform, let us give you some essential context about Flywire.
00:36Flywware is built to process the world most complex payments across global industries like education, healthcare, travel, and business services.
00:46Today, we serve over 5,000 clients across more than 240 countries around the world.
00:53Because we process highv value international transactions, regulatory compliance, privacy and security are not optional add-ons.
01:01They are core requirements of our business.
01:04We operate under strict compliance frameworks like PCI, DSS and so 2. This means that customer data, personal identification and payment details must remain completely safe.
01:17We cannot simply take sensitive context and pass it to third party services without strict governance and architectural boundaries.
01:25To give you an idea of our engineering organization, we have around 400 software engineers distributed globally across our teams.
01:34We maintain diverse technology stacks including Java, Ruby, Go, Python, .NET, and PHP.
01:43Almost everything we build and run is deployed on Victoria, our internal platform as a service that operates on top of GitLab and AWS infrastructure.
01:51Victoria provides standardized build pipelines and consistent deployment practices.
01:57So how did our AI journey begin?
02:00About 6 months ago, back in April, we formally established the dedicated AI platform experience team.
02:08Even before that, passionate engineers across Flywire were already experimenting with AI tools, creating skills, building MCP servers, and assembling an early code review bot.
02:20That organic activity proved that our engineers wanted AI automation.
02:26It also showed that we needed a dedicated team to provide infrastructure for safe and reliable agendic solutions.
02:33Since then, we have moved fast with close to a thousand commits to our internal tool.
02:40But the industry does not stop.
02:42In the same six months, Cloud Code alone published more than 160 releases, almost one per day, and the major AI labs release more than 20 new models.
02:54Every week brings a new model and a new tool or a new way of working.
02:59Our challenge is not only to build fast but to keep up.
03:04This bring us to Orbit, our own toil killer.
03:07Orbit is an internal platform that eliminates daily engineering repetitive tasks by including LMS directly into structural workflows.
03:17Instead of letting AI agents run freely in open-ended loops, Orbit pairs deterministic code with a small LLM steps.
03:26We crystallize repetitive procedural logic into code and we call a model only when reasoning is actually needed.
03:35When we evaluated whether to build or buy, building inhouse made complete sense for our environment.
03:42Frontier models made feature development really fast which drastically lowers the cost of a custom platform.
03:50More importantly, as a financial enterprise handling PCI, PII and PHI data, keeping execution inside our existing Victoria infrastructure ensures that sensitive data never leaves our control environment.
04:05We also skip long vendor security reviews and extra compliance work.
04:11So what does a workflow actually look like?
04:14Every agent in orbit is defined in a single JML file that lives in a repository and goes through the same merge request review as any other code.
04:26Take our code review agent as an example.
04:29It starts with a trigger, a Gilda web hook that fires when an engineer mentions the review bot on a merge request.
04:37From there, a sequence of deterministic activities, fetches the merge request metadata, classifies its risk tier, loads prior findings and clones the repository only.
04:49Then we call the model in an LLM step that can read the code but has no permission to modify it and that must answer in a strict output schema.
04:58Finally, another deterministic step post the findings back to the mer request.
05:04This is what we mean by workflows, not autopilot.
05:06Code handles the predictable parts and the model is called only where reasoning is actually needed.
05:14We choose to start with JAML files and a request review process on purpose.
05:18YAML is a format our engineers already know and the review works the way they already work every day.
05:25It also keeps the codebase AI friendly.
05:28Coding agents can read, write, and review workflow definitions as easily as people can.
05:33However, not every use case comes from an engineering team.
05:38For corporate use cases, we are considering an authoring tool in the orbit UI where people can build a workflow by dragging and dropping from a set of predefined activities
05:49without writing a jam.
05:51A workflow can start in three ways.
05:54Engineers and other system can call Orbit's internal API directly or using our internal CLI.
06:01Web hooks from GitLab, Jira, Sentry and AWS start a workflow from the moment something happens such as a new merge request or a new production error.
06:12Schedule jobs run workflows on a fixed cadence like our daily translation syncs, feature flag cleanups, and knowledgebased refreshes.
06:20Beyond these entry points, we are currently building an MCP server for Orbit so that the other AI services and tools we use across FlyWire can start workflows directly.
06:31For example, an engineer will be able to ask cloth in Slack through CL tag to run an orbit workflow without ever leaving the conversation.
06:41We started small with clear high impact use cases.
06:45First the daily engineering work automated code reviews on mer request and our Victoria upgrades agent which fixes Victoria interf findings automatically.
06:56Second translations one workflow now handles the full cycle.
07:00It fetches the new strings, translates them, updates the files and notifies the teams with the updated docs.
07:07Third production errors.
07:08A sentry web hook starts the sentry investigator.
07:12It assesses the issue, post the assessment in Slack and shares the mer request with a fix in the same place.
07:20On the support side, ticket triage classifies new tickets and routes them to the right engineering area.
07:27AI reading is a scorecards measures how ready a repository is for AI assisted development and a fixer acts on the findings.
07:37Finally, the J implement takes a ticket and turns it into a mer.
07:42It works like a small toward factory.
07:45One of the biggest benefits that we found is that we can make changes in bulk across many repositories that lets us do software engineering at scale.
07:56It is very convenient for migrations, library updates and deprecations.
08:02And we also use it to run cloud doctor and to launch our own AR readiness scorecard.
08:09Even with all this automation, the mer request is still our unit of work and it is where a human stays in the loop.
08:18We operate in a regulated environment, so people still review and approve those merge requests.
08:25Let's take a deeper look at the technical architecture of Orbit.
08:29Originally we began with a simple container setup on AWS ECS Fargate where the orchestration layer harness and the runtime environment lived inside the same container.
08:40As our needs grew, we evolved towards separating the orchestration layer, the agent runtime environment and the session state into distinct components.
08:49This improves isolation, maintainability and resource allocation.
08:54It also makes the system more resilient because an error in the effort does not kill the sandbox and it lets us scale each piece independently and adapt to changing requirements.
09:05Fargate lets us scale task elastically so we can run a very large number of agents in parallel.
09:12It was also really easy to deploy orbit because we already had Victoria our past that scales applications without any effort.
09:21But that scale needs guard rails.
09:23so the platform is not misused.
09:25First, timeouts.
09:27Every agent stops after a configurable amount of time.
09:30Second, red limits, which are aggressive by default.
09:34A broken API integration could start spawning thousands of agents and make our build much bigger.
09:40The limits keep that under control.
09:42Third, concurrency control.
09:44Our control plane uses locks to prevent duplicate executions of the same task.
09:50For example, if a code review is already running, we drop the duplicated request.
09:56Running agents in a Victoria comes with container limits.
09:59Victoria uses standard Fargate task definitions.
10:02So the containers are lightweight and tightly restricted.
10:05Our baseline agent container includes the common language runtimes.
10:09But it intentionally leaves out package managers like Bundler, Maven, and Composer.
10:14So heavy agents like our Jer implement cannot easily run native builds or unit tests inside the container.
10:22When we started orbit in April, AWS Lambda microVMs did not exist yet.
10:28AWS released them in June.
10:30So our first version run only on Fargate.
10:33These are managed microVMs that work like Lambda functions resumable processes.
10:39So they can pause and continue where they stop.
10:42that fits the nature of agentic execution really well.
10:45So we now use them with a runtime chosen by project type.
10:49Each tech stack gets its own tool chain quickly in an isolated environment.
10:54This is not full sandboxing yet and we are actively moving there.
10:58The goal is to support setups like dockering docker or multiple microvms so agents can run test or more complex verifications on any repository.
11:07Inside each LLM step, we did not build our own harness.
11:11Instead, we reuse the clot agents SDK, the same harness that powers cloud code.
11:17This gives us a mature, well- tested harness with tool usage, sub aents, and skills out of the box.
11:24So, we can focus on the workflow rather than on the plumbing.
11:28What we add on top is control.
11:30We restrict four different things for every step.
11:33the network the agent can reach, the tools it can call, the skills it can load, and the MCP servers it can use down to the specific tools allowed from each server.
11:43All of this is declared in the YAML file itself.
11:46A code review step, for example, can read and search the code, but cannot edit files or push changes.
11:51And our central investigator can query traces through our MCP server, but only with a short list of readonly tools.
11:58In practice, each LM step runs with only the axis it needs and the axis is visible and reviewable like any other code.
12:07Skills themselves come from a single place.
12:10At Flywire, we maintain a central repository of skills that is published internally as a cloud code plug-in marketplace and orbit connects to that marketplace at build time.
12:22This means the same skills works in two places.
12:26An engineer can run it in cloud code on their laptop and orbit can run it inside an automated workflow.
12:32That share foundation also shapes our recommendation to teams.
12:37Start with the skill.
12:38Skill is cheap to write, easy to iterate on, and useful from day one.
12:43Once a team is sure that a task is worth automating, we move it into orbit and turn it into a workflow.
12:50The predictable parts become deterministic code.
12:53The model is called only where it adds value, and each run becomes more reliable and less expensive than an open-ended session.
13:01Workflows also run in the background, so engineers do not have to sit with an open session and guided step by step.
13:09That means less context switching and more time for the work that needs them.
13:14Beyond runtime isolation, we also rely heavily on structure output to guarantee systemic reliability.
13:22every LM step in an orbit workflow enforces strict JSON schemas for model responses.
13:28By constraining output formats, we ensure that subsequent execution branches, error handling, and state transitions are driven by terministic control flow rather than unpredictable string parsing.
13:40If an NLM response fails validation against the expected JSON schema, Orbit automatically triggers a targeted retry or fallback pass before proceeding.
13:51Another crucial pattern we rely on is isolated component evaluation.
13:56Evaluating complex end to end agentic workflows in synthetic benchmark environments is often noisy, expensive, and difficult to reproduce consistently.
14:07Instead of trying to evaluate entire workflows from start to finish, our evaluation harness isolates individual LLM steps.
14:14By injecting exact historical state and context directly into specific workflow nodes, we can evaluate model output quality, measure PROM adjustments, and catch performance regressions quickly and cost effectively.
14:27So, how do we know that an agent is good enough?
14:30Today, a new agent usually reaches production after a b check.
14:34The team tries it on a few real cases and confirms that it behaves as expected.
14:39Once it runs in production, we trace every execution end to end with open telemetry, including each activity, model call, and tool call.
14:48We then extract traces from real executions and add them to a golden data set.
14:54For now, each golden data set is version in the repository next to the code and we run it on demand.
15:00Based on those results, we improve the prompt or change the model and we run the data set again to compare.
15:07We do not have online evaluations yet, but we are working on them internally.
15:12Our goal is to run the whole evaluation loop inside orbit from the UI or through the MCP server.
15:18First, you deploy your agent.
15:20Then, you monitor and evaluate its execution in production.
15:25Finally, you improve the agent based on those evaluations without the need to deploy it again.
15:30That is our Roma for evaluations.
15:33One place to ship, measure and improve every workflow.
15:38The operational impact of this approach has been significant across our engineering organization.
15:44In the past 30 days alone, our automated code review agent completed over 9,700 runs, averaging under three minutes per merge request.
15:53Beyond co- review, our audit fixer open 159 merge requests to fix vulnerable dependencies and the AI readiness fixer improve around 80 repositories.
16:05The Victoria llinter fixer fixed 413 projects and the merch conflict resolver run 921 times.
16:13Across these workflows, we run over 11,000 executions with a 99% success rate.
16:21Reflecting on our journey, adopting a self-service model was one of the best decisions we've made.
16:26A standardized guard rays allow engineers across SR, security, IT, and support to build custom workflow steps easily and safely.
16:35We also learned that cost control starts with model selection.
16:40Not every step needs the most capable model.
16:42So each LLM step in a workflow declares its own model.
16:46For example, a planning step and an implementation step can use different models.
16:51Matching the model to the task keeps cost under control and it is why a cost aware model router is in our road map.
16:58On the other hand, managing data separation remains an ongoing responsibility.
17:02We maintain a strict distillation between engineering SDLC data, corporate operational data and sensitive payment production data.
17:11Additionally, because demand for automation is high, maintaining focus and prioritizing high impact work is essential.
17:21Looking ahead, our road map focuses on expanding our platform features and increasing overall architecture flexibility.
17:27We are actively moving towards a complete sandboxing so agents can handle complex builds, local environments, and test suits seamlessly across all project types.
17:37Moving forward, we are also introducing resumable sessions to handle complex multi-step task.
17:42We are also building more evaluation tools directly into orbit so teams can measure the quality of their workflows on the platform itself.
17:50In parallel, we are refactoring orchestration layer to be model agnostic, enabling flexible routting across different model vendors, including an automated model routter that selects the most cost-ffective model for its task.
18:04that will help us control cost as we keep scaling the platform.
18:08We also uh want to make the platform harness agnostic so that we are not tied to a single harness.
18:16To wrap up, here are three things we learn.
18:19First, start small.
18:20Pick a repetitive well-defined task and prove it works before you scale it.
18:25Second, use the model only where it adds value.
18:28Everything predictable stays as code and that is what keeps the workflows reliable.
18:33And third, keep guard rails and humans in the loop.
18:37Owners limits and may request abuse.
18:40That is how we move from AI experiments to workflows our engineerings use every day.
18:46Thank you all so much for your time and attention today.
18:48Feel free to reach out to us in case you have further questions or comments.