Using RL Agent to Detect and Remediate ETL Pipeline Failures - Anna Marie Benzon
https://www.youtube.com/watch?v=LrGCT7G_rU8
AI Engineer · 14 min · 136 sentences · sentence-level transcript from YouTube's caption track; each line begins at the time shown

[00:00] Imagine you're this engineer.
[00:03] A production data job failed.
[00:05] You went to the dashboard and spent all day checking the logs, the schema, and the upstream data.
[00:12] And now it's passed.
[00:14] The same question chips.
[00:17] Coming back, what changed?
[00:19] The failure itself may be small, but the expensive part is everything around it: inspection, diagnosis, choosing a safe, running the job, and confirming that we didn't make the data error.
[00:35] Hi, I'm Anamari Bazhan.
[00:37] In this talk, I'll show an ETL-guided system that selects bounded remittance actions for ETL failures.
[00:45] The central question is simply whether an agent can act, but whether it can act usefully, explainably, and within the boundaries that an operation would actually trust.
[00:59] Cloud failures.
[00:59] Early arrival errors.
[01:00] We're able sources, daytime in compatibility.
[01:08] Now reach spikes, type changes, and run-time errors that don't do much anything in the run book.
[01:15] Usual response is a human work flow: Inspect the logs, form, diagnose a temporary job, and validate the output.
[01:24] Each step is reasonable.
[01:26] The latency comes from the head in complete contact and the need to avoid and unsafe fix in the caption.
[01:35] Evaluate the manual recovery baseline was modeled at roughly 2.5 working days.
[01:41] This represents an incident moving through normal queueing, investigation, and approval.
[01:47] So the engineering objective is specific criteria that look for routine recognizable failures while escalating the cases that are uncertain, novel, and high risk.
[01:59] This diagram shows the end-to-end AWS architecture from my cupstone.
[02:03] An existing AWS Glue initial job emits a job fail event.
[02:09] Amazon Event Bridge catches that event and triggers the Lamda function that runs the agent.
[02:15] Lamda gathers evidence from two read-only sources.
[02:19] CloudWatch provides the error log.
[02:22] Glue data catalog provides the current schema.
[02:25] Metadata: The system uses these signals to classify the failure.
[02:29] Assess the data quality and operational risk and construct the state.
[02:34] Passed to the RL decision engine, the policy then proposes an appropriate response.
[02:39] The safety layer takes that proposal before the executor can use the Glue API to retrigger the job and apply and approve remediation.
[02:50] Amazon stores its artifacts and audits the log.
[02:54] Quarantine Outputs: Finally, the job is run and validated.
[03:00] So this is closed.
[03:02] Operational look: Monitor, diagnose, score, decide, check, safety, act, and verify.
[03:11] Recovery: The caption implementation uses synthetic data provided by the client.
[03:17] The public repository uses this pattern through sanitize.
[03:20] Generalize deployment.
[03:22] The intelligence layer deliberately separates three concerns: the terminal anomaly rules.
[03:29] Establish observable facts.
[03:30] Fields disappear after a type change and a data rate exceeds thresholds.
[03:35] The learning policy handles context action selection.
[03:38] Given the current incident state, the system should retry the schema.
[03:43] Roll back the quarantine exit and simply log the event.
[03:48] Then a safety override sets outside the learned policy.
[03:52] For example, if the anomaly is critical, the policy proposes a passive action such as logging.
[03:58] The override converts that choice into an escalation.
[04:00] This separation is the result of the project rules for facts.
[04:06] Learning for bounded choices and guards for authority before selecting an action The system has established what actually happened The schema profiler extracts structure types, statistical and knowledge statistics The drift detector compares the current profiler with the baseline and identifies additions,
[04:27] removals, and type changes The data quality analyzer measures completeness, validity, and consistency The error classifier measures log parts in failure families, and the risk scorer measures signals in an operational risk level
[04:45] These components are deterministic by using directly observable data conditions An explicit rule is easier to validate than an explicit rule with a richer and more representative incident history Some classifiers could become learned components, but ML ready is not the same as ML requires the simplest reliable component Should on each decision The policy receives the compact state Failure
[05:13] category Risk level Countability and data quality conditions These six actions Retry covers Roll back Quarantine Escalate and L I use RQ learning Because the state and action spaces are small,
[05:30] the Q tables are small and every decision can be inspected directly for this state, this action values, and this action one Technically, each incident is modeled as a single- step contextual decision Implemented
[05:44] with learning Rather than as a long horizontal control task that formulates the system to choose one safe operational response from an unbounded action Set the value of the learn policy There is no sophistication for its own sake It structures
[06:02] way to learn action preferences from outcomes While retaining a decision service that an engineer can specify The learn policy does not have final authority It proposes an action The safety layer The proposal against the Normally, the system's operational constraints are passed.
[06:25] Passive actions are overlooked for critical conditions and higher.
[06:29] Unknown cases are escalated.
[06:32] Every proposal is overlooked.
[06:34] The execution result and validation outcome is written to an audit record.
[06:39] Notice that escalation is included in the action space that is not the agent giving up.
[06:46] It is the system correctly recognizing the boundary of its evidence and authority for an operational agent.
[06:53] The ability to say I should not do this automatically is the capability if success is measured only by non- escalation.
[07:02] The optimization target is wrong.
[07:04] There is one failure path.
[07:06] The agent receives the Glue Style job failure event.
[07:09] The Love classifier.
[07:10] The text is a daytime format incompatibility with 0.9 confidence base on the state.
[07:18] The policy prop schema is correct.
[07:21] The safety overhead does not fire because this is not classified as a critical event, but the execution Then discover that automatic coverage is not available for this specific case.
[07:34] The system does not pretend that the fix should happen.
[07:39] It proposes an action.
[07:41] Reports that execution was unavailable and identifies the incident for manual review.
[07:46] This example shows to distinguish controls, policy, safety, and implementation capability.
[07:54] An action can be safe in principle and still be unavailable in the current environment.
[08:00] A ROWAS agent must represent both conditions explicitly to make the work independently reviewable without exploding the client context.
[08:10] I build the public benchmark around the generalized AWS Lambda-style architecture.
[08:17] The CUPS implementation uses client- provided synthetic data.
[08:21] The public repository uses newly generalized synthetic schemas.
[08:25] Records logs and incident scenarios.
[08:28] It contains no client documents.
[08:31] Infrastructure identifiers and business-specific values ​​ are used for control, experiment groups, and Repeated the robustness evaluation across 36 from 42 to 71.
[08:43] The reported aggregates include 95% confidence intervals.
[08:49] This process uses the system design and experimental logic in a firm that other engineers can inspect and run while maintaining the confidence.
[08:59] Confidentiality boundary on the control benchmark: the rule base abnormally.
[09:04] The detector achieved a precision of one recall of 0.8 and an NF score of 0.889.
[09:09] This means that the detector was conservative.
[09:14] Normally, it flagged errors correctly in this benchmark, but it still missed some positive cases for operations that required distinction.
[09:23] Perfect precision does not mean perfect detection for cases where the RL-guided work flow resolved the incident successfully.
[09:31] The mean resolution time was about 5.24 minutes across the 30 runs.
[09:37] The simulated success rate was 74.63% ( plus or minus 1.51 percentage points).
[09:43] The non-escalation rate was 88.63% ( plus or minus 0.89 points).
[09:50] The chart compares that minute scale result with the model's manual baseline of 2 and 1/2 working hours.
[09:57] That is 260 seconds.
[10:01] In the benchmark, that is approximately a 99.85% reduction in MTTR.
[10:08] This figure qualifies performance against the control benchmark.
[10:12] In that scope, they show that the architecture can automate the first path for non-failure conditions.
[10:18] Production validation is the next evaluation.
[10:20] The boundary resolution results are, in my view, the most useful part of the project.
[10:26] The RL policy matches the equivalent deterministic policy.
[10:30] The difference of 0.19 percentage points (with a 0.19 point confidence interval) on this compact state space.
[10:38] The learned policy can maintain the Same Success Level as the Hand Define Policy by Contrast the Terministic Action Section vs Random Selection by 15.63 points and Enabling the Safety of Ride Reduce Nuisance by about 15.03 points.
[10:56] That decrease is intentional.
[10:58] The garden system escalates more often when its autonomy would be inappropriate.
[11:05] So why did the reliability come from primarily from the initial state?
[11:11] Sensible decision logic and external safety constraints, not from RL alone.
[11:15] That is a useful engineering result in the current benchmark.
[11:21] RL provides an inspectable learned decision service rather than an immediate success rate advantage.
[11:28] Its value becomes more significant as the incident history becomes richer.
[11:33] Action outcomes by manually maintaining preference becomes more difficult.
[11:37] This slide defines the current validation boundary.
[11:44] The results come from synthetic scenarios.
[11:46] The agent responds after a failure signal.
[11:49] It does not predict a failure before it happens.
[11:52] Real incident diversity meets the current state space.
[11:57] Some remediation actions are simulated and deliberately bounded.
[12:01] An online learning in a production environment.
[12:04] Req approval gets version policy rollback support and continues monitoring.
[12:10] The result is a credible feasibility demonstration of the system.
[12:14] This is the clear path over production validation.
[12:19] The next step is a shadow mode deployment based on representations in incident traces.
[12:24] Where recommendations can be made, decisions can be made before the agent traces.
[12:29] Execution authority.
[12:29] There are five takeaways: I would leave an engineering team.
[12:35] First, use the terministic logic for facts that can be measured directly.
[12:40] Second, use learning only.
[12:42] Contextual action selection is of real value.
[12:46] Third, place safety constraints outside the learned policy, so a policy update cannot silently redefine its own authority.
[12:53] Fourth, trade escalation and post-action validation as first-class outcomes.
[13:00] Not exception parts and fifth, effect across repeated steps and compare against a simple baseline.
[13:06] A single favorable run is a demo, not evidence.
[13:09] A practical self-healing system does not need the largest possible model.
[13:14] It needs a clear, state-bound, action-reproducible, evaluation observable.
[13:20] Decisions and the discipline stop when uncertainty exists; its authority this brings the suspect to the engineer in the opening video; the goal is not to eliminate human judgment; it is to stop spending that judgment and the same recognizable failure at two in the morning before the response;
[13:40] manual log inspection schema; tracking delay dashboards and recovery process measured in working days after the retained path becomes event trigger diagnosis and RL guided but safety constraint action; explicit validation and recovery measured in minutes; when the case is supported by teams and high- risk failures are still going to the humans, that is
[14:06] the point; human attention is to serve for incident contact; trade-off and authority are genuinely required; the code, synthetic benchmarks, experiments, scripts, and representativeness instructions are available in the GI Hub; record story on screen if you work on agent reliability, date, quality, and protection; incident automation I would Specially call your feedback on state representation
[14:35] diz and safety boundary.
[14:36] Thank you for watching.