Using RL Agent to Detect and Remediate ETL Pipeline Failures - Anna Marie Benzon

AI Engineer · 14 min · 136 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

1. 00:00Imagine you're this engineer.
2. 00:03A production data job failed.
3. 00:05You went to the dashboard and spent all day checking the logs, the schema, and the upstream data.
4. 00:12And now it's passed.
5. 00:14The same question chips.
6. 00:17Coming back, what changed?
7. 00:19The failure itself may be small, but the expensive part is everything around it: inspection, diagnosis, choosing a safe, running the job, and confirming that we didn't make the data error.
8. 00:35Hi, I'm Anamari Bazhan.
9. 00:37In this talk, I'll show an ETL-guided system that selects bounded remittance actions for ETL failures.
10. 00:45The central question is simply whether an agent can act, but whether it can act usefully, explainably, and within the boundaries that an operation would actually trust.
11. 00:59Cloud failures.
12. 00:59Early arrival errors.
13. 01:00We're able sources, daytime in compatibility.
14. 01:08Now reach spikes, type changes, and run-time errors that don't do much anything in the run book.
15. 01:15Usual response is a human work flow: Inspect the logs, form, diagnose a temporary job, and validate the output.
16. 01:24Each step is reasonable.
17. 01:26The latency comes from the head in complete contact and the need to avoid and unsafe fix in the caption.
18. 01:35Evaluate the manual recovery baseline was modeled at roughly 2.5 working days.
19. 01:41This represents an incident moving through normal queueing, investigation, and approval.
20. 01:47So the engineering objective is specific criteria that look for routine recognizable failures while escalating the cases that are uncertain, novel, and high risk.
21. 01:59This diagram shows the end-to-end AWS architecture from my cupstone.
22. 02:03An existing AWS Glue initial job emits a job fail event.
23. 02:09Amazon Event Bridge catches that event and triggers the Lamda function that runs the agent.
24. 02:15Lamda gathers evidence from two read-only sources.
25. 02:19CloudWatch provides the error log.
26. 02:22Glue data catalog provides the current schema.
27. 02:25Metadata: The system uses these signals to classify the failure.
28. 02:29Assess the data quality and operational risk and construct the state.
29. 02:34Passed to the RL decision engine, the policy then proposes an appropriate response.
30. 02:39The safety layer takes that proposal before the executor can use the Glue API to retrigger the job and apply and approve remediation.
31. 02:50Amazon stores its artifacts and audits the log.
32. 02:54Quarantine Outputs: Finally, the job is run and validated.
33. 03:00So this is closed.
34. 03:02Operational look: Monitor, diagnose, score, decide, check, safety, act, and verify.
35. 03:11Recovery: The caption implementation uses synthetic data provided by the client.
36. 03:17The public repository uses this pattern through sanitize.
37. 03:20Generalize deployment.
38. 03:22The intelligence layer deliberately separates three concerns: the terminal anomaly rules.
39. 03:29Establish observable facts.
40. 03:30Fields disappear after a type change and a data rate exceeds thresholds.
41. 03:35The learning policy handles context action selection.
42. 03:38Given the current incident state, the system should retry the schema.
43. 03:43Roll back the quarantine exit and simply log the event.
44. 03:48Then a safety override sets outside the learned policy.
45. 03:52For example, if the anomaly is critical, the policy proposes a passive action such as logging.
46. 03:58The override converts that choice into an escalation.
47. 04:00This separation is the result of the project rules for facts.
48. 04:06Learning for bounded choices and guards for authority before selecting an action The system has established what actually happened The schema profiler extracts structure types, statistical and knowledge statistics The drift detector compares the current profiler with the baseline and identifies additions,
49. 04:27removals, and type changes The data quality analyzer measures completeness, validity, and consistency The error classifier measures log parts in failure families, and the risk scorer measures signals in an operational risk level
50. 04:45These components are deterministic by using directly observable data conditions An explicit rule is easier to validate than an explicit rule with a richer and more representative incident history Some classifiers could become learned components, but ML ready is not the same as ML requires the simplest reliable component Should on each decision The policy receives the compact state Failure
51. 05:13category Risk level Countability and data quality conditions These six actions Retry covers Roll back Quarantine Escalate and L I use RQ learning Because the state and action spaces are small,
52. 05:30the Q tables are small and every decision can be inspected directly for this state, this action values, and this action one Technically, each incident is modeled as a single- step contextual decision Implemented
53. 05:44with learning Rather than as a long horizontal control task that formulates the system to choose one safe operational response from an unbounded action Set the value of the learn policy There is no sophistication for its own sake It structures
54. 06:02way to learn action preferences from outcomes While retaining a decision service that an engineer can specify The learn policy does not have final authority It proposes an action The safety layer The proposal against the Normally, the system's operational constraints are passed.
55. 06:25Passive actions are overlooked for critical conditions and higher.
56. 06:29Unknown cases are escalated.
57. 06:32Every proposal is overlooked.
58. 06:34The execution result and validation outcome is written to an audit record.
59. 06:39Notice that escalation is included in the action space that is not the agent giving up.
60. 06:46It is the system correctly recognizing the boundary of its evidence and authority for an operational agent.
61. 06:53The ability to say I should not do this automatically is the capability if success is measured only by non- escalation.
62. 07:02The optimization target is wrong.
63. 07:04There is one failure path.
64. 07:06The agent receives the Glue Style job failure event.
65. 07:09The Love classifier.
66. 07:10The text is a daytime format incompatibility with 0.9 confidence base on the state.
67. 07:18The policy prop schema is correct.
68. 07:21The safety overhead does not fire because this is not classified as a critical event, but the execution Then discover that automatic coverage is not available for this specific case.
69. 07:34The system does not pretend that the fix should happen.
70. 07:39It proposes an action.
71. 07:41Reports that execution was unavailable and identifies the incident for manual review.
72. 07:46This example shows to distinguish controls, policy, safety, and implementation capability.
73. 07:54An action can be safe in principle and still be unavailable in the current environment.
74. 08:00A ROWAS agent must represent both conditions explicitly to make the work independently reviewable without exploding the client context.
75. 08:10I build the public benchmark around the generalized AWS Lambda-style architecture.
76. 08:17The CUPS implementation uses client- provided synthetic data.
77. 08:21The public repository uses newly generalized synthetic schemas.
78. 08:25Records logs and incident scenarios.
79. 08:28It contains no client documents.
80. 08:31Infrastructure identifiers and business-specific values ​​ are used for control, experiment groups, and Repeated the robustness evaluation across 36 from 42 to 71.
81. 08:43The reported aggregates include 95% confidence intervals.
82. 08:49This process uses the system design and experimental logic in a firm that other engineers can inspect and run while maintaining the confidence.
83. 08:59Confidentiality boundary on the control benchmark: the rule base abnormally.
84. 09:04The detector achieved a precision of one recall of 0.8 and an NF score of 0.889.
85. 09:09This means that the detector was conservative.
86. 09:14Normally, it flagged errors correctly in this benchmark, but it still missed some positive cases for operations that required distinction.
87. 09:23Perfect precision does not mean perfect detection for cases where the RL-guided work flow resolved the incident successfully.
88. 09:31The mean resolution time was about 5.24 minutes across the 30 runs.
89. 09:37The simulated success rate was 74.63% ( plus or minus 1.51 percentage points).
90. 09:43The non-escalation rate was 88.63% ( plus or minus 0.89 points).
91. 09:50The chart compares that minute scale result with the model's manual baseline of 2 and 1/2 working hours.
92. 09:57That is 260 seconds.
93. 10:01In the benchmark, that is approximately a 99.85% reduction in MTTR.
94. 10:08This figure qualifies performance against the control benchmark.
95. 10:12In that scope, they show that the architecture can automate the first path for non-failure conditions.
96. 10:18Production validation is the next evaluation.
97. 10:20The boundary resolution results are, in my view, the most useful part of the project.
98. 10:26The RL policy matches the equivalent deterministic policy.
99. 10:30The difference of 0.19 percentage points (with a 0.19 point confidence interval) on this compact state space.
100. 10:38The learned policy can maintain the Same Success Level as the Hand Define Policy by Contrast the Terministic Action Section vs Random Selection by 15.63 points and Enabling the Safety of Ride Reduce Nuisance by about 15.03 points.
101. 10:56That decrease is intentional.
102. 10:58The garden system escalates more often when its autonomy would be inappropriate.
103. 11:05So why did the reliability come from primarily from the initial state?
104. 11:11Sensible decision logic and external safety constraints, not from RL alone.
105. 11:15That is a useful engineering result in the current benchmark.
106. 11:21RL provides an inspectable learned decision service rather than an immediate success rate advantage.
107. 11:28Its value becomes more significant as the incident history becomes richer.
108. 11:33Action outcomes by manually maintaining preference becomes more difficult.
109. 11:37This slide defines the current validation boundary.
110. 11:44The results come from synthetic scenarios.
111. 11:46The agent responds after a failure signal.
112. 11:49It does not predict a failure before it happens.
113. 11:52Real incident diversity meets the current state space.
114. 11:57Some remediation actions are simulated and deliberately bounded.
115. 12:01An online learning in a production environment.
116. 12:04Req approval gets version policy rollback support and continues monitoring.
117. 12:10The result is a credible feasibility demonstration of the system.
118. 12:14This is the clear path over production validation.
119. 12:19The next step is a shadow mode deployment based on representations in incident traces.
120. 12:24Where recommendations can be made, decisions can be made before the agent traces.
121. 12:29Execution authority.
122. 12:29There are five takeaways: I would leave an engineering team.
123. 12:35First, use the terministic logic for facts that can be measured directly.
124. 12:40Second, use learning only.
125. 12:42Contextual action selection is of real value.
126. 12:46Third, place safety constraints outside the learned policy, so a policy update cannot silently redefine its own authority.
127. 12:53Fourth, trade escalation and post-action validation as first-class outcomes.
128. 13:00Not exception parts and fifth, effect across repeated steps and compare against a simple baseline.
129. 13:06A single favorable run is a demo, not evidence.
130. 13:09A practical self-healing system does not need the largest possible model.
131. 13:14It needs a clear, state-bound, action-reproducible, evaluation observable.
132. 13:20Decisions and the discipline stop when uncertainty exists; its authority this brings the suspect to the engineer in the opening video; the goal is not to eliminate human judgment; it is to stop spending that judgment and the same recognizable failure at two in the morning before the response;
133. 13:40manual log inspection schema; tracking delay dashboards and recovery process measured in working days after the retained path becomes event trigger diagnosis and RL guided but safety constraint action; explicit validation and recovery measured in minutes; when the case is supported by teams and high- risk failures are still going to the humans, that is
134. 14:06the point; human attention is to serve for incident contact; trade-off and authority are genuinely required; the code, synthetic benchmarks, experiments, scripts, and representativeness instructions are available in the GI Hub; record story on screen if you work on agent reliability, date, quality, and protection; incident automation I would Specially call your feedback on state representation
135. 14:35diz and safety boundary.
136. 14:36Thank you for watching.