WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=LrGCT7G_rU8 NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/LrGCT7G_rU8.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:00.560 --> 00:00:03.064 Imagine you're this engineer. s2 00:00:03.064 --> 00:00:05.484 A production data job failed. s3 00:00:05.484 --> 00:00:12.453 You went to the dashboard and spent all day checking the logs, the schema, and the upstream data. s4 00:00:12.453 --> 00:00:14.905 And now it's passed. s5 00:00:14.905 --> 00:00:17.039 The same question chips. s6 00:00:17.039 --> 00:00:19.439 Coming back, what changed? s7 00:00:19.439 --> 00:00:35.037 The failure itself may be small, but the expensive part is everything around it: inspection, diagnosis, choosing a safe, running the job, and confirming that we didn't make the data error. s8 00:00:35.037 --> 00:00:37.520 Hi, I'm Anamari Bazhan. s9 00:00:37.520 --> 00:00:45.328 In this talk, I'll show an ETL-guided system that selects bounded remittance actions for ETL failures. s10 00:00:45.328 --> 00:00:59.440 The central question is simply whether an agent can act, but whether it can act usefully, explainably, and within the boundaries that an operation would actually trust. s11 00:00:59.440 --> 00:00:59.964 Cloud failures. s12 00:00:59.964 --> 00:01:00.750 Early arrival errors. s13 00:01:00.750 --> 00:01:08.816 We're able sources, daytime in compatibility. s14 00:01:08.816 --> 00:01:15.136 Now reach spikes, type changes, and run-time errors that don't do much anything in the run book. s15 00:01:15.136 --> 00:01:24.651 Usual response is a human work flow: Inspect the logs, form, diagnose a temporary job, and validate the output. s16 00:01:24.651 --> 00:01:26.799 Each step is reasonable. s17 00:01:26.799 --> 00:01:35.425 The latency comes from the head in complete contact and the need to avoid and unsafe fix in the caption. s18 00:01:35.425 --> 00:01:41.332 Evaluate the manual recovery baseline was modeled at roughly 2.5 working days. s19 00:01:41.332 --> 00:01:47.517 This represents an incident moving through normal queueing, investigation, and approval. s20 00:01:47.517 --> 00:01:59.759 So the engineering objective is specific criteria that look for routine recognizable failures while escalating the cases that are uncertain, novel, and high risk. s21 00:01:59.759 --> 00:02:03.856 This diagram shows the end-to-end AWS architecture from my cupstone. s22 00:02:03.856 --> 00:02:09.249 An existing AWS Glue initial job emits a job fail event. s23 00:02:09.249 --> 00:02:15.724 Amazon Event Bridge catches that event and triggers the Lamda function that runs the agent. s24 00:02:15.724 --> 00:02:19.599 Lamda gathers evidence from two read-only sources. s25 00:02:19.599 --> 00:02:22.114 CloudWatch provides the error log. s26 00:02:22.114 --> 00:02:25.616 Glue data catalog provides the current schema. s27 00:02:25.616 --> 00:02:29.533 Metadata: The system uses these signals to classify the failure. s28 00:02:29.533 --> 00:02:34.179 Assess the data quality and operational risk and construct the state. s29 00:02:34.179 --> 00:02:39.328 Passed to the RL decision engine, the policy then proposes an appropriate response. s30 00:02:39.328 --> 00:02:50.080 The safety layer takes that proposal before the executor can use the Glue API to retrigger the job and apply and approve remediation. s31 00:02:50.080 --> 00:02:54.119 Amazon stores its artifacts and audits the log. s32 00:02:54.119 --> 00:03:00.868 Quarantine Outputs: Finally, the job is run and validated. s33 00:03:00.868 --> 00:03:02.772 So this is closed. s34 00:03:02.772 --> 00:03:11.770 Operational look: Monitor, diagnose, score, decide, check, safety, act, and verify. s35 00:03:11.770 --> 00:03:17.120 Recovery: The caption implementation uses synthetic data provided by the client. s36 00:03:17.120 --> 00:03:20.959 The public repository uses this pattern through sanitize. s37 00:03:20.959 --> 00:03:22.159 Generalize deployment. s38 00:03:22.159 --> 00:03:29.200 The intelligence layer deliberately separates three concerns: the terminal anomaly rules. s39 00:03:29.200 --> 00:03:30.640 Establish observable facts. s40 00:03:30.640 --> 00:03:35.039 Fields disappear after a type change and a data rate exceeds thresholds. s41 00:03:35.039 --> 00:03:38.416 The learning policy handles context action selection. s42 00:03:38.416 --> 00:03:43.920 Given the current incident state, the system should retry the schema. s43 00:03:43.920 --> 00:03:48.078 Roll back the quarantine exit and simply log the event. s44 00:03:48.078 --> 00:03:52.168 Then a safety override sets outside the learned policy. s45 00:03:52.168 --> 00:03:58.044 For example, if the anomaly is critical, the policy proposes a passive action such as logging. s46 00:03:58.044 --> 00:04:00.715 The override converts that choice into an escalation. s47 00:04:00.715 --> 00:04:06.781 This separation is the result of the project rules for facts. s48 00:04:06.781 --> 00:04:27.520 Learning for bounded choices and guards for authority before selecting an action The system has established what actually happened The schema profiler extracts structure types, statistical and knowledge statistics The drift detector compares the current profiler with the baseline and identifies additions, s49 00:04:27.520 --> 00:04:45.199 removals, and type changes The data quality analyzer measures completeness, validity, and consistency The error classifier measures log parts in failure families, and the risk scorer measures signals in an operational risk level s50 00:04:45.199 --> 00:05:13.967 These components are deterministic by using directly observable data conditions An explicit rule is easier to validate than an explicit rule with a richer and more representative incident history Some classifiers could become learned components, but ML ready is not the same as ML requires the simplest reliable component Should on each decision The policy receives the compact state Failure s51 00:05:13.967 --> 00:05:30.960 category Risk level Countability and data quality conditions These six actions Retry covers Roll back Quarantine Escalate and L I use RQ learning Because the state and action spaces are small, s52 00:05:30.960 --> 00:05:44.720 the Q tables are small and every decision can be inspected directly for this state, this action values, and this action one Technically, each incident is modeled as a single- step contextual decision Implemented s53 00:05:44.720 --> 00:06:02.400 with learning Rather than as a long horizontal control task that formulates the system to choose one safe operational response from an unbounded action Set the value of the learn policy There is no sophistication for its own sake It structures s54 00:06:02.400 --> 00:06:25.195 way to learn action preferences from outcomes While retaining a decision service that an engineer can specify The learn policy does not have final authority It proposes an action The safety layer The proposal against the Normally, the system's operational constraints are passed. s55 00:06:25.195 --> 00:06:29.536 Passive actions are overlooked for critical conditions and higher. s56 00:06:29.536 --> 00:06:32.160 Unknown cases are escalated. s57 00:06:32.160 --> 00:06:34.000 Every proposal is overlooked. s58 00:06:34.000 --> 00:06:39.727 The execution result and validation outcome is written to an audit record. s59 00:06:39.727 --> 00:06:46.426 Notice that escalation is included in the action space that is not the agent giving up. s60 00:06:46.426 --> 00:06:53.267 It is the system correctly recognizing the boundary of its evidence and authority for an operational agent. s61 00:06:53.267 --> 00:07:02.079 The ability to say I should not do this automatically is the capability if success is measured only by non- escalation. s62 00:07:02.079 --> 00:07:04.074 The optimization target is wrong. s63 00:07:04.074 --> 00:07:06.479 There is one failure path. s64 00:07:06.479 --> 00:07:09.865 The agent receives the Glue Style job failure event. s65 00:07:09.865 --> 00:07:10.785 The Love classifier. s66 00:07:10.785 --> 00:07:18.278 The text is a daytime format incompatibility with 0.9 confidence base on the state. s67 00:07:18.278 --> 00:07:21.032 The policy prop schema is correct. s68 00:07:21.032 --> 00:07:34.502 The safety overhead does not fire because this is not classified as a critical event, but the execution Then discover that automatic coverage is not available for this specific case. s69 00:07:34.502 --> 00:07:39.312 The system does not pretend that the fix should happen. s70 00:07:39.312 --> 00:07:41.440 It proposes an action. s71 00:07:41.440 --> 00:07:46.620 Reports that execution was unavailable and identifies the incident for manual review. s72 00:07:46.620 --> 00:07:54.412 This example shows to distinguish controls, policy, safety, and implementation capability. s73 00:07:54.412 --> 00:08:00.975 An action can be safe in principle and still be unavailable in the current environment. s74 00:08:00.975 --> 00:08:10.372 A ROWAS agent must represent both conditions explicitly to make the work independently reviewable without exploding the client context. s75 00:08:10.372 --> 00:08:17.280 I build the public benchmark around the generalized AWS Lambda-style architecture. s76 00:08:17.280 --> 00:08:21.140 The CUPS implementation uses client- provided synthetic data. s77 00:08:21.140 --> 00:08:25.559 The public repository uses newly generalized synthetic schemas. s78 00:08:25.559 --> 00:08:28.520 Records logs and incident scenarios. s79 00:08:28.520 --> 00:08:31.199 It contains no client documents. s80 00:08:31.199 --> 00:08:43.971 Infrastructure identifiers and business-specific values ​​ are used for control, experiment groups, and Repeated the robustness evaluation across 36 from 42 to 71. s81 00:08:43.971 --> 00:08:49.600 The reported aggregates include 95% confidence intervals. s82 00:08:49.600 --> 00:08:59.277 This process uses the system design and experimental logic in a firm that other engineers can inspect and run while maintaining the confidence. s83 00:08:59.277 --> 00:09:04.720 Confidentiality boundary on the control benchmark: the rule base abnormally. s84 00:09:04.720 --> 00:09:09.919 The detector achieved a precision of one recall of 0.8 and an NF score of 0.889. s85 00:09:09.919 --> 00:09:14.640 This means that the detector was conservative. s86 00:09:14.640 --> 00:09:23.440 Normally, it flagged errors correctly in this benchmark, but it still missed some positive cases for operations that required distinction. s87 00:09:23.440 --> 00:09:31.800 Perfect precision does not mean perfect detection for cases where the RL-guided work flow resolved the incident successfully. s88 00:09:31.800 --> 00:09:37.108 The mean resolution time was about 5.24 minutes across the 30 runs. s89 00:09:37.108 --> 00:09:43.932 The simulated success rate was 74.63% ( plus or minus 1.51 percentage points). s90 00:09:43.932 --> 00:09:50.080 The non-escalation rate was 88.63% ( plus or minus 0.89 points). s91 00:09:50.080 --> 00:09:57.980 The chart compares that minute scale result with the model's manual baseline of 2 and 1/2 working hours. s92 00:09:57.980 --> 00:10:01.328 That is 260 seconds. s93 00:10:01.328 --> 00:10:08.640 In the benchmark, that is approximately a 99.85% reduction in MTTR. s94 00:10:08.640 --> 00:10:12.240 This figure qualifies performance against the control benchmark. s95 00:10:12.240 --> 00:10:18.879 In that scope, they show that the architecture can automate the first path for non-failure conditions. s96 00:10:18.879 --> 00:10:20.640 Production validation is the next evaluation. s97 00:10:20.640 --> 00:10:26.795 The boundary resolution results are, in my view, the most useful part of the project. s98 00:10:26.795 --> 00:10:30.026 The RL policy matches the equivalent deterministic policy. s99 00:10:30.026 --> 00:10:38.315 The difference of 0.19 percentage points (with a 0.19 point confidence interval) on this compact state space. s100 00:10:38.315 --> 00:10:56.493 The learned policy can maintain the Same Success Level as the Hand Define Policy by Contrast the Terministic Action Section vs Random Selection by 15.63 points and Enabling the Safety of Ride Reduce Nuisance by about 15.03 points. s101 00:10:56.493 --> 00:10:58.449 That decrease is intentional. s102 00:10:58.449 --> 00:11:05.360 The garden system escalates more often when its autonomy would be inappropriate. s103 00:11:05.360 --> 00:11:11.520 So why did the reliability come from primarily from the initial state? s104 00:11:11.520 --> 00:11:15.986 Sensible decision logic and external safety constraints, not from RL alone. s105 00:11:15.986 --> 00:11:21.440 That is a useful engineering result in the current benchmark. s106 00:11:21.440 --> 00:11:28.799 RL provides an inspectable learned decision service rather than an immediate success rate advantage. s107 00:11:28.799 --> 00:11:33.519 Its value becomes more significant as the incident history becomes richer. s108 00:11:33.519 --> 00:11:37.920 Action outcomes by manually maintaining preference becomes more difficult. s109 00:11:37.920 --> 00:11:44.064 This slide defines the current validation boundary. s110 00:11:44.064 --> 00:11:46.656 The results come from synthetic scenarios. s111 00:11:46.656 --> 00:11:49.412 The agent responds after a failure signal. s112 00:11:49.412 --> 00:11:52.928 It does not predict a failure before it happens. s113 00:11:52.928 --> 00:11:57.036 Real incident diversity meets the current state space. s114 00:11:57.036 --> 00:12:01.066 Some remediation actions are simulated and deliberately bounded. s115 00:12:01.066 --> 00:12:04.360 An online learning in a production environment. s116 00:12:04.360 --> 00:12:10.080 Req approval gets version policy rollback support and continues monitoring. s117 00:12:10.080 --> 00:12:14.345 The result is a credible feasibility demonstration of the system. s118 00:12:14.345 --> 00:12:19.920 This is the clear path over production validation. s119 00:12:19.920 --> 00:12:24.426 The next step is a shadow mode deployment based on representations in incident traces. s120 00:12:24.426 --> 00:12:29.440 Where recommendations can be made, decisions can be made before the agent traces. s121 00:12:29.440 --> 00:12:29.640 Execution authority. s122 00:12:29.640 --> 00:12:35.855 There are five takeaways: I would leave an engineering team. s123 00:12:35.855 --> 00:12:40.128 First, use the terministic logic for facts that can be measured directly. s124 00:12:40.128 --> 00:12:42.240 Second, use learning only. s125 00:12:42.240 --> 00:12:46.240 Contextual action selection is of real value. s126 00:12:46.240 --> 00:12:53.039 Third, place safety constraints outside the learned policy, so a policy update cannot silently redefine its own authority. s127 00:12:53.039 --> 00:13:00.159 Fourth, trade escalation and post-action validation as first-class outcomes. s128 00:13:00.159 --> 00:13:06.791 Not exception parts and fifth, effect across repeated steps and compare against a simple baseline. s129 00:13:06.791 --> 00:13:09.600 A single favorable run is a demo, not evidence. s130 00:13:09.600 --> 00:13:14.944 A practical self-healing system does not need the largest possible model. s131 00:13:14.944 --> 00:13:20.880 It needs a clear, state-bound, action-reproducible, evaluation observable. s132 00:13:20.880 --> 00:13:40.079 Decisions and the discipline stop when uncertainty exists; its authority this brings the suspect to the engineer in the opening video; the goal is not to eliminate human judgment; it is to stop spending that judgment and the same recognizable failure at two in the morning before the response; s133 00:13:40.079 --> 00:14:06.399 manual log inspection schema; tracking delay dashboards and recovery process measured in working days after the retained path becomes event trigger diagnosis and RL guided but safety constraint action; explicit validation and recovery measured in minutes; when the case is supported by teams and high- risk failures are still going to the humans, that is s134 00:14:06.399 --> 00:14:35.120 the point; human attention is to serve for incident contact; trade-off and authority are genuinely required; the code, synthetic benchmarks, experiments, scripts, and representativeness instructions are available in the GI Hub; record story on screen if you work on agent reliability, date, quality, and protection; incident automation I would Specially call your feedback on state representation s135 00:14:35.120 --> 00:14:36.932 diz and safety boundary. s136 00:14:36.932 --> 00:14:40.160 Thank you for watching.