WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=u1yaOeEX4e8

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/u1yaOeEX4e8.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.309 --> 00:00:03.309
[music]

s2
00:00:13.599 --> 00:00:14.080
Hi.

s3
00:00:14.080 --> 00:00:23.279
Uh thanks thanks and uh hope everyone is out of uh the lunch coma and we'll survive this talk.

s4
00:00:23.279 --> 00:00:38.000
So uh yeah myself Ritik I uh lead the payments team in uh JP Morgan and uh today I'll be talking about learn execution graphs how these graphs can help to

s5
00:00:38.000 --> 00:00:55.440
uh detect any anomaly and uh drifts also how we can automate few things around that and uh you know u uh at the same time if we can reduce

s6
00:00:55.440 --> 00:01:11.920
the manual you know detection work and uh going on that side uh so whenever we hear about graph uh there are persistence graph and property graphs uh which Neo4j

s7
00:01:11.920 --> 00:01:24.400
and you know other products uh we use for them we query those uh graphs and get the answers out of it what I'm talking about today is execution graph.

s8
00:01:24.400 --> 00:01:26.320
It's short-lived graph.

s9
00:01:26.320 --> 00:01:40.880
Uh and idea here is holistically try to identify how the request processing happens and if there is any deviation on that and how to detect that and how to fix that.

s10
00:01:41.759 --> 00:01:43.840
So here is a simple example.

s11
00:01:43.840 --> 00:01:47.920
Uh say we have set of applications.

s12
00:01:47.920 --> 00:02:02.000
uh you have one edge layer uh the first layer where you know request comes in and then uh you have some gateways uh if k is there you have ingress layer on top of it then authentication authorization

s13
00:02:02.000 --> 00:02:16.959
happens after that there is some orchestration layer and uh few other systems which could be called in parallel uh once everything is done you are notifying your client that what's the update on that request right

s14
00:02:16.959 --> 00:02:30.400
So here the idea is representing the uh overall request processing as DAG and using tag simplifies most of the things here.

s15
00:02:30.640 --> 00:02:42.400
one now you know that uh in what order service execution will be happening right so that's one of the thing the other thing is uh you know the context

s16
00:02:42.400 --> 00:02:54.239
that in at what node what context will be there and what will be passed to the next node uh in that way it will be very um u ordered and simplified

s17
00:02:54.239 --> 00:03:09.120
u uh simply can be represented uh there are few other uh use cases could be uh in terms of retries and uh the loops etc. uh the idea here is

s18
00:03:09.120 --> 00:03:14.720
uh every loop uh to put in the graph as a separate entity itself.

s19
00:03:14.720 --> 00:03:20.720
So uh in that way it could be tracked uh easily.

s20
00:03:23.680 --> 00:03:44.879
how we can make this system uh more uh reliable at the same time not using uh most of the resources right so in the tier one check or it's your first check it's like uh going to airport and you you you know it's just boarding pass is some someone is looking at the boarding pass

s21
00:03:44.879 --> 00:03:53.680
and let you go so uh now if you know the baseline of your request execution end to end if uh everything looks Good.

s22
00:03:53.680 --> 00:03:57.200
You don't need to go to the tier two uh or next tier of check.

s23
00:03:57.200 --> 00:03:57.920
Right?

s24
00:03:57.920 --> 00:04:02.720
Once if you find that there is some delay.

s25
00:04:02.720 --> 00:04:06.400
So now you need to check that what changed here.

s26
00:04:06.400 --> 00:04:11.040
One of the uh the drift here could be because of the structural change.

s27
00:04:11.040 --> 00:04:22.400
So if any new node or new step added which you are not aware of that could be one of the thing or one of the step which is removed that could be the another reason right.

s28
00:04:22.400 --> 00:04:40.880
Uh once you know about that then further uh further analysis could be done uh in terms of scale deviations uh divergence and uh exponential ma uh so in simpler terms

s29
00:04:40.880 --> 00:04:53.680
if you know that client A's request is taking this much time normally and uh client B's request could take might take more time than the client A um because of say

s30
00:04:53.680 --> 00:05:01.520
one client is local to you and one client is uh you know the request is coming from outside and uh there are a few more checks needs to be done.

s31
00:05:01.520 --> 00:05:15.120
So in in that case the baseline will change client to client and now you know that uh what your threshold it and uh how you can u reduce the noise of such alerts.

s32
00:05:16.960 --> 00:05:22.320
Uh so here the idea is very simple.

s33
00:05:22.320 --> 00:05:29.440
First you uh represent the entire request processing as DAG.

s34
00:05:29.440 --> 00:05:31.199
You come up with the baseline.

s35
00:05:31.199 --> 00:05:38.400
You find out the deviation and then you try to find out where exactly the issue is.

s36
00:05:38.400 --> 00:05:46.720
Once you localize that then you compare that based on your system that whether it is uh within the threshold or not.

s37
00:05:46.720 --> 00:05:59.280
If it is within the threshold yeah you don't need to u you know do the alerts or automate anything but if it is out of the threshold then certain action needs to be taken.

s38
00:06:00.400 --> 00:06:19.039
Uh coming back to our example here say overall uh request processing from all the different nodes within our system uh is happening but somehow the foreign transaction uh rate service is taking more time than the usual.

s39
00:06:19.039 --> 00:06:32.479
Now if you represented this whole uh request processing in multiple nodes you know where the problem is or where the issue is and correspondingly you will uh you know now you can

s40
00:06:32.479 --> 00:06:34.479
uh exactly know where the problem is.

s41
00:06:34.479 --> 00:06:47.680
So you can solve it uh that what how to uh how FX system or what all different uh cases were there in the past where FX FX rate system was failing.

s42
00:06:47.680 --> 00:07:04.800
Here is one of the example uh for benchmark uh open telemetry and that star bench were used and say for 7 days of the time millions of uh traces were

s43
00:07:04.800 --> 00:07:20.560
um you know injected or uh in the system then you inject the problem or uh anomaly there and based on that you train your system before anything goes on live.

s44
00:07:21.199 --> 00:07:29.120
So uh again uh basic thing here is what is anomaly and what is drift right?

s45
00:07:29.120 --> 00:07:43.039
So say you are driving from your home to office every day and uh one of and usually it takes 1 hour but uh one fine day it took you more time than 1 hour.

s46
00:07:43.039 --> 00:07:50.880
Uh the reason might be some traffic or you know um car accident or anything.

s47
00:07:50.880 --> 00:08:01.680
But uh he this is one of the incidents and uh based on your system and criticality of your system uh you can decide how to address that.

s48
00:08:01.680 --> 00:08:16.639
The other part is one fine day you are uh taking sip of coffee around 4 p.m. and realized that a year back it used to take one hour for you from your home to office but nowadays it it is taking 20 more minutes

s49
00:08:16.639 --> 00:08:30.560
right so what happened in Bay Area in number of carc so this is over the time what you are seeing is pattern changed and that's where you might need to come up with

s50
00:08:30.560 --> 00:08:47.600
uh the new baseline itself so uh that's that's the drift that over the time you start seeing some delays or you know some uh performance deviation then once you know that there is a drift

s51
00:08:47.600 --> 00:09:02.240
uh you can further categorize it first category is structural uh so say you somehow in the system a new node is added or one of the node is removed as I mentioned earlier

s52
00:09:02.240 --> 00:09:14.640
for example you like uh again you know there is a shop where you like uh drinking coffee and uh one fine day they are ask start asking you about membership so they added one more

s53
00:09:14.640 --> 00:09:28.959
uh step in it now every day they might ask you for hey do you have membership with us if you have then there is there are a special discount for you if you don't have membership then the regular prices will be there so in that way uh you know

s54
00:09:28.959 --> 00:09:40.640
same way in our service processing or a request processing if new node is added that means uh Now you need to consider that step also in your old baselines

s55
00:09:40.640 --> 00:09:43.200
and new alerts.

s56
00:09:43.200 --> 00:10:01.040
The other one is uh say because of the volume of request uh one of your service is taking more time or it's not you know u cannot serve the request

s57
00:10:01.040 --> 00:10:04.560
or the volume which you are expecting now over the time.

s58
00:10:04.560 --> 00:10:15.120
So yeah, such kind of drifts you might need to treat differently because now you need either you need to scale up those services or instances of those services and or

s59
00:10:15.120 --> 00:10:27.519
you need to either make it asynchronous call or based on based on the use case you know what whatever uh works there uh co-variate is a different uh one of the category

s60
00:10:27.519 --> 00:10:42.880
say when you started the business uh you were seeing around 60% of local request but uh and 40% uh request from you know out of the country and that's where you might need

s61
00:10:42.880 --> 00:10:54.480
to change the currency or you know one extra step is there so you now you know that what is the baseline for your uh request in US dollar but uh what is the baseline for

s62
00:10:54.480 --> 00:11:05.680
uh any of the other currency over the time what happens is your product is so popular that you started getting more uh request from the outside.

s63
00:11:05.680 --> 00:11:06.720
So nothing changed.

s64
00:11:06.720 --> 00:11:08.959
Your system is working fine, right?

s65
00:11:08.959 --> 00:11:15.200
But now you need to come up with the criteria and reassess your baselines again.

s66
00:11:15.200 --> 00:11:31.920
Sorry here where uh either you need to come up with two different uh you know uh graphs to compare that one is for local and one is for um outside uh request from outside

s67
00:11:31.920 --> 00:11:38.320
or what you can do is you can increase your uh average request time baseline.

s68
00:11:38.320 --> 00:11:42.320
So once you know the pattern uh you know the solution.

s69
00:11:42.320 --> 00:11:48.000
So uh that's where you need to categorize uh this drifts.

s70
00:11:48.000 --> 00:11:55.440
One category could be for the same request uh now you are seeing the different behavior itself.

s71
00:11:55.440 --> 00:12:06.560
Then probably if and when it's needed you might need to roll back such uh changes or either you need to reconsider that.

s72
00:12:06.560 --> 00:12:18.639
So where I'm going with this is in that way you need to re-evaluate and reassess your system before uh identifying that what action needs to be taken.

s73
00:12:18.639 --> 00:12:25.680
This whole talk is mostly about statistical uh uh you know part of uh the solution.

s74
00:12:25.680 --> 00:12:41.279
uh it's it's uh part of bigger neuro uh specific uh um algorithms and system in a way but this is just one of the module which uh I'm talking about here now

s75
00:12:41.279 --> 00:12:58.639
so once you know the drift or deviation and here is the simple uh tag for how uh this whole system would work.

s76
00:12:58.639 --> 00:13:02.240
Open telemetry will will keep feeding the data.

s77
00:13:02.240 --> 00:13:11.920
Once you have that data, uh root cause analysis could be taken based on uh once your system knows about all the data points.

s78
00:13:11.920 --> 00:13:17.279
Now you know which type of drift it is and what solution could be there.

s79
00:13:17.279 --> 00:13:21.839
Then you identify what action needs to be taken.

s80
00:13:21.839 --> 00:13:35.360
Once you know what action needs to be taken further you need to also uh find out that what is the risk if we go with this approach or if we automate this uh solution right so once you know the risk

s81
00:13:35.360 --> 00:13:50.959
either you can go with uh roll out that system uh um solution for say 5% or 10% of your uh machines monitor it verify everything looks good and then you roll out for your 100% of the nodes Here

s82
00:13:54.000 --> 00:13:55.440
are a couple of example.

s83
00:13:55.440 --> 00:14:09.120
uh again uh say generally overall request processing takes 700 millisecond but uh on the left side what you are seeing is u in the graph approach itself how it could help you is

s84
00:14:09.120 --> 00:14:19.920
uh now you know that uh which specific node is taking more time suddenly you get alert on that or otherwise if you are seeing that uh the delay is all across

s85
00:14:19.920 --> 00:14:35.920
then something which is common which you need to fix here all these things because in the payments and the real time uh payment processing we want to keep it very faster right so

s86
00:14:35.920 --> 00:14:48.560
we don't want delay the actual request processing the solution which we generally use uh everyone in the industry is asynchronously feeding the data to uh opal telemetry from there

s87
00:14:48.560 --> 00:14:56.480
u some kafka could be used and stream assessment uh could be used on top of it.

s88
00:14:56.480 --> 00:14:58.320
There could be two different paths.

s89
00:14:58.320 --> 00:15:07.440
One is say hot path where you can take a decision very faster and uh um work on the solution or automate that solution.

s90
00:15:07.440 --> 00:15:18.320
The other one is more recon kind of solution where it might take some time but more more accurate could it could be

s91
00:15:18.975 --> 00:15:20.975
[snorts]

s92
00:15:21.199 --> 00:15:26.800
few of uh uh other challenges which uh we need to fine-tune here.

s93
00:15:26.800 --> 00:15:35.680
So all I talked about is hey you have seven nodes in your system and every node is feeding the data to your uh telemetry.

s94
00:15:35.680 --> 00:15:40.720
What if one of the system is delaying the event?

s95
00:15:41.440 --> 00:15:53.759
Should we consider it as a structural change because now what you have data in your system is for six nodes and the seven uh uh the data from the seven node is delayed already.

s96
00:15:53.759 --> 00:16:02.560
So we need to fine-tune that uh that those numbers also that when to consider that uh there is a structural change or not.

s97
00:16:02.560 --> 00:16:16.959
So basically here we are uh trying to reduce uh any false alarm based on the use cases and here in this use case u we should go with tail by based system because

s98
00:16:16.959 --> 00:16:30.320
what we are trying to track here is uh or in this specific example is when the service request started and when it ended right so uh for each and every node

s99
00:16:30.320 --> 00:16:39.279
the other part is the cold start uh if there is a new endpoint consider the new baseline don't make it very generic.

s100
00:16:39.759 --> 00:16:55.199
So in that way on the detect side any MMD or KL uh could be used uh and once you confirm uh this with admin and then classify uh the problem

s101
00:16:55.199 --> 00:17:06.079
uh that will give you where what exact solution needs to be done and next step whatever uh if we can automate it we'll automate it.

s102
00:17:06.079 --> 00:17:16.319
So what we see uh in general here is uh mean time to uh discovery reduced a lot to make it very real time.

s103
00:17:16.319 --> 00:17:28.960
Instead of comparing or waiting for multiple windows of uh the time duration it was uh a single window which we uh uh you know which we identified uh helped a lot

s104
00:17:28.960 --> 00:17:33.360
uh to fix the issues fast.

s105
00:17:33.520 --> 00:17:38.000
uh few of other things which we might need to make sure.

s106
00:17:38.000 --> 00:17:49.200
One is uh the labels when it comes to uh learnings the labels helps a lot but uh at the same time uh we need to make sure that uh

s107
00:17:49.200 --> 00:17:51.919
the system is very fine tuned in terms of that.

s108
00:17:51.919 --> 00:18:04.320
Uh the other part is instead of saying that all the post requests should have you know this is a baseline for all the post request uh try to come up with very

s109
00:18:04.320 --> 00:18:14.880
uh a number which works for you post for payments for real-time payment or post for wire payments or u based on you know uh your use cases.

s110
00:18:14.880 --> 00:18:16.960
So that that would help a lot.

s111
00:18:16.960 --> 00:18:30.400
Um again uh if anything is u um you are considering a structural change or something uh the window uh should be well defined for each and every client.

s112
00:18:30.400 --> 00:18:38.080
In my previous uh example which I talked about uh if you can come up with the new baseline would really help uh to reduce the noise.

s113
00:18:38.080 --> 00:18:44.160
Uh explanability uh all the data should be well explained.

s114
00:18:44.160 --> 00:18:50.320
If you go to the doctor and doctor says your health score is 22, it doesn't make much sense to you.

s115
00:18:50.320 --> 00:19:01.919
So, uh yeah, the actual uh more data uh can explain you know more things to you and uh can we can take the informative decision on top of it.

s116
00:19:01.919 --> 00:19:05.120
Uh whole system should be aware of the new deployment.

s117
00:19:05.120 --> 00:19:12.480
Uh so u u based on that you can take either roll back decision or not.

s118
00:19:12.480 --> 00:19:14.559
So yeah that's that's about it.

s119
00:19:14.559 --> 00:19:15.919
Uh thanks everyone.

s120
00:19:15.919 --> 00:19:18.080
Uh I would like to connect with you all.

s121
00:19:18.080 --> 00:19:19.840
Uh here is my LinkedIn.

s122
00:19:19.840 --> 00:19:22.559
I can answer it.

s123
00:19:35.083 --> 00:19:37.083
[music]