WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=tZycGA_Fbfo

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/tZycGA_Fbfo.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:03.200 --> 00:00:04.560
Hello everyone.

s2
00:00:04.560 --> 00:00:11.800
Um, I would like to walk you through that how we can transition in AI agents from an experiments to the real enterprise environment.

s3
00:00:11.800 --> 00:00:14.560
So, before I begin, a quick introduction about me.

s4
00:00:14.560 --> 00:00:15.880
My name is Sharish Gokar.

s5
00:00:15.880 --> 00:00:24.080
I'm a senior staff software engineer at Intuit Credit Karma where I built and own the AI analytics infrastructure and services.

s6
00:00:24.080 --> 00:00:35.920
Um, and beside that, um, I'm advisor, uh, author, researchers, um, and also a program committee members for various organizations where I mentor the young researchers as well.

s7
00:00:35.920 --> 00:00:37.760
All right.

s8
00:00:37.760 --> 00:00:41.600
So, let's, uh, walk through the, um, agenda here.

s9
00:00:41.600 --> 00:00:55.640
So, first we would like to talk about what are the challenges we have when we build an AI agents and deploy into the production systems and how we, uh, can build a different framework to address that, uh, different challenges here.

s10
00:00:55.640 --> 00:00:58.640
Um, then how the architecture looks like for it.

s11
00:00:58.640 --> 00:01:04.920
Um, then, um, how you can build your own MCP servers, um, and MCP gateway.

s12
00:01:04.920 --> 00:01:08.680
And then some monitoring on top of that gateway as well.

s13
00:01:08.680 --> 00:01:18.720
Um, then further we're going to talk about what are the different type of agent framework you can have, um, and then, um, some additional framework around managing the memories,

s14
00:01:18.720 --> 00:01:28.280
tracking and eval, um, how you can block the PII data, uh, going before the LLMs, and then context optimizer to save your token, right?

s15
00:01:28.280 --> 00:01:31.720
So, without delay, let's, um, let me walk you through the challenge.

s16
00:01:31.720 --> 00:01:38.720
Um, of course, like there are many agents, uh, uh, generally get failed in productions because their foundation controls are missing.

s17
00:01:38.720 --> 00:01:49.080
Um, and the key reason behind is that, um, you know, like, oh, most of the people are missing the gateway to make sure that agent can, you know, communicate to the

s18
00:01:49.080 --> 00:01:56.200
gateway to get authenticated first with the right auth providers so that they can redirect to the right MCP servers, right?

s19
00:01:56.200 --> 00:02:01.440
Second, that, um, MCP servers are missing the right tool to execute your query.

s20
00:02:01.440 --> 00:02:13.360
Third, do we have like enough tracking and evaluation system to make sure that every query when you are sending to the agents, whether they've been tracked or not, and are you evaluating the quality response or not, right?

s21
00:02:13.360 --> 00:02:25.000
Do we have any in memory management system or not in terms of let's say you are doing the interaction with agent for long or for many days, is it able to recall that what you have been asked in the past, right?

s22
00:02:25.000 --> 00:02:36.959
And then there are other framework, of course, like do you have a context of the measures to save the token, or do you have inner production in place to make sure that your any secrets should not go into LLM or tracking or tracing

s23
00:02:36.959 --> 00:02:39.320
or any memory DB or so, right?

s24
00:02:39.320 --> 00:02:41.680
So, a lot of lot of open questions.

s25
00:02:41.680 --> 00:02:49.320
So, in order to resolve this, we are not going to, you know, like build a magical agent here in this demo.

s26
00:02:49.320 --> 00:02:54.520
We're going to talk about like different framework that you can build to address this.

s27
00:02:54.520 --> 00:02:56.680
So, first is the agent framework.

s28
00:02:56.680 --> 00:03:04.040
This is where like we use the like we choose to use the LangGraph framework, which is provided by LangChain community.

s29
00:03:04.040 --> 00:03:17.760
This is help us in building the task-specific node based on your questions, and which is easy to plug in any like marketplace plugins or skills and prompts to make sure that agents

s30
00:03:17.760 --> 00:03:20.160
respond efficiently based on your query.

s31
00:03:20.160 --> 00:03:31.959
Further, the second part is we are also building the MCP gateway to make sure that whenever agent make the call or any service account from that agent make the call, make sure that they are securely get authenticated

s32
00:03:31.959 --> 00:03:37.239
to that MCP gateway, and then get redirected to the right MCP servers.

s33
00:03:37.239 --> 00:03:44.800
Third is the customized MCP, which means that you are building your own MCP servers with the right tool in place to execute your query.

s34
00:03:44.800 --> 00:03:50.200
And then finally some audit on top of what actually operation you're performing, right?

s35
00:03:50.200 --> 00:03:59.160
And then we talk about some additional framework around you know like how you can block the PII secrets, how you going to store the tracking eval into the cloud,

s36
00:03:59.160 --> 00:04:14.040
and how you going to process the memories, especially any threads and checkpoints on every context, and then also how you can optimize overall your context to make sure that you don't don't burn too many tokens

s37
00:04:14.040 --> 00:04:15.760
when you run your query.

s38
00:04:15.760 --> 00:04:17.680
All right?

s39
00:04:17.680 --> 00:04:20.840
So, we talk about now the agent infrastructure.

s40
00:04:20.840 --> 00:04:24.440
So, the Firstly, we'll go through the local development.

s41
00:04:24.440 --> 00:04:29.960
The way the approach should be is that you know like whenever you build an agent, right?

s42
00:04:29.960 --> 00:04:32.880
First you have to register your agent into some system.

s43
00:04:32.880 --> 00:04:46.200
You usually tell that hey look, this is my the use case and these are the use case going to use for this particular vertical or system, and you feed all those information and generate a ID for your agent, right?

s44
00:04:46.200 --> 00:04:55.160
Once you have that, then you can have a dev portal, internal portal where you can ingest all the details saying that hey look, this is my agent use case, this is my agent ID,

s45
00:04:55.160 --> 00:04:58.880
this is the framework I'm going to use either LangGraph, LangChain or so.

s46
00:04:58.880 --> 00:05:10.120
These are my language I'm going to use either Python or TypeScript, and then you can choose you know like your skills or prompts, and then you submit your information, right?

s47
00:05:10.120 --> 00:05:10.800
To this.

s48
00:05:10.800 --> 00:05:22.000
Once you submit this, the dev portal is going to create a repository, which is going to contains not just the agent framework like your LangGraph agent framework, but is also going to contains

s49
00:05:22.000 --> 00:05:25.160
your memory framework, your tracking eval framework, right?

s50
00:05:25.160 --> 00:05:30.360
and then your context optimizer framework, and your PII blocking framework as well.

s51
00:05:30.360 --> 00:05:39.200
So, the reason for that is because not necessarily all your developers are taking care of all this framework, right?

s52
00:05:39.200 --> 00:05:45.919
They just sometimes you know create the framework what generally like AI assistant tools are generally the right?

s53
00:05:45.919 --> 00:05:47.840
Or they stick to one or two framework, right?

s54
00:05:47.840 --> 00:06:02.280
So, we just want to make sure that when you roll out such framework, make sure they are consistent across the company um and they can take care of uh and they can be centralized in one place so you can visualize their results as well, right?

s55
00:06:02.280 --> 00:06:14.240
So, once you have that repository, so developers basically, you know, clone that repository, they can build their agent as a Docker image and containers and test their agent input and output through the query.

s56
00:06:14.240 --> 00:06:18.720
And they can also traces on their local memory, maybe cloud span evaluator here.

s57
00:06:18.720 --> 00:06:27.680
And you can also store this tracking and eval in some of your DB, maybe SQLite or maybe BigQuery um to see to make sure that agents are fully working on your local, right?

s58
00:06:27.680 --> 00:06:35.040
So, once your agents are running in local, um then you're ready to go ahead and generate a PR and then you can once you get your two PR approval,

s59
00:06:35.040 --> 00:06:37.840
then it's go to the CICD pipeline, right?

s60
00:06:37.840 --> 00:06:49.200
So, the PR is basically going to firstly is going to uh register your agent into some centralized agent registry system, which provide a length with URL to it to make sure that you see all the traces.

s61
00:06:49.200 --> 00:06:57.520
And then the CICD pipeline is basically, you know, whatever the local Docker container you built here is going to build that container into production system.

s62
00:06:57.520 --> 00:07:00.680
Uh it will scan that image, make sure that there's no vulnerability.

s63
00:07:00.680 --> 00:07:07.040
Um if everything looks great, security approved, you can move that image back into the registry, right?

s64
00:07:07.040 --> 00:07:16.600
Further, you have your own deployment platform or orchestrator platform which going to deploy that agent into the Kubernetes cluster in a given namespace, right?

s65
00:07:16.600 --> 00:07:22.600
So, the good part here is that this dev portal is not just creating the repository, um right?

s66
00:07:22.600 --> 00:07:29.840
It also do create um bunch of other production resources, for example, the workload identity, um right?

s67
00:07:29.840 --> 00:07:36.520
And then memory DB or tracking and eval tables and then agent secret management, right?

s68
00:07:36.520 --> 00:07:45.240
Um so, it based on whatever the name you provide, it not just again it like not just repository, it all these different resources as well.

s69
00:07:45.240 --> 00:07:54.000
So, what happened is when this internal orchestrator when it deploy the agents, the agent already know that hey, what workload identity I supposed to work with, right?

s70
00:07:54.000 --> 00:07:56.080
That binding should already happen.

s71
00:07:56.080 --> 00:08:02.280
So, once this K to GSA binding happens, the agent will start communicating to all these resources.

s72
00:08:02.280 --> 00:08:14.120
Once everything in place, um agent is further going to communicate to the rest of your endpoint services like for example, AI gateway or MCP servers or length fuse for tracing or some other API endpoints.

s73
00:08:14.120 --> 00:08:16.120
Right?

s74
00:08:16.160 --> 00:08:20.480
So, now firstly, we talk about the MCP server as well.

s75
00:08:20.480 --> 00:08:21.560
So, we talk about architecture.

s76
00:08:21.560 --> 00:08:23.280
Now, we're going to the different component.

s77
00:08:23.280 --> 00:08:33.440
So, MCP server is something that you can definitely use something which is outside available, but if you're working on air gap environment, it's highly recommended that you build your own MCP servers.

s78
00:08:33.440 --> 00:08:38.560
So, um yeah, uh and you can use the uh I will not go with the definition.

s79
00:08:38.560 --> 00:08:39.719
Everyone knows about it.

s80
00:08:39.719 --> 00:08:52.960
So, the way the framework has been built here is that uh you you can leverage the fast MCP server package um or wrapper to um you know, build that MCP servers and you can create a different set of tools

s81
00:08:52.960 --> 00:08:58.600
inside that a fast MCP wrapper to construct your um you know, the overall servers.

s82
00:08:58.600 --> 00:09:12.800
And then you make sure that, you know, like every tool when you define in MCP server, they had to be fully reviewed by your own internal security team to make sure that only authorized tool can be go in production.

s83
00:09:12.800 --> 00:09:25.360
And that's how you can you know, enforce some filters, limits, and block the fields which is really not required to go to LLM and get you the information which is you're not supposed to get, right?

s84
00:09:25.360 --> 00:09:35.680
Um and third part is like a of course, you need to make sure that you build your MCP server in a way that whenever the traffic is coming from your MCP client to MCP gateway to MCP servers,

s85
00:09:35.680 --> 00:09:47.880
either it's coming from user service account, they has to be authenticated to the MCP gateway first and so that they can be access your MCP tool to get the right information.

s86
00:09:48.080 --> 00:09:49.040
All right.

s87
00:09:49.040 --> 00:09:51.000
Now, the gateway part.

s88
00:09:51.000 --> 00:09:52.240
So, the how the gateway works.

s89
00:09:52.240 --> 00:09:58.400
So, this is again, you can build it your own internal MCP gateway if you are again working on your air gap environment.

s90
00:09:58.400 --> 00:10:05.400
So, the way it work is you are working on cloud assist like local, you know, code assistance tool or maybe your agent, right?

s91
00:10:05.400 --> 00:10:13.080
So, the moment you submit your first query to your agent, it will send that one-time authentication request to the MCP gateway, right?

s92
00:10:13.080 --> 00:10:16.320
MCP gateway recognized and take your request.

s93
00:10:16.320 --> 00:10:19.960
First, it will save your session state into the Redis, right?

s94
00:10:19.960 --> 00:10:26.120
And then it will, you know, return back the JWT access token back to the your client, right?

s95
00:10:26.120 --> 00:10:32.120
Once client will receive that JWT token, it will know that what MCP server needs to call, right?

s96
00:10:32.120 --> 00:10:43.680
So, it will submit that tool {slash} call request on a given name space {slash} MCP, meaning let's say Artifactory or GitHub {slash} MCP with the auth bearer JWT token

s97
00:10:43.680 --> 00:10:44.400
into that.

s98
00:10:44.400 --> 00:10:58.080
So, once MCP gateway receive that, it will understand that what name space, meaning MCP server needs to look up for, and then it will unwrap that JWT token, and then it will attach that user token

s99
00:10:58.080 --> 00:11:01.480
in that request and send it to the respective MCP servers.

s100
00:11:01.480 --> 00:11:11.839
Now, once MCP server receive it, it will make that respective API calls to that endpoint of that, you know, to the service and then it will exchange that ID token

s101
00:11:11.839 --> 00:11:16.960
and return the response based on whatever scope that user is authorized to see, right?

s102
00:11:16.960 --> 00:11:27.720
So, once the user receive the token, you know, MCP server basically, you know, unwrap that all the data, make the payload which is compatible for MCP gateway to receive,

s103
00:11:27.720 --> 00:11:33.080
and then finally the final payload will send back to the client to see the data points.

s104
00:11:33.080 --> 00:11:45.480
So, what it shows is that, you know, like we build this MCP gateway to make sure that we have a centralized authentication and routing hub that securely connect any coding assistant tool or agents to

s105
00:11:45.480 --> 00:11:48.440
the endpoint via this MCP gateway and MCP server.

s106
00:11:48.440 --> 00:11:53.720
This There should not be direct access provided um on any endpoints.

s107
00:11:53.720 --> 00:12:01.800
They has to go through the MCP gateway through proper authentications uh before they're reaching to the MCP server and the endpoints.

s108
00:12:01.800 --> 00:12:03.280
All right?

s109
00:12:03.280 --> 00:12:06.920
So, now the part is the gateway monitoring, right?

s110
00:12:06.920 --> 00:12:11.320
We talk about again, we talk about architecture, we talk about MCP server, MCP gateway.

s111
00:12:11.320 --> 00:12:12.400
Now, it's a monitoring.

s112
00:12:12.400 --> 00:12:17.560
Again, the AI agent cannot be trusted without continuous monitoring, evaluation, and feedback, right?

s113
00:12:17.560 --> 00:12:21.240
This is why we are building the MCP gateway uh monitoring.

s114
00:12:21.240 --> 00:12:30.000
So, the way the monitoring work is usually like the moment you submit any calls to your agent, uh it will send all the operational log into the log store.

s115
00:12:30.000 --> 00:12:33.120
You can use ELK, Loki, or CloudWatch also, right?

s116
00:12:33.120 --> 00:12:40.280
And generally, this contains the information like show me like all last 401s in my last 10 minutes call, right?

s117
00:12:40.280 --> 00:12:44.040
And what did OAuth do when the token refresh failed, right?

s118
00:12:44.040 --> 00:12:46.920
So, those kind of requests can goes to the log store.

s119
00:12:46.920 --> 00:12:53.880
Now, there are some, you know, traces and metrics data um like real-time data that goes to the OpenTelemetry as well.

s120
00:12:53.880 --> 00:13:05.200
Um OpenTelemetry again is open source solutions, which going to collect all your traces um and metrics and it's going to process the data and redirect to the back-end system, right?

s121
00:13:05.200 --> 00:13:07.400
And in this case, the traces will go to the Tempo.

s122
00:13:07.400 --> 00:13:18.120
This is again an open source solutions, which going to store all your traces in terms of like traces in the sense like, you know, like information around like which tool was slow on this request,

s123
00:13:18.120 --> 00:13:24.960
whether this call is coming from agent or IDE, uh what was the exact error um on this failed call or so, right?

s124
00:13:24.960 --> 00:13:28.840
So, these kind of data are going to store into the Tempo.

s125
00:13:28.840 --> 00:13:29.320
My bad.

s126
00:13:29.320 --> 00:13:30.080
Okay.

s127
00:13:30.080 --> 00:13:42.160
And then the metrics is about the Prometheus is all about the information about like what are P95 latencies for a given tool, right?

s128
00:13:42.160 --> 00:13:47.160
Whether my auth rejection is spiking on a particular time or not, right?

s129
00:13:47.160 --> 00:13:50.240
So, those kind of metrics is going to store in Prometheus.

s130
00:13:50.240 --> 00:14:00.600
So, once you get all these data point like traces and metrics, this entire aggregate data actually save into the graph now for your visualizations.

s131
00:14:01.800 --> 00:14:02.160
All right.

s132
00:14:02.160 --> 00:14:06.080
So, the MCP server gateway and monitoring is covered.

s133
00:14:06.080 --> 00:14:09.000
Now, we're going to talk about the framework.

s134
00:14:09.000 --> 00:14:10.760
So, the agent framework, right?

s135
00:14:10.760 --> 00:14:20.400
So, we use LangGraph for most of the cases and is basically LangGraph again it's a provided by LangChain community.

s136
00:14:20.400 --> 00:14:29.640
It's provide a graph based architecture that support loops, branching, state management for complex multi-agent AI workflows, right?

s137
00:14:29.640 --> 00:14:43.000
And the way it work is like the moment you submit your first query, we do interpret you know, like your question first, like what type of questions it is and what response is looking for

s138
00:14:43.000 --> 00:14:46.640
and based on that it's create a plan, right?

s139
00:14:46.640 --> 00:14:57.920
Then it will send for classify classify area where it will determine that hey, whether it is just like a read operations or whether it is just like a you know, modification level changes

s140
00:14:57.920 --> 00:15:01.040
coming up or whether you really want to delete something, right?

s141
00:15:01.040 --> 00:15:02.800
Or some sensitive operations.

s142
00:15:02.800 --> 00:15:08.280
Based on that it adjust the overall structure and then plan accordingly.

s143
00:15:08.280 --> 00:15:14.680
The next step is plan is actually itself is a sub graph where you do the major of your operations.

s144
00:15:14.680 --> 00:15:23.400
Meaning that in the plan like once you know what operation you're performing, it goes to the like let's say in a fetch evidence section where it actually call that

s145
00:15:23.400 --> 00:15:34.680
whatever the question you ask here, after all interpret and classify, um it actually go ahead and make the call to the MCP gateway, uh make sure they get authenticated,

s146
00:15:34.680 --> 00:15:42.960
um and then it make the call to the MCP server using the right tool, um and get the data, and then generate that evidence, right?

s147
00:15:42.960 --> 00:15:46.840
Once you validate all your finding, then it will check the policies.

s148
00:15:46.840 --> 00:15:53.760
Uh it will check whether it is a read-only, then go ahead and execute, and I will save all the record points into our audit logs.

s149
00:15:53.760 --> 00:16:00.560
But if it is a right and sensitive operations, definitely it will go through the HITL process where you wait for an approval.

s150
00:16:00.560 --> 00:16:02.480
If it is approved, you go ahead and execute.

s151
00:16:02.480 --> 00:16:07.120
If it is get rejected, in either case it's going to save into the audit log.

s152
00:16:07.120 --> 00:16:08.640
Right?

s153
00:16:08.640 --> 00:16:17.040
So, if we combine all this together, like MCP gateway, MCP server, um and the agent, right?

s154
00:16:17.040 --> 00:16:18.280
This is how things looks like.

s155
00:16:18.280 --> 00:16:29.200
So, ultimately, whatever the agent you're building, whatever the call uh you're submitting, it will go through the MCP gateway for authentications, and then it will redirect to the right MCP servers

s156
00:16:29.200 --> 00:16:37.440
to execute uh you know, your query on a given tool, uh so that it make the API call to that endpoint and get the response.

s157
00:16:37.440 --> 00:16:38.839
Right?

s158
00:16:38.839 --> 00:16:39.200
All right.

s159
00:16:39.200 --> 00:16:42.839
If it If we think from the infrastructure perspective, this is how it looks like.

s160
00:16:42.839 --> 00:16:49.800
So, agent is communicating to the MCP gateway, goes to the MCP server, and to the endpoint.

s161
00:16:49.800 --> 00:16:50.960
All right?

s162
00:16:50.960 --> 00:16:59.360
So, now the uh we talked a lot about MCP gateway, MCP servers, um the monitoring, um and the agent framework.

s163
00:16:59.360 --> 00:17:07.520
Now we talk about uh three and four additional framework that we should have to take care of when building the end-to-end ecosystem, right?

s164
00:17:07.520 --> 00:17:10.240
One is them is agent memory management.

s165
00:17:10.240 --> 00:17:19.920
Again, it's a uh you know, it's a it's refer It's basically a process where you can save and restore the state of your workflow over the time, right?

s166
00:17:19.920 --> 00:17:26.839
Ensuring that it's provide you a durability and seamless recovery across across your every query run.

s167
00:17:26.839 --> 00:17:34.720
So, the way it work is like whenever you submit any query to your agent, it actually generate a thread, right?

s168
00:17:34.720 --> 00:17:40.880
On that given thread, when you ask your first question, it will going to create a run ID for that.

s169
00:17:40.880 --> 00:17:48.800
So, the moment you ask some deeper questions, it will go on a particular node in that langraph and execute that query.

s170
00:17:48.800 --> 00:17:53.640
And on that node, it going to save that respective checkpoint here, right?

s171
00:17:53.640 --> 00:17:58.520
Now, if you ask a follow-up questions on the same thread, it going to create another run ID.

s172
00:17:58.520 --> 00:18:04.840
And again, depend on your follow-up question, it go to a specific node and save the checkpoint over there, right?

s173
00:18:04.840 --> 00:18:13.960
And all this data for your local development, you can use the SQLite SQLite DB, but in production, unless in GCP, you can use Cloud Spanner.

s174
00:18:13.960 --> 00:18:20.280
And in respective databases, you can use in Azure Azure and AWS as well, right?

s175
00:18:20.280 --> 00:18:25.120
So, just walk you through the quick demo here on the memory.

s176
00:18:25.120 --> 00:18:36.000
So, for example, I have, you know, like just build this agent locally and then, you know, trying to run this query on SQLite DB to see if there is any data or not.

s177
00:18:36.000 --> 00:18:39.200
So, I'm just, you know, initially I don't have any data.

s178
00:18:39.200 --> 00:18:44.560
Now, I submit my query, you will see that I received the one record, which is saying that, "Hey, this is my thread ID.

s179
00:18:44.560 --> 00:18:46.520
This is the question I have been asked."

s180
00:18:46.520 --> 00:18:50.840
And exact time, you know, when this query was executed.

s181
00:18:50.840 --> 00:18:56.760
Now, if I run the next query to see that, "Hey, how many messages I've been produced on a given thread?"

s182
00:18:56.760 --> 00:19:01.280
You can get this information by running the further queries, right?

s183
00:19:01.280 --> 00:19:11.480
Now, if I would like to run to see that, you know, like all the metadata level information on my query, I can run my query again and And can see the detail

s184
00:19:11.480 --> 00:19:20.080
description around, you know, like how many tool I get invoked, what is my thread ID, what is my run ID, um you know, like and all those informations.

s185
00:19:20.080 --> 00:19:29.680
Now, if I want to also see that, hey, what tool have been executed when I run this query, um you can go ahead and see the exact tool name as well on that given MCP servers.

s186
00:19:29.680 --> 00:19:40.240
Now, further I will also want to see that how the progression was happened on my lang graph framework, that also you can see by um running this type of query on your memory DB.

s187
00:19:40.240 --> 00:19:48.400
You can see that here like different kind of, you know, nodes been called when actually, um you know, that query has been run.

s188
00:19:48.680 --> 00:19:50.000
All right?

s189
00:19:50.000 --> 00:19:51.880
Now, the tracking and eval.

s190
00:19:51.880 --> 00:20:00.960
So, just to make sure that um there is no confusion here like the we talked about earlier the MCP gateway monitoring, this is the agent level tracking.

s191
00:20:00.960 --> 00:20:13.240
So, we do this because we really want to track that what query you have been submitting to the agent um and what the response is coming, what alarm model you have been used, what's the cost of your token,

s192
00:20:13.240 --> 00:20:19.080
um how many token you have been burned, and what's your, you know, evaluation criteria looks like.

s193
00:20:19.080 --> 00:20:21.920
What is the quality of your response or not, right?

s194
00:20:21.920 --> 00:20:25.000
So, you need to store that data from the day one.

s195
00:20:25.000 --> 00:20:37.360
And imagine that if you have a thousand plus agent, every agents will have their own um you know, um agent tracking and eval um project and the tables, right?

s196
00:20:37.360 --> 00:20:45.280
So, if you combine, let's say if you have 100 agents, you will have 100 different projects and each project has their own respective tracking and eval tables.

s197
00:20:45.280 --> 00:20:58.360
Now, tomorrow, 6 months down the line, if somebody ask you that how your agent has been performed, what was the response, what's was the quality, how much token you have been burned, you can aggregate all this data and you can create executive dashboards around it to see

s198
00:20:58.360 --> 00:21:03.040
um with to see the what is the ROI in building so many agents, right?

s199
00:21:03.040 --> 00:21:07.720
So, let me walk you through the the tracking and eval um demo here.

s200
00:21:07.720 --> 00:21:17.600
So, um here like if I look at uh this one, so first like, you know, like if you see that there are six tables are listed here.

s201
00:21:17.600 --> 00:21:27.360
One is a job tracking table, um and which is contains most of this like information around what query like what is your query, what's the response and everything.

s202
00:21:27.360 --> 00:21:35.760
And there are a bunch of other tables as well like your quality related and your tool invocations and then node level executions and all that.

s203
00:21:35.760 --> 00:21:46.960
So, in this case I'm going to submit my first query on my agent um and you will see that um there will not be uh any data in the first place uh because it takes some time.

s204
00:21:46.960 --> 00:21:49.880
So, I will refresh this again.

s205
00:21:50.000 --> 00:22:00.160
And this time I will see that when I refresh, I do see that for a given query I get my request ID, my session ID, my agent name, um you know, what question I've been asked,

s206
00:22:00.160 --> 00:22:10.440
um and what was the output um of that uh question, uh whether it has completed or not, and then what is the time it took, what is the model I've been used,

s207
00:22:10.440 --> 00:22:15.560
um and then some of the middle level uh middle level informations, right?

s208
00:22:15.560 --> 00:22:20.880
Further, if I look for the quality, again there is a eight quality checks per run, right?

s209
00:22:20.880 --> 00:22:33.960
And you can see that this one row each for, you know, like success, like latency, cost, tool accuracy, safety, etc. Like all this has been stored here, right?

s210
00:22:33.960 --> 00:22:35.480
Now, the eval scores.

s211
00:22:35.480 --> 00:22:38.240
This is again this going to be a one row.

s212
00:22:38.240 --> 00:22:48.360
This is going to be a scoreboard for your given run, which is basically varies between zero to one, which is scores uh you know, like all the quality dimension in a single row,

s213
00:22:48.360 --> 00:22:49.200
right?

s214
00:22:49.200 --> 00:22:50.400
Now, the quality metrics.

s215
00:22:50.400 --> 00:22:53.360
So, again it's a deep dive on that particular one run.

s216
00:22:53.360 --> 00:23:02.760
Basically a nested field for the response quality, safety, latency, cost, uh and the tool stat in one white row, right?

s217
00:23:02.760 --> 00:23:11.240
Now, the tool invocations, so every tool call, which MCP tool ran, how long it took, and whether it succeeded or not.

s218
00:23:11.240 --> 00:23:11.840
Right?

s219
00:23:11.840 --> 00:23:25.080
And now the tool agent trajectory, which is talk about step-by-step story of the run, each LLM call, and the graph state in order, like plan, summarize, etc. All right.

s220
00:23:25.080 --> 00:23:30.480
So, now uh the next is the agent PIR treatment.

s221
00:23:30.480 --> 00:23:40.640
This is very, very important because many times it happens that when you're working in agents or uh this thing, um sometimes there's a possibility that there are some secrets being ingested

s222
00:23:40.640 --> 00:23:42.040
as a part of your query, right?

s223
00:23:42.040 --> 00:23:50.080
Let's say you're uploading some doc which is has some sensitive informations um and sometimes while typing you may have some API keys or so.

s224
00:23:50.080 --> 00:23:56.280
So, we have a framework which define that different sort of data types and their severity level.

s225
00:23:56.280 --> 00:24:05.000
If your severity is above 90% or so, uh we will block and redact that uh secrets uh immediately.

s226
00:24:05.000 --> 00:24:09.920
So, that uh make sure that that secret is never goes to LLM models, right?

s227
00:24:09.920 --> 00:24:18.000
Neither it will goes to your memory DB, or nor it will go to tracking and eval tables, or tracing uh systems as well.

s228
00:24:18.000 --> 00:24:23.840
So, the goal here is to just block this sensitive data as it is the moment it get ingested, right?

s229
00:24:23.840 --> 00:24:26.120
Moment moment you type.

s230
00:24:26.120 --> 00:24:27.000
All right.

s231
00:24:27.000 --> 00:24:32.080
Um the last framework is all about now is the context optimizer.

s232
00:24:32.080 --> 00:24:39.640
This is one of my favorite area where like um you know, uh whenever you submit, it's very important, right?

s233
00:24:39.640 --> 00:24:48.000
Whenever you submit too many queries, sometimes you upload a lot of the um data points and you upload bunch of file systems or so, right?

s234
00:24:48.000 --> 00:24:54.160
It's very important that time is um you know, like you make sure that you save your token in every query.

s235
00:24:54.160 --> 00:25:00.680
So, we build in context optimizer framework where you actually shape your query and then you prune it before it go to LLM.

s236
00:25:00.680 --> 00:25:07.440
In the pruning system, uh we do select the payload based on the token limit.

s237
00:25:07.440 --> 00:25:15.360
Then we re-rank that, "Hey, which is the most relevant relevant information that we should target for when you submit this huge giant, you know, context?"

s238
00:25:15.360 --> 00:25:26.360
And then we dedupe that, uh remove all the duplicates, and create a very unique set of uh very curated data like uh I mean the main data for the LLM.

s239
00:25:26.360 --> 00:25:30.280
And then we finally compress this large payload before it goes to the LLM.

s240
00:25:30.280 --> 00:25:33.600
So, let's see this uh demo here.

s241
00:25:33.600 --> 00:25:34.760
Uh this will be quick.

s242
00:25:34.760 --> 00:25:40.960
So, for example, let's say I will submit my first query, "Hey, find me all the artifacts in my JFrog Artifactory."

s243
00:25:40.960 --> 00:25:46.320
Um you see that first when before even I not submitted any query yet, there is no data here.

s244
00:25:46.320 --> 00:25:57.840
So, the moment I submit this first query, and if I refresh my query here, um and then you can see that the record has been added.

s245
00:25:57.840 --> 00:26:00.840
So, you can see there is a 24,000 token has been used.

s246
00:26:00.840 --> 00:26:05.320
I will go back a little bit um just to show the thing.

s247
00:26:05.320 --> 00:26:09.080
So, you can see this 24,000 token has been used for given questions.

s248
00:26:09.080 --> 00:26:16.800
Now, if I ask uh now I enable the context optimizer behind the scene and I'm rerun the exact same query again.

s249
00:26:16.800 --> 00:26:25.720
So, this time you will see that uh you know, for the same query, if I refresh this run, and you will see that uh this time it used just 2,400

s250
00:26:25.720 --> 00:26:26.840
token.

s251
00:26:26.840 --> 00:26:27.440
Right?

s252
00:26:27.440 --> 00:26:30.120
For exact same query.

s253
00:26:30.120 --> 00:26:32.360
So, which is almost like 90% saving.

s254
00:26:32.360 --> 00:26:41.200
I can rerun the same demo on the Slack bot as well where I'm asking Slack bot, "Hey, tell me all my Docker images across all the repositories in Artifactory."

s255
00:26:41.200 --> 00:26:43.360
When I run this query, there is no data.

s256
00:26:43.360 --> 00:26:46.160
Now, I'm going to submit my query.

s257
00:26:46.160 --> 00:26:53.440
So, once this query will run, um and if I run my query again here, and you will see the record get created.

s258
00:26:53.440 --> 00:26:57.760
It's again used like 24,000 for this this particular query.

s259
00:26:57.760 --> 00:27:04.360
Now, what I will do is I will turn on my optimizer behind the scene and I rerun the same query in the Slack bot.

s260
00:27:04.360 --> 00:27:13.600
Um and then if I run this query again, um it will show like it exactly used 1,500 almost token, which is again like 90% saving.

s261
00:27:13.600 --> 00:27:17.560
In most of the cases, we get like 70 to 78% token saves.

s262
00:27:17.560 --> 00:27:33.360
So, imagine that you have thousands of um you know, like um um if you have thousands of like agents, which is running like you know, many queries per day, you imagine how much token you can save with this framework.

s263
00:27:33.360 --> 00:27:33.680
All right.

s264
00:27:33.680 --> 00:27:43.720
So, now the key takeaway here is uh you know, like um for us is like make sure that we build an enterprise framework, um not just an agent, right?

s265
00:27:43.720 --> 00:27:46.120
Which make sure that you have all the framework underneath.

s266
00:27:46.120 --> 00:27:49.400
Don't try to be in rush to building some agent quickly and deploy ship it.

s267
00:27:49.400 --> 00:27:59.240
Make sure that it has to go through all the framework all together, so that you should not realize a letter state that, you know, I should have this framework in place.

s268
00:27:59.240 --> 00:27:59.520
Right?

s269
00:27:59.520 --> 00:28:01.240
In in the very first place.

s270
00:28:01.240 --> 00:28:04.880
Second is like your Make sure that you have a proper gateway system.

s271
00:28:04.880 --> 00:28:14.120
Make sure that every tool call request either you are doing this from your local or through agent, it has to go through some sort of centralized MCP gateway.

s272
00:28:14.120 --> 00:28:18.560
Uh make sure it authenticated before it reaching to the MCP servers.

s273
00:28:18.560 --> 00:28:28.920
And then make sure that your MCP servers take a control on what tool um you know, uh you should use uh to execute a particular query, right?

s274
00:28:28.920 --> 00:28:33.840
And make sure that you have monitoring in place and complete auditing in place for that tool executions.

s275
00:28:33.840 --> 00:28:42.360
And of course, you should um you know, keep the always human in the loop process for every um every action items, every high-risk items.

s276
00:28:42.360 --> 00:28:48.560
Um make sure that there is a observability um and you know, um auditing has to be in place.

s277
00:28:48.560 --> 00:28:57.600
And last but not least, make sure that context optimizer should always be there to justify your cost on your agents, right?

s278
00:28:57.600 --> 00:29:03.760
So, just want to share is that I released my MCP book recently on Amazon.

s279
00:29:03.760 --> 00:29:04.920
So, feel free to download.

s280
00:29:04.920 --> 00:29:06.960
It's free in Kindle.

s281
00:29:06.960 --> 00:29:11.000
And it's hard copy is also available.

s282
00:29:11.000 --> 00:29:12.080
All right.

s283
00:29:12.080 --> 00:29:16.440
Thank you so much for watching this video.

s284
00:29:16.440 --> 00:29:23.440
And if you do have any questions, feel free to, you know, like reach out to me over LinkedIn or Twitter.

s285
00:29:23.440 --> 00:29:25.000
I'm happy to answer.

s286
00:29:25.000 --> 00:29:27.280
Thank you again.