WEBVTT

NOTE Sentence-level transcript of https://www.youtube.com/watch?v=Akm1sqvWG4A

NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/Akm1sqvWG4A.html. A cue ends where the next begins, or 2 s after its last word.

s1
00:00:01.760 --> 00:00:03.680
Hello everyone.

s2
00:00:03.680 --> 00:00:05.480
I'm Abid Matini.

s3
00:00:05.480 --> 00:00:09.680
I'm senior backend developer at OGOV.

s4
00:00:09.880 --> 00:00:17.960
Thanks to AI Engineer World Fair 2026 for giving me a time slot for online track.

s5
00:00:17.960 --> 00:00:24.360
And I'm going to walk you through bypassing the multimodal text today.

s6
00:00:24.360 --> 00:00:28.960
How we can have a have a framework free hybrid rag.

s7
00:00:28.960 --> 00:00:32.720
Withdraw SQL RF and live telemetry.

s8
00:00:32.720 --> 00:00:39.400
As you can see, I have got my live demo in the right side of the screen.

s9
00:00:39.400 --> 00:00:39.955
So,

s10
00:00:39.955 --> 00:00:40.640
[snorts]

s11
00:00:40.640 --> 00:00:50.520
any slide that I pass and if there is anything related related to my presentation, I can show you.

s12
00:00:50.520 --> 00:01:00.560
I have few tabs also ready here for you to show and walk you through uh the slides and the demos.

s13
00:01:04.839 --> 00:01:07.480
Okay, let's begin.

s14
00:01:07.960 --> 00:01:17.600
So, we have two problems for every document we have two problems that we try to solve today.

s15
00:01:17.600 --> 00:01:34.680
One is when we start to chat with any LLM and if we have to upload a document, we usually going to just drag and drop a PDF or a Word doc or an image and ask the chatbot to get ready for the questions that going to come up

s16
00:01:34.680 --> 00:01:36.760
after we upload those documents.

s17
00:01:36.760 --> 00:01:53.960
The issue with that is as soon as we upload these documents, we going to basically lose some of our tokens that we supposed to have only for uh processing these documents without even asking a question.

s18
00:01:53.960 --> 00:02:01.040
So, we already uh spend some tokens there without even asking any questions.

s19
00:02:01.040 --> 00:02:03.560
That would be the first issue.

s20
00:02:03.560 --> 00:02:15.040
And then, if we're building a chatbot and it's uh we want to have vector database there, we want to have keyword search and semantic search there, and we want to just add more tools.

s21
00:02:15.040 --> 00:02:19.800
This is going to be too many tools uh uh combined all together.

s22
00:02:19.800 --> 00:02:35.800
So, in the production uh in the production basically chatbot, we're going to have uh too many tools, and it's going to be complicated for us to manage our production chatbot properly.

s23
00:02:35.800 --> 00:02:44.760
So, today I'm going to uh walk you through in uh three main uh sections.

s24
00:02:44.760 --> 00:02:56.200
One is how we can uh upload our documents and get our documents and make it ready bef- uh for our chatbot before uh users going to chat.

s25
00:02:56.200 --> 00:03:06.840
So, on the on the right side, I have I do have a FAQ assistant chatbot that's running locally now for me.

s26
00:03:06.840 --> 00:03:15.720
And um I'm going to show you the dashboard, the dashboard that I've created for it, and how we can chat, and uh how we get the results.

s27
00:03:15.720 --> 00:03:26.480
So, this is going to be a uh supposedly FAQ um assistant for a sample employee handbook.

s28
00:03:26.480 --> 00:03:39.560
So, imagine that you have HR company, and this HR has uploaded the the the the handbook for uh for uh for the employees to ask their question, any question they have about

s29
00:03:39.560 --> 00:04:00.720
leave, about uh uh like uh different kinds of leaves, and sickness, or uh parent like uh parental leave, or any other questions related to our, uh uh, possibly an HR would have as a common question that they get every day.

s30
00:04:00.720 --> 00:04:06.520
So, um, that would be one sample that we're going to use to upload.

s31
00:04:06.520 --> 00:04:18.760
So, all documents can get can get in the different formats or we can upload in different formats of PDF, uh, like PowerPoint documents and Word documents or even image.

s32
00:04:18.760 --> 00:04:20.760
So, we're going to talk about that.

s33
00:04:20.760 --> 00:04:25.640
We're going to talk about how we have to chunk these because we're talking about drag here.

s34
00:04:25.640 --> 00:04:32.080
We need to know how we can chunk this information uh, properly based on our the documents.

s35
00:04:32.080 --> 00:04:37.600
So, we're going to talk about uh, different type of chunking here as well.

s36
00:04:37.600 --> 00:04:48.919
We can, uh, we show you in the admin how we can choose different uh, different strategies to uh, to chunk the documents.

s37
00:04:48.919 --> 00:04:57.160
Then we're going to see how the search is going to work, how the embedding is going to work, how did our database is going to be, how the search going to be based on the

s38
00:04:57.160 --> 00:05:00.240
keyword search or the semantic search.

s39
00:05:00.240 --> 00:05:06.080
And we're going to see about the how our F in the Python going to work.

s40
00:05:06.080 --> 00:05:16.400
And then we're going to talk about the observability, how we going to check the safety, how we going to observe, uh, all the process with LangFuse and how, uh,

s41
00:05:16.400 --> 00:05:27.160
we can the prevent, uh, prompt injection pattern or, um, any risky question that might come to LLM.

s42
00:05:27.160 --> 00:05:42.040
I'm planning this talk for about under an hour and this, uh, the whole thing, the plan is the entire code base can be easily will be running on the GitHub code code space which you can just pull from the repo and

s43
00:05:42.040 --> 00:05:50.240
one command run and it's going to be ready to test on GitHub code spaces.

s44
00:05:51.960 --> 00:06:03.920
Um so, the stack we're using here are Python, FastAPI for the back end, React for the front end, uh PostgreSQL for our database.

s45
00:06:03.920 --> 00:06:15.600
Uh Docker for just easily being able to have the containers and we can just re- re- produce it anywhere and rerun it anywhere, for example, in code spaces or

s46
00:06:15.600 --> 00:06:19.640
code spaces or any other uh environment we want.

s47
00:06:19.640 --> 00:06:30.560
Then, we have Ollama and local language models and local language models and embedding models that running only locally that can be run on any uh server as well.

s48
00:06:30.560 --> 00:06:32.000
It doesn't need a GPU.

s49
00:06:32.000 --> 00:06:34.760
The CPU is going to be good enough for it.

s50
00:06:34.760 --> 00:06:43.320
So, uh any staging server, any code space uh uh going going to be fine for it.

s51
00:06:43.320 --> 00:06:57.000
And LangFuse for uh tracking what's going on and how we can see the chat and the latency uh so, we can improve our chats better.

s52
00:06:57.160 --> 00:07:03.320
Um This is the end-to-end blueprint of uh what we're going to talk about.

s53
00:07:03.320 --> 00:07:12.240
So, we're going to have a document few documents as a source of truth for our chatbot.

s54
00:07:12.240 --> 00:07:15.440
So, we're going to upload some documents.

s55
00:07:15.440 --> 00:07:25.520
So, what we can do with Python is we're using Doc Link to convert those uh raw documents uh to markdown files.

s56
00:07:25.520 --> 00:07:31.840
And then, we're going to chunk those markdown files to uh in our into our uh PostgreSQL.

s57
00:07:31.840 --> 00:07:35.360
And then, we can also do some um scan as well here.

s58
00:07:35.360 --> 00:07:37.760
So, uh that would happen as well.

s59
00:07:37.760 --> 00:07:40.800
Then, we would have some queries from the user.

s60
00:07:40.800 --> 00:07:46.680
So, the employee is asking questions, HR-related questions, and it's going to be retrieved from the database.

s61
00:07:46.680 --> 00:07:52.200
Is the the the result relevant results with the top rank would be passed through the LLM.

s62
00:07:52.200 --> 00:07:58.760
And LLM would uh return the in the in our chatbots that we have here.

s63
00:07:58.760 --> 00:08:09.920
Uh in the format of chat, so we can see and get our questions answered here, and we can observe in the length views what's going on in our system.

s64
00:08:11.280 --> 00:08:16.200
So, um optimizing data ingest.

s65
00:08:16.200 --> 00:08:19.320
So, we have two different ways of optimizing.

s66
00:08:19.320 --> 00:08:34.159
So, if we first uh going to just throw it into the We have a document, and if we just uploading it on our um uh cloud chatbot anywhere, the issue we just

s67
00:08:34.159 --> 00:08:39.800
uh having of the one issue we have is we're spending tokens before we doing anything.

s68
00:08:39.800 --> 00:08:50.520
Another issue is you don't know how that document looks li- looks like there and that chunking uh how it's being chunked and how uh basically that LLM is seeing it.

s69
00:08:50.520 --> 00:08:52.360
So, that would be a risk quality.

s70
00:08:52.360 --> 00:08:57.000
If if there is a table that we don't know how it's being read there, so we kind of don't see.

s71
00:08:57.000 --> 00:09:06.400
We're trying to uh have more clarity about how the how the document's being stored in our database and how it's being retrieved.

s72
00:09:06.400 --> 00:09:12.320
So, another way would be a structure first, so we use the local CPU.

s73
00:09:12.320 --> 00:09:24.000
If you're running it here, we we can uh upload a document or doc link would convert it to uh markdown file, and we would know the better chunk uh how the chunking is going to work.

s74
00:09:24.000 --> 00:09:26.080
I'm going to show you a few examples.

s75
00:09:26.080 --> 00:09:29.040
And uh of course, it's going to be a cheaper prompt.

s76
00:09:29.040 --> 00:09:39.200
If after that you want to talk to a lo- um an online cloud any- but if you want to If you want to have it locally, then it would be actually no cost

s77
00:09:39.200 --> 00:09:40.440
for you.

s78
00:09:40.440 --> 00:09:51.560
Uh, so the DocLink pipeline would be uh, we would be running it It's like a local locally running it on our server.

s79
00:09:51.560 --> 00:09:55.040
The The The Python is going to get that document.

s80
00:09:55.040 --> 00:10:00.280
It's going to export it to the markdown and save that markdown file.

s81
00:10:00.280 --> 00:10:14.320
Then the get this going to chunk it based on different strategy and uh, we're using embedding models to embed it to our database and insert that on to our

s82
00:10:14.320 --> 00:10:16.120
vector database.

s83
00:10:16.120 --> 00:10:29.120
Uh, so if we save while local because uh, imagine if if one or two pages the cost is next to nothing, but if you have 200 pages uh, then the cost going to be too much.

s84
00:10:29.120 --> 00:10:43.000
And if you have a big company with lots of employees and they want to just us keep asking the same questions or uh, if they want to upload documents then it's going to be

s85
00:10:43.000 --> 00:10:46.120
uh, going to be a big cost.

s86
00:10:46.120 --> 00:10:50.839
So, um, there are four ways of chunking a document.

s87
00:10:50.839 --> 00:10:58.839
Let me first before I show you the four ways, let me uh, show you the admin page that what I'm talking about.

s88
00:10:58.839 --> 00:11:05.240
So, imagine here you have We are We're having our FAQ admin page.

s89
00:11:05.240 --> 00:11:16.000
So, uh, imagine you have Let me delete few of the documents that I have chunked before here.

s90
00:11:16.000 --> 00:11:25.920
So, uh, you want to upload a new FAQ uh, file for your HR as for for your HR policy.

s91
00:11:25.920 --> 00:11:33.040
I want to show uh, give your chatbot uh, data updated with our AI system.

s92
00:11:33.040 --> 00:11:42.080
So, imagine you have um you have some question answers here.

s93
00:11:42.080 --> 00:11:46.360
Uh before I do that, let me show you here what I mean.

s94
00:11:46.360 --> 00:11:46.753
So,

s95
00:11:46.753 --> 00:11:47.480
[snorts]

s96
00:11:47.480 --> 00:11:50.400
um this is the heading FAQ, yeah.

s97
00:11:50.400 --> 00:12:01.640
So, one way that you can feed our chatbot with information is if we actually create a question answer uh file.

s98
00:12:01.640 --> 00:12:07.840
So, that would be easier to digest for our uh system to understand what's going on.

s99
00:12:07.840 --> 00:12:13.480
So, part of that is uh imagine we're having a this the entire FAQ file.

s100
00:12:13.480 --> 00:12:18.840
So, the entire uh basically handbook.

s101
00:12:18.840 --> 00:12:26.200
If we want to chunk this uh handbook and just upload it, imagine uh we're uploading the entire PDF.

s102
00:12:26.200 --> 00:12:30.760
This is going to be the way our PDF is going to look like after chunking.

s103
00:12:30.760 --> 00:12:37.440
So, it started by acknowledgement, then they see a few chunk that doesn't have any meaning.

s104
00:12:37.440 --> 00:12:40.040
And then something starts here.

s105
00:12:40.040 --> 00:12:49.440
There are some chunks that doesn't have any Uh for example, here signature date, that's basically kind of useless.

s106
00:12:49.440 --> 00:12:54.080
So, this going to slow down and reduce the accuracy of our chat.

s107
00:12:54.080 --> 00:12:56.000
It's going to increase hallucination.

s108
00:12:56.000 --> 00:13:16.400
So, um instead of uploading uh straight uh 28 pages, we can if we can divide our documents to different chunks and just to different files, relevant files, and clean up that data a bit as much as we can,

s109
00:13:16.400 --> 00:13:19.560
uh and then upload it, then we will have a better result.

s110
00:13:19.560 --> 00:13:31.320
So, uh one way of uh uploading in the FAQ case is to actually have your your turning your data to question and answers.

s111
00:13:31.320 --> 00:13:41.520
So, uh So, talking about working hours, talking about different situations here, so you can have it based on questions and answers.

s112
00:13:41.520 --> 00:13:52.200
Then this way you know when you reference your documents, you're having the uh correct reference, and it won't be any confusing.

s113
00:13:52.200 --> 00:13:57.560
And there are uh different way to uh upload your documents.

s114
00:13:57.560 --> 00:14:03.240
One way is you have the question and answer, and then you would like to upload it.

s115
00:14:03.240 --> 00:14:05.800
This is my FAQ PDF file.

s116
00:14:05.800 --> 00:14:14.080
Uh same PDF file, and I'd like to upload it based on uh heading-based chunking strategy.

s117
00:14:14.080 --> 00:14:20.800
That means well, while I'm explaining the the parsing where Docling is happening in the background.

s118
00:14:20.800 --> 00:14:29.400
So, that means each uh chunking would be based on the heading that Docling would find.

s119
00:14:29.400 --> 00:14:31.440
And the answer that comes based on that.

s120
00:14:31.440 --> 00:14:38.360
So, each heading and answer is going to heading as a question and answer is going to become a chunk.

s121
00:14:38.360 --> 00:14:46.000
So, we know our chunking is clean, and it's easy to uh reference.

s122
00:14:46.040 --> 00:14:57.880
So, when this chunking finishes in a seconds, we would see, okay, this is the first FAQ, it could chunk successfully to uh based on the headings.

s123
00:14:57.880 --> 00:15:05.720
So, it recognized the heading question, uh and then the relevant answer.

s124
00:15:05.720 --> 00:15:10.320
So, each chunk has its own uh question and answer here.

s125
00:15:10.320 --> 00:15:20.000
So, if we say uh if we say, for example, uh

s126
00:15:20.000 --> 00:15:36.640
Um as our questions here Uh do we do we have or something like this.

s127
00:15:36.640 --> 00:15:42.360
So, we should expect a kind of answer with the reference that's coming to that chunk.

s128
00:15:42.360 --> 00:15:50.320
So, we talking about heading chance going to understand the documents based on uh headings that you have.

s129
00:15:50.320 --> 00:15:54.120
That's how it's going to chunk each document heading by heading.

s130
00:15:54.120 --> 00:16:04.800
So, as you can see here, um our question was about how much This was the question we prepared, and this was the answer we prepared.

s131
00:16:04.800 --> 00:16:14.360
Uh sample Coca-Cola 90% of the individual employee premium and 75% dependent premium.

s132
00:16:14.360 --> 00:16:30.320
What we see here as a reference, we getting that answer, and not only that, you getting the correct answer, you also getting the Uh you see that it's been this answer has been generated based on this document, based on this chunk.

s133
00:16:30.320 --> 00:16:31.800
So, it's easy to reference.

s134
00:16:31.800 --> 00:16:37.080
It's easy, you know, where the issue is if there is anything that's not working, you know why.

s135
00:16:37.080 --> 00:16:41.760
Then you can go track it why this right chunk hasn't been retrieved.

s136
00:16:41.760 --> 00:16:49.360
So, in this way you have more uh more uh clear path towards how you can debug it.

s137
00:16:49.360 --> 00:17:02.520
Because I was uh in our system, I was asking for um return top two chunks, it would uh suggesting me I'll talk about all these models here.

s138
00:17:02.520 --> 00:17:05.839
Um it would bring top two response.

s139
00:17:05.839 --> 00:17:16.079
So, then it was also checking that sample employee handbook that big file here, sample employee handbook.

s140
00:17:16.079 --> 00:17:18.959
As you can see here, that's a big chunk.

s141
00:17:18.959 --> 00:17:30.600
And if actually it's a good example because if you compare these two, uh because we just drag uh uploaded the entire chunk here, you can see although it's returned some information,

s142
00:17:30.600 --> 00:17:35.440
but it's not that accurate and you don't know.

s143
00:17:35.440 --> 00:17:46.760
Of course, it went through that file and found something, but uh you don't know how accurate that information could be and you can't follow because the chunk is is just

s144
00:17:46.760 --> 00:17:50.440
uh uh untrackable that easily.

s145
00:17:50.440 --> 00:17:57.040
So, one way of chunking is chunking based on the headings.

s146
00:17:57.040 --> 00:17:59.680
Second way of chunking is based on the paragraph.

s147
00:17:59.680 --> 00:18:11.200
So, technically when you upload your documents, you would say uh just chunk each paragraph you catch, have it as a separate chunk.

s148
00:18:11.200 --> 00:18:14.680
Doesn't matter there is a heading or not heading or anything.

s149
00:18:14.680 --> 00:18:25.240
So, uh if I want to go back to the to our to our documents and say, let me first choose I want to do a paragraph chunking.

s150
00:18:25.240 --> 00:18:36.480
I want to say, okay, uh for paragraph um Let me find the paragraph general policy for paragraph.

s151
00:18:36.480 --> 00:18:38.560
I want to upload that.

s152
00:18:38.560 --> 00:18:51.280
Um So, for general policy, it's uploading here as you would see the whole file is just been paragraph by paragraph being chunked.

s153
00:18:51.280 --> 00:18:54.880
So, there is no heading, there is no no separate information.

s154
00:18:54.880 --> 00:19:18.640
And if you look at the uh if you look at that information as well, if I can find it here quickly, um general general paragraph uh policy paragraph as you can see the information is just paragraph by paragraph.

s155
00:19:18.640 --> 00:19:25.080
So, you kind of cleaned up your data again, but you're dividing it based on the different system.

s156
00:19:25.080 --> 00:19:29.160
So, instead of heading you're talking about paragraphs.

s157
00:19:29.160 --> 00:19:36.720
That's another way of uh basically feeding your rag for your information.

s158
00:19:36.720 --> 00:19:40.440
Another way would be a fixed 512 of character.

s159
00:19:40.440 --> 00:19:54.680
You can say, "Okay, one of the best practices for rag is you just divide based on 512 character, and you have 64% overlap between the chunks, so you won't lose the track."

s160
00:19:54.680 --> 00:20:08.240
So, if your data is just too is not organized, if there's just random data that you have and you can't clean that up, that's another way you can upload your document.

s161
00:20:08.240 --> 00:20:26.440
So, that would be instead of this, you can choose another file, and you can say basically there's another paragraph for global travel uh file here.

s162
00:20:26.440 --> 00:20:35.880
So, that's kind of when it's kind of overlap here when it's going to see here that the about this chunking strategy, you would see okay this started for chunking and then

s163
00:20:35.880 --> 00:20:49.640
it ended in the 512, and then it's again uh using that uh I think it was 64 uh I was 64 overlap, yeah.

s164
00:20:49.640 --> 00:20:52.760
So, it's using the 64 overlap here again.

s165
00:20:52.760 --> 00:21:02.640
So, you would see it's basically uh chopped it from here.

s166
00:21:02.640 --> 00:21:15.480
And so, from here to here is 512, and from here to here is 32 then another 32 from here to here.

s167
00:21:15.480 --> 00:21:19.600
So, this is a 32 32 64 in total.

s168
00:21:19.600 --> 00:21:22.080
So, that's how it's relating each chunks together.

s169
00:21:22.080 --> 00:21:24.480
It's happening same for here.

s170
00:21:24.480 --> 00:21:27.480
As you can see it's happening same for here.

s171
00:21:27.480 --> 00:21:35.640
It's the way that But you see the issue with that is it's coming here to assist the next system.

s172
00:21:35.640 --> 00:21:36.600
So, it's breaking.

s173
00:21:36.600 --> 00:21:40.720
It's not the best but in some cases it works for you.

s174
00:21:40.720 --> 00:21:47.240
Another way that we can do is sentence base.

s175
00:21:47.240 --> 00:21:51.560
So, it can be based on first counting based on the number of sentences and it chop it there.

s176
00:21:51.560 --> 00:21:53.760
There are different use case for each ones.

s177
00:21:53.760 --> 00:22:00.880
So, another interesting one is imagine that you There's some maintenance coming.

s178
00:22:00.880 --> 00:22:21.240
You're receiving some email and you just want to quickly upload the document in your chat without without cuz you do you don't have the it's just a screenshot and you can't do anything.

s179
00:22:21.240 --> 00:22:34.560
Imagine you just receiving an email and says, "Okay, there is a maintenance plan for this day from this hour and you just wants to upload this in your chat

s180
00:22:34.560 --> 00:22:35.840
for the for the user.

s181
00:22:35.840 --> 00:22:41.200
So, if me as a employee ask, "Okay, what is is there any maintenance plan for this weekend?"

s182
00:22:41.200 --> 00:22:44.080
Then I would know immediately.

s183
00:22:44.080 --> 00:22:50.320
So, another way that we can easily upload that screenshot as well.

s184
00:22:53.040 --> 00:22:58.440
Let me find easily upload that.

s185
00:23:02.880 --> 00:23:03.360
Oh, sorry.

s186
00:23:03.360 --> 00:23:05.640
That wasn't our screenshots.

s187
00:23:05.640 --> 00:23:23.120
So, the screenshots uh you can upload then we're using another um uh model that just turns that uh image to text and text to .md and then we're using

s188
00:23:23.120 --> 00:23:28.800
this uh uh basically uh what's the name?

s189
00:23:28.800 --> 00:23:37.760
Uh the sentence model for the emails that can convert uh that image to a text and then it can feed.

s190
00:23:37.760 --> 00:23:52.280
So, we can see we're using the chunking strategy a sentence group is turned that image to text and it's already uh good for the uh for our chatbot.

s191
00:23:52.280 --> 00:23:56.560
So, we can uh we can index it into our knowledge base now.

s192
00:23:56.560 --> 00:24:01.080
So, then it's going to be a next one that's been uploaded here screenshot.

s193
00:24:01.080 --> 00:24:04.560
We can see how it's been chunked because this is just the email.

s194
00:24:04.560 --> 00:24:05.480
It doesn't have heading.

s195
00:24:05.480 --> 00:24:08.880
It doesn't necessarily have all the paragraph chunk properly.

s196
00:24:08.880 --> 00:24:13.360
So, and if it's a shortage of time it's a small message temporary change.

s197
00:24:13.360 --> 00:24:16.680
You don't need to sit and clean up a data for here.

s198
00:24:16.680 --> 00:24:25.160
You just want to drag and drop updated for a weekend and everybody knows okay uh this weekend we're having a maintenance plan.

s199
00:24:25.160 --> 00:24:35.600
So, uh when they will come here and ask is there any maintenance plan for this weekend?

s200
00:24:35.600 --> 00:24:44.840
Then technically we should get the result of okay, this is the maintenance plan that's been happening and the reference documents.

s201
00:24:44.840 --> 00:24:59.360
So, uh the maintenance uh this is the exact maintenance uh window that we got and actually the reference you can see that's the screenshot that uh we uploaded just now.

s202
00:24:59.360 --> 00:25:11.160
So, if we do if we had I done the screen the file name better, there would have been even a more relevant and somebody says, "Okay, according to this information

s203
00:25:11.160 --> 00:25:19.200
that I got from here, we have a maintenance plan tomorrow from 6:00 p.m. onwards till midnight."

s204
00:25:19.200 --> 00:25:26.800
So, and I can easily archive and delete any of these and then it won't be visible as well.

s205
00:25:26.800 --> 00:25:31.000
So, these are basically different ways that you can chunk.

s206
00:25:31.000 --> 00:25:40.240
That's one of the that's basically the one of the most important part of the whole system because most of the LLMs would just work.

s207
00:25:40.240 --> 00:25:52.680
The but the way you want to feed that data and you want to clean that data and not not having just drag and drop the entire handbook here and have a messy data,

s208
00:25:52.680 --> 00:26:00.160
that would help you a lot to have a more successful in this case FAQ chatbot.

s209
00:26:00.160 --> 00:26:05.000
Uh Yeah, so these different strategies would help you as well.

s210
00:26:05.000 --> 00:26:13.760
And um Yeah, let's move on to the next slide.

s211
00:26:13.760 --> 00:26:30.640
Um So, we're using we're using Postgres and um we are also having a few agents that they are just Python codes.

s212
00:26:30.640 --> 00:26:36.320
They are not really any We're not calling another LLM as an agent to do something for us.

s213
00:26:36.320 --> 00:26:41.080
The real one reason for that is the speed cuz I'm running this locally.

s214
00:26:41.080 --> 00:26:43.720
I'm having running this on Ollama.

s215
00:26:43.720 --> 00:26:54.840
I don't have I don't need to three four agents to loop three times four times because then I'll have to wait I have to wait 20 30 seconds for it and user would lose the interest.

s216
00:26:54.840 --> 00:26:58.160
You just want one natural language to run.

s217
00:26:58.160 --> 00:27:05.960
The rest if something can be done by a Python function, let's a Python function then run it for you.

s218
00:27:05.960 --> 00:27:08.320
Then you have also the full control over the function.

s219
00:27:08.320 --> 00:27:14.800
Won't be any hallucination because you need to you're going to write your test suite so to just cover most of the situations.

s220
00:27:14.800 --> 00:27:17.680
So that would help you.

s221
00:27:17.680 --> 00:27:25.360
The agents uh I just want to show you something here in the settings what can it be.

s222
00:27:25.360 --> 00:27:32.360
So I can I can uh off and have it agents off and on as well.

s223
00:27:32.360 --> 00:27:39.840
So I can say if somebody wants the current date then this function is it can be called and retrieve.

s224
00:27:39.840 --> 00:27:42.480
You don't need the LLM to bring you back the current date.

s225
00:27:42.480 --> 00:27:57.960
You don't need the LLM to calculate for you something cuz also another plan for expanding this expanding this chatbot is having it for customer service for that's going to talk about the products and the prices and calculations.

s226
00:27:57.960 --> 00:28:04.280
So that's another thing product info or even having another deeper search on the agent mode.

s227
00:28:04.280 --> 00:28:11.480
But the point with that is then cuz it's going to be another search then you would lose that referencing on that.

s228
00:28:11.480 --> 00:28:13.080
That would be important.

s229
00:28:13.080 --> 00:28:22.040
So while I'm here, I'll show you what do I have as a LLM actually I'm using here.

s230
00:28:22.040 --> 00:28:36.160
I'm using the smallest Qwen 2.5. So it's 0.5 billion is parameters instruct model and it's I think it's 400 megabytes only.

s231
00:28:36.160 --> 00:28:49.640
So it's that small and I'm using this model also for for the agent mode if I want to run so I would run and two more I would run another one and I have to

s232
00:28:49.640 --> 00:28:53.080
run the agent mode is going to make it slightly slower.

s233
00:28:53.080 --> 00:28:59.760
Um and then Um yeah, let's go back to the slides here.

s234
00:28:59.760 --> 00:29:09.840
Um So Yeah, this uh way of working you have a full visibility on your poison function.

s235
00:29:09.840 --> 00:29:11.040
That's what's going on.

s236
00:29:11.040 --> 00:29:17.680
So, you only pass the in needed needed information clear information to LLM and receive a clear information.

s237
00:29:17.680 --> 00:29:22.520
So, it would reduce the chance of hallucination dramatically.

s238
00:29:22.520 --> 00:29:32.000
Um also showing you here about uh vectors and things on the in the database.

s239
00:29:32.000 --> 00:29:37.680
So, um So, when we're going to chunk, we're going to chunk the information.

s240
00:29:37.680 --> 00:29:44.520
This is based on the ID and the section um and the what kind of vector embedding are we doing?

s241
00:29:44.520 --> 00:30:02.720
And then um we have our chunk, we have we can see our basically how it's going to find the nearest neighbor here and um Also, the keyword match at the same time because if if you

s242
00:30:02.720 --> 00:30:09.800
uh having a customer service product chatbot, you don't have no need the nearest information.

s243
00:30:09.800 --> 00:30:12.160
You need the exact information.

s244
00:30:12.160 --> 00:30:20.200
If it comes to the number as well, if it's come to the product as well, you need exact uh if it's a medical chatbot, you need the exact medication.

s245
00:30:20.200 --> 00:30:22.800
You need don't need something similar or close to it.

s246
00:30:22.800 --> 00:30:33.160
So, we need to have both both basically uh keyword search and um and the semantic search together.

s247
00:30:33.160 --> 00:30:37.080
So, we'll have a hybrid search that giving you a better result.

s248
00:30:37.080 --> 00:30:42.760
And we're talking about the number number that we're going to have.

s249
00:30:42.760 --> 00:30:48.920
I have it in the system limited to two because in this case you don't need more than that.

s250
00:30:48.920 --> 00:31:03.040
But, if you want to retrieve some information, if it's a product, you might have you don't want to just uh bring top two products that's being there for the user because you might have the same brand like a 20 product or 50 product of the same brand.

s251
00:31:03.040 --> 00:31:11.000
If you don't let the window your window to pull those information, those products are never going to be sold and never going to be shown in the chatbot.

s252
00:31:11.000 --> 00:31:14.760
So, it's important how many you can fetch.

s253
00:31:14.880 --> 00:31:25.960
Uh and then there the more about the retrieval that when we're using the BM25 to get the exact keywords.

s254
00:31:25.960 --> 00:31:40.000
So, the previous slide was about to get the most closest with the cosine distance that we have uh because when we upload the information with the in our vector in our in our vector database,

s255
00:31:40.000 --> 00:31:45.880
the information with the similar meanings are going to sit next to each other.

s256
00:31:45.880 --> 00:32:06.040
And then when we user search for something and ask for something, then uh our information and our our search is going to convert that uh question that query uh in that database and go find a similar

s257
00:32:06.040 --> 00:32:09.640
closest information to that question and will return it.

s258
00:32:09.640 --> 00:32:11.760
So, that's the reason we can't just one.

s259
00:32:11.760 --> 00:32:16.800
We need to return more than those so we have multiple answers and more closer answers.

s260
00:32:16.800 --> 00:32:21.280
But, here in the smart retrieval, we need the exact answer.

s261
00:32:21.280 --> 00:32:32.480
So, we would need to we would need to filter it based on the language, for example, we need to based on the ID or exact product name, SKUs, or

s262
00:32:32.480 --> 00:32:34.600
uh brand name or something.

s263
00:32:34.600 --> 00:32:44.000
So, that would be the the keyword uh retrieval uh for us.

s264
00:32:44.000 --> 00:32:57.640
And then how we do the ranking based on which is very important because when we have the both vector and the BM25, we want to mix these and still bring the top

s265
00:32:57.640 --> 00:33:06.200
uh top two or top five relevant answers that we need for us.

s266
00:33:06.200 --> 00:33:16.640
So, more accurate if we if we bring too many if we bring top 20 here and uh bring shows top 20 answers is going to be quite confusing for the user.

s267
00:33:16.640 --> 00:33:27.080
So, you just want to have make sure you bringing the right amount based on your use case based based on your use case.

s268
00:33:27.080 --> 00:33:29.160
So, if you're having products, you would bring more.

s269
00:33:29.160 --> 00:33:30.840
You retrieve more.

s270
00:33:30.840 --> 00:33:34.280
If it's something medical, you retrieve less.

s271
00:33:34.280 --> 00:33:39.760
Uh you don't want to information to be all over the place.

s272
00:33:39.760 --> 00:33:45.720
You want to accurate more accurate information because you would have more liability obviously on that.

s273
00:33:45.720 --> 00:34:03.520
Um another point here is um So, this is uh this is more about reranker.

s274
00:34:03.520 --> 00:34:23.000
So, when we have our reranker here, uh that would also um bring the information bring that uh And so, after that score uh how many of those information that shows by default.

s275
00:34:23.000 --> 00:34:25.480
So, I'm only showing two there.

s276
00:34:25.480 --> 00:34:32.000
Um so, I've I've briefly talked about the agent mode vs. direct rag.

s277
00:34:32.000 --> 00:34:39.000
The direct rag is is just uh following the it's a fixed pipeline.

s278
00:34:39.000 --> 00:34:41.120
It's not going to go calling anything.

s279
00:34:41.120 --> 00:34:44.840
You know, the the path is quite clear.

s280
00:34:44.840 --> 00:34:50.440
You have embed embedded query, you have a hybrid retriever.

s281
00:34:50.440 --> 00:34:54.240
And you just bring up the answer.

s282
00:34:54.240 --> 00:34:57.560
It's good if compliance is quite important for you.

s283
00:34:57.560 --> 00:35:03.120
But agent mode, you have a few extra tools that going to be called.

s284
00:35:03.120 --> 00:35:05.360
So, a search is a compare product.

s285
00:35:05.360 --> 00:35:08.120
These can bring more information back.

s286
00:35:08.120 --> 00:35:10.160
It's less in your control.

s287
00:35:10.160 --> 00:35:18.480
So, uh Oh, that's quite important and it takes longer because you have another agent that going to work.

s288
00:35:18.480 --> 00:35:24.440
So, basically it's getting this information and then you have agent mode based on your request is going to call.

s289
00:35:24.440 --> 00:35:30.640
Um So, that would that would take slightly longer slightly longer.

s290
00:35:32.480 --> 00:35:37.560
About that about the telemetry and application guidelines.

s291
00:35:37.560 --> 00:35:46.760
So, um can use LangChain Fuse for we integrated LangChain Fuse uh in our functions.

s292
00:35:46.760 --> 00:35:52.200
So, we can uh see what kind of answers uh we have.

s293
00:35:52.200 --> 00:35:54.440
It's quite easy to everything is local.

s294
00:35:54.440 --> 00:35:57.240
It's no outside calling or anything.

s295
00:35:57.240 --> 00:36:02.240
So, it's easy to track the information that was chatting with the widget anonymous.

s296
00:36:02.240 --> 00:36:04.000
I wasn't logged in.

s297
00:36:04.000 --> 00:36:06.240
It's a FAQ FAQ chat.

s298
00:36:06.240 --> 00:36:12.200
That's FAQ agent is name of the the whole projects that I'm running there.

s299
00:36:12.200 --> 00:36:15.520
So, each conversation has a ID.

s300
00:36:15.520 --> 00:36:24.200
Uh you can track if there was any maintenance for this weekend and you can track the answer for it.

s301
00:36:24.200 --> 00:36:27.040
You can see what model I've been using.

s302
00:36:27.040 --> 00:36:35.840
You can see that I've returned the top two answers and the cuz it was top two is referenced twice on that image.

s303
00:36:35.840 --> 00:36:39.040
Um page at most was off.

s304
00:36:39.040 --> 00:36:41.280
Uh it was a chance.

s305
00:36:41.280 --> 00:36:49.560
Two chunk of information was returned and it took this long milliseconds and uh yeah.

s306
00:36:49.560 --> 00:36:52.960
That's basically the information we have.

s307
00:36:52.960 --> 00:36:58.000
If you have external models, you can even estimate the cost that you're spending.

s308
00:36:58.000 --> 00:37:03.360
That's another thing you can uh you can use that for.

s309
00:37:03.360 --> 00:37:07.800
And uh if you have different sessions, different users, you can track the logins.

s310
00:37:07.800 --> 00:37:17.000
So, um that's give you a good insight of how users are treating you uh with your chat.

s311
00:37:17.000 --> 00:37:21.320
So, um Yeah.

s312
00:37:21.320 --> 00:37:30.960
So, as I said about the user sessions and IDs, um and then you would have the guardrails um as well.

s313
00:37:30.960 --> 00:37:37.880
So, something because of you using functions for it, we can uh have a better control over it.

s314
00:37:37.880 --> 00:37:45.480
So, if I ask is chatbot, how do I uh treat flu?

s315
00:37:45.840 --> 00:38:02.400
I have created a section uh to say without going to chatbot, the point of lots of these guardrails is that the issue should stop before we um the prompt injection or anything that we have uh

s316
00:38:02.400 --> 00:38:06.400
or a question that is we're not supposed to respond.

s317
00:38:06.400 --> 00:38:10.400
We shouldn't even send it to the LLM to generate anything.

s318
00:38:10.400 --> 00:38:22.200
So, before even in the code, I can have So, we have created a section here uh to say if there is a medical escalation, bring this medical message back.

s319
00:38:22.200 --> 00:38:35.480
Uh and you can see how the system prompt is quite small because I'm basically prompting everything in the code, so uh, although the system prompt looks quite small and just few sentences,

s320
00:38:35.480 --> 00:38:41.160
uh, I'm actually prompting way more in the uh, code.

s321
00:38:41.160 --> 00:38:44.640
Instead of prompt, we use I'm using the code to say what do's and don'ts.

s322
00:38:44.640 --> 00:38:54.400
And uh, another example is the medical escalation and you can add more based on your uh, concerns you would have for your product.

s323
00:38:54.400 --> 00:38:56.560
So, um, yeah.

s324
00:38:56.560 --> 00:38:58.560
So, this is the example for that.

s325
00:38:58.560 --> 00:39:03.640
Um, can be uh, can needs to be expanded with different keywords, with different kind of sentences.

s326
00:39:03.640 --> 00:39:07.320
Should be cured and see how many times somebody could pass this or not.

s327
00:39:07.320 --> 00:39:16.160
But technically, that would be example of how we should block uh, I have our guard there should block any query that's not related.

s328
00:39:16.160 --> 00:39:21.200
Um, can do the same with the injection and so on.

s329
00:39:21.200 --> 00:39:25.840
Um, what else do we have here?

s330
00:39:25.840 --> 00:39:35.240
We talked about the length fuse and uh, we just need to create a generate a key and you have it in your uh, system.

s331
00:39:35.240 --> 00:39:42.840
I would I would work is just uh, a little bit about the observability that I've talked about.

s332
00:39:42.840 --> 00:39:49.320
Um, and then um, then the injection as well.

s333
00:39:49.320 --> 00:39:54.760
So, the tools for prompt injection again, it shouldn't go to the LLM and be stopped there.

s334
00:39:54.760 --> 00:40:01.720
It should be stopped before before uh, it's being sent there.

s335
00:40:01.760 --> 00:40:10.760
And yeah, we're using intent rejects and term dictionaries and LLM classifier to stop any of these issues.

s336
00:40:10.760 --> 00:40:12.720
And of course, we can expand it more.

s337
00:40:12.720 --> 00:40:16.800
But the good thing about that, because it's just a code, it doesn't it It need the long prompt.

s338
00:40:16.800 --> 00:40:21.000
It doesn't need anything uh Uh outcome would be obvious.

s339
00:40:21.000 --> 00:40:23.920
You know what you uh blocking and what you're not blocking.

s340
00:40:23.920 --> 00:40:33.080
So, it's not like uh uh LM that sometimes likes to listen to you as you're uh as the instruction and sometimes just escape and does its own thing.

s341
00:40:33.080 --> 00:40:44.400
So, part of that integrating all of these in your Python before sending it is that you can have rigid test and to say, "Okay, this is what I'm blocking and this is why."

s342
00:40:44.400 --> 00:40:46.440
And it passes all the tests.

s343
00:40:46.440 --> 00:40:52.320
So, you can be more assured uh about that.

s344
00:40:52.480 --> 00:40:55.520
Um So, yeah.

s345
00:40:55.520 --> 00:41:08.840
It's just a small slide about uh We can just create your the and I'm having it as a um I'm having it here as a as a ba- uh as a widget.

s346
00:41:08.840 --> 00:41:16.000
It could be also done here um instead of um in a bigger chat as well.

s347
00:41:16.000 --> 00:41:24.240
So, another thing is you have to in your chat as a part of the rules, you should say also if somebody doesn't understand the terms that they're using.

s348
00:41:24.240 --> 00:41:36.800
So, if not, the person wouldn't allow me to start the chat and not and uh Also, I gave the the widget capability if I want.

s349
00:41:36.800 --> 00:41:39.360
So, user, you can reset the consent as well.

s350
00:41:39.360 --> 00:41:42.400
It's just for my demo purposes.

s351
00:41:42.400 --> 00:41:46.200
So, and then the the widget would show here.

s352
00:41:46.200 --> 00:41:48.400
Uh if I say no, this wouldn't work.

s353
00:41:48.400 --> 00:41:53.240
If I reset the consent, uh if I say yes, then it will start.

s354
00:41:53.240 --> 00:41:56.960
Uh So, yeah.

s355
00:41:56.960 --> 00:41:58.960
So, that's also about the widgets.

s356
00:41:58.960 --> 00:42:06.320
And is cuz I'm using React, it can be used anything else um um any other uh framework as well.

s357
00:42:06.320 --> 00:42:08.640
I just use it for simplicity.

s358
00:42:08.640 --> 00:42:13.160
Um here saying about Okay, the models I used.

s359
00:42:13.160 --> 00:42:15.640
So, I've used two models.

s360
00:42:15.640 --> 00:42:25.960
Uh I pulled the quant 2.5. Um Actually, first I've had the 7 billion, but it's um took longer.

s361
00:42:25.960 --> 00:42:33.720
So, for a smaller server, I've changed it to uh Let me show you the admin here.

s362
00:42:39.840 --> 00:42:42.320
uh 0.5. Otherwise, I have different options.

s363
00:42:42.320 --> 00:42:45.120
So, I was trying with different models.

s364
00:42:45.120 --> 00:42:48.200
If you have a bigger model, it's just going to take longer.

s365
00:42:48.200 --> 00:42:49.960
And it's might be too wordy.

s366
00:42:49.960 --> 00:42:54.720
Interesting finding from my side was you don't need the biggest model.

s367
00:42:54.720 --> 00:43:07.640
You if you can vet your data before sending it to your LLM, a smaller smallest model can sort you out and get a good uh uh answer for you.

s368
00:43:07.640 --> 00:43:10.400
And uh it would give you less hallucination.

s369
00:43:10.400 --> 00:43:15.000
It would uh it's safer because it's not going to make up information.

s370
00:43:15.000 --> 00:43:19.440
If it doesn't have information, it's simply it's going to say no, I don't have that information.

s371
00:43:19.440 --> 00:43:28.880
So, that's another way of guard railing um to not uh to not have that issue of your LLM randomly generating something.

s372
00:43:28.920 --> 00:43:33.760
Um I have BGM embedding models.

s373
00:43:33.760 --> 00:43:42.880
So, when I upload anything here that uh upload anything here that embedding models basically that embedding models do the embedding for me.

s374
00:43:42.880 --> 00:43:47.120
Uh so, technically you need two models only.

s375
00:43:47.120 --> 00:44:02.760
Smallest chat model and uh one one of the smallest uh uh one of the smallest model for uh embedding.

s376
00:44:02.760 --> 00:44:07.320
And then, if you have it on Docker, one click, everything's It's to be running.

s377
00:44:07.320 --> 00:44:17.200
You need your uh Python requirements to be installed and then you have your React working on the front end.

s378
00:44:17.200 --> 00:44:19.600
And then that's all you need.

s379
00:44:19.600 --> 00:44:22.520
Uh one more thing that I should be adding here.

s380
00:44:22.520 --> 00:44:27.920
I haven't the Langfuse also uh should be added here.

s381
00:44:27.920 --> 00:44:29.720
That's going to be running in a different port.

s382
00:44:29.720 --> 00:44:31.840
That's missing from here.

s383
00:44:31.840 --> 00:44:37.280
So, the key takeaway for um this is the structure markdown first.

s384
00:44:37.280 --> 00:44:41.680
So, we use Docling for uh to do the heavy job.

s385
00:44:41.680 --> 00:44:47.240
And then we use the Postgres vector database to just store everything.

s386
00:44:47.240 --> 00:44:50.760
Uh didn't use any specific framework.

s387
00:44:50.760 --> 00:44:55.920
Uh we just used uh Llama for our chats.

s388
00:44:55.920 --> 00:44:59.120
And no other uh paid framework anywhere else.

s389
00:44:59.120 --> 00:45:02.040
We used Lang Langfuse for our telemetry.

s390
00:45:02.040 --> 00:45:05.240
That was also free to use.

s391
00:45:05.240 --> 00:45:09.840
And this can be uh escalated to different products as well.

s392
00:45:09.840 --> 00:45:14.960
It can be document catalog or from any database.

s393
00:45:14.960 --> 00:45:18.680
Um Thank you very much uh for your time.

s394
00:45:18.680 --> 00:45:30.160
I would like to get connected with you and uh if you have any questions, if you have uh any kind of concern or something that you figure that would be

s395
00:45:30.160 --> 00:45:36.520
uh better to integrate, please share your idea with me either in the comments or message me on LinkedIn.

s396
00:45:36.520 --> 00:45:38.960
Um Yeah.

s397
00:45:38.960 --> 00:45:40.520
Thank you so much.

s398
00:45:40.520 --> 00:45:43.640
Uh thank you AI Engineer for the opportunity.

s399
00:45:43.640 --> 00:45:45.760
And uh I'll see you next time.

s400
00:45:45.760 --> 00:45:47.760
Cheers.