Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

AI Engineer · 45 min · 400 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01Hello everyone.
00:03I'm Abid Matini.
00:05I'm senior backend developer at OGOV.
00:09Thanks to AI Engineer World Fair 2026 for giving me a time slot for online track.
00:17And I'm going to walk you through bypassing the multimodal text today.
00:24How we can have a have a framework free hybrid rag.
00:28Withdraw SQL RF and live telemetry.
00:32As you can see, I have got my live demo in the right side of the screen.
00:39So,
00:39[snorts]
00:40any slide that I pass and if there is anything related related to my presentation, I can show you.
00:50I have few tabs also ready here for you to show and walk you through uh the slides and the demos.
01:04Okay, let's begin.
01:07So, we have two problems for every document we have two problems that we try to solve today.
01:17One is when we start to chat with any LLM and if we have to upload a document, we usually going to just drag and drop a PDF or a Word doc or an image and ask the chatbot to get ready for the questions that going to come up
01:34after we upload those documents.
01:36The issue with that is as soon as we upload these documents, we going to basically lose some of our tokens that we supposed to have only for uh processing these documents without even asking a question.
01:53So, we already uh spend some tokens there without even asking any questions.
02:01That would be the first issue.
02:03And then, if we're building a chatbot and it's uh we want to have vector database there, we want to have keyword search and semantic search there, and we want to just add more tools.
02:15This is going to be too many tools uh uh combined all together.
02:19So, in the production uh in the production basically chatbot, we're going to have uh too many tools, and it's going to be complicated for us to manage our production chatbot properly.
02:35So, today I'm going to uh walk you through in uh three main uh sections.
02:44One is how we can uh upload our documents and get our documents and make it ready bef- uh for our chatbot before uh users going to chat.
02:56So, on the on the right side, I have I do have a FAQ assistant chatbot that's running locally now for me.
03:06And um I'm going to show you the dashboard, the dashboard that I've created for it, and how we can chat, and uh how we get the results.
03:15So, this is going to be a uh supposedly FAQ um assistant for a sample employee handbook.
03:26So, imagine that you have HR company, and this HR has uploaded the the the the handbook for uh for uh for the employees to ask their question, any question they have about
03:39leave, about uh uh like uh different kinds of leaves, and sickness, or uh parent like uh parental leave, or any other questions related to our, uh uh, possibly an HR would have as a common question that they get every day.
04:00So, um, that would be one sample that we're going to use to upload.
04:06So, all documents can get can get in the different formats or we can upload in different formats of PDF, uh, like PowerPoint documents and Word documents or even image.
04:18So, we're going to talk about that.
04:20We're going to talk about how we have to chunk these because we're talking about drag here.
04:25We need to know how we can chunk this information uh, properly based on our the documents.
04:32So, we're going to talk about uh, different type of chunking here as well.
04:37We can, uh, we show you in the admin how we can choose different uh, different strategies to uh, to chunk the documents.
04:48Then we're going to see how the search is going to work, how the embedding is going to work, how did our database is going to be, how the search going to be based on the
04:57keyword search or the semantic search.
05:00And we're going to see about the how our F in the Python going to work.
05:06And then we're going to talk about the observability, how we going to check the safety, how we going to observe, uh, all the process with LangFuse and how, uh,
05:16we can the prevent, uh, prompt injection pattern or, um, any risky question that might come to LLM.
05:27I'm planning this talk for about under an hour and this, uh, the whole thing, the plan is the entire code base can be easily will be running on the GitHub code code space which you can just pull from the repo and
05:42one command run and it's going to be ready to test on GitHub code spaces.
05:51Um so, the stack we're using here are Python, FastAPI for the back end, React for the front end, uh PostgreSQL for our database.
06:03Uh Docker for just easily being able to have the containers and we can just re- re- produce it anywhere and rerun it anywhere, for example, in code spaces or
06:15code spaces or any other uh environment we want.
06:19Then, we have Ollama and local language models and local language models and embedding models that running only locally that can be run on any uh server as well.
06:30It doesn't need a GPU.
06:32The CPU is going to be good enough for it.
06:34So, uh any staging server, any code space uh uh going going to be fine for it.
06:43And LangFuse for uh tracking what's going on and how we can see the chat and the latency uh so, we can improve our chats better.
06:57Um This is the end-to-end blueprint of uh what we're going to talk about.
07:03So, we're going to have a document few documents as a source of truth for our chatbot.
07:12So, we're going to upload some documents.
07:15So, what we can do with Python is we're using Doc Link to convert those uh raw documents uh to markdown files.
07:25And then, we're going to chunk those markdown files to uh in our into our uh PostgreSQL.
07:31And then, we can also do some um scan as well here.
07:35So, uh that would happen as well.
07:37Then, we would have some queries from the user.
07:40So, the employee is asking questions, HR-related questions, and it's going to be retrieved from the database.
07:46Is the the the result relevant results with the top rank would be passed through the LLM.
07:52And LLM would uh return the in the in our chatbots that we have here.
07:58Uh in the format of chat, so we can see and get our questions answered here, and we can observe in the length views what's going on in our system.
08:11So, um optimizing data ingest.
08:16So, we have two different ways of optimizing.
08:19So, if we first uh going to just throw it into the We have a document, and if we just uploading it on our um uh cloud chatbot anywhere, the issue we just
08:34uh having of the one issue we have is we're spending tokens before we doing anything.
08:39Another issue is you don't know how that document looks li- looks like there and that chunking uh how it's being chunked and how uh basically that LLM is seeing it.
08:50So, that would be a risk quality.
08:52If if there is a table that we don't know how it's being read there, so we kind of don't see.
08:57We're trying to uh have more clarity about how the how the document's being stored in our database and how it's being retrieved.
09:06So, another way would be a structure first, so we use the local CPU.
09:12If you're running it here, we we can uh upload a document or doc link would convert it to uh markdown file, and we would know the better chunk uh how the chunking is going to work.
09:24I'm going to show you a few examples.
09:26And uh of course, it's going to be a cheaper prompt.
09:29If after that you want to talk to a lo- um an online cloud any- but if you want to If you want to have it locally, then it would be actually no cost
09:39for you.
09:40Uh, so the DocLink pipeline would be uh, we would be running it It's like a local locally running it on our server.
09:51The The The Python is going to get that document.
09:55It's going to export it to the markdown and save that markdown file.
10:00Then the get this going to chunk it based on different strategy and uh, we're using embedding models to embed it to our database and insert that on to our
10:14vector database.
10:16Uh, so if we save while local because uh, imagine if if one or two pages the cost is next to nothing, but if you have 200 pages uh, then the cost going to be too much.
10:29And if you have a big company with lots of employees and they want to just us keep asking the same questions or uh, if they want to upload documents then it's going to be
10:43uh, going to be a big cost.
10:46So, um, there are four ways of chunking a document.
10:50Let me first before I show you the four ways, let me uh, show you the admin page that what I'm talking about.
10:58So, imagine here you have We are We're having our FAQ admin page.
11:05So, uh, imagine you have Let me delete few of the documents that I have chunked before here.
11:16So, uh, you want to upload a new FAQ uh, file for your HR as for for your HR policy.
11:25I want to show uh, give your chatbot uh, data updated with our AI system.
11:33So, imagine you have um you have some question answers here.
11:42Uh before I do that, let me show you here what I mean.
11:46So,
11:46[snorts]
11:47um this is the heading FAQ, yeah.
11:50So, one way that you can feed our chatbot with information is if we actually create a question answer uh file.
12:01So, that would be easier to digest for our uh system to understand what's going on.
12:07So, part of that is uh imagine we're having a this the entire FAQ file.
12:13So, the entire uh basically handbook.
12:18If we want to chunk this uh handbook and just upload it, imagine uh we're uploading the entire PDF.
12:26This is going to be the way our PDF is going to look like after chunking.
12:30So, it started by acknowledgement, then they see a few chunk that doesn't have any meaning.
12:37And then something starts here.
12:40There are some chunks that doesn't have any Uh for example, here signature date, that's basically kind of useless.
12:49So, this going to slow down and reduce the accuracy of our chat.
12:54It's going to increase hallucination.
12:56So, um instead of uploading uh straight uh 28 pages, we can if we can divide our documents to different chunks and just to different files, relevant files, and clean up that data a bit as much as we can,
13:16uh and then upload it, then we will have a better result.
13:19So, uh one way of uh uploading in the FAQ case is to actually have your your turning your data to question and answers.
13:31So, uh So, talking about working hours, talking about different situations here, so you can have it based on questions and answers.
13:41Then this way you know when you reference your documents, you're having the uh correct reference, and it won't be any confusing.
13:52And there are uh different way to uh upload your documents.
13:57One way is you have the question and answer, and then you would like to upload it.
14:03This is my FAQ PDF file.
14:05Uh same PDF file, and I'd like to upload it based on uh heading-based chunking strategy.
14:14That means well, while I'm explaining the the parsing where Docling is happening in the background.
14:20So, that means each uh chunking would be based on the heading that Docling would find.
14:29And the answer that comes based on that.
14:31So, each heading and answer is going to heading as a question and answer is going to become a chunk.
14:38So, we know our chunking is clean, and it's easy to uh reference.
14:46So, when this chunking finishes in a seconds, we would see, okay, this is the first FAQ, it could chunk successfully to uh based on the headings.
14:57So, it recognized the heading question, uh and then the relevant answer.
15:05So, each chunk has its own uh question and answer here.
15:10So, if we say uh if we say, for example, uh
15:20Um as our questions here Uh do we do we have or something like this.
15:36So, we should expect a kind of answer with the reference that's coming to that chunk.
15:42So, we talking about heading chance going to understand the documents based on uh headings that you have.
15:50That's how it's going to chunk each document heading by heading.
15:54So, as you can see here, um our question was about how much This was the question we prepared, and this was the answer we prepared.
16:04Uh sample Coca-Cola 90% of the individual employee premium and 75% dependent premium.
16:14What we see here as a reference, we getting that answer, and not only that, you getting the correct answer, you also getting the Uh you see that it's been this answer has been generated based on this document, based on this chunk.
16:30So, it's easy to reference.
16:31It's easy, you know, where the issue is if there is anything that's not working, you know why.
16:37Then you can go track it why this right chunk hasn't been retrieved.
16:41So, in this way you have more uh more uh clear path towards how you can debug it.
16:49Because I was uh in our system, I was asking for um return top two chunks, it would uh suggesting me I'll talk about all these models here.
17:02Um it would bring top two response.
17:05So, then it was also checking that sample employee handbook that big file here, sample employee handbook.
17:16As you can see here, that's a big chunk.
17:18And if actually it's a good example because if you compare these two, uh because we just drag uh uploaded the entire chunk here, you can see although it's returned some information,
17:30but it's not that accurate and you don't know.
17:35Of course, it went through that file and found something, but uh you don't know how accurate that information could be and you can't follow because the chunk is is just
17:46uh uh untrackable that easily.
17:50So, one way of chunking is chunking based on the headings.
17:57Second way of chunking is based on the paragraph.
17:59So, technically when you upload your documents, you would say uh just chunk each paragraph you catch, have it as a separate chunk.
18:11Doesn't matter there is a heading or not heading or anything.
18:14So, uh if I want to go back to the to our to our documents and say, let me first choose I want to do a paragraph chunking.
18:25I want to say, okay, uh for paragraph um Let me find the paragraph general policy for paragraph.
18:36I want to upload that.
18:38Um So, for general policy, it's uploading here as you would see the whole file is just been paragraph by paragraph being chunked.
18:51So, there is no heading, there is no no separate information.
18:54And if you look at the uh if you look at that information as well, if I can find it here quickly, um general general paragraph uh policy paragraph as you can see the information is just paragraph by paragraph.
19:18So, you kind of cleaned up your data again, but you're dividing it based on the different system.
19:25So, instead of heading you're talking about paragraphs.
19:29That's another way of uh basically feeding your rag for your information.
19:36Another way would be a fixed 512 of character.
19:40You can say, "Okay, one of the best practices for rag is you just divide based on 512 character, and you have 64% overlap between the chunks, so you won't lose the track."
19:54So, if your data is just too is not organized, if there's just random data that you have and you can't clean that up, that's another way you can upload your document.
20:08So, that would be instead of this, you can choose another file, and you can say basically there's another paragraph for global travel uh file here.
20:26So, that's kind of when it's kind of overlap here when it's going to see here that the about this chunking strategy, you would see okay this started for chunking and then
20:35it ended in the 512, and then it's again uh using that uh I think it was 64 uh I was 64 overlap, yeah.
20:49So, it's using the 64 overlap here again.
20:52So, you would see it's basically uh chopped it from here.
21:02And so, from here to here is 512, and from here to here is 32 then another 32 from here to here.
21:15So, this is a 32 32 64 in total.
21:19So, that's how it's relating each chunks together.
21:22It's happening same for here.
21:24As you can see it's happening same for here.
21:27It's the way that But you see the issue with that is it's coming here to assist the next system.
21:35So, it's breaking.
21:36It's not the best but in some cases it works for you.
21:40Another way that we can do is sentence base.
21:47So, it can be based on first counting based on the number of sentences and it chop it there.
21:51There are different use case for each ones.
21:53So, another interesting one is imagine that you There's some maintenance coming.
22:00You're receiving some email and you just want to quickly upload the document in your chat without without cuz you do you don't have the it's just a screenshot and you can't do anything.
22:21Imagine you just receiving an email and says, "Okay, there is a maintenance plan for this day from this hour and you just wants to upload this in your chat
22:34for the for the user.
22:35So, if me as a employee ask, "Okay, what is is there any maintenance plan for this weekend?"
22:41Then I would know immediately.
22:44So, another way that we can easily upload that screenshot as well.
22:53Let me find easily upload that.
23:02Oh, sorry.
23:03That wasn't our screenshots.
23:05So, the screenshots uh you can upload then we're using another um uh model that just turns that uh image to text and text to .md and then we're using
23:23this uh uh basically uh what's the name?
23:28Uh the sentence model for the emails that can convert uh that image to a text and then it can feed.
23:37So, we can see we're using the chunking strategy a sentence group is turned that image to text and it's already uh good for the uh for our chatbot.
23:52So, we can uh we can index it into our knowledge base now.
23:56So, then it's going to be a next one that's been uploaded here screenshot.
24:01We can see how it's been chunked because this is just the email.
24:04It doesn't have heading.
24:05It doesn't necessarily have all the paragraph chunk properly.
24:08So, and if it's a shortage of time it's a small message temporary change.
24:13You don't need to sit and clean up a data for here.
24:16You just want to drag and drop updated for a weekend and everybody knows okay uh this weekend we're having a maintenance plan.
24:25So, uh when they will come here and ask is there any maintenance plan for this weekend?
24:35Then technically we should get the result of okay, this is the maintenance plan that's been happening and the reference documents.
24:44So, uh the maintenance uh this is the exact maintenance uh window that we got and actually the reference you can see that's the screenshot that uh we uploaded just now.
24:59So, if we do if we had I done the screen the file name better, there would have been even a more relevant and somebody says, "Okay, according to this information
25:11that I got from here, we have a maintenance plan tomorrow from 6:00 p.m. onwards till midnight."
25:19So, and I can easily archive and delete any of these and then it won't be visible as well.
25:26So, these are basically different ways that you can chunk.
25:31That's one of the that's basically the one of the most important part of the whole system because most of the LLMs would just work.
25:40The but the way you want to feed that data and you want to clean that data and not not having just drag and drop the entire handbook here and have a messy data,
25:52that would help you a lot to have a more successful in this case FAQ chatbot.
26:00Uh Yeah, so these different strategies would help you as well.
26:05And um Yeah, let's move on to the next slide.
26:13Um So, we're using we're using Postgres and um we are also having a few agents that they are just Python codes.
26:30They are not really any We're not calling another LLM as an agent to do something for us.
26:36The real one reason for that is the speed cuz I'm running this locally.
26:41I'm having running this on Ollama.
26:43I don't have I don't need to three four agents to loop three times four times because then I'll have to wait I have to wait 20 30 seconds for it and user would lose the interest.
26:54You just want one natural language to run.
26:58The rest if something can be done by a Python function, let's a Python function then run it for you.
27:05Then you have also the full control over the function.
27:08Won't be any hallucination because you need to you're going to write your test suite so to just cover most of the situations.
27:14So that would help you.
27:17The agents uh I just want to show you something here in the settings what can it be.
27:25So I can I can uh off and have it agents off and on as well.
27:32So I can say if somebody wants the current date then this function is it can be called and retrieve.
27:39You don't need the LLM to bring you back the current date.
27:42You don't need the LLM to calculate for you something cuz also another plan for expanding this expanding this chatbot is having it for customer service for that's going to talk about the products and the prices and calculations.
27:57So that's another thing product info or even having another deeper search on the agent mode.
28:04But the point with that is then cuz it's going to be another search then you would lose that referencing on that.
28:11That would be important.
28:13So while I'm here, I'll show you what do I have as a LLM actually I'm using here.
28:22I'm using the smallest Qwen 2.5. So it's 0.5 billion is parameters instruct model and it's I think it's 400 megabytes only.
28:36So it's that small and I'm using this model also for for the agent mode if I want to run so I would run and two more I would run another one and I have to
28:49run the agent mode is going to make it slightly slower.
28:53Um and then Um yeah, let's go back to the slides here.
28:59Um So Yeah, this uh way of working you have a full visibility on your poison function.
29:09That's what's going on.
29:11So, you only pass the in needed needed information clear information to LLM and receive a clear information.
29:17So, it would reduce the chance of hallucination dramatically.
29:22Um also showing you here about uh vectors and things on the in the database.
29:32So, um So, when we're going to chunk, we're going to chunk the information.
29:37This is based on the ID and the section um and the what kind of vector embedding are we doing?
29:44And then um we have our chunk, we have we can see our basically how it's going to find the nearest neighbor here and um Also, the keyword match at the same time because if if you
30:02uh having a customer service product chatbot, you don't have no need the nearest information.
30:09You need the exact information.
30:12If it comes to the number as well, if it's come to the product as well, you need exact uh if it's a medical chatbot, you need the exact medication.
30:20You need don't need something similar or close to it.
30:22So, we need to have both both basically uh keyword search and um and the semantic search together.
30:33So, we'll have a hybrid search that giving you a better result.
30:37And we're talking about the number number that we're going to have.
30:42I have it in the system limited to two because in this case you don't need more than that.
30:48But, if you want to retrieve some information, if it's a product, you might have you don't want to just uh bring top two products that's being there for the user because you might have the same brand like a 20 product or 50 product of the same brand.
31:03If you don't let the window your window to pull those information, those products are never going to be sold and never going to be shown in the chatbot.
31:11So, it's important how many you can fetch.
31:14Uh and then there the more about the retrieval that when we're using the BM25 to get the exact keywords.
31:25So, the previous slide was about to get the most closest with the cosine distance that we have uh because when we upload the information with the in our vector in our in our vector database,
31:40the information with the similar meanings are going to sit next to each other.
31:45And then when we user search for something and ask for something, then uh our information and our our search is going to convert that uh question that query uh in that database and go find a similar
32:06closest information to that question and will return it.
32:09So, that's the reason we can't just one.
32:11We need to return more than those so we have multiple answers and more closer answers.
32:16But, here in the smart retrieval, we need the exact answer.
32:21So, we would need to we would need to filter it based on the language, for example, we need to based on the ID or exact product name, SKUs, or
32:32uh brand name or something.
32:34So, that would be the the keyword uh retrieval uh for us.
32:44And then how we do the ranking based on which is very important because when we have the both vector and the BM25, we want to mix these and still bring the top
32:57uh top two or top five relevant answers that we need for us.
33:06So, more accurate if we if we bring too many if we bring top 20 here and uh bring shows top 20 answers is going to be quite confusing for the user.
33:16So, you just want to have make sure you bringing the right amount based on your use case based based on your use case.
33:27So, if you're having products, you would bring more.
33:29You retrieve more.
33:30If it's something medical, you retrieve less.
33:34Uh you don't want to information to be all over the place.
33:39You want to accurate more accurate information because you would have more liability obviously on that.
33:45Um another point here is um So, this is uh this is more about reranker.
34:03So, when we have our reranker here, uh that would also um bring the information bring that uh And so, after that score uh how many of those information that shows by default.
34:23So, I'm only showing two there.
34:25Um so, I've I've briefly talked about the agent mode vs. direct rag.
34:32The direct rag is is just uh following the it's a fixed pipeline.
34:39It's not going to go calling anything.
34:41You know, the the path is quite clear.
34:44You have embed embedded query, you have a hybrid retriever.
34:50And you just bring up the answer.
34:54It's good if compliance is quite important for you.
34:57But agent mode, you have a few extra tools that going to be called.
35:03So, a search is a compare product.
35:05These can bring more information back.
35:08It's less in your control.
35:10So, uh Oh, that's quite important and it takes longer because you have another agent that going to work.
35:18So, basically it's getting this information and then you have agent mode based on your request is going to call.
35:24Um So, that would that would take slightly longer slightly longer.
35:32About that about the telemetry and application guidelines.
35:37So, um can use LangChain Fuse for we integrated LangChain Fuse uh in our functions.
35:46So, we can uh see what kind of answers uh we have.
35:52It's quite easy to everything is local.
35:54It's no outside calling or anything.
35:57So, it's easy to track the information that was chatting with the widget anonymous.
36:02I wasn't logged in.
36:04It's a FAQ FAQ chat.
36:06That's FAQ agent is name of the the whole projects that I'm running there.
36:12So, each conversation has a ID.
36:15Uh you can track if there was any maintenance for this weekend and you can track the answer for it.
36:24You can see what model I've been using.
36:27You can see that I've returned the top two answers and the cuz it was top two is referenced twice on that image.
36:35Um page at most was off.
36:39Uh it was a chance.
36:41Two chunk of information was returned and it took this long milliseconds and uh yeah.
36:49That's basically the information we have.
36:52If you have external models, you can even estimate the cost that you're spending.
36:58That's another thing you can uh you can use that for.
37:03And uh if you have different sessions, different users, you can track the logins.
37:07So, um that's give you a good insight of how users are treating you uh with your chat.
37:17So, um Yeah.
37:21So, as I said about the user sessions and IDs, um and then you would have the guardrails um as well.
37:30So, something because of you using functions for it, we can uh have a better control over it.
37:37So, if I ask is chatbot, how do I uh treat flu?
37:45I have created a section uh to say without going to chatbot, the point of lots of these guardrails is that the issue should stop before we um the prompt injection or anything that we have uh
38:02or a question that is we're not supposed to respond.
38:06We shouldn't even send it to the LLM to generate anything.
38:10So, before even in the code, I can have So, we have created a section here uh to say if there is a medical escalation, bring this medical message back.
38:22Uh and you can see how the system prompt is quite small because I'm basically prompting everything in the code, so uh, although the system prompt looks quite small and just few sentences,
38:35uh, I'm actually prompting way more in the uh, code.
38:41Instead of prompt, we use I'm using the code to say what do's and don'ts.
38:44And uh, another example is the medical escalation and you can add more based on your uh, concerns you would have for your product.
38:54So, um, yeah.
38:56So, this is the example for that.
38:58Um, can be uh, can needs to be expanded with different keywords, with different kind of sentences.
39:03Should be cured and see how many times somebody could pass this or not.
39:07But technically, that would be example of how we should block uh, I have our guard there should block any query that's not related.
39:16Um, can do the same with the injection and so on.
39:21Um, what else do we have here?
39:25We talked about the length fuse and uh, we just need to create a generate a key and you have it in your uh, system.
39:35I would I would work is just uh, a little bit about the observability that I've talked about.
39:42Um, and then um, then the injection as well.
39:49So, the tools for prompt injection again, it shouldn't go to the LLM and be stopped there.
39:54It should be stopped before before uh, it's being sent there.
40:01And yeah, we're using intent rejects and term dictionaries and LLM classifier to stop any of these issues.
40:10And of course, we can expand it more.
40:12But the good thing about that, because it's just a code, it doesn't it It need the long prompt.
40:16It doesn't need anything uh Uh outcome would be obvious.
40:21You know what you uh blocking and what you're not blocking.
40:23So, it's not like uh uh LM that sometimes likes to listen to you as you're uh as the instruction and sometimes just escape and does its own thing.
40:33So, part of that integrating all of these in your Python before sending it is that you can have rigid test and to say, "Okay, this is what I'm blocking and this is why."
40:44And it passes all the tests.
40:46So, you can be more assured uh about that.
40:52Um So, yeah.
40:55It's just a small slide about uh We can just create your the and I'm having it as a um I'm having it here as a as a ba- uh as a widget.
41:08It could be also done here um instead of um in a bigger chat as well.
41:16So, another thing is you have to in your chat as a part of the rules, you should say also if somebody doesn't understand the terms that they're using.
41:24So, if not, the person wouldn't allow me to start the chat and not and uh Also, I gave the the widget capability if I want.
41:36So, user, you can reset the consent as well.
41:39It's just for my demo purposes.
41:42So, and then the the widget would show here.
41:46Uh if I say no, this wouldn't work.
41:48If I reset the consent, uh if I say yes, then it will start.
41:53Uh So, yeah.
41:56So, that's also about the widgets.
41:58And is cuz I'm using React, it can be used anything else um um any other uh framework as well.
42:06I just use it for simplicity.
42:08Um here saying about Okay, the models I used.
42:13So, I've used two models.
42:15Uh I pulled the quant 2.5. Um Actually, first I've had the 7 billion, but it's um took longer.
42:25So, for a smaller server, I've changed it to uh Let me show you the admin here.
42:39uh 0.5. Otherwise, I have different options.
42:42So, I was trying with different models.
42:45If you have a bigger model, it's just going to take longer.
42:48And it's might be too wordy.
42:49Interesting finding from my side was you don't need the biggest model.
42:54You if you can vet your data before sending it to your LLM, a smaller smallest model can sort you out and get a good uh uh answer for you.
43:07And uh it would give you less hallucination.
43:10It would uh it's safer because it's not going to make up information.
43:15If it doesn't have information, it's simply it's going to say no, I don't have that information.
43:19So, that's another way of guard railing um to not uh to not have that issue of your LLM randomly generating something.
43:28Um I have BGM embedding models.
43:33So, when I upload anything here that uh upload anything here that embedding models basically that embedding models do the embedding for me.
43:42Uh so, technically you need two models only.
43:47Smallest chat model and uh one one of the smallest uh uh one of the smallest model for uh embedding.
44:02And then, if you have it on Docker, one click, everything's It's to be running.
44:07You need your uh Python requirements to be installed and then you have your React working on the front end.
44:17And then that's all you need.
44:19Uh one more thing that I should be adding here.
44:22I haven't the Langfuse also uh should be added here.
44:27That's going to be running in a different port.
44:29That's missing from here.
44:31So, the key takeaway for um this is the structure markdown first.
44:37So, we use Docling for uh to do the heavy job.
44:41And then we use the Postgres vector database to just store everything.
44:47Uh didn't use any specific framework.
44:50Uh we just used uh Llama for our chats.
44:55And no other uh paid framework anywhere else.
44:59We used Lang Langfuse for our telemetry.
45:02That was also free to use.
45:05And this can be uh escalated to different products as well.
45:09It can be document catalog or from any database.
45:14Um Thank you very much uh for your time.
45:18I would like to get connected with you and uh if you have any questions, if you have uh any kind of concern or something that you figure that would be
45:30uh better to integrate, please share your idea with me either in the comments or message me on LinkedIn.
45:36Um Yeah.
45:38Thank you so much.
45:40Uh thank you AI Engineer for the opportunity.
45:43And uh I'll see you next time.
45:45Cheers.