WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=iKQ78wyJEXU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/iKQ78wyJEXU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:01.309 --> 00:00:03.309 [music] s2 00:00:12.800 --> 00:00:14.160 Hello everyone. s3 00:00:14.160 --> 00:00:16.000 Good afternoon. s4 00:00:16.000 --> 00:00:23.359 Today I'm going to talk about how we vetted 2,000 AI skills before they reach a developers. s5 00:00:23.359 --> 00:00:30.160 But before I before of that, I'm Lucas Palma, but many people call me LP. s6 00:00:30.160 --> 00:00:48.239 I'm the product security manager at New Bank, the product security structures, uh structure that's within security, looking upon how we make code safe and supporting engineers, product managers and everybody to making our products safer. s7 00:00:48.239 --> 00:01:04.160 I have uh over a decade of experience in financial services engineering background also a lot of years working here at security and a close relationship with the part that I love which is innovation. s8 00:01:04.479 --> 00:01:11.760 So before beginning I believe I want to bring to you uh why are we here. s9 00:01:11.760 --> 00:01:29.600 So one thing that's important for all of you to understand that the now that we are using AI everywhere even though even with uh coding one thing that uh is important that the AI skills are being part s10 00:01:29.600 --> 00:01:46.720 of the developer workflow and that's this might bring some risks because although they look like configuration they behave like supply chain dependence like uh for example libraries and others. s11 00:01:46.720 --> 00:01:58.159 So what we made here was to build a security review system in order to check if these skills were safe or not to be used before deploying them. s12 00:01:58.159 --> 00:02:10.080 So the lesson that I want to bring you here by the end of this presentation is that we should be protecting the whole workflow not only the code that's being generated. s13 00:02:10.080 --> 00:02:12.000 All right. s14 00:02:12.000 --> 00:02:23.120 So what I mean about the supply chain part is that uh traditionally the supply chain has uh package containers, models and so on. s15 00:02:23.120 --> 00:02:26.879 But now in the AI era, it doesn't have only that. s16 00:02:26.879 --> 00:02:45.040 It still have the traditional part, but it will it also includes skills, plugins, MCP servers, agent rules and much more things to be acting as supply chain and where AI skill fits into this. s17 00:02:45.040 --> 00:02:53.760 Uh I believe that before I go into that it's important for everybody be on the same page on what is an AI skill. s18 00:02:53.760 --> 00:03:10.959 So an AI skill has there's normally the developer is using AI tools in order to generate an output which will be code most of the case and within this AI tool there are a bunch of things that can be embedded. s19 00:03:10.959 --> 00:03:14.000 One of them are the AI skill. s20 00:03:14.000 --> 00:03:28.720 So with this skill we can have a capability to a model or to an agent uh bundling some instructions some context in order to have better guidance over what it can be done. s21 00:03:28.720 --> 00:03:35.280 But there is also an impact over that because somebody can create their own skill and share with others. s22 00:03:35.280 --> 00:03:54.480 So when we do that this first person is guiding over the code that's being generated by the other person and then that can be dangerous and since we are here talking in the AI in finance track it's also important for us to s23 00:03:54.480 --> 00:03:58.640 understand that we are in a regulated environment. s24 00:03:58.640 --> 00:04:15.840 So from one side there are are developers wanting better faster coding more context to have less repetitive work but but from the other side even more because of the regulate part we need to be s25 00:04:15.840 --> 00:04:30.479 aware of the auditability of looking upon credentials safety by default and many other security aspects and keeping that balances is hard, right? s26 00:04:30.479 --> 00:04:36.960 So, some people might say like are AI skills dangerous? s27 00:04:36.960 --> 00:04:44.240 So, I brought here a few examples of what do I mean by AI skills being dangerous? s28 00:04:44.240 --> 00:05:00.320 So first uh one thing that can happen is that when people are describing what they skill can or cannot do it can it can ask for it to retrieve a token or something and it will begin using that token s29 00:05:00.320 --> 00:05:08.080 hardcoded which will go to logs and so on and it can generate a data leak in the future. s30 00:05:08.080 --> 00:05:20.560 Another thing that can happen is also the person to h instruct the AI to use shell comments and then this skill will be used by another person and when they use on their shell s31 00:05:20.560 --> 00:05:28.400 a lot of dangerous things that that can happen and a lot of files be modified and so on and there's also permissions. s32 00:05:28.400 --> 00:05:46.160 So depending on how the skill was configured, it might have excessive permissions much more than what was needed and even a typo can make some dangerous stuff depending on who is using that such skill. s33 00:05:47.919 --> 00:06:00.160 So first thing first what we did initially is that how do we share skills among ourselves how the engineers would be sharing the skills. s34 00:06:00.160 --> 00:06:04.479 So uh we went through the marketplace solution. s35 00:06:04.479 --> 00:06:13.199 So the skills are being canonically shared among marketplace with the plugins included the skills among them. s36 00:06:13.199 --> 00:06:22.960 So it's a internal marketplace where people can discover new skills and that's our boundary where we are trying to make it safer. s37 00:06:22.960 --> 00:06:36.000 So what happens is that when someone creates an skill it uh will open the pull request and normally it will go to the marketplace but we made a step before that like a CI step s38 00:06:36.000 --> 00:06:47.120 where we created a tool that's called skill vector and this is this what this tool does is to check if this skill is safe or not to be used s39 00:06:47.120 --> 00:06:58.880 uh using a lot of assessments that I will bring it here and also classify those risks and request remediation and so on. s40 00:06:58.880 --> 00:07:13.199 So what skill vector does in a single page is that when a skill is created or changed not on the during the creation phase one thing that's important is that the engineers are able to s41 00:07:13.199 --> 00:07:22.720 use it locally and also be iterating until the skill is being considered safe before uploaded it and after them upload the skill. s42 00:07:22.720 --> 00:07:32.479 We also runs it again because we can ensure that the engineer has run locally or has run the most updated version. s43 00:07:32.479 --> 00:07:37.199 So we also be scanning that after the upload. s44 00:07:37.199 --> 00:07:48.720 And then we have some determinate checks for the uh easiest parts to check some uh easy risks using regular regular expressions and so on. s45 00:07:48.720 --> 00:07:53.680 After that when we check that we need better context we then use LLM. s46 00:07:53.680 --> 00:08:07.360 Uh it's important to have this hybrid approach with LLM checking the the context but also with the determinist because you know how LLM is depending on the temperature that was set. s47 00:08:07.360 --> 00:08:12.160 Sometimes it will check that it's a risk sometimes it might not. s48 00:08:12.160 --> 00:08:17.360 And then all of these findings are reporting the PR that was open to upload the skill. s49 00:08:17.360 --> 00:08:27.280 So it will improve the usability since the engineer will have the in the same PR what has to be changed before uploading the skill. s50 00:08:27.280 --> 00:08:42.320 And another good thing that we made that's important is to have a serif with all of these so it can be consumed by our security tools as well and generate a report on the risks and be part of our vulnerability s51 00:08:42.320 --> 00:08:44.480 management program. s52 00:08:44.480 --> 00:08:58.160 So depending on the severity, depending on the policy, uh the skill can require some remediation, can be blocked, all of this before the marketplace distribution. s53 00:08:59.040 --> 00:09:10.080 So there is the local scan, the p request, the determinist scanner, then there is the LLM review, PR feedback, serif, and then the decision. s54 00:09:10.080 --> 00:09:11.200 Will we use it? s55 00:09:11.200 --> 00:09:13.519 We will allow it, will we allow it? s56 00:09:13.519 --> 00:09:17.760 but it requires remediation and so on. s57 00:09:18.399 --> 00:09:22.080 A few examples of what we have ex scanned here. s58 00:09:22.080 --> 00:09:25.120 It's uh a non-exhaustive list. s59 00:09:25.120 --> 00:09:29.360 So we are looking upon if there are some unsafe instructions. s60 00:09:29.360 --> 00:09:39.680 If there are some drift be within the behavior that the agent has if there are some destructive shell comments that I commented earlier if there are some file modifications s61 00:09:39.680 --> 00:09:41.680 that shouldn't be there. s62 00:09:41.680 --> 00:09:44.880 uh credential requests, how are they being done? s63 00:09:44.880 --> 00:09:54.080 Some data being exposed uh unintentionally, if there are permissions that are over broad, risky, MCP usage and much more. s64 00:09:54.080 --> 00:09:57.519 These these are the main ones. s65 00:09:58.240 --> 00:10:05.279 And so getting back to the title, we have scanned on that over 2,000 skills. s66 00:10:05.279 --> 00:10:19.440 uh now there is much more than that but this is the baseline that I brought for you on this presentation uh inside this we have identified uh more than 1,000 and half s67 00:10:19.440 --> 00:10:22.480 uh risks. s68 00:10:22.480 --> 00:10:36.000 So not that 1,00 skills had risk because a single skill can has many risks but these were the total risks that we identified over uh this amount of skills s69 00:10:36.000 --> 00:10:50.399 and 1,000 of them were probably remediated right after and there are few of them that were really hky that we were able to block before going to the marketplace. s70 00:10:50.399 --> 00:11:03.120 So we also had made a b uh historical scan looking upon the skills that were created before the skill v implemented. s71 00:11:03.120 --> 00:11:12.692 Uh over there we were able to identify new uh risks as well and put it them into the vulnerability management program so s72 00:11:12.692 --> 00:11:12.800 [snorts] s73 00:11:12.800 --> 00:11:16.079 it can be could be remediated. s74 00:11:17.360 --> 00:11:20.399 A few lessons that I want to bring here as well. s75 00:11:20.399 --> 00:11:39.839 So what things that work well is having both the the terminist scanners for non-risk patterns but also LLM review for uh behavior checking upon the destructive comments uh looking upon the s76 00:11:39.839 --> 00:11:50.240 credentials checks as well having the output in serif and adding comments on PRs and things that needed improvement. s77 00:11:50.240 --> 00:12:02.560 And we worked during the process were also there were some risks like comments that we were treating equally but depending on the comment it can be more or less risky. s78 00:12:02.560 --> 00:12:11.680 Also some signals that were weak and didn't have much context that were uh more troublesome than helpful. s79 00:12:11.680 --> 00:12:14.880 There is also the prompt level ask for confirmation. s80 00:12:14.880 --> 00:12:18.160 I there's a next slide about that that I will go deeper. s81 00:12:18.160 --> 00:12:20.160 That's an important one. s82 00:12:20.160 --> 00:12:29.120 Also, uh there were some warnings that seemed uh harmless, but only if it was running locally. s83 00:12:29.120 --> 00:12:37.120 If there were going to production, then they could be impactful and that we had also to look up on that. s84 00:12:37.120 --> 00:12:42.480 Uh if the finding had hadn't some clear guidance was troublesome as well. s85 00:12:42.480 --> 00:12:47.760 And last but not least, we know that other people could create other marketplace. s86 00:12:47.760 --> 00:12:56.160 So how can we proactively scan check there is a new marketplace and put skill vector into it as well. s87 00:12:56.160 --> 00:13:10.480 So regarding the prompt level that's something that's important for you to know people sometimes will add the instruction like you need to ask for confirmation but the AI may ask confirmation for itself. s88 00:13:10.480 --> 00:13:27.120 So from your perspective there is a human in the loop but for the AI perspective there is has been a confirmation and that's okay another has confirmed then let's go so that's something that we were scanning as well looking up on having proper human in the loop s89 00:13:27.120 --> 00:13:45.839 looking the tool that's executing if it's going through the approval gates and so on having hooks and within this as I said it's uh plug-in marketplace skill is one among many things that there is into that. s90 00:13:45.839 --> 00:13:50.320 So there are things that we can reuse from this lesson. s91 00:13:50.320 --> 00:13:55.760 So for example, treating these as supply chain is important. s92 00:13:55.760 --> 00:14:00.240 Reviewing what's being uploaded to the marketplace before goes there. s93 00:14:00.240 --> 00:14:14.720 letting developers to run these checks locally, enforcing these checks that are being run locally also in the CI having the termination checks together with the LLM checks and looking upon s94 00:14:14.720 --> 00:14:22.320 dangerous actions and prompting uh and having enforcement when they happen. s95 00:14:23.600 --> 00:14:35.760 So next steps over here is that I'm talking a lot about skills here but a lot of these as I said could be applied to plugins to MCP servers rules hooks. s96 00:14:35.760 --> 00:14:48.240 So all of this that I'm saying here we also have the MCP vector the rules checks and so on that's also applicable here but with different risks uh having also s97 00:14:48.240 --> 00:15:03.360 different gates depending on policies that were implemented depending on the marketplace as well have some enforcements on tool level here enforcing that there are audit logs trusted gateways and so on s98 00:15:03.360 --> 00:15:11.600 and also last but not least Having the trusted trusted AI marketplace is very important. s99 00:15:11.600 --> 00:15:28.160 So we can have a canonical way to scan and share knowing that then are being safe and that's not about only about the skills that are being created by people but it also includes the third party skills or plugins and so on. s100 00:15:28.160 --> 00:15:34.399 So if someone downloads something and wants to use it's important to upload it on the marketplace. s101 00:15:34.399 --> 00:15:44.399 So all of this scanning can be done and check if it's safe or not to be used and also that way allow other people to use in a safe way. s102 00:15:45.680 --> 00:15:47.199 And that's it. s103 00:15:47.199 --> 00:15:49.600 Uh I'm sharing here my contact. s104 00:15:49.600 --> 00:15:52.000 There's my link in profile. s105 00:15:52.000 --> 00:15:58.160 If anybody wants to contact talk more about that the QR code will bring you to my profile. s106 00:15:58.160 --> 00:16:01.680 If you don't want to type, no problem at all. s107 00:16:01.680 --> 00:16:06.480 And I hope you you've enjoyed the talk.