How AI Agents Let GTM Teams Scale — Justin Joyce, Cloudflare

AI Engineer · 19 min · 156 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:01[music]
00:12Well, thank you everyone for joining.
00:13I'm hoping that you guys had a great time so far at this conference and you guys have a lot of takeaways back, you know, to your company.
00:20Um, I don't know if any of you guys saw the uh AGI pills downstairs.
00:24Yeah, well, I just took some.
00:26So, if I say any phrases about that's the uh the the gun What's that What's that the phrase?
00:32That's the uh burning gun or the smoking gun or I start hallucinating in general, well, please snap me back.
00:38That's probably just the AGI pills.
00:41All right.
00:42So, without further ado, let's get started.
00:45Uh my name is Justin Joyce.
00:46I'm a principal sales operations and strategy manager at Cloudflare.
00:50Um and I work with the go-to-market team as part of the revenue operations organization, specifically on the teams that uh produce leads for the sales teams, as well as the customer experience team, which works on
01:04uh the customer experience after the sale the sales have been done.
01:09And uh just a little background about me.
01:13As Moda said, I started in sales operations uh and sales and then I moved to sales uh to the machine learning side about the last 7 years at Grainger.
01:23And I really wanted to do that to be able to learn how to uh have prescriptive analysis and prescriptive uh prediction so I can help the business better to
01:33make decisions and to understand what's the next step next best step.
01:39So, uh 6 months ago, I had an opportunity to move back into sales operations uh because I really wanted to use all the skills that I'd been learning from machine learning as well as from sales
01:52operations in general.
01:57So, what's the general problem?
01:59The general problem is that traditional go-to-market does not scale.
02:04There's a few um facets to this.
02:07The first facet is that usually teams on the back office side, uh they're either doing work in Excel or sheets at worst, um building analysis each week, multiple hours a week,
02:20and as they take on multiple projects, it gets exponentially long with how many of those analysis that they're doing.
02:26Uh at best, they are producing dashboards, um providing information to the uh leadership and executive team, um which, you know, meets the needs of most teams, um but not all of them.
02:39And that needs meets that needs uh means that in general, not all the requirements of the go-to-market teams are met.
02:47They're not able to really provide all the information that the teams need when um they need it.
02:55The second problem, which is more on the go-to-market side with the sales and uh the sales teams and the other teams that I mentioned that I support, is they have two gaps.
03:06Essentially, the first gap is the context gap, meaning when a salesperson is uh talking to a prospect um one call and talking to a current customer in the next call, or
03:18um talking at an adoption conversation the call after that, they have to constantly switch contexts and they have to gather information for those specific calls, which is good.
03:28They really need to get that information to have those calls and understand how to approach the situation, but they have to do all that work in between.
03:37Um the second one what is like what I like to call is the expert gap, which is the gap between how your expert salesperson or expert go-to-market um sales individual,
03:49how he would approach a situation, how he would talk to a prospect, how he would uh work on adoption call, how he would handle a customer satisfaction issue, and
03:59between also the a new salesperson or someone that's just ramping up.
04:04So, ideally like everyone working at the same operational level, so you have consistency in execution, consistency in messaging of how to assess a a customer's problems and how your product can help fit that portfolio.
04:17So, with these two problems with manual work as well as uh salespeople not having enough information and having the gap of having to get all the information they need
04:29and gather that as well as not be able to execute the same level, it really creates an inefficiency in the go-to-market organization.
04:37And so, for the last 6 months or so since I've joined Cloudflare, I've been really focusing on how can I make this operation efficient from back to front and
04:47there's a lot of great things that we've been doing at our company in general that's helped me enable that and I really want to share some of those findings with you.
04:56So, the framework that I'm proposing here, which I think is is it's going to really be effective in as we flesh this out in the future is a three-pillar
05:06approach.
05:07The first pillar approach is how can we scale analysis and the ability of operations team to meet the data needs of executives, leadership, as well as be able to build applications
05:20using business context.
05:21How can they take things that would take 2 hours to do down to 5 minutes?
05:26Um back in well, not back in January of this year, I after I joined I joined the company, I had built these skills and I'd started asking questions of the data directly
05:37and I was able to get answers immediately while doing other things and I saw the huge power of how if we can scale the analysis and the operations of the teams, we can actually focus on the second part of my job, which is strategy.
05:50The second pillar is how can is to scale insight.
05:54There's story in the data, and how can we provide that to the team?
05:58How can we provide that team at, you know, the weekly level, at the different levels of management?
06:02How can we provide that information, that insight, that story to every customer that the sales teams are talking to?
06:11And the third one, which is arguably the biggest one, is uh to provide self-service capabilities to the go-to-market team.
06:19When these sales individuals, when they're talking to a customer, how can they get the expert-level information that they need to interact with that customer and to best assess, you know, how they should approach the situation, how to handle rejections,
06:34how to upsell them, and how to handle customer satisfaction issues.
06:39Uh this is a huge part of what I've seen we've done at Cloudflare, and I'll share a little bit what that looks like.
06:47So, as it relates to scaling the analytical capability, the back-office operations, what we have done as we have built role-specific skill files, which have the context of the business information tying it to the data.
07:01This is for both technical and non-technical users.
07:04Technical users, you could say the ones who are building SQL and being able to data engineer a lot of solutions.
07:10And then the non-technical people would be more individuals who are closer to the business with the sales people who may not know how to write SQL.
07:18And so, we have skill files that they're able to use to ask questions of the data to get answers fairly quickly while doing other tasks.
07:27And one example here is in those skill files, we also, through testing, we've included the types of questions that the business would ask of the data.
07:38In this case, looking at closed date changes and opportunities, as well as uh changes in the amount of the opportunities, so that we can answer essentially 80% or or more of the questions,
07:49where the other 20% might be more uh complex strategic questions.
07:54And so, overall, this allows the teams to be able to embed all of the logic into the skill files and get answers uh fairly quickly.
08:02So, I've seen users who do not know any SQL, and the essentially their request in the past we bottleneck to someone who knows data and can write SQL for complex queries, be able to just ask questions of the data
08:14and get answers.
08:15And this is very useful also for what I show later on on the third pillar is for building skills for these go-to-market teams, so that they um can and ask questions of their data
08:28and and get answers.
08:30Also, uh in our team we've used these same skill files to build multiple applications.
08:34Uh when usually, you know, that is uh done in IT and bottlenecked in those areas, we're able to use the semantic information about the business um knowledge, as well as the columns table to build these applications
08:48rather quickly.
08:48So, this allows us to free up our time, so that we can focus on strategy and enablement.
08:57All right, for the second pillar, uh what I mentioned earlier, there's a story in the data and they really shouldn't have to search for it.
09:03Uh what I'm showing you here is synthetic data on the right.
09:07Uh we have a weekly summary that goes out, which highlights how the business is doing, how they're pacing to their goals, and then highlighting um trends, uh standouts, as well as watches.
09:18So, this uh we provide this information to the business, so they can, as you just like you can open your phone now uh on Gemini, if you have it, and you can see your notes for the day or the things that you need to do,
09:30giving that same level of information to the go-to-market team, so they can just go along their day, and if they do need to look at some of the reports or dashboards, they can
09:39to drill in, um but we bring the story to them.
09:42And I'll pull this together why I think this is really important.
09:46Um you know, of course there's a place for dashboards and standard information, but there's different level of adoption of the KPI metrics at any given company.
09:54You're going to have people who are love dashboards, people are never going to look at them.
09:58So, I think you really need to have a way to scaffold that across um the business.
10:03So, how do we do this automated analysis?
10:05So, a big part of this is simplifying the data so that the AI agents can actually um analyze the data in a very consistent and clean way.
10:14Here what we do is we transform the data by the dimension of time, also slice of the logical part of the business, which is manager, theater, and finally the metric.
10:24Here we have data uh that is wide.
10:26You could also go um from wide to long.
10:29Uh our trend information that I showed you, um uh that data is long, and then we do some pre-processing on that data um to then highlight trends.
10:38So, the the uh embedding of the logic of how you would filter this data to even analyze it, um as well as the logical um aggregations the business want to see is all engineered
10:49up front.
10:50This from my experience, this handles uh 80 or plus percent of the requests is just getting information about the performance of the teams and how they're doing.
10:59You can always go down to the raw data, but this last uh pillar, which I'll go over in a minute, um allows them to do that.
11:05To be able to orchestrate this uh effectively and be able to rely on it, we have a multi-agent workflow where we first get the data, and then we do a first pass draft on the data calling our MCPs,
11:18um and then this we have a second reviewer agent who checks the veracity of the data, and then we have a third um agent, which is a tone agent, who using a multi-shot prompt
11:27um is able to just craft the message and highlight the risks and opportunities um equally.
11:33And with every run, we have observability into each of the LLM calls, so we can see what is passed and what is the response that is going on there.
11:41And so, this architecture we tested for about 2-3 months and, you know, looking every single run to see what is going wrong.
11:50And this is the the model that we had set up that is really working for us.
11:54And we really hope to expand this beyond just what I've shown you for multiple teams, but also down to the customer level like I was just talking to you about.
12:03The third part is the self-service model.
12:06And what I'm showing you here is our internal tool called Cloudflare OS, which is an agentic workspace that is running on Cloudflare where the go-to-market team can come in here.
12:17It spins up their own compute and their own persistent environment using Cloudflare workers as well as durable objects, which is basically a storage sort of like S3.
12:29And so, the sales people can come in here and get the data they need it when they need it.
12:36So, some use cases that these teams are using it for is doing a forecast brief, building QBR decks, building a purchase deck on what the customer that they're onboarding has purchased,
12:46doing account planning, general queries of the data, as well as renewal preparation.
12:51How are they going to look at what the customer has used and either upsell them or figure out how they can get them adopting their product more.
13:01So, just a little bit more into that Cloudflare S setup that I just showed you.
13:05The AI agent workspace is where that screen I was I was showing you.
13:09And through the the three-part piece of the skills in the lower left, which is our like expert level information, as well as the MCP connection and the AI gateway,
13:22they're able to have conversations in this agentic workspace to pull data they need and using expert-level skills, which is curated, um so that they're able to execute the jobs that they need to do when they need to do it.
13:36And so, a little more information about the skill repository, uh we have a central alias where skills are presented uh to uh the central team, curated by the uh go-to-market team, as well as by operations team, and they're reviewed, so we can make sure that we're not having a proliferation
13:52of skills, and we have an expert-level knowledge skill at every level, so that they can really get all the information they need for how to approach uh any customer situation.
14:04And so, just a few more uh um images here of uh them using it.
14:07Here, uh we have them building a prescriptive plan for their daily work.
14:12They're asking a question, it's their um you can see the agent is responding by looking into the MCP and starting to pull the data together.
14:20And related to the QBR deck, uh here's a slide of generating a custom slide deck for a customer call.
14:29And so, this has really, I think, unlocked the ability of the go-to-market teams to be able to really have all their information uh serviced to them.
14:39And so, bring it together with the three pillars that I talked about.
14:42Um if you really don't have all these, I think you have issues with serving the go-to-market needs uh in terms of uh using optimizing the use of Agentyc uh systems.
14:53With self-service, you allow them to be able to pull data when they need it for whatever situation they need it with the expert-level information.
14:59The the second is by pushing the uh the insights to the business, you're able to surface the generalized standardized information of how these teams are doing, and also um
15:11yeah, so so that there's no and also uh so they're aligning with source of truth on performance.
15:15Um and then, the the third one is the scaling of the analytical team for them to be able to answer queries and build applications for the teams, which really unlocks a lot
15:25um because the opportunity cost of that team being overloaded and being able to help is that the meet the needs of the go-to-market team um is not met.
15:34So, some findings um and the future.
15:37So, uh the first thing is skill curation is the basis for all of this agentic workforce.
15:43If you're able to embed the knowledge of the business into the skill files as well as the skills uh uh for the an analyst to be able to build
15:52uh answer questions or build applications as well as the skills that I showed you in the Cloudflare OS, you're really able to uh give them the ability to use the agentic systems in a more predictable and deterministic way so that they can execute
16:07um evenly across the board.
16:09The second thing is through this whole process, the feedback loop is very important.
16:14Just like uh a company would try to sell a product externally and get feedback, with these internal teams, uh the feedback loop is very important to be able to see is is what you're building is it actually useful?
16:25What are some issues that they're having?
16:26And how can you make this uh work more efficiently?
16:30And the third thing is the layering of those uh three pillars that I talked about.
16:35Being able to answer questions where the team comes to you, some of the go-to-market team, that's how they like to interface with the operations team is to be able to ask questions.
16:43Um and then the pushing of information and then self-serviceability.
16:47Through that, you're able to uh interweave all the needs of the team to be able to be met by this uh agentic run uh team.
16:55So, through all these uh different pillars that I talked about, we've really been able to 2x our efficiency and be able to serve the teams as well as allowing them to be able to get the information that they need to do their job.
17:08And some some things um that I see going into the future.
17:11Number one is a deeper integration with uh our systems that we work in.
17:16So, for example, those QBR decks and um renewal call skills, uh how can we set up meetings for the go-to-market team and embed those artifacts in those meetings so they don't have to actually pull them.
17:30We can allow them to that self-service portal to be more ad hoc in what they need.
17:35But, that requires some information or some security setup and how can we do that?
17:40And then also, another example is getting meeting notes from those calls, which you have to set up that across the board.
17:48So, there's some system side thing that we have to work on there.
17:51The second thing is harder problems around quoting and approvals and updating the CRM itself.
17:59Uh we use Salesforce and we're just in the midst of building the connections and the ability for us to update Salesforce with these Agenty systems.
18:08And I see that being um set up in a way that I set up with that automated analysis where you have workflows uh to just make sure that everything is getting
18:19um done right.
18:20And the second thing is we're sort of reached the Cambrian stage of using Agenty systems, which means there's an explosion of excitement and skills and finding out ways to solve anything with AI.
18:32But, I see as we get to this uh fuller integration and standardization, we're going to uh want to come back and and and not really limit, but just figure out a really strategic approach for allowing each team to use the Agenty system so that the source of truth in all the systems are aligning.
18:50All right.
18:51Well, thank you for joining this talk and I appreciate you, um you know, coming here.
18:55Hope you have a great conference.
18:57[applause]
19:12[music]