WEBVTT NOTE Sentence-level transcript of https://www.youtube.com/watch?v=13YZ1EB9hjU NOTE One cue per sentence. Cue ids are the line anchors on /transcripts/13YZ1EB9hjU.html. A cue ends where the next begins, or 2 s after its last word. s1 00:00:12.960 --> 00:00:13.920 Hi everyone. s2 00:00:13.920 --> 00:00:18.080 Uh before we get started, I actually have a quick question for all of you. s3 00:00:18.080 --> 00:00:24.800 How many of you actually today use a closed source model like chat GPD or Anthropic? s4 00:00:24.800 --> 00:00:25.439 Okay. s5 00:00:25.439 --> 00:00:30.640 How many of you feel like it's getting really expensive really quickly? s6 00:00:30.960 --> 00:00:34.320 Okay, that's that's a good percentage of you all. s7 00:00:34.320 --> 00:00:49.360 And uh just like all of you might be thinking of switching from closed source model to open source models for cost reasons, performance reasons, there uh other customers and users that are going through the same journey. s8 00:00:49.360 --> 00:01:02.079 And today I'm going to talk about where these open source models shine and how to make the performance or quality of these open source models match the closed source models itself. s9 00:01:02.079 --> 00:01:07.760 And before I get into that quick introduction of myself uh I work at fireworks. s10 00:01:07.760 --> 00:01:10.400 I lead part of the applied machine learning team. s11 00:01:10.400 --> 00:01:14.240 My name is Jatashri but feel free to call me Jet if that's easier. s12 00:01:14.240 --> 00:01:25.280 Um let's talk about the whole train to deploy loop that that the open source models are taking today when users are switching from closed source to open source models. s13 00:01:25.439 --> 00:01:33.040 But even before we get into that let's talk about some fundamental truths that are true today in the AI ecosystem. s14 00:01:33.040 --> 00:01:40.560 So the three fundamental truths that are true uh are one open source models for the most part are production ready. s15 00:01:40.560 --> 00:01:48.560 If any of you have used GLM 5.2 two, you know that that is true especially for the coding use cases. s16 00:01:48.560 --> 00:02:04.960 Two, if open source models don't like beat or match the quality of close source models, the differentiating factor uh comes from how you customize, train and improve these open source models to match the performance and quality of closed source models. s17 00:02:04.960 --> 00:02:17.040 And three, the AI stack itself for most of our customers and users is shifting from let's just serve a model once once and that's done to a more continuous s18 00:02:17.040 --> 00:02:24.640 loop that is improving from training to inferencing uh the same open source model itself. s19 00:02:24.640 --> 00:02:27.440 So that's where fireworks kind of comes in. s20 00:02:27.440 --> 00:02:39.599 It makes the whole process of you having to go through the continuous journey of training these open source models as well as hosting these open source models at scale much more easier and also much more cost effective as well. s21 00:02:39.599 --> 00:02:46.720 And I'll get into how we actually do the training itself and different flavors of training that we offer towards the end of the presentation. s22 00:02:46.720 --> 00:02:55.760 But I did also want to quickly talk about inference before we go to the training portion of it which has become really popular among our customers and user base. s23 00:02:55.760 --> 00:03:12.400 And before we also talk about inference and training, I just wanted to give you a quick overview of how uh inference looks like especially at scales and what are the kind of production impact that we have seen among some of our customers and user base itself. s24 00:03:12.400 --> 00:03:19.360 So today at in production at scale uh we serve about 30 trillion tokens and it's increasing. s25 00:03:19.360 --> 00:03:22.800 Um this number was actually preGLM 5.2 release. s26 00:03:22.800 --> 00:03:28.239 And I'm sure this has pretty much kind of increased by about at least 5 trillion tokens a day. s27 00:03:28.239 --> 00:03:39.120 Now with the GLM 5.2 release, we support more than 10,000 companies globally and process about 200k requests a particular second itself. s28 00:03:39.120 --> 00:03:46.080 And the kind of production impact that we are seeing within our customer and user base is very significant. s29 00:03:46.080 --> 00:03:57.519 So cursor uh if for those of you who already use cursor and cursor models u the composer 2 model for cursor was trained and also hosted on our platform today s30 00:03:57.519 --> 00:04:08.480 and um it is able to achieve 13x faster inference because of the fire attention engine that we have and I'll speak a little bit about that in the next slide as well. s31 00:04:08.480 --> 00:04:12.080 Uh Versell is also another big customers uh of ours. s32 00:04:12.080 --> 00:04:20.320 We help them actually fine-tune and also host the model on our platform and they see a 40x latency improvement. s33 00:04:20.320 --> 00:04:23.840 And another big use case study is actually genpark. s34 00:04:23.840 --> 00:04:31.840 Um genpark lovables are in very similar field where they actually let you design systems with just prompts itself. s35 00:04:31.840 --> 00:04:40.320 And uh quality is a big bottleneck or was a big bottleneck for them when they were trying to switch from closed source to open source models. s36 00:04:40.320 --> 00:04:41.840 and we helped them fine-tune. s37 00:04:41.840 --> 00:04:43.759 We did uh RFD for them. s38 00:04:43.759 --> 00:04:55.600 We helped them clean the data set, define evaluator functions and at the end of the RFT training job itself, we saw that there was 12% quality improvement for the Gen Spark use case. s39 00:04:55.600 --> 00:05:11.199 Um also we are a trusted platform among all our other customers and user base as well and we have raised around 327 million CDC uh a while ago and we are about 200 people working at fireworks. s40 00:05:11.199 --> 00:05:13.360 That brings me to the next portion of it. s41 00:05:13.360 --> 00:05:23.680 I know in the previous slide I did mention that we made latency faster for a couple of our customers such as versell and how we do it is quite simple s42 00:05:23.680 --> 00:05:25.600 at a at a higher level. s43 00:05:25.600 --> 00:05:30.000 Um at fireworks we have our own inference engine called fire attention. s44 00:05:30.000 --> 00:05:41.120 So you have a model you have GPUs and the way the model sits on top of the GPUs is through a translation layer called CUDA kernels and as part of fire attention we build our own CUDA kernel. s45 00:05:41.120 --> 00:05:54.240 So when a customer comes to us uh with a workload and a performance target in mind then we can optimize this particular inference engine and the whole AI stack including that CUDA kernel for that specific use case. s46 00:05:54.240 --> 00:06:08.880 So most of the time customer like versel in the previous slide come to us and tell us uh they have x input tokens y output tokens uh they have zprompt to uh prompt cache hit rate and they want to meet so and so latency targets s47 00:06:08.880 --> 00:06:22.400 and we help them optimize this whole inference stack including that cuda kernel so that we can serve this model uh at a cheaper price but also at a that can meet their performance target and for their specific workload itself. s48 00:06:22.400 --> 00:06:36.880 uh as part of fire attention and inference optimization, we do also implement the very well-known techniques in the AI field which is things like speculative decoding, disagregation, quantization. s49 00:06:36.880 --> 00:06:43.440 In terms of speculative decoding itself, we also allow customers to train their draft models on our platform. s50 00:06:43.440 --> 00:06:50.960 Uh so they can even upload the data set very similar to how they do for fine-tuning for uh training a draft model. s51 00:06:50.960 --> 00:07:04.080 uh which helps them uh make the model the main model uh perform much faster uh meet latency requirements also gives you better throughput if the draft model is trained uh very well s52 00:07:04.080 --> 00:07:16.240 and um so once people actually people actually get to this stage mainly once they're happy with model quality but when customers are switching from closed source to open source models s53 00:07:16.240 --> 00:07:30.400 the biggest bottleneck or the gap is quality and the The way we we try to solve for it is through our training product and our training product itself comes in three different flavors for different segments s54 00:07:30.400 --> 00:07:32.880 uh within our customer and user base. s55 00:07:32.880 --> 00:07:35.280 One being the training agent. s56 00:07:35.280 --> 00:07:45.120 This is very popular among our product managers and builders that get that got into this AI inferencing and training field very recently. s57 00:07:45.120 --> 00:08:00.800 So the agent can be accessed through our UI and uh you can you you just need to link it the data set that you have uploaded uh on the fireworks platform as well as description on what you're trying to do as part of this training run itself. s58 00:08:00.800 --> 00:08:02.879 Uh it will even let you choose the model. s59 00:08:02.879 --> 00:08:04.720 It will clean the data set for you. s60 00:08:04.720 --> 00:08:14.400 It will also help you do the hyperparameter uh fine-tuning itself because it's not always easy to get the right uh hyperparameters itself. s61 00:08:14.400 --> 00:08:18.400 um when you are doing training for your specific workload. s62 00:08:18.400 --> 00:08:29.280 The next flavor also one of our most popular flavors among our customer and user base is our managed training and this is what most of our customers use today. s63 00:08:29.280 --> 00:08:37.120 They can use our managed training platform one through API and second through the UI itself. s64 00:08:37.120 --> 00:08:43.120 It is as simple as you inputting the hyperparameters that you think will be good for your model. s65 00:08:43.120 --> 00:08:47.920 choosing a model, uploading the data set itself and then clicking a button. s66 00:08:47.920 --> 00:08:52.800 That's how simple we make it on uh fireworks for the manage training itself. s67 00:08:52.800 --> 00:08:59.120 And uh I know one challenge a lot of people have is observability once the training run itself kicks in. s68 00:08:59.120 --> 00:09:09.360 So we also have a good UI dashboard that shows you how the training uh is happening, how well your model is uh learning at each epoch, what does the loss function look like. s69 00:09:09.360 --> 00:09:24.160 Additionally, you can also plug in u your weights and biases API key into the manage training service itself uh and export some of the observability metrics to platforms like weights and biases if you're already using that. s70 00:09:24.160 --> 00:09:32.320 And the third and the kind of the most advanced flavor of training that we offer is the training API itself. s71 00:09:32.320 --> 00:09:41.680 And the training API is very popular among some uh of the ML engineers within uh different organizations or within our customer base. s72 00:09:41.680 --> 00:09:52.240 And in this in this particular this particular offering of the product lets uh ML engineers define their own training loop and loss functions. s73 00:09:52.240 --> 00:09:57.600 And this becomes important when you want to customize your training job according to your needs. s74 00:09:57.600 --> 00:10:11.279 that is that goes well above beyond the managed training service that we offer itself and these these different uh flavors or the products within the training ecosystem that we offered s75 00:10:11.279 --> 00:10:18.320 are offered for supervised finetuning reinforcement fine-tuning and even DPO as well. s76 00:10:18.320 --> 00:10:30.240 Um nowadays I think RFT has become extremely extremely popular and if you ever need help in helping define a good evaluator function then we have worked with customers like Genpark s77 00:10:30.240 --> 00:10:41.680 that I was talking to you about in the previous slide to help them define an evaluator function clean the data set um and also choose the right hyperparameters especially for RFD. s78 00:10:41.680 --> 00:10:45.200 Unfortunately the training agent itself does not support RF. s79 00:10:45.200 --> 00:10:55.519 So it does not uh help you clean the data set or even do the hyperparameter tweaking but we are more than happy to help you do it if you have a big use case in mind. s80 00:10:55.519 --> 00:11:07.920 So now that we know all about the different um training offerings that we have I also wanted to show you a quick demo of each of these the training agent manage training and the training API itself. s81 00:11:07.920 --> 00:11:15.200 Uh but before we get into that I just wanted to close off by saying how the data flywheel looks like today on our platform. s82 00:11:15.200 --> 00:11:28.800 I know in the first slide I was talking to you about the realities of AI today where most people prefer doing a continuous loop where they train host train host as they get new and new data and feedback from their users s83 00:11:28.800 --> 00:11:38.480 and that is what we are trying to make more seamless on our platform and that's also why we have different training products different offerings as well as making it easier s84 00:11:38.480 --> 00:11:44.560 uh for you to host and make it also cheaper and faster for you to host those models at scale itself. s85 00:11:44.560 --> 00:11:48.240 And the way this flywheel works is kind of very simple. s86 00:11:48.240 --> 00:11:55.040 Once these uh models are trained, I will show you uh a demo in the next slide where the models get trained. s87 00:11:55.040 --> 00:12:00.320 It's it's you just are one click away from deploying these models into production. s88 00:12:00.320 --> 00:12:02.720 And once you've deployed, you can uh monitor. s89 00:12:02.720 --> 00:12:12.240 We have u monitoring metrics for the number of tokens, cash it rate, how your speculator is working, if you've trained a speculator. s90 00:12:12.240 --> 00:12:26.480 uh we help you collect gather uh feedback from all the all the metrics that you have collected and if you choose to also retrain on the same platform from the end from the checkpoint that you were using to a new checkpoint itself. s91 00:12:26.480 --> 00:12:35.440 So some of some capabilities on the platform that makes these data flywheel possible is one definitely having a host of uh GPUs. s92 00:12:35.440 --> 00:12:44.320 We do have we do partner with different cloud and neocloud providers which makes it easier for us to provide compute and also host these models at scale. s93 00:12:44.320 --> 00:13:00.399 Another thing is we offer is hotloading and this has become really popular especially when we were training models for cursor and this was actually a feature we implemented specifically for cursor and then we thought would be useful for general public while they were training models as well. s94 00:13:00.399 --> 00:13:09.040 uh what hot loading essentially means is when you have a new checkpoint uh you you can directly go to live deployment without any downtime itself. s95 00:13:09.040 --> 00:13:19.760 So if you have a deployment and you've gone through this continuous loop and trained a new checkpoint then you can just click a button and swap out the weights itself that you were using s96 00:13:19.760 --> 00:13:22.720 uh uh previously with the new endpoints. s97 00:13:22.720 --> 00:13:28.399 Another thing that makes it uh easier to train on our platform is the numerical fidelity. s98 00:13:28.399 --> 00:13:38.880 So a lot of the time customers use platform A for training and host on platform B. But what happens when you do that is you'll have issues in quality of the model. s99 00:13:38.880 --> 00:13:46.800 And the reason you have those issues is humix is not always compatible when you train on a different platform and host on a different platform. s100 00:13:46.800 --> 00:14:00.240 So that's why we offer training and inferencing on the same platform and ensure that uh you won't find any numerical differences after you finish training and once you decide to host this particular model on the platform itself. s101 00:14:00.240 --> 00:14:06.720 And um one other thing that that makes it more seamless is because training and inference is on the same platform. s102 00:14:06.720 --> 00:14:12.800 You're just one click away from uh deploying this model at production scale. s103 00:14:12.800 --> 00:14:22.480 Now that we have learned all about uh the inference and training, I just wanted to show you quick demo of different training products and how they work on our platform. s104 00:14:22.480 --> 00:14:27.920 The first portion of the demo is showing you how you can use the training agent itself. s105 00:14:27.920 --> 00:14:34.720 Over here you can uh input where your training data set lives within the fireworks what it's called. s106 00:14:34.720 --> 00:14:40.800 Uh you can also uh give it the eval data set itself. s107 00:14:40.800 --> 00:14:45.199 uh I know people have the train and eval or a test data set separately. s108 00:14:45.199 --> 00:14:54.959 So you give the mo you give the agent both the data sets and then you ask it uh to fine-tune and even select the hyperparameters and the model for you. s109 00:14:54.959 --> 00:15:02.480 The the agent then goes on and tells you the step that it will it will take to accomplish the task that you have given it. s110 00:15:02.480 --> 00:15:05.120 So it it details the SFT plan. s111 00:15:05.120 --> 00:15:07.600 It tells you why it selected the base model. s112 00:15:07.600 --> 00:15:11.120 In this case, it's selected quen 38B. s113 00:15:11.120 --> 00:15:19.120 And it'll also give you some information on the hyperparameters and what the whole training process can look like at a high level. s114 00:15:19.120 --> 00:15:28.800 And once you're done uh once you are uh you feel okay about the output that the agent has given, you can approve whatever the agent has uh the a the agent has outlined. s115 00:15:28.800 --> 00:15:37.120 And once you approve it, it will actually go through different steps and keep you posted on if it finishes step two, what does uh it look like? s116 00:15:37.120 --> 00:15:40.959 If it successfully finishes step three, what does that look like now? s117 00:15:40.959 --> 00:15:55.759 And at the end of the whole training run, it will also give you a summary of uh the config, the base model um and uh different hyperparameters and how that impacts it as well. s118 00:15:59.519 --> 00:16:06.959 And once it's once the whole model training is done, it actually gives you metrics such as accuracy and F1 score. s119 00:16:06.959 --> 00:16:13.279 Uh and it also summarizes kind of resummarizes and gives you an overview of everything that uh that it did. s120 00:16:13.279 --> 00:16:20.959 And one cool feature is it also gives you recommended next step as part of u the training run itself. s121 00:16:20.959 --> 00:16:28.240 Now the next portion of uh this demo actually talks about how we can use our uh manage training platform. s122 00:16:28.240 --> 00:16:37.360 I did mention that you just have to select the model data set and the hyperparameters and then you're just a button click away from actually training a model. s123 00:16:37.360 --> 00:16:41.040 Say you've uploaded uh the data set, chosen a model. s124 00:16:41.040 --> 00:16:52.240 Uh all you'll have to do is then uh go to the go to the u evaluator settings, choose the evaluator data set and also you get to choose the hyperparameter s125 00:16:52.240 --> 00:17:00.800 as part of optional settings if you decide to have a more fine grained hyperparameters for your uh training run itself. s126 00:17:00.800 --> 00:17:11.679 And once you're happy with all the configurations including the model and the data set itself, you just need to create uh you just need to click on the button that says create. s127 00:17:11.679 --> 00:17:18.240 Um and as soon as you click on the button that says create, a training job then kick kicks out. s128 00:17:18.240 --> 00:17:25.360 And you can see that uh the UI looks similar to this when a training job actually gets completed. s129 00:17:26.799 --> 00:17:38.000 And if you click on that particular training job itself, it'll give you information on the data set it used, the hyperparameters it used, what the model loss itself looks like when the model was training. s130 00:17:38.000 --> 00:17:46.400 Um, and then you can even clone these training runs and reproduce it with different settings and configurations itself. s131 00:17:47.600 --> 00:18:01.360 And once you're happy with the fine-tuned model, you can go to the deployment section, choose the custom model that just got uh trained on our platform, and then click on deploy once you're happy with the deployment configurations. s132 00:18:01.360 --> 00:18:09.440 Or you can choose a set of deployment configurations that we have such as throughput, fast, minimal. s133 00:18:09.440 --> 00:18:22.880 uh and this is uh kind of a template that that works well for generic workloads that we have optimized based on throughput latency uh time to first token and other such metrics s134 00:18:23.840 --> 00:18:39.840 and uh I unfortunately don't have an entire demo for our training API because the setup itself differs from customer to customer but we do have a lot of cookbooks on our websites that you can see and follow especially if you're interested in s135 00:18:39.840 --> 00:18:44.400 customizing your own training loops and loss functions itself. s136 00:18:44.400 --> 00:18:47.039 Uh and that's all I have for you for today. s137 00:18:47.039 --> 00:18:52.640 I just wanted to give you a brief overview of how the whole training to inferencing loop looks like. s138 00:18:52.640 --> 00:18:53.440 Thank you very much. s139 00:18:53.440 --> 00:18:59.200 And if you have any questions, I'm happy to take them in the next minute or so that we have.