Stop Rationing Tokens: Let the Harness Pick the Model — Kimchi by Cast AI

AI Engineer · 18 min · 220 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:13All right, guys.
00:14Very nice to meet you all.
00:16My name is Laura.
00:16I'm the co-founder and president of Castai, who is the inventor of Kimchi, which I'm going to show you in a few seconds.
00:24Um the talk is going to be about preparatory models for edge case but use open source for the rest and you're going to see some great result we have with that with the coding agent called Kimchi.
00:35I have the pleasure to be here with Zilinas.
00:38Hey guys, I am Julianas.
00:39I am leading engineering for Kimchi coding platform.
00:42So you have Zilinas who run the entire engineering team for Kimchi.
00:45I'm so excited to be here with you.
00:48All right, let's go.
00:48So um you you may have seen this in the news uh two two piece of information that came up in the last few weeks.
00:56The first one is there's a company in India that spent $500 million on anthropic alone in one months which is a kind of a good example of tokens that are completely out of control.
01:08The other example is a famous tweet from Uber, the CTO of Uber, who say that in the last four months he actually used the entire budget of anthropic for the entire year.
01:20And this is how much the token um the token maxing or the token cost is a thing for the enterprises.
01:28And you're going to see how we fix this.
01:30Companies responded this by limiting the use of token.
01:34You may have heard this many many times with our friends developers and as a developer when you limit the amount of token it feels like this.
01:45It's like hey you can use your laptop you have an hour battery and you can only charge once once once a day.
01:54That's how it feels to limit the amount of token for a developer.
02:00We at Kimchi think this is wrong.
02:03Our jobs as manager is to do the opposite is to ensure that we want to make the tokens unlimited and inexpensive.
02:17Our job is not to prevent the developer to use coding agent.
02:21Our job is to make sure they can use it as much as they want for as long as they want in a completely unlimited fashion.
02:29This is why we started Kimchi.
02:32At Kimchi, we have 300 developers.
02:35Oh, sorry.
02:36300 employees.
02:38Twothird of those are developers.
02:40The Zeven has run that.
02:42And we've been using this coding agent for the last three months.
02:46I'm going to show you what we built, how we built it, and the result of it.
02:52And then Zilvenat will show you a few more things that make the coding really, really cool.
03:00It's coming now.
03:03We built an entire token optimization based on this idea.
03:08What you see on screen is the true cost of LLM model for the same task.
03:15It was made by the famous white paper in a university.
03:19And here's what they say.
03:21On one side, you see the cost per token.
03:25On the other side, you see the cost per task for the same task with the same quality.
03:32That is the important thing.
03:33Otherwise, you cannot compare Apple with Apple.
03:35And here's what is written here.
03:37First, they are all different, but also look at Gemini 3 Flash, our friends at Google.
03:44What it seems to be a relatively inexpensive model at $3.5 per million token.
03:51It's a blended average at the time this study was made, but actually the task to complete with Geminina 3 was 305, sorry, $75.
04:03And look at the others.
04:04Minimax 2.7 is one of my favorite.
04:07The cost per token is 1.5 and the task the t the cost for the task is $148.
04:15It means the models are not the same.
04:18They don't cost the same, but they also not the same per task.
04:23And that's why we built our coding agent.
04:27Would it be nice that you have an automated fashion, automated engine, automated harness that will select the right model for the task at the right time based on the outcome
04:39of the task.
04:41That is what we did with Kimy Coding.
04:44That is what Zilmanas did with his team.
04:48And here's the results.
04:50We've been using it for three months.
04:522.5 times in the amount of savings we have overloaded for the period.
04:58This is a graph of what would have been our cloud bill by day.
05:05The purple one on top is what would have been cloud cost over time.
05:12The blue line is what we paid.
05:17same outcome.
05:19But look at this.
05:20The number of tokens increased by 1.5 times.
05:26Okay, we all experienced that.
05:28The cost to code decreased by 1.5 times.
05:34This is our job as managers.
05:36We provide to our team a coding agent that is essentially unlimited because of that coding agent is growing 3x less 2.5 time less than the gross of token for the task.
05:52There's another result the multimodel I showed you before.
05:57I'm going to show you the evolution of the selection of the model over time by the harness.
06:05And here it looks like you see there is a lot of orange early June and there's a lot of yellow late June.
06:14There's a complete shift meaning the harness is selecting a very different model.
06:19It happened the 12th of June.
06:23And this is what happened the 12th of The 3rd of June Kim Kim 2.6 is winning by far the most popular of the harness.
06:3421st of June the one winning is Minimat 3. Us as human probably we can find and understand and test and be comfortable with a new model coming in.
06:46Do you think we look at this as human?
06:50Absolutely not.
06:51But an autonomous harness, an automated coding agent that is obsessed with cost is going to do just that.
07:03That is what I mean when I say our responsibility is to think about providing our developers with the right cost per token for the same outcome.
07:16It is our responsibility to do it.
07:19Zinas is going to tell you a little bit more of how it works because whatever you're going to see it's open source.
07:27Go Zas.
07:30Okay.
07:31So one more thing to mention.
07:32So Laura mentioned that token maxing is no longer an option for everyone and uh as a manager at the company as most of you guys probably heard the having unlimited amount of tokens to spend is no longer looking to be like a case and uh we had to build this hardness for ourselves.
07:51So meaning like we ended up spending too much on cloud code like half a year ago.
07:55We saw this uh cloud bill skyrocketing and we started working on our own hardness.
07:59behind the scenes it's using like pono SSDK but uh the goal of that is like to take the open source experience still contribute back to open source by providing
08:09our own harnesses open source building constructs like ferment which fits into the kimchi theme which is meant for longunning tasks where on average we estimate the human envelope for these tasks is uh
08:21typically happening only like every two hours every three hours and etc. So the goal is uh you use Kim to harness you run a firmment process.
08:30It asks you a bunch of questions and autonomously runs a coding task for like more than two hours.
08:35The goal of that is that because it breaks it into milestones.
08:39It knows how to implement it and at the end of the cycle you would get scoring for your output.
08:44So you would actually know what's happening.
08:46And going back to previously discussed like the amount of models used with uh Kimchi harness the scoring is there to prove you that whatever model we will pick for you you will always have like
08:57good enough scoring for the output on the task you're implementing.
09:01The way whole Kimchi platform works the goal is like uh we are not just providing you the coding harness.
09:07We want to provide you the full software development life cycle solution.
09:11So you do a change it runs a build for you.
09:15It checks if something broke.
09:16If something broke, it will go back, do a change again, rebuild again.
09:20And that's uh how this fermentation process looks.
09:23If everything is complete, uh there is no extra limitation needs to be done.
09:27The changes will be deployed to the staging environment.
09:31And you see what what I like with the way explain it is the harness select the model fment is constantly checking the quality of the code and going back
09:44meaning there is a a higher order model that is checking that quality and thinking that quality is not good enough please do it again this is what I was saying earlier by
09:54it looks at the outcome and it's optimizing the token based on the quality of the outcome.
10:02That is what is super cool with FET.
10:04Yep.
10:04And artifact is only considered complete once like the output of the scoring like at least B as a score.
10:11And even for that like if you don't like the score you can ask agent to go back fix it up and like it will try to achieve the A as a scoring.
10:21The intention of stuffing at the staging is currently by design.
10:25We still believe there's like some little human that needs to be in the loop before going to the production.
10:29At least at this point we all know that the ultimate f future is to avoid that.
10:34You go to production, you look at the metrics, you look at the observability data, the solo metrics you have in your environment.
10:41If you're running on Kubernetes, you want to check if the pots are not crashing, if application health checks are passing.
10:46So all of these things need to be done before it ship it shipped to production automatically.
10:51So this is like one of the things that are coming.
10:53So the current limitation we stop at staging but uh we'll go to production and and further on that what we believe is that reading code is not enough anymore.
11:04So like uh probably most of you noticed like across this year like uh the more vibe engineering or agentic engineering we do in our coding environments the more code changes we get in our chain sets and it kind of becomes unbearable to read especially as you keep bringing like a little bit of a less technical personnel into
11:23your pipelines.
11:24So imagine like product manager doing the coding or my dear colleague Laura starting to do some coding and suddenly he comes up with a PR which is like 2,000 lines long and you need to actually review not just the differences but the actual intent which he wanted to achieve and the specifications
11:41which he provided to the coding agent.
11:44So yeah moving on uh one thing we built more on top of that is the thing we called kimchi teleport.
11:52So Kimchi teleport allows you to close your laptop.
11:54It spins up sandbox in the remote environment.
11:57It's a secure sandbox where you don't need to care about anything.
12:00We can do that both on our SAS platform.
12:02We can ship it to your on premise deployments if you're like a financial technology company and uh things like that.
12:09So we will manage this agent sessions for you.
12:11We will make sure they are running.
12:12We are accessible both from your laptop, from mobile device if we are running in the background.
12:17So it goes back and ties to the story where I mentioned that the human in the loop is only appearing like every two or three hours.
12:24So teleport emphasizes that.
12:26You run your process, you teleported to the remote environment.
12:30The y spins up in the background.
12:32You don't need to care about it.
12:33We will make sure it's the right size, it's the right security requirements and everything else.
12:37And it will keep on running until the task is done or until the human in the loop needs to be involved.
12:43Get notifications for some clarification of specs and things like that.
12:48Guys, I really like this idea.
12:50Look at the this this is a screenshot of teleport.
12:54I am puzzled that sometimes innovations are always the simplest one.
13:00Like this was born out of the idea that one of us was in the plane coding and then Wi-Fi breaks.
13:08And what do you do when Wi-Fi breaks?
13:10It breaks the coding because it lives inside your thing.
13:15No, I think it was not the Wi-Fi.
13:17It was the battery that broke, not the Wi-Fi.
13:20So you lost.
13:21So he That's it.
13:22The guy was stopped there.
13:25What teleport does it it takes all your environment.
13:28It synchronizes with a container that live inside one of the hyperscaler.
13:33In our case, it's Google and locally on your laptop.
13:38It feels exactly the same.
13:39It just doesn't run inside your laptop.
13:42It feels like it.
13:45It runs somewhere else in your environment in a container in a Kubernetes cluster inside the place at Google that will continue running even if you close your laptop.
13:57how simple that is.
14:00And yet our favorite tool is this one.
14:03I think the the last one I I I looked at it, 62% of our engineers are only using teleport when they are coding now for that problem.
14:12They go home, it continues.
14:14They're in transit.
14:15It continues.
14:17They on holiday, they don't look at it anymore.
14:19It continues.
14:22That's so cool, huh?
14:23Yep.
14:24So very smart to that.
14:26So we built another tool called Kimchi studio.
14:29If that teleport is something typically individual uses, studio is kind of like a teleport for teams and enterprises.
14:36The goal behind the studio is uh that you have as kind of like board on your environment.
14:41It again runs in the teleport sandboxes in the secure environment.
14:45The thing here is like you're supposed to share your tasks and work in progress and plans which are made by the Kimchi coding harness with open source or proprietary models with your teams.
14:54So for example imagine you as an engineer you are like working on a task uh you started implementing asking for a plan the plan can be easily reviewed by your colleagues like by your pe engineer by your product manager or someone else who cares what's happening and
15:08this is how you kind of work with your whole team on collaborating on this agentic engineering journey instead of like saying oh I'm going to type this prompt I'm going to try to execute it to the production
15:19only later on to learn that the specification and requirements were incorrect the output is not what was actually wanted by the task and things like that.
15:28And this is yet another thing we built for ourself, we find it so cool, we decided to make it a product.
15:37How do you work with your colleague?
15:40If what you have is a C is a is a terminal, a CLI, how do you share that with the others?
15:47Well, first he has to live somewhere else.
15:49So teleport is the base of what we call studio and studio is a very nice way to visualize all the sessions that are running including those that are asking
16:01for a review and then someone in the teams can say I take it and I answer the uh I I answer the question that is for review.
16:11Then you answer it.
16:12It goes back to in progress until the next one goes into review.
16:17And you can have many many of those in a team in a pizza team, a team of five to 10 guys.
16:24Think about it.
16:25Yet yet a very simple innovation that is so essential in how our engineers are coding.
16:33They can now share in a very simple way, right?
16:36It's a it's a browser.
16:37It's graphical.
16:38They can share and they can take over the task according to the comban style of what is in review, what is in progress.
16:48They can start new ones with backlog.
16:50And you know what?
16:51If they close the laptop,
16:53it's continues.
16:55This is so cool.
16:57Yep.
16:58This is really cool.
16:59Thank you.
17:01Yeah, we forgot to show.
17:03Oh, this is this is the one that we took over, right?
17:06That's the one,
17:07guys.
17:08You can see this at our booth if you want to uh to see more of it.
17:12We are in U4.
17:13It's exactly this way across a few streets.
17:17But have a look at it.
17:18It's really good.
17:19It is open source.
17:20Most of it at least the harness is entirely open source.
17:24Uh studio and teleport.
17:25You need a Google account.
17:26It's a little bit more com uh more complicated, but it's a a five minute install.
17:32And then you get your teleport setup as many sessions as you want inside your Google account and then you can share with share it with your colleagues with Kimchi Studio.
17:43So come see us.
17:45We'll be delighted to speak to you.
17:47Thank you guys.
17:48Thank you guys.