One Operator, Many Drones: Inside Skydio's Autonomy Stack — Suchet Bargoti, Skydio

AI Engineer · 20 min · 213 sentences · from YouTube's caption track

Each timecode opens YouTube at the start of that sentence. Line anchors (#s42) are the cue ids in the WebVTT, and every line carries its start and end seconds. All transcripts has every talk, and the whole corpus as one file.

00:12Thanks everyone for coming.
00:13So we'll be talking about how um to use Agentic sort of orchestration to command many many different drones to achieve the different tasks that we have ahead of us.
00:23and just going to rethink this presentation a bit and not just jump straight into slides, but instead we're going to fly.
00:30Um, so what we have here on the left hand side is a real time view.
00:36What we're going to be doing right now, am I showing anything?
00:41Well, I'm flying.
00:42There you go.
00:44Sweet.
00:44Okay.
00:45Um, we are somewhere here in the city down south, which where where our headquarters is.
00:50Uh we have a few drones up and I'm going to hit launch.
00:55Uh and these are drones that are in docked stations.
00:59So we calling this drones as infrastructure uh where we actually have thousands of these drones now planted around the country with power utilities with uh public safety uh with construction companies.
01:11And what I'm doing right now is using my keyboard to simply fly here.
01:16Uh for those of you that know the area, this is San Mateo.
01:19And perhaps if I kind of zoom in here, we may see a very faint render of the uh SF skyline.
01:25Although the uh the fog city is always there.
01:28So we might just say hello to um the uh SFO airport here.
01:33Some planes are launching.
01:34If anyone has a plane tracker app, they can sort of see this is uh realtime stuff there.
01:39So what we have been building is the full autonomy stack behind like how does a vehicle operate autonomously?
01:45How does the cloud system operate here?
01:47How do the cloud servers work?
01:49uh and where does the uh different levels of intelligence and automation needs to happen for us to make this happen robustly such that when I'm here and saying hey oh there's an incident happening here I need to go respond
02:00I could at the same time go back here and say hey what about that other thing that's happening on the other side of the country what if we launch that instead so while that's happening
02:10I'm now going to launch something in Colorado so this is a imagine some sort of fire incident has happen on your on the power lines and we wanted to go out and
02:21inspect those things.
02:22So, a dock is now opening up in Colorado while the first drone is still flying safely.
02:28Um, and I'm just going to uh let it do all the safety checks that it needs to do before the launch.
02:33And this is all happening in sort of conference Wi-Fi traffic.
02:36So, you can imagine I can shut down my laptop right now and everything needs to safely happen behind the scenes.
02:42Um, so I'm going to do this while the first part is still happening.
02:47Let me let's go back to the first drone.
02:52Might give it another set of instruction.
02:54Let's have a look around here on the first drone.
02:56The second drone has sort of kick started off and maybe there's some cars sort of going around there um that we can perhaps track.
03:04So, let's track and what's this car doing here?
03:09All right.
03:09So, we're now tracking this car.
03:10Maybe this is a runaway car that we needed to follow.
03:13And I'm hands off right now.
03:14This is uh the autonomous system kind of taking over um through the various interactions that I've given uh for it to do.
03:22And at the same time while this is happening we can do the final thing which is yet another sort of drone in the system uh back at our HQ
03:34and we can say hey let's why don't we run a third drone.
03:38So what we're building towards is how do we enable autonomy at scale where traditionally how it started off historically 15 years ago that you might have a drone at home.
03:50It's a hobbyist drone and you might play around with it.
03:52You will tinker with it.
03:53You will work with the controlling software.
03:55And then about 10 years ago drones started to become a lot more available and they started to become a tool.
04:01A lot of industries out there started to use them.
04:03They will carry them with them in the truck, go out there, deploy it.
04:07And the next era that we're working towards is drones as infrastructure.
04:10Imagine these systems.
04:12And for the sake of this conference, I'm going to call them agents.
04:14These are physical emboded agents that are kind of available at all times at anywhere for the different sort of use cases that we're interested in that can automatically uh uh launch, execute, and do their tasks.
04:27And what is the minimum amount of autonomy that needs to be begged in?
04:30And what does the future interface look like today?
04:32Everyone needs to be a dedicated pilot.
04:35I go went through a certification exercise.
04:36I need to think about the safety standards here.
04:38But you can imagine in a few years time the safety is going to be determined by the autonomous system and the interface becomes really high level.
04:47It could be a little slack bot that says, "Hey, something's happened here.
04:50Why don't we go send a drone?"
04:52And you might not even know that the drone launched.
04:55So, while this is happening, I'm going to tell all the drones to pause and return to doc.
05:03So, we'll write this.
05:05They're all going to start returning to doc.
05:06And while that's happening, I'm going to start on the presentation.
05:14So, what I've shown you here is not just concept.
05:17These are literally systems that are being used in production.
05:20Uh as was mentioned we are the largest manufacturer of drones in the US and we want to give people superpowers uh through this technology.
05:28Let's take a quick look at what some of the people are doing here.
05:31This is um in the northeast coast of the US uh where our client has set up uh the system next to a power station and they kind of used this to do normal patrols and as they flew around
05:42they found that some of the there was a the pole was burning from the inside and it was really starting to show up here and this could have fallen at any time and created a fire risk that they would have otherwise
05:52not caught without having to send people there which itself is quite uh expensive.
05:59Moving on to the sort of next use case is on the public safety side.
06:03This is San Francisco.
06:04Uh we work very closely with SFPD.
06:07Um normally when a car gets stolen, you'll see a high-speed chase happening in the city.
06:11Quite dangerous.
06:12But what if you could deploy a drone instead where the uh people in the car don't even know that there's a drone following them.
06:18Here's a person who has stolen the car on the right hand side and they're about to change their license plates uh on that.
06:26So they go here, get out their tools.
06:29They come back, luckily they point the license plate up so the drone can see it and we know exactly what they're doing.
06:34But they're now replacing the plate uh in the car uh with a new one.
06:38And all this time they don't know that there's they're being chased.
06:41The the the how they behave in the uh public, how they behave out there is very different and it's a lot safer uh in how the operations are done.
06:49They're now going to go ahead and tint the windows.
06:52uh but this allows the police to like strategically position themselves in the most safest way form to intervene at the right time rather than doing a high-speed chase outside.
07:04So these are being used across many different industries and we are really starting to treat this as infrastructure that can operate day in day out nighttime rain sunshine.
07:16We've got a few of these docs deployed in Alaska, so very cold weathers.
07:19Few of these docks deployed in uh Texas, so extreme heat.
07:22And these need to be reliable down to 99.9999% uh where we do our sort of simulation and testing to be able to prove that.
07:29And today, uh we have about 16 million people living within 2 mi radius of this infrastructure that the uh um public safety, the power companies can kind of use this technology to be able to respond to such incidents without having to travel there.
07:45So, what really happens when this happens at scale?
07:48Can I get a hands up of people that have flown a drone before?
07:52A couple of hands up.
07:52When I started to fly it, it was like kind It took me a few hours to like really figure it out.
07:57Then I put on the FPV, that was even more tricky, but it felt good to get get that expertise up, but it's kind of like a skill that you develop
08:04and you develop that skill over time and you say, "Okay, for each skilled person, we're going to put them next to a drone and they're going to start working."
08:12But now you want more of these.
08:14So, uh, police companies, infrastructure companies need to start hiring these people more.
08:18And at some stage, this really starts to break down.
08:20The more 911 calls come that come in, the more alerts that can come in, it doesn't really scale.
08:26So, we kind of rethinking as to what this means in terms of this like initial firsterson view engagement with these systems to how do we convert this to a more strategic
08:35uh, multi- aent view that you can kind of command the entire fleet with an objective in mind without having to worry about the flight.
08:42Let me just go back and see if that was all working well.
08:44Great.
08:45They all landed.
08:46I am still pleased when that happens successfully.
08:48Although it's meant to happen all the time.
08:51All right, let's go back to this.
08:55Oh, all right.
08:58We're just going to carry on.
09:02So, what does that mean when we start to launch different things?
09:05You saw me launch them.
09:06I was still kind of thinking about it.
09:08Okay, I need to launch this one, that one, that one.
09:11And even myself in this like uh who's used to this, I'm going to have some cognitive challenges where our vision is to be able to launch uh many many of them uh in a potentially unsupervised
09:21way.
09:21So what sort of commands these things?
09:23How do we get it to like just hey get out there, launch, search in this area, find a missing person or look out for this type of car and hold your position there.
09:32And in order to do that, we really need to think about how do we get like really like the many nines of reliability that we need in autonomous flight.
09:41And that's where we get an edge in the industry because at Sky we're sort of controlling the hardware, the software, the cloud, the user interface to be able to manage all that and specifically the autonomy.
09:52uh allowing us to think about how our underlying vision system should work to see the environment to behave in the environment correctly whether it's at high altitudes whether it's cloudy
10:02whether it's at high speeds how do we deal in the rain on the bottom left we're showing how do we navigate in cities how do we plan large scale and be able to do that and for anyone that has worked with any sort of GPS device in the city even our phones they kind of suck
10:16uh so how do we robustly do that and how do we also like do tracking when there's a lot of occlusion These are the all the different places where we're thinking about how to train AI systems quote unquote models.
10:28The word model itself is uh has different meanings in different places and I'll discuss a little bit on what that means for us.
10:37But in order for us to you like really harness this is we are learning on the go.
10:41This is a learning flywheel that we're getting out there, we're collecting data, we're operating, and we're coming back and doing that so that each flight we can log the data.
10:50Uh kind of like Google Street View where we need to think about sanitizing that data, make sure there's no private information left there.
10:56Uh and make sure we don't sort of like uh uh the customers know exactly what they're sharing with us.
11:01But if we can once we do that, we have access to a huge amounts of data that we can kind of learn from every single time we instructed this but the drone did this.
11:08every single time.
11:09We thought this was going to happen, but this happened.
11:11That can come back to our uh learning agents, our reinforcement learning ecosystems to be able to retrain, evaluate, and send it back out there and kind of continue that flywheel
11:21that allows us to uh have that robust framework.
11:25And the other added advantage that we have is it's not just about having autonomy on the drone.
11:30As I mentioned earlier, we have the luxury now to have autonomy on the edge device, but also have autonomy on the cloud.
11:37what I was showing you earlier, all that video feed, all the telemetry that's going through a cloud server.
11:41We could set up GPUs and we could set up inference engines there to be able to have that heavier lifting, maybe that longerterm planning there, whereas the immediate autonomous actions happen on the drone and kind of always thinking about the trade-off that we need to make uh to make that successful.
11:57One thing is true, however, that uh once you do start thinking about having your agents from the cloud, this the amount of data coming to the cloud really matters.
12:06Uh here's us sort of investing in how we think about enabling the best um uh video quality coming up through to the servers uh in uh low lower low sort of bandwidth areas.
12:18Uh being able to sort of optimize that being able to un uh um encode that information into smaller sizes and be able to decode into something clear that allows us to have much higher quality exact same sort of network conditions here.
12:32uh and it's the some of the areas of investments that we make so that we can have this more cloud-based infrastructure uh to be able to manage this at scale.
12:41So once we do that I want to sort of explore at a high level some of the uh models that we um uh have in our system uh that allows us to orchestrate all of this.
12:53Firstly a model this is a section about world models.
12:56I want to sort of talk about world models from a context of maps.
12:59Not too dissimilar to how uh Whimo works.
13:02They have a map of the world and they kind of navigate in that world.
13:05They do local perception, but also think about global planning.
13:08If I want to go from part A of the city to part B, I can't just like keep hitting every building and kind of navigating around them.
13:14I need to think about what's the optimal path uh along the way.
13:18So, we start off with a lot of prior information.
13:20uh um and we merge that with uh not just sort of building data, but maybe there's uh vector data such as where the power lines are, where the roads are, if you want different behavior
13:32in these areas.
13:32And we think about how to combine these resources to ultimately build a map that we can plan and navigate around.
13:39And the drone has knowledge of this map uh at all times to be able to go around that.
13:45But like any map, maps can go out of date.
13:48Uh luckily we have so many eyes in the sky to think about how to maintain and update these maps along the way.
13:56On the left hand side uh we are rendering our knowledge of the world in the points onto our video feed.
14:05However, that doesn't line up perfectly everywhere.
14:08Uh there's some sections here that um if I sort of zoom out here, there's some sections here that a new construction site had set up uh that we did not know about.
14:18Uh however as a drone now starts to fly and not this is not just one drone but your fleet of drone they're now observing these things that we can feed back into our
14:27uh map syncing process that can come back land give the data and now once uh in the next iteration all the drones in the fleet have this most updated map of the world that they can uh do all the planning in.
14:44A second type of model is uh perhaps today a more conventional sort of uh machine learning uh inference model uh which is the ability to uh be able to track understand objects in the scene but be able to track them be able to track them behind occlusions.
15:02So the uh this implicit representation behind the scene is some sort of world representation of the object that hey it's gone behind this building and it might come out on the other side.
15:11So I should navigate myself so I can kind of follow it there or I should move myself in a different direction to be able to do that.
15:18Maybe 5 years ago this would be more done uh in a more conventional way.
15:21You have to like manually think about how to move.
15:24But now you can think about uh more reinforcement learning style techniques or more learned end to-end approaches that can really help out here without you having to engineer all the edge cases
15:33uh that can go in.
15:35And then obviously like how do we do this robustly rain, snow, day, night um uh using um vision only.
15:44One other thing that we're sort of doing here on the tracking side is we have some minimal set of tracking that's happened on device on the edge but we can do some higher level tracking that happens on the cloud.
15:54So perhaps it can reason more about your entire map.
15:56Perhaps you can reason more about use heavier models.
15:59Use VLMs uh uh with lower which have u which don't respond as quickly uh at a rate of like let's say 7 to 10 hertz but can give you feedback at a 1 to two second uh latency
16:10but that's good enough for us to make uh broad decisions about um where to move.
16:16For a lot of our um infrastructure customers uh we're doing a lot of semantic reasoning.
16:22So what is there in the scene?
16:24Uh here is an illustration of us thinking about um uh utility poles and we want to uh when a instruction comes in hey go look at this line there's something gone wrong the drone needs to go there it needs to understand the scene then it needs to take actions within that scene and we're constantly looking at how
16:40to build these primitives these tools ultimately uh that uh we today code but uh at any time an agent can access these tools to better understand uh what the drone is seeing and what it could do about
16:54So an example of uh the agentic sort of system in action is a visual language model on the top left side.
17:01The user here is typing in look for a white jeep uh and it's doing a sort of find and follow.
17:07It in it uh accesses a drone API to command a certain sort of trajectory that it should take.
17:13While that is happening the detection head is kind of uh the V uh the VLM is kind of running here to say hey what's what in the scene am I looking for?
17:21it finds something, then it has access to the tools that allow the drone to track and follow.
17:26And that's without any specific coding of that law rules, but instead uh having a more sort of agentic uh giving the agents the all the tools that it needs to be able to um understand the drone state and make decisions given the uh information and the context that's available there.
17:45And then there's uh always the sort of long-term vision uh that's often there in the self-driving community here right now or any sort of robotic system.
17:53What if uh we could just give it raw sensor data and out comes uh the perfect uh results.
18:01Uh perhaps that's the actuation that happens.
18:03Perhaps it's where the drone is pointing.
18:05Perhaps it's where the drone goes.
18:06And we're definitely sort of doing a lot of testing and triing with reinforcement learning on what that looks like if we have multiple uh instantiations of uh this um behavior.
18:16Does it get to the right um end spectrum?
18:19The main consideration whenever we work with a physical system is that you're often looking at really high volumes of reliability as I was saying many nines of reliability.
18:30uh and the um doing a completely end to-end system uh does have its challenges in the sense that the observability of what's going wrong and the guarantees on reliability is very difficult today.
18:41So while it's a direction that we're continuously taking and exploring, it's kind of figuring out which segments of your end to-end chunk need to move to a more uh um
18:51like sort of world model representation of it.
18:54Uh, and it's specifically the things where we always find ourselves, okay, I need to handgineer this.
18:59I need to code this in specifically.
19:00I need to look at these rules.
19:01Like search and rescue is one of those things like, oh, go look for these trees, but if you don't find the trees, look under here or then turn on thermal or but then if you didn't find it here, look there.
19:10We're trying to get away from having to have all these if statements and the code and these branching strategies and kind of let the agent have some highle tools
19:18uh to be able to instruct these very high level commands uh for the drone.
19:26So um I talked about a quadcopter today, but we're kind of doing this similar with a much smaller from form factor quadcopter and we're now starting to look at this what this looks like from a fixed wing as well.
19:36all in the world of infrastructure.
19:38Uh so they can be launched from anywhere, recovered from anywhere and ultimately the sweet spot where we can kind of start to very quickly build on the cloud uh to orchestrate these things.
19:47And the way we thinking about it is allowing these systems to have basic APIs of interactivity that the cloud agents can come in and uh tap into and make decisions on
19:58and ultimately allow for very high level thinking when we uh uh work with these drones.
20:04So like many talks here we are hiring uh as as I said we have full stack sort of uh we do the end toend thing hardware software autonomy full stack front end back end
20:15uh wireless networking everything all the technologies on our mobile phones we're now making them fly um so uh come say hi or look at our website uh would love to talk more
20:27thank