As a roboticist, my dream has been to make an autonomous vehicle (AV) that could be let loose on the world, without needing human guidance. It would be an independent, robotic road companion: imagine the lovable Pixar character Baymax as a minivan and you have the spirit of this dream.
Motional’s AVs are driving further on their own and handling more situations independently everyday. But, at least for the near future, there are going to be occasions where the AVs need some assistance seeing, understanding, and reacting to unusual situations, for example, when a water main bursts and floods the roadway.
Enter Motional’s Remote Vehicle Assistance platform, which is designed to help our AVs plot a plan forward so they can continue their journeys - quickly and safely. Today this platform, called RVA, is staffed by humans. But as our AVs drive more and collect more data, the platform will use machine learning principles to become smarter thus requiring less direct human intervention over time.
Teaching vehicles to drive autonomously
Robots in pop culture seem capable of handling anything that happens to them. Rosie, the Jetsons’ robotic housemaid, for example, never seemed flummoxed despite the chaos around her.
But as roboticists, we know that robots can get stuck when encountering scenarios they haven’t experienced before. It would be impractical to refuse to deploy robots and automated vehicles unless they could solve every possible scenario on their own. Instead, companies like Motional consider the “appropriate levels of automation.” This means we can launch a fleet of robotaxis capable of safely operating on public roadways, and providing a public benefit, without also demanding they can handle every unique scenario perfectly.
Think about your morning orange juice. The vast majority of an orange’s juice can be extracted with little to moderate effort. But those last couple of drops are especially difficult to liberate and probably aren't worth the effort.
It’s the same principle with AVs: they can currently handle the vast majority of what they encounter safely, such as driving down highways and switching lanes. But also think about all the variables we encounter while driving over the course of a year: accidents, weather changes, basic human unpredictability. Getting enough real-world data to train AVs to handle a rare scenario they might encounter once a year - such as a mattress falling off a truck and blocking a travel lane - is difficult.
Perfect can’t be the enemy of progress. Like squeezing out those last few drops of orange juice, there needs to be consideration given to effort vs. frequency.
RVA provides backup while learning
Every day our vehicles are on the road, they are gaining experience. Our AV stack collects more training data on a daily basis, which is fed into a Continuous Learning Framework to get smarter with every mile.
But even with an updated, high-performing stack, the AV may still encounter a situation where it isn’t sure what to do. For example, if a water main break floods a street, cars may need to shift into reverse and start backing up. Even the most experienced human driver may need a moment to process what is happening, and resolving the situation could still involve some creative hand signals and language.
So if our goal is to have vehicles capable of independently handling the vast majority of the scenarios they encounter, we still need a solution in place for those truly rare events.
The need for remote assistance
RVA merges the classic goal of teleoperation with our need for appropriate levels of automation. Motional’s approach to RVA is to quickly provide our vehicles with a safe plan forward so they can continue their journeys. Contrary to popular imagination, RVA doesn’t involve team members with USB steering wheels looking at monitors driving vehicles remotely like in a video game. Doing so would require zero latency and constant high-quality network connectivity, neither of which are currently possible.
Instead of taking over the controls, we can send high-level commands to the vehicle’s onboard compute that guide our AV toward a solution. We can manually draw a path for the vehicle or suggest alternative routes and behaviors. But the AV system always remains in control.
Improving RVA using AI
This is where the technology gets exciting. By applying AI and ML principles, we can continually make RVA work better and smarter. Not only will RVA be able to help vehicles figure out solutions on their own, but also learn how to prioritize requests for assistance, improving the ride experience for passengers.
For instance, one of the tools our remote agents use most frequently is path drawing. This involves a human agent drawing a line that the AV follows until it’s clear of the situation. Once this happens, we now have a human-generated path through a previously unknown/unsolved scenario. Because it was executed in a real-world scenario, we know with the utmost confidence that the human solution worked. (And if a human-drawn scenario doesn’t resolve the situation, we can use that outcome to train the car what not to do). These are great data points to have.
In addition to the solution, we also have all the perception data from the vehicle (including what it thought the different agents were, and how far away the human-drawn path stayed from them). This information can help train a convolutional Neural Network path generation model: no further annotation or data labeling required. And the next time one of our AVs get stuck in a similar situation, our ML model can generate a recommended path for the RVA operator to review and approve.
This video shows an RVA operator helping a Motional AV navigate a busy pick up/drop off area at a Las Vegas casino.
What’s more, we can use the remote agents to help screen and verify the updated AV stack models. Instead of loading the new models straight into the vehicle, we can use them to suggest solutions to the remote agent, and have the remote agent verify if the AI-generated solution is good or not. This provides an extra level of testing and screening before the models are pushed out to the fleet. By taking agent guidance and feeding it straight back into training the system, we can reduce the need for assistance in the future.
In this image, the AV’s path through the intersection is blocked by another vehicle. When it becomes clear the stuck vehicle isn’t moving, the AV will connect with RVA.
In this image, an RVA operator starts drawing a path for the AV around the stuck vehicle.
And this image shows the finished, safe, alternate path that the AV will follow around the stuck vehicle. Note: the RVA operator never takes control of the AV; the AV remains fully autonomous.
How RVA accelerates robotaxi commercialization
Aside from creating training data, RVA is very much connected to our success as a robotaxi fleet provider. A ride-hail passenger can currently ask the human driver when something seems wrong. But a sudden unexplained stop can impact a passenger’s opinion of the ride. A bad enough experience could cause that passenger to choose a human driver next time.
AI can also improve response time. When a vehicle requests assistance, there are a number of factors we consider to establish priority. Some situations are high priority because there are safety implications: these include car accidents and scenarios involving emergency vehicles. But there are other considerations as well: Does the AV have a passenger? How long has the vehicle been stuck? Has the vehicle stopped because it’s in a traffic jam? An AI-based algorithm can weigh all these factors and prioritize requests for assistance, making sure that agents are providing assistance where it’s needed most.
Quickly resuming the driverless ride
The robots we see in pop culture may appear to have been launched fully functional, capable of handling whatever it encounters. But the reality around AVs is that the unpredictability of the environment around them is going to throw a curveball once in awhile.
AVs have come a long way and are ready to take their place on public roadways besides human drivers. Like any human driver, they are going to encounter odd, unique scenarios they haven't seen before.
When that happens, our RVA system will make sure our AVs can get some human guidance quickly so it can resume its journey safely.
