The main focus of the nuPlan planning challenge is to evaluate a motion planning system in realistic driving scenarios, using multiple performance metrics. In this challenge, it is assumed that a planner will consume a top-down semantic representation of detected traffic participants (vehicles, bicycles, etc.) and static obstacles and plan the vehicle’s future trajectory for a specific time horizon. The challenge is organized into three increasingly complex modes:
- Open-loop
- Closed-loop non-reactive agents
- Closed-loop reactive agents
- January 30th, 2023 - Warm-up phase and Test phase opens for submission
- May 26th, 2023 - Competition closes - Finalists are announced and invited to share their code with us for verification
- June 2th, 2023 - Winners are announced
- June 18th, 2023 - The competition results are presented at the End-to-End Autonomous Driving workshop at CVPR 2023
- Sign up and download the dataset from https://www.nuscenes.org/nuplan.
- Clone the devkit. Feel free to ask any questions by raising an issue on our GitHub: https://github.com/motional/nuplan-devkit
- Create a team on EvalAI.
- Go through the nuPlan submission tutorial. The tutorial will show how to create a valid Docker image.
- Submit your Docker image through EvalAI.
- Send the following information to nuScenes@motional.com after submitting on EvalAI
- Team name
- Method name
- Authors
- Affiliations
- Country (Of the individual or team's institution)
- Method description (5+ sentences)
- Project URL (If applicable)
- Paper URL (If applicable)
- Submission ID
Participants of the nuPlan planning challenge will submit their code to the challenge’s cloud-based evaluation server through EvalAI - an open-source evaluation platform. The code will be submitted in the form of a Docker image and will be executed for each of the challenge modes (open-loop, non-reactive closed-loop and reactive closed-loop simulation) against the hidden test set of the challenge.
After submission, metrics that were generated for each simulation will be aggregated and sent to the public leaderboard. A final scoring function will be used to rank all submissions in each challenge. In addition, granular metrics and error analysis plots will be available for each submission on the challenge’s page of the nuPlan official website (coming soon). For each of the three planning challenges, the following set of rules applies:
- A fixed set of scenario simulations for each scenario type (e.g. lane change) will be run to evaluate a submission.
- Each scenario simulation will start from a fixed point in time that represents the beginning of the scenario (e.g. the start of an unprotected turn).
- The current and past (up to 2s) scene information for each simulation iteration will be passed to the planner - that includes ego pose, ego route, agents, and static/dynamic map information.
- Each planner will have a fixed time budget of 1s for each simulation iteration, after which the simulation will time out.
- The simulation horizon will be up to 15s for each scenario and the simulation frequency will be 10Hz.
- There will be a strict limit of three submissions for each participant for this competition.
- Every submission should provide a short description of the method used.
- Submissions that obfuscate code in the submitted Docker image will not be considered for the competition and therefore winning prizes. However, participants are still welcome to submit their "top secret" planner
- The top performing submissions will be inspected for compliance with the challenge’s rules - any attempt to circumvent these rules will result in a permanent ban of the team or company from all future nuPlan challenges.
The warm-up phase allows participants to test the submission pipeline. During this phase, participants may submit up to 10 submissions. The submissions will be run on all three challenges. The scenario types used for the warm-up phase are the same as the ones that will be used for the test phase. However, the number of scenarios run will be smaller, and the data is not taken from the test split. The metrics and the overall scores will be displayed on the leaderboard.
Start: January 30th, 2023
End: May 26th, 2023
The results from the test phase will be used to determine the winners of each challenge. All submissions will be evaluated on a larger set of scenarios from the test split. The scores from the test phases will be used to shortlist finalists. Finalists will then be invited to share their code with the organizers. Once the submissions from the finalists are verified by the organizers, the same scores are used to determine the winners of each challenge. Participants will be allowed to submit a maximum of three submissions.
Start: January 30th, 2023
End: May 26th, 2023
Warning: The following requirements must be met for a submission to be considered valid. Failing to meet these requirements will result in your submission failing.
| Planner Trajectory Requirements | Value |
|---|---|
| Expected minimum horizon length | 8s |
| Expected minimum horizon steps | 2 |
| Expected signals |
|
| Configuration | Value |
|---|---|
| Simulation iteration time budget | 1s |
| Max submissions per each user/team | 3 |
| Instance type |
|
All challenges will be run with the following common configurations
| Common Challenge Configuration | Value |
|---|---|
| Frequency | 10Hz |
| Rollout horizon | 15s |
| Past observation horizon | 2s |
In this challenge, the planning system is tasked with imitating the expert driver in open-loop. For every time step, the planner’s predicted trajectory is scored based on a set of predefined open-loop metrics. Namely, it is compared directly against the driven ground-truth trajectory. As the name suggests, the planned trajectory will not be used to alter the state of the simulation.
| Configuration | Value |
|---|---|
| Metric computation horizon | 3s, 5s, 8s @ 1Hz sampling |
| Controller | N/A |
| Observations |
|
The overall score of challenge 1 considers the following factors:
- Average Displacement Error (ADE)
- Final Displacement Error (FDE)
- Average Heading Error (AHE)
- Final Heading Error (FHE)
- Miss Rate
The details of each metric can be found here. The details of the scoring hierarchy can be found here
In this challenge, the planner outputs a planned trajectory using the information available at each time step, similar to the previous case. The predicted trajectory is used as a reference for a tracking controller to simulate the vehicle’s state at the next time step. Nonetheless, the other agents are replayed from a log according to their observed states in open-loop - hence, the name non-reactive. Most importantly, the ground-truth is no longer applicable to the evaluation of the planner. Instead, a suite of closed-loop metrics will be used to judge the performance of a planner
| Configuration | Value |
|---|---|
| Controller | LQR |
| Observations |
|
The overall score of challenge 2 considers the following factors:
- At-fault collision
- Drivable area compliance
- Driving direction compliance
- Making progress
- Time to collision (TTC)
- Speed limit compliance
- Ego progress along the expert's route ratio
- Comfort
The details of each metric can be found here. The details of the scoring hierarchy can be found here
In the final and most complex challenge, the vehicles in the scene become reactive. In other words, the vehicles will react to the actions of the planner-controlled vehicle along with the behavior of other agents, the policy determining the behavior of agents is an Intelligent Driver Model (IDM) policy. The rest of the observations are still replayed in open-loop. Similar to the non-reactive closed-loop challenge, a suite of metrics will be used to judge the performance of a planner in closed-loop.
| Configuration | Value |
|---|---|
| Controller | LQR |
| Observations |
|
The overall score of challenge 3 considers the following factors:
- At-fault collision
- Drivable area compliance
- Driving direction compliance
- Making progress
- Time to collision (TTC)
- Speed limit compliance
- Ego progress along the expert's route ratio
- Comfort
The details of each metric can be found here. The details of the scoring hierarchy can be found here
All three challenges will be run on the following scenario types.
| Scenario Type | Description |
|---|---|
| starting_straight_traffic_light_intersection_traversal | Ego at the start of a traversal going straight across an intersection area controlled by traffic lights while not stopped |
| high_lateral_acceleration | Ego high ego acceleration (1.5 < acceleration < 3 m/s^2) across the lateral axis with high yaw rate while not turning |
| changing_lane | Ego at the start of a lane change towards an adjacent lane |
| high_magnitude_speed | Ego high velocity magnitude with low acceleration (velocity > 9 m/s) |
| low_magnitude_speed | Ego low ego velocity magnitude (0.3 < velocity < 1.2 m/s) with low acceleration while not stopped |
| starting_left_turn | Ego at the start of a traversal turning left across an intersection area while not stopped |
| starting_right_turn | Ego at the start of a traversal turning right across an intersection area while not stopped |
| stopping_with_lead | Ego starting to decelerate (acceleration magnitude < -0.6 m/s^2, velocity magnitude < 0.3 m/s) with a leading vehicle ahead (distance < 6 m) at any area |
| following_lane_with_lead | Ego following (velocity > 3.5 m/s) its current lane with a moving leading vehicle ahead on the same lane (velocity > 3.5 m/s, longitudinal distance < 7.5 m) |
| near_multiple_vehicles | Ego nearby (distance < 8 m) of multiple (>6) moving vehicles while ego is moving (velocity > 6 m/s) |
| traversing_pickup_dropoff | Ego during the traversal of a pickup/drop-off area while not stopped |
| behind_long_vehicle | Ego behind (3 m < longitudinal distance < 10 m) a long (length > 8 m) vehicle in the same lane as ego (lateral distance < 0.5 m) |
| waiting_for_pedestrian_to_cross | Ego waiting for a nearby (distance < 8 m, time to intersection < 1.5 m) pedestrian to cross a crosswalk area while ego is not stopped and the pedestrian is not at a pickup/drop-off area |
| stationary_in_traffic | Ego is stationary with multiple (>6) vehicles nearby (distance < 8 m) |
The submissions are ranked based on the mean overall score across all three challenges.
- 1st place: USD $10,000
- 2nd place: USD $8,000
- 3rd place: USD $5,000
- Innovation Prize: USD $5,000
- The Innovation Prize will be awarded to the most innovative submission judged by a panel of Motional experts.
- Internship opportunity
- Eligible individual participants may be considered for internships at Motional based on their performance on the challenges. Any offers for internships are subject to the Competition Terms, the participant successfully completing the Motional application process, and any other terms and conditions solely determined by Motional.
Terms and conditions apply.
To contact the organizers, please make an issue on our GitHub page or email us at nuScenes@motional.com