Skip to content

nuplan_submission_tutorial.md

Latest commit

 

History

History
89 lines (63 loc) · 7.23 KB

nuplan_submission_tutorial.md

File metadata and controls

89 lines (63 loc) · 7.23 KB

Submission Tutorial

This tutorial will walk you through how to generate a valid submission, and send it to the evaluation server.

Prerequisites

In this guide, it is assumed you followed the nuplan_framework tutorial and are familiar with the nuPlan environment. It is suggested, but not required, to first go through this tutorial with the example Planner (SimplePlanner) and then make the necessary modifications to use your custom planner.

Another requirement is to have a local Docker installation. You can follow the official documentation here. Furthermore, for local testing docker-compose is needed, and a version >=1.28.0 is required. On a linux machine you can install the latest version (at the moment of writing) with:

sudo apt remove docker-compose
sudo curl -L "https://github.com/docker/compose/releases/download/v2.9.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Protocol specifications

The client/server communication is made using gRPC, the configuration files and code related to the communication can be found at ~/nuplan_devkit/nuplan/submission. In order for submissions to work as expected, the protocol files (protos/) and the autogenerated (challenge_pb2.py and challenge_pb2_grpc.py) files MUST NOT be modified, as we will use our version of them.

Similarly, you should not modify submission_container.py and submission_planner.py, to avoid invalid submissions. You should modify the specified section of entrypoint_submission.sh.

Creating a valid submission

To be able to run your planner, the basic requirement is that it needs to inherit from AbstractPlanner, and implement the relevant interfaces.

Checkpoint Run a simulation with your custom planner using the run_simulation script, as exemplified in the tutorials.

Once you can run your planner, to be able to submit your planner for remote execution you will have to package it in a Docker container. Here are the steps:

  1. Dockerfile.submission is the starting point. You might not need to edit it directly, however, feel free to change it as you see fit. If you need to edit the system dependencies you can do so by adding apt targets as needed in Dockerfile.submission. By default, anything under the nuplan_devkit/nuplan directory will be present in the Docker image at the absolute path /nuplan_devkit/nuplan.
  2. To add additional pip requirements you may need, please append them to requirements_submission.txt.
  3. You can copy checkpoints and other files inside your container, as long as they are in the same directory or in a subdirectory from Dockerfile.submission. To do so follow the example present in Dockerfile.submission.
  4. run_submission_planner.py is where your planner is instantiated. This is done through hydra, which allows you to pass any arguments required to instantiate your planner from a config file.
    • For reference, see the SimplePlanner hydra config. For example, if you want to instantiate SimplePlanner but modify max_velocity, you could make a config file simple_planner2.yaml in nuplan/planning/script/config/simulation/planner containing: 
      simple_planner2:
    _target_: nuplan.planning.simulation.planner.simple_planner.SimplePlanner
    _convert_: 'all'
    horizon_seconds: 10.0
    sampling_time: 0.25
    acceleration: [0.0, 0.0] # x (longitudinal), y (lateral)
    max_velocity: 7.0  # modified
    steering_angle: 0.0
      and then set planner=simple_planner2 in entrypoint_submission.sh. More examples are resent in the other tutorials.
    • If you run into problems using hydra, you can instantiate the planner manually in run_submission_planner.py. See the included comments for pointers.

At this point, you need to make sure that you can generate a valid Docker image from the Dockerfile you edited.

Checkpoint Create a Docker image using the edited Dockerfile.submission. If you are unfamiliar with Docker, the command below should be a good starting point (to be run within nuplan_devkit/):

docker build --network host -f Dockerfile.submission . -t nuplan-evalservice-server:test.contestant

Once you created the docker image, you need to test its behavior in the sever-client architecture, to make sure everything works fine. To do so you can use docker-compose which should take care of everything for you.

docker-compose.yml mounts the dataset and maps to the submission and simulation containers, as well as writing results from the simulation locally on your machine. By default, they mount inside the containers the directories where $NUPLAN_DATA_ROOT, $NUPLAN_MAPS_ROOT and $NUPLAN_EXP_ROOT point to, so you need to make sure these are set. The following command will create both the submission and the simulation images, and will start the simulation (check the entrypoint files for details).

Checkpoint Run the simulation by running the command below:

docker-compose up --build

You can inspect the results in your $NUPLAN_EXP_ROOT directory after simulation.

Submitting a solution

To submit through to EvalAI you will have to register on the EvalAI website. Then you will have to install the CLI and push your submission, with for example:

evalai push <image>:<tag> --phase random-dev-1856

If you run into problems, the exact command can be found on your competition submission page. The results will be visible on the EvalAI leaderboard.