Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions

Shiyu Fang

[Project web]

Updates

2025/12/10: This work has been published in IEEE Transactions on Intelligent Transportation System. If you find it helpful, please consider citing our work.

Getting started 🚀

1. Install Ollama

Before you start, make sure to have ollama installed and a working environment. If you don’t have ollama installed, check here Ollama. After installing Ollama, run the following command to install llama3:

ollama run llama3

You can choose any other LLM model provided by Ollama—just replace "llama3" with the model you want to use and change llm_agent.py line10 accordingly. Check here for more details. (better with python3.10)

2. Install the dependent package

pip install -r requirements.txt

3. Start training and testing

python train.py
python test.py
python test_with_listener.py

First, run train.py to generate an interaction experience database (you may need to activate line24&25 in memory.py if you do not have a relative embedding model). After that, you can use test.py or test_with_listener.py for testing. The simulation will run at a frequency of 10Hz. In test_with_listener.py, if your device has a microphone, you can act as an HDV and provide human instructions to the CAV. Have fun 😊

Overview 🔍

Repo description 📄

In this repository, you can expect to find the following features: Included:

• Code for Actor-Reasoner (including roundabout, merge, intersection)
• Video and Data of our experiment

Not included:

• Code for Comparison Algorithm

Files description 📂

Main Functions:

• train.py: Runs only the Reasoner to train the Actor’s interaction experience database. This must be executed first when running the project for the first time.
• test.py: Uses both the Reasoner and Actor. Since most users don’t have access to real vehicles or driving simulators, this function simulates actual operations in a virtual environment with an update frequency of 10Hz.
• test_with_listener.py: An upgraded version of test.py. If your device has a microphone, this function allows you to provide human instructions via voice input to observe the CAV’s behavior.

Vehicle Behavior Generation:

• llm_agent.py: Takes scene and conflict information as input and generates semantic actions.
• idm_controller.py: Converts semantic-level actions generated by the LLM into acceleration.
• bayesian_game_agent.py: Decision model for heterogeneous HDVs.
• vehicle.py: Defines the vehicle class.

Environment & Tools:

• params.py: Contains hyperparameter settings for vehicles and the environment.
• tools.py: Includes utility functions such as conflict detection and risk assessment.
• prompt_llm.py: Handles prompt design for LLM interactions.
• listener.py: Listens to real-time microphone input.
• memory.py: Provides tools for managing the interaction experience database.

If you have any questions, feel free to contact Shiyu (fangshiyu@tongji.edu.cn) 📧.

Citation

Our paper has been pre-printed! If you find our work helpful, please consider citing us using the following reference 😊:

@misc{fang2025interactinstructimprovellmdriven,
      title={Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions}, 
      author={Shiyu Fang and Jiaqi Liu and Chengkai Xu and Chen Lv and Peng Hang and Jian Sun},
      year={2025},
      eprint={2503.00502},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2503.00502}, 
}