Reversal Q-Learning (RQL)

Paper  

Website  

Installation

RQL requires Python 3.9+ and is based on JAX. The main dependencies are jax >= 0.4.26, ogbench == 1.2.0, and gymnasium == 0.29.1. To install the full dependencies, simply run:

pip install -r requirements.txt

Usage

The main implementation of RQL is in agents/rql.py.

Tuned hyperparameters for each environment and agent are provided in the paper. Complete list of RQL commands here: hyperparameters.sh

# RQL

python main.py 
    --agent=agents/rql.py 
    --env_name=humanoidmaze-large-navigate-singletask-v0 
    --agent.alpha=10 
    --agent.expectile=0.9 
    --agent.ensemble_ct=10 
    --agent.rho=0.0 
    --agent.h=1 
    --agent.discount=0.995 
    --offline_steps=1000000 
    --online_steps=0 
    --agent.batch_size=256

Using larger datasets

The paper uses 100m-sized datasets for the OGBench puzzle-4x4 & cube-quadruple environments. These datasets can be downloaded with the following commands (see this section of the OGBench repository for more diverse 100M-sized datasets available):

# cube-quadruple-play-100m (100 datasets * 1000 length-1000 trajectories).
wget -r -np -nH --cut-dirs=2 -A "*.npz" https://rail.eecs.berkeley.edu/datasets/ogbench/cube-quadruple-play-100m-v0/
# puzzle-4x4-play-100m (100 datasets * 1000 length-1000 trajectories).
wget -r -np -nH --cut-dirs=2 -A "*.npz" https://rail.eecs.berkeley.edu/datasets/ogbench/puzzle-4x4-play-100m-v0/

Acknowledgments

This codebase is built on top of reference implementations from Flow Q-Learning.