LEGOEval

LEGOEval is a toolkit for dialogue system evaluation via crowdsourcing, see our demo video.

Install LEGOEval

LegoEval supports Python 3.6+. Windows is not supported currently.

1. Installing via pip

If you have installed npm >= 7.7.6 and Node.js >= 15.13.0, you can install LegoEval via pip.

Clone the Repo

In your terminal app, paste the following command.

git clone https://github.com/yooli23/LEGOEval.git

Installing the libarary and dependencies

Installing the library and dependencies is simple using pip.

cd app
pip3 install -r requirements.txt

2. Installing via Docker

We also provide a dockerfile which makes it easy to distribute the environment.

Once you have installed Docker, you can run the following commands to get the environment.

1. Build

sudo docker build https://github.com/yooli23/LEGOEval.git#main -t legoeval:1

2. Run docker

sudo docker run -it legoeval:1

3. Clone the code

git clone https://github.com/yooli23/LEGOEval.git

Documentation

Check out our docs for the following steps, here.

Citing

If you use LEGOEval in your research, please cite LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing.

@misc{li2021legoeval,
      title={LEGOEval: An Open-Source Toolkit for Dialogue System Evaluation via Crowdsourcing}, 
      author={Yu Li and Josh Arnold and Feifan Yan and Weiyan Shi and Zhou Yu},
      year={2021},
      eprint={2105.01992},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}

Contact Information

For help or issues using LEGOEval, please submit a GitHub issue. For other communications related to LEGOEval, please contact Yu Li (yooli@ucdavis.edu)