Automated Long-Running Experiments #31
Description
At the moment, I execute all experiments manually. This process means that every config change requires a manual effort to SSH into a machine, change the checkpoint path, change the hyperparameters, etc. Instead, a fail-safe automated system could allow us to run these things without manual intervention, without it ever making a typo or forgetting to change a variable. Such an automated system would free up time to do other things, such as research or engineering.
This issue tracks the progress of implementing such a CI pipeline.