Clarification on dataset splits, critic types, and feedback settings for reproduction

[junho328]

Hello,
I am trying to reproduce the results from your paper, and I encountered a few points that are somewhat unclear to me. I opened this issue to ask for clarification so that I can faithfully follow the experimental setup.

1. Dataset splits (train / eval / test)
   First, I would like to clarify the exact train, evaluation, and test splits used in the experiments.

According to Appendix C.1 of the paper, for the writing collaboration task (TLDR):

• Training set: TLDR[0:1000]
• Test set: TLDR[1000:1100]

However, when I checked magrpo_tldr_config.yaml, I noticed that:

• train_split is set to train[:1100]
• eval_split is set to test[:1100]

This seems slightly different from what is described in the paper.

Additionally, the trl-lib/tldr dataset already provides train, validation, and test splits.
In this context, does TLDR[1000:1100] refer to:

• indices [1000:1100] from the train split, or
• indices [1000:1100] from the test split?

I would really appreciate clarification on this point.

Similarly, for the Minecraft experiments, in house_build_magrpo_config.yaml, I see that:

• train_split is set to [:]

whereas the paper mentions using [0:8] for training.
I was wondering whether this discrepancy was an oversight during code release, or if there is another intended explanation.

2. Critic types used in main experiments

For the Actor-Critic–based algorithms, could you clarify which critic type was used for each method in the main experimental results reported in the paper?

Specifically, were the critic types exactly those specified in the released configuration files, or were there any differences between the paper experiments and the public code?

3. Feedback types in Minecraft experiments

For the Minecraft tasks, I assume that performance may vary depending on the feedback type.
Could you please let me know which feedback types were used when training StrBuild and HouseBuild, respectively, for the results reported in Table 1?

⸻

Sorry for bothering you again with many questions
I genuinely think your work is excellent, and I would really like to run the experiments myself to better understand and verify the results.

Thank you very much for your time and for sharing such great research!!

[LovelyBuggies]

Hi @junho328 , for your questions:

1. YAML files just provide an example for setting up training, and some may be outdated/updated (as you see, there are 2 papers related to this repo, and maybe more). You will find all the exact parameters we used in the paper Appendix C. 2. Some clarifications about your questions in the tldr train and eval, different from code generation based on unit tests, since the reward function in writing collaboration depends purely on scores, so you can use whichever data you mentioned, and the performance would be similar.
2. The critic we used is as shown in Appendix C. 2, though, as far as I tested, once you have the critic model as any agent model, the results should also be fine. While it's indeed an issue that we should make them more flexible and the interface should be more integrated, we just fixed it and allowed better interfaces for model loading in PR fix: heterogenous loading and critic conflict handling #60. Please try it with pip install -e . Once that PR is merged, you can refer to the model loading section in the updated docs to see how to load models as changed. I'll notify you and close this issue when that pr is merged.
3. The feedback type for str and house build are position feedback and score feedback respectively.

I notice you have forked this repo and also made your own. An efficient practice I strongly suggest is to click "Sync this repo" in your forked repo. It can help to sync the latest version on time, and help us better to track your repo, find your issues if exists, and also help use to improve (ofc also good for CoMLRL's credits :) Since I intend to make it a long-term Python library rather than just for a one-paper project, the config keys and files would change a lot over time for coordination, and we will make some significant changes in next season.

Thanks again for trying our repo!

[LovelyBuggies]

Hi, @junho328 , we just made a new release v1.3.6. You should be able to re-implement easily with pip install comlrl==v1.3.6. The new interface has been changed; see the commands, results, and estimated time and resources that I tested for #60. Feel free to reopen this issue if you have any questions.