troubles when reproducing

[HanGuangXin]

Hi, thanks for the wonderful work.

But I had some troubles when trying to reproduce the results with command:

python3 projects/IDOL/train_net.py --config-file projects/IDOL/configs/ytvis19_r50.yaml --num-gpus 8 MODEL.WEIGHTS projects/IDOL/weights/cocopretrain_R50.pth SOLVER.IMS_PER_BATCH 16

The results is 46.96, which is lower than provided results 49.5. I'm using Torch 1.9.0, and batchsize was set to 16 instead of 32.

Is there something I missed? Looking forward to your reply.

[NingYuanxiang]

Hi, thanks for the wonderful work.

But I had some troubles when trying to reproduce the results with command:

python3 projects/IDOL/train_net.py --config-file projects/IDOL/configs/ytvis19_r50.yaml --num-gpus 8 MODEL.WEIGHTS projects/IDOL/weights/cocopretrain_R50.pth SOLVER.IMS_PER_BATCH 16

The results is 46.96, which is lower than provided results 49.5. I'm using Torch 1.9.0, and batchsize was set to 16 instead of 32.

Is there something I missed? Looking forward to your reply.

跑的这么快，几张卡啊大哥。STEPS、 MAX_ITER增大了吗？

[HanGuangXin]

@NingYuanxiang I then doubled MAX_ITER as I cut IMS_PER_BATCH by half. The results is 0.4710, which is still much lower than provided results 49.5. I'll take a look at STEPS.

[wjf5203]

Thanks for your attention!

After seeing your issue, I conducted more experiments to provide some reference results, hoping it can help.
First of all, in order to achieve the results reported in the paper, we need to infer at 480p resolution according to the description in Table 3, so please set 'MIN_SIZE_TEST' to 480.
Directly reducing batchsize will result in insufficient usage of the dataset, so I have made three settings for IMS_PER_BATCH and MAX_ITER, and each setting is repeated five times.
The settings and results are as follows:

Original:
IMS_PER_BATCH: 32
STEPS: (8000,)
MAX_ITER: 12000
Results (mAP): [50.5, 49.4, 49.8, 49.7, 48.8 ]

Setting1:
IMS_PER_BATCH: 16
STEPS: (8000,)
MAX_ITER: 12000
Results (mAP): [ 47.5, 47.1, 47.9, 47.4, 48.1]

Setting2:
IMS_PER_BATCH: 16
STEPS: (16000,)
MAX_ITER: 24000
Results (mAP): [49.0, 47.9, 48.5, 49.1, 48.6]

So, the reported results can be stably reproduced by default setting.
If you reduce the batchsize from 32 to 16, you may need to adjust the number of iterations appropriately.
Welcome to continue exploring the impact of different settings on training results and discuss here.