ScanNet evaluation doesn't match paper

[mrharicot]

Hi,
Thank you for sharing your code!
I ran test on scannet following the instructions but I get much worse results.
Do you know what might be happening?

Total test num: 10434
['abs_rel', 'abs_diff', 'sq_rel', 'rms', 'log_rms', 'a1', 'a2', 'a3']
[0.26725563624661075, 0.4768538641935877, 0.18805316813831266, 0.5750360812767931, 0.3188030923313161, 0.4764993420147505, 0.8155114483545208, 0.9528344624353279]

Here's the first output for scene0707_00

[thuzhaowang]

Hi,

Thanks for your attention!

I checked the code and it runs well here for me. May I ask, are your testing environments the same as the those in environment.yml? We used an old Pytorch version (1.1.0) since we built the code based on NAS. It might have some inconsistency with the latest version.

[mrharicot]

I am using pythorch 1.7.1 as my GPU only supports CUDA 11.0 and up.
However I have used newer versions of pytorch on older ones and never had issues.
Are the pretrained weights on google drive the correct ones?

[thuzhaowang]

I see. I have tried to use Pytorch 1.7.1 right now and it indeed produces bad results, while the original environment works well.
I will investigate where the inconsistency is, in these days. Before that, probably you could find another environment using the same environment (Pytorch 1.1.0) to run the evaluation.

[mrharicot]

Thanks for your help!
I see that someone seems to have a similar issue for NAS on pytorch 1.3.1 so it's possible the backwards compatibility breaking change happened between 1.1. and 1.3.

[thuzhaowang]

Hi, I just found the change of torch.nn.functional.grid_sample is the problem. It affects the construction of feature cost volume.

"Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details."

So change the the grid sample in https://github.com/thuzhaowang/idn-solver/blob/6b5dcfd94f35cc118c5dee0f98401e4848e670e3/core/utils/inverse_warp.py#L108 and https://github.com/thuzhaowang/idn-solver/blob/6b5dcfd94f35cc118c5dee0f98401e4848e670e3/core/utils/inverse_warp_d.py#L123 with align_corners=True will solve the problem for higher version of Pytorch (1.3.0 or later).

[mrharicot]

This looks a lot better, thanks!