Questions regarding Bidirectional Depth Alignment and Noise Modeling

[Lebron-233]

While I was reading this part of the code, I noticed that the "Depth Inpainting" step was not present in the script file. I'm not sure if I missed the corresponding section. I look forward to your response.

[zst1406217]

Hi there,

The "Depth Inpainting" module is designed specifically for real-world depth images. For the exact implementation, please check the instinct_onboard repository.

Regarding the training configs: We don't release the full training configurations of the noisy depth camera, because they are highly dependent on the specific camera and environment used. All the models are provided in InstinctLab, you will likely need to adjust the noise modeling parameters yourself to get the best performance on your hardware.

[Lebron-233]

Could you please tell me in which project I should operate if I want to first reproduce the process of "Bidirectional Depth Alignment and Noise Modeling"? I'm looking forward to your reply.

[zst1406217]

You should add noise model in Training, i.e. InstinctLab Project.

[Lebron-233]

In the noise model of the InstinctLab project, there is no Depth Inpainting processing of the real end that you mentioned in the paper. As you previously replied to me, the relevant processing was in the instinct_onboard project. If I want to perform the corresponding operation, I need to integrate the files from both projects. Thank you again for your reply.

[zst1406217]

The hole noise is only appear in real depth image, so only onboard deployment need "Depth Inpainting". Please refer to our paper for more details

[Lebron-233]

Sorry, I would like to ask you another question. What are the differences between the InstinctLab and instinct_onboard projects? Thank you very much for your reply. It's really a bother to you.

[zst1406217]

InstinctLab is for policy training in simulation, instinct_onboard is for policy deployment on real robot.

[Lebron-233]

I currently need to process the camera data from the simulation end, but I haven't found the code related to Out-of-Distribution (OOD) Perturbation. Could you please tell me the corresponding location and where to find it?

[zst1406217]

Please check https://github.com/project-instinct/InstinctLab/blob/main/source/instinctlab/instinctlab/utils/noise/noise_model.py#L579

[Lebron-233]

I also have a question regarding the processing of the real-end camera. How can we determine that the post-processing results of your depth camera (the real-end) meet the requirements?Thank you very much for constantly answering my questions.

[zst1406217]

You can run the onboard script with the --depth_vis flag and record a rosbag. This allows you to inspect the processed depth images during playback.

[ZZQ0316]

Please check https://github.com/project-instinct/InstinctLab/blob/main/source/instinctlab/instinctlab/utils/noise/noise_model.py#L579

Hello, i am also confused about it. The function 'random_gaussian_noise' seems to be different from the content of OOD in paper.