How to achieve a good performance of 4DGaussians on Technicolor Dataset

[liyw420]

Hello! Thank you for your great work!

Recently I've been reproducing the results of Dynamic GS models on dynamic scene datasets. Your work is really brilliant!

When I test the model of 4DGaussians [1] on the technicolor dataset, I found that the final results are really poor. Although I cannot guarantee I handle all the inputs well (camera parameters, point clouds, and images), I guess the poor results are caused by the settings of hyperparameters. Given that you've reproduced this model on this dataset, I was wondering if you could give me some suggestions on this issue?

Thank you!!!

Poor rendered image:
Train view:

Test view:

Reference: 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering, CVPR 2024.

[chenj02]

Hello! Thank you for your great work!

Recently I've been reproducing the results of Dynamic GS models on dynamic scene datasets. Your work is really brilliant!

When I test the model of 4DGaussians [1] on the technicolor dataset, I found that the final results are really poor. Although I cannot guarantee I handle all the inputs well (camera parameters, point clouds, and images), I guess the poor results are caused by the settings of hyperparameters. Given that you've reproduced this model on this dataset, I was wondering if you could give me some suggestions on this issue?

Thank you!!!

Reference: 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering, CVPR 2024.

hi, I met the same problem too, and I found the point cloud generate by colmap is very different with the Ex4DGS pretrianed model. For example, in the dataset Technicolor Train, the red box is my point cloud my_point.ply using pre_techicolor.py, the green box is the Ex4DGS pretrained model's input.ply, and the yellow box is the Ex4DGS pretrained model's final point cloud point_cloud.ply. The position and scale are very different between my_point.ply and input.ply, and also between input.ply and point_cloud.ply. I don't know why? Do you meet this problem?

[juno181]

Hello, I think I might have an issue with COLMAP. Since COLMAP does not compute values in an absolute coordinate system (e.g., in meters), there is some randomness involved. Therefore, it's better to use the camera positions and point cloud generated from a single run. However, as far as I know, if the camera pose is accurate, training is still feasible even if there are differences in the point cloud. Based on my experience, this type of issue tends to occur in the Technicolor dataset when the principal point offset is incorrect. I’ll check the preprocessing for the Technicolor dataset. Upon checking, I see that I am using COLMAP version 3.9, and you can build it directly from COLMAP.

[liyw420]

Hello! Thank you for your great work!
Recently I've been reproducing the results of Dynamic GS models on dynamic scene datasets. Your work is really brilliant!
When I test the model of 4DGaussians [1] on the technicolor dataset, I found that the final results are really poor. Although I cannot guarantee I handle all the inputs well (camera parameters, point clouds, and images), I guess the poor results are caused by the settings of hyperparameters. Given that you've reproduced this model on this dataset, I was wondering if you could give me some suggestions on this issue?
Thank you!!!
Reference: 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering, CVPR 2024.

hi, I met the same problem too, and I found the point cloud generate by colmap is very different with the Ex4DGS pretrianed model. For example, in the dataset Technicolor Train, the red box is my point cloud my_point.ply using pre_techicolor.py, the green box is the Ex4DGS pretrained model's input.ply, and the yellow box is the Ex4DGS pretrained model's final point cloud point_cloud.ply. The position and scale are very different between my_point.ply and input.ply, and also between input.ply and point_cloud.ply. I don't know why? Do you meet this problem?

Hello, currently I didn't reproduce the code of Ex4DGS, so I didn't meet the same problem with you. But I cannot achieve good results of 4DGaussians on Technicolor Dataset. I use the point clouds and camera poses estimated by the pipline from E-D3DGS. While E-D3DGS runs successfully, 4D Gaussians always fail. I don't know why.

[liyw420]

Hello, I think I might have an issue with COLMAP. Since COLMAP does not compute values in an absolute coordinate system (e.g., in meters), there is some randomness involved. Therefore, it's better to use the camera positions and point cloud generated from a single run. However, as far as I know, if the camera pose is accurate, training is still feasible even if there are differences in the point cloud. Based on my experience, this type of issue tends to occur in the Technicolor dataset when the principal point offset is incorrect. I’ll check the preprocessing for the Technicolor dataset. Upon checking, I see that I am using COLMAP version 3.9, and you can build it directly from COLMAP.

Dear author,

Thank you for your great work.

I cannot achieve good results of 4DGaussians on Technicolor Dataset. Could you provide the preprocessing and training details of 4DGaussians on the Technicolor Dataset? Thank you!

[juno181]

Hello, I think I might have an issue with COLMAP. Since COLMAP does not compute values in an absolute coordinate system (e.g., in meters), there is some randomness involved. Therefore, it's better to use the camera positions and point cloud generated from a single run. However, as far as I know, if the camera pose is accurate, training is still feasible even if there are differences in the point cloud. Based on my experience, this type of issue tends to occur in the Technicolor dataset when the principal point offset is incorrect. I’ll check the preprocessing for the Technicolor dataset. Upon checking, I see that I am using COLMAP version 3.9, and you can build it directly from COLMAP.

Dear author,

Thank you for your great work.

I cannot achieve good results of 4DGaussians on Technicolor Dataset. Could you provide the preprocessing and training details of 4DGaussians on the Technicolor Dataset? Thank you!

For 4DGaussians, the original code does not use the principle point. However, in the Technicolor dataset, training does not work without using it. Therefore, modifications need to be made to the dataloader and projection matrix to accommodate this. Pay attention to the variables cxr and cyr in the following two classes:

https://github.com/hustvl/4DGaussians/blob/843d5ac636c37e4b611242287754f3d4ed150144/scene/dataset_readers.py#L33

Ex4DGS/scene/dataset_readers.py

Line 35 in cb7b497

class CameraInfo(NamedTuple):

[liyw420]

Hello, I think I might have an issue with COLMAP. Since COLMAP does not compute values in an absolute coordinate system (e.g., in meters), there is some randomness involved. Therefore, it's better to use the camera positions and point cloud generated from a single run. However, as far as I know, if the camera pose is accurate, training is still feasible even if there are differences in the point cloud. Based on my experience, this type of issue tends to occur in the Technicolor dataset when the principal point offset is incorrect. I’ll check the preprocessing for the Technicolor dataset. Upon checking, I see that I am using COLMAP version 3.9, and you can build it directly from COLMAP.

Dear author,
Thank you for your great work.
I cannot achieve good results of 4DGaussians on Technicolor Dataset. Could you provide the preprocessing and training details of 4DGaussians on the Technicolor Dataset? Thank you!

For 4DGaussians, the original code does not use the principle point. However, in the Technicolor dataset, training does not work without using it. Therefore, modifications need to be made to the dataloader and projection matrix to accommodate this. Pay attention to the variables cxr and cyr in the following two classes:

https://github.com/hustvl/4DGaussians/blob/843d5ac636c37e4b611242287754f3d4ed150144/scene/dataset_readers.py#L33

Ex4DGS/scene/dataset_readers.py

Line 35 in cb7b497

class CameraInfo(NamedTuple):

I've tried your suggestion and it works. Thank you very much!

[chenj02]

Thank you very much, I have solved my problem!