[WIP] Add latest Gaussian Splatting techiques to 2DGS #151
Conversation
- Introduced `AppearanceNetwork` and `UpsampleBlock` classes in `scene/appearance_model.py` - Added decoupled appearance parameter and new learning rates in `ModelParams` and `OptimizationParams` - Adjusted lambda values in `OptimizationParams` for better optimization control
- Introduced `edge_aware_normal_gradient_loss` for better edge handling - Integrated `AppearanceModel` to support decoupled appearance handling - Enhanced normal handling with expected normal calculation in renderer
hugoycj
changed the title
|
Hello, the work you have done is impressive. I can't wait to try the code you submitted. While I am waiting for the results, could you share the comparison before and after the improvement, such as chaotic areas, textureless areas, illumination varying areas, etc. |
|
@CanCanZeng The result heavily depends on the prior from our latest work StableNormal, you could take a loot at the StableNormal video demo for reference https://www.youtube.com/watch?v=sylXTxG_U2U |
- Add new parameters to OptimizationParams for better propagation control - Refactor patchmatch.py to improve normal rendering and consistency checking - Update depth and normal processing logic to handle new parameters
- Added 'rend_depth' to rendering outputs in gaussian_renderer - Integrated generate_K function for camera intrinsic matrix in cameras - Implemented generate_K to calculate focal lengths and principal points
- Implement normal2rotation and rotmat2quaternion functions - Modify densify_from_depth_propagation to include propagated_normal - Enhance detail and filter mask downsampling from 1/8 to 1/4
…e hypothesis - Add normal vector handling during input initialization - Refactor plane hypothesis and cost calculation with normals - Simplify and consolidate cost propagation logic
|
@CanCanZeng @hbb1 We have added a modified Propagation-guided Densification to solve the hole issue in the large scenes. Currently, the patch match algorithm input rendered normals to initialize the planar instead of randomly initialized. During the propagation, these normals remain fixed. Additionally, I've enhanced the densification step by computing scale and rotation based on both depth and normal information. Much faster and more robust than calculating the scaling using knn. We're seeking peer review of these modifications and would greatly appreciate any feedback to further refine our method." |
|
Awesome for contributing a stronger baseline with 2DGS. I am overwhelmed by some stuffs but once I am free, I will review the code. |
feat: Add alpha loss
- Re-enabled storing normals and depth normals in GaussianExtractor - Fixed background masking issue preventing redundant RGB clipping - Adjusted normal and depth normal visualization saving logic
- Use ground truth mask for valid pixel processing in losses - Adjust normal calculations to avoid NaN values - Ensure accurate weighting by valid pixel count
Integrated Level of Gaussians used in https://github.com/zju3dv/LoGIntegrated Geometry Regularization used in https://github.com/Anttwo/SuGaRIntegrated GES used in https://github.com/ajhamdi/ges-splattingAdapted the hyperparameter modification used in https://github.com/zju3dv/PGSR for better reconstructionIntegrated Multi-View Geometric Consistency used in https://github.com/zju3dv/PGSR