diff --git a/README.md b/README.md index c825749..c4057c6 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,28 @@ + +![Method](https://raw.githubusercontent.com/IIT-PAVIS/DiffAssemble/release/page/DiffAssemble.png) + Reassembly tasks play a fundamental role in many fields and multiple approaches exist to solve specific reassembly problems. In this context, we posit that a general unified model can effectively address them all, irrespective of the input data type (images, 3D, etc.). We introduce DiffAssemble, a Graph Neural Network (GNN)-based architecture that learns to solve reassembly tasks using a diffusion model formulation. Our method treats the elements of a set, whether pieces of 2D patch or 3D object fragments, as nodes of a spatial graph. Training is performed by introducing noise into the position and rotation of the elements and iteratively denoising them to reconstruct the coherent initial pose. DiffAssemble achieves state-of-the-art (SOTA) results in most 2D and 3D reassembly tasks and is the first learning-based approach that solves 2D puzzles for both rotation and translation. Furthermore, we highlight its remarkable reduction in run-time, performing 11 times faster than the quickest optimization-based method for puzzle solving. We will release the code upon paper acceptance. - [Webpage](https://iit-pavis.github.io/DiffAssemble/) - [Code](https://github.com/IIT-PAVIS/DiffAssemble) -- [Arxiv](soon) +- [Arxiv](https://arxiv.org/abs/2402.19302") # Method -![Method](https://raw.githubusercontent.com/IIT-PAVIS/DiffAssemble/release/page/DiffAssemble.png) +![Architecture](https://raw.githubusercontent.com/IIT-PAVIS/DiffAssemble/release/page/architecture.jpg) Following the Diffusion Probabilistic Models formulations, we model a Markov chain where we inject noise into the pieces’ position and orientation. At timestep t = 0, the pieces are in their correct position, and at timestep t = T , they are in a random position with random orientation. At each timestep t, our attention-based GNN takes as input a graph where each node contains an equivariant feature that describes a particular piece and its position and orientation. The network then predicts a less noisy version of the piece’s position and orientation. +# Visuals + +## 2D +![2d](https://raw.githubusercontent.com/IIT-PAVIS/DiffAssemble/release/page/viz.gif) + +## 3D + +![3d](https://raw.githubusercontent.com/IIT-PAVIS/DiffAssemble/release/page/bottle_new2_0001-0350.gif) + # Datasets ## Puzzles @@ -46,12 +58,11 @@ and orientation. The network then predicts a less noisy version of the piece’s # Cite us! ``` -@misc{giuliari2023positional, - title={Positional Diffusion: Ordering Unordered Sets with Diffusion Probabilistic Models}, - author={Francesco Giuliari and Gianluca Scarpellini and Stuart James and Yiming Wang and Alessio {Del Bue}}, - year={2023}, - eprint={2303.11120}, - archivePrefix={arXiv}, - primaryClass={cs.CV} +@InProceedings{scarpellini2024diffassemble, + author = {Gianluca Scarpellini and Stefano Fiorini and Francesco Giuliari and Pietro Morerio and Alessio Del Bue}, + title = {DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly}, + booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, + month = {June}, + year = {2024}, } ``` diff --git a/_config.yaml b/_config.yaml index a416ad5..0e6b697 100644 --- a/_config.yaml +++ b/_config.yaml @@ -1,5 +1,5 @@ remote_theme: pages-themes/cayman@v0.2.0 plugins: - jekyll-remote-theme -title: DiffAssemble -description: Official repository for DiffAssemble-A Unified Graph-Diffusion Model for 2D and 3D Reassembly accepted at CVPR2024 +title: DiffAssemble-A Unified Graph-Diffusion Model for 2D and 3D Reassembly +description: Accepted at CVPR2024 diff --git a/page/architecture.jpg b/page/architecture.jpg new file mode 100644 index 0000000..751f069 Binary files /dev/null and b/page/architecture.jpg differ diff --git a/page/bottle_new2_0001-0350.gif b/page/bottle_new2_0001-0350.gif new file mode 100644 index 0000000..78350ac Binary files /dev/null and b/page/bottle_new2_0001-0350.gif differ diff --git a/page/celeba_14_1.gif b/page/celeba_14_1.gif new file mode 100644 index 0000000..5172c27 Binary files /dev/null and b/page/celeba_14_1.gif differ diff --git a/page/celeba_14_2.gif b/page/celeba_14_2.gif new file mode 100644 index 0000000..6b8d5b0 Binary files /dev/null and b/page/celeba_14_2.gif differ diff --git a/page/wikiart_6_1.gif b/page/wikiart_6_1.gif new file mode 100644 index 0000000..368024a Binary files /dev/null and b/page/wikiart_6_1.gif differ