Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features [CVPR 2024]

Project Webpage | Paper

Setup

conda env create -f environment.yaml
conda activate diff3f

Additional prerequisites

Install pytorch3d

conda install -c fvcore -c iopath -c conda-forge fvcore iopath

You might face difficulty in installing pytorch3d or encounter the error ModuleNotFoundError: No module named 'pytorch3d during run time. Unfortunately, this is because pytorch3d could not be installed properly. Please refer here for alternate ways to install pytorch3d.

Usage

Please check the example notebook test_correspondence.ipynb for details on computing features for a mesh and finding correspondence/part segmentations.

Datasets

SHREC'19 remeshed dataset | SHREC'19 GT | This is a more challenging version of the dataset for topology dependent methods

Original SHREC'19 dataset | Please use this version of the dataset when using point cloud renders as it has higher res meshes

TOSCA and SHREC'07

SHREC'20

Evaluation

Much of the code for evaluation is based upon SE-ORNet with some tweaks to decouple it from the method and make it work for precomputed correspondence (there is no change in the computation of core metrics). Building the environment for evaluation can be painful as it involves building multiple CUDA packages for 3D, for installation please refer to SE-ORNet or DPC. If you can get their code to run, this will also work flawlessly. The main issue is usually to get diffusers and pytorch3d working with the dependencies mentioned in SE-ORNet/DPC but this is not essential as you can keep two separate environemnts-- one to extract mesh features and another to perform the evaluation. Therefore, building the environment from SE-ORNet/DPC alone might be enough.

I have attched my eval_environment but it may not work for you.

# Extract features for SHREC'19 using
python extract_shrec.py

# Once features are extracted, run evaluation using
python evaluate_pipeline_shrec.py

# Extract features for TOSCA using
python extract_tosca.py

# Once features are extracted, run evaluation using
python evaluate_pipeline_tosca.py

Additional Notes

The meshes provided in the meshes directory are provided as examples from various sources and we do not claim any copyright.

BibTeX

If you find our research useful, please consider citing it as follows.

@InProceedings{Dutt_2024_CVPR,
    author    = {Dutt, Niladri Shekhar and Muralikrishnan, Sanjeev and Mitra, Niloy J.},
    title     = {Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {4494-4504}
}