First commit

Removing test data

Update Readme to add instructions for downloading test_data

Add gitattributes for test_data

.gitattributes

@@ -1,2 +1,2 @@
-
 test_data/** filter= diff= merge= -text
+

.gitignore

@@ -0,0 +1,6 @@
+results
+checkpoints/*
+*.npz
+**/*.pyc
+**/__pycache__
+.vscode

README.md

@@ -0,0 +1,134 @@
+# Contact-GraspNet 
+
+### Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes 
+Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter Fox 
+ICRA 2021 
+
+[paper](https://arxiv.org/abs/2103.14127), [project page](https://research.nvidia.com/publication/2021-03_Contact-GraspNet%3A--Efficient), [video](http://www.youtube.com/watch?v=qRLKYSLXElM)
+
+<p align="center">
+ <img src="examples/2.gif" width="640" title="UOIS + Contact-GraspNet"/>
+</p>
+
+## Installation
+
+This code has been tested with python 3.7, tensorflow 2.2, CUDA 10.1, and CUDNN 7.6.0
+
+Create the conda env
+```
+conda env create -f contact_graspnet_env.yml
+```
+
+### Troubleshooting
+
+- Recompile pointnet2 tf_ops, see [here](pointnet2/tf_ops/HowTO.md) 
+
+### Hardware
+Training: 1x Nvidia GPU >= 24GB VRAM, >=64GB RAM 
+Inference: 1x Nvidia GPU >= 8GB VRAM (might work with less)
+
+## Download Models and Data
+### Model
+Download trained models from [here](https://drive.google.com/drive/folders/1tBHKf60K8DLM5arm-Chyf7jxkzOr5zGl?usp=sharing) and copy them into the `checkpoints/` folder.
+### Test data
+Download the test data from [here](https://drive.google.com/drive/folders/1v0_QMTUIEOcu09Int5V6N2Nuq7UCtuAA?usp=sharing) and copy them them into the `test_data/` folder.
+
+## Inference
+
+
+Contact-GraspNet can directly predict a 6-DoF grasp distribution from a raw scene point cloud. However, to obtain object-wise grasps, remove background grasps and to achieve denser proposals it is highly recommended to use (unknown) object segmentation [e.g. [1](https://github.com/chrisdxie/uois), [2](https://arxiv.org/abs/2103.06796)] as preprocessing and then use the resulting segmentation map to crop local regions and filter grasp contacts.
+
+Given a .npy/.npz file with a depth map (in meters), camera matrix K and (optionally) a 2D segmentation map, execute:
+
+```shell
+python contact_graspnet/inference.py \
+ --np_path=test_data/*.npy \
+ --local_regions --filter_grasps
+```
+
+<p align="center">
+ <img src="examples/7.png" width="640" title="UOIS + Contact-GraspNet"/>
+</p>
+--> close the window to go to next scene
+
+Given a .npy/.npz file with just a 3D point cloud (in meters), execute [for example](examples/realsense_crop_sigma_001.png):
+```shell
+python contact_graspnet/inference.py --np_path=/path/to/your/pc.npy \
+ --forward_passes=5 \
+ --z_range=[0.2,1.1]
+```
+
+`--np_path`: input .npz/.npy file(s) with 'depth', 'K' and optionally 'segmap', 'rgb' keys. For processing a Nx3 point cloud instead use 'xzy' and optionally 'xyz_color' as keys. 
+`--ckpt_dir`: relative path to checkpooint directory. By default `checkpoint/scene_test_2048_bs3_hor_sigma_001` is used. For very clean / noisy depth data consider `scene_2048_bs3_rad2_32` / `scene_test_2048_bs3_hor_sigma_0025` trained with no / strong noise. 
+`--local_regions`: Crop 3D local regions around object segments for inference. (only works with segmap) 
+`--filter_grasps`: Filter grasp contacts such that they only lie on the surface of object segments. (only works with segmap) 
+`--skip_border_objects` Ignore segments touching the depth map boundary. 
+`--forward_passes` number of (batched) forward passes. Increase to sample more potential grasp contacts. 
+`--z_range` [min, max] z values in meter used to crop the input point cloud, e.g. to avoid grasps in the foreground/background(as above). 
+`--arg_configs TEST.second_thres:0.19 TEST.first_thres:0.23` Overwrite config confidence thresholds for successful grasp contacts to get more/less grasp proposals 
+
+
+## Training
+
+### Download Data 
+
+Download the Acronym dataset, ShapeNet meshes and make them watertight, following these [steps](https://github.com/NVlabs/acronym#using-the-full-acronym-dataset).
+
+Download the training data consisting of 10000 table top training scenes with contact grasp information from [here](https://drive.google.com/drive/folders/1eeEXAISPaStZyjMX8BHR08cdQY4HF4s0?usp=sharing) and extract it to the same folder:
+
+```
+acronym
+├── grasps
+├── meshes
+├── scene_contacts
+└── splits
+```
+
+### Train Contact-GraspNet
+
+When training on a headless server set the environment variable
+```shell
+export PYOPENGL_PLATFORM='egl'
+```
+
+Start training with config `contact_graspnet/config.yaml`
+```
+python contact_graspnet/train.py --ckpt_dir checkpoints/your_model_name \
+ --data_path /path/to/acronym/data
+```
+
+### Generate Contact Grasps and Scenes yourself (optional)
+
+The `scene_contacts` downloaded above are generated from the Acronym dataset. To generate/visualize table-top scenes yourself, also pip install the [acronym_tools]((https://github.com/NVlabs/acronym)) package in your conda environment as described in the acronym repository.
+
+In the first step, object-wise 6-DoF grasps are mapped to their contact points saved in `mesh_contacts`
+
+```
+python tools/create_contact_infos.py /path/to/acronym
+```
+
+From the generated `mesh_contacts` you can create table-top scenes which are saved in `scene_contacts` with
+
+```
+python tools/create_table_top_scenes.py /path/to/acronym
+```
+
+Takes ~3 days in a single thread. Run the command several times to process on multiple cores in parallel.
+
+You can also visualize existing table-top scenes and grasps
+
+```
+python tools/create_table_top_scenes.py /path/to/acronym \
+ --load_existing scene_contacts/000000.npz -vis
+```
+
+## Citation
+
+```
+@article{sundermeyer2021contact,
+ title={Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes},
+ author={Sundermeyer, Martin and Mousavian, Arsalan and Triebel, Rudolph and Fox, Dieter},
+ booktitle={2021 IEEE International Conference on Robotics and Automation (ICRA)},
+ year={2021}
+}
+```

contact_graspnet/config.yaml

@@ -0,0 +1,186 @@
+DATA:
+ scene_contacts_path: scene_contacts
+ gripper_width: 0.08
+ input_normals: false
+ use_uniform_quaternions: False
+ view_sphere:
+ elevation: [30,150]
+ distance_range: [0.9,1.3]
+ train_on_scenes: True
+ labels:
+ to_gpu: False
+ bin_weights:
+ - 0.16652107
+ - 0.21488856
+ - 0.37031708
+ - 0.55618503
+ - 0.75124664
+ - 0.93943357
+ - 1.07824539
+ - 1.19423112
+ - 1.55731375
+ - 3.17161779
+ filter_z: true
+ k: 1
+ max_radius: 0.005
+ num_neg_contacts: 0
+ num_pos_contacts: 8000
+ offset_bins:
+ - 0
+ - 0.00794435329
+ - 0.0158887021
+ - 0.0238330509
+ - 0.0317773996
+ - 0.0397217484
+ - 0.0476660972
+ - 0.055610446
+ - 0.0635547948
+ - 0.0714991435
+ - 0.08
+ z_val: -0.1
+ pc_augm:
+ occlusion_nclusters: 0
+ occlusion_dropout_rate: 0.0
+ sigma: 0.000
+ clip: 0.005
+ depth_augm:
+ sigma: 0.001
+ clip: 0.005
+ gaussian_kernel: 0
+ raw_num_points: 20000
+ ndataset_points: 20000
+ num_point: 2048
+ use_farthest_point: false
+ train_and_test: false
+ num_test_scenes: 1000
+ intrinsics: 'realsense'
+LOSS:
+ min_geom_loss_divisor: 1.0
+ offset_loss_type: sigmoid_cross_entropy
+ too_small_offset_pred_bin_factor: 0
+ topk_confidence: 512
+MODEL:
+ bin_offsets: true
+ dir_vec_length_offset: false
+ grasp_conf_head:
+ conv1d: 1
+ dropout_keep: 0.5
+ grasp_dir_head:
+ conv1d: 3
+ dropout_keep: 0.7
+ joint_head:
+ conv1d: 4
+ dropout_keep: 0.7
+ joint_heads: false
+ asymmetric_model: true
+ model: contact_graspnet
+ pointnet_fp_modules:
+ - mlp:
+ - 256
+ - 256
+ - mlp:
+ - 256
+ - 128
+ - mlp:
+ - 128
+ - 128
+ - 128
+ pointnet_sa_module:
+ group_all: true
+ mlp:
+ - 256
+ - 512
+ - 1024
+ pointnet_sa_modules_msg:
+ - mlp_list:
+ - - 32
+ - 32
+ - 64
+ - - 64
+ - 64
+ - 128
+ - - 64
+ - 96
+ - 128
+ npoint: 2048
+ nsample_list:
+ - 32
+ - 64
+ - 128
+ radius_list:
+ - 0.02
+ - 0.04
+ - 0.08
+ - mlp_list:
+ - - 64
+ - 64
+ - 128
+ - - 128
+ - 128
+ - 256
+ - - 128
+ - 128
+ - 256
+ npoint: 512
+ nsample_list:
+ - 64
+ - 64
+ - 128
+ radius_list:
+ - 0.04
+ - 0.08
+ - 0.16
+ - mlp_list:
+ - - 64
+ - 64
+ - 128
+ - - 128
+ - 128
+ - 256
+ - - 128
+ - 128
+ - 256
+ npoint: 128
+ nsample_list:
+ - 64
+ - 64
+ - 128
+ radius_list:
+ - 0.08
+ - 0.16
+ - 0.32
+ pred_contact_approach: false
+ pred_contact_base: false
+ pred_contact_offset: true
+ pred_contact_success: true
+ pred_grasps_adds: true
+ pred_grasps_adds_gt2pred: false
+OPTIMIZER:
+ max_epoch: 16
+ batch_size: 3
+ learning_rate: 0.001
+ optimizer: adam
+ momentum: 0.9
+ adds_gt2pred_loss_weight: 1
+ adds_loss_weight: 10
+ approach_cosine_loss_weight: 1
+ dir_cosine_loss_weight: 1
+ offset_loss_weight: 1
+ score_ce_loss_weight: 1
+ bn_decay_clip: 0.99
+ bn_decay_decay_rate: 0.5
+ bn_decay_decay_step: 200000
+ bn_init_decay: 0.5
+ decay_rate: 0.7
+ decay_step: 200000
+TEST:
+ first_thres: 0.23
+ second_thres: 0.19
+ allow_zero_margin: 0
+ bin_vals: max
+ center_to_tip: 0.0
+ extra_opening: 0.005
+ max_farthest_points: 150
+ num_samples: 200
+ with_replacement: false
+ filter_thres: 0.0001

contact_graspnet/config_utils.py

@@ -0,0 +1,66 @@
+import os
+import yaml
+
+def recursive_key_value_assign(d,ks,v):
+ """
+ Recursive value assignment to a nested dict
+
+ Arguments:
+ d {dict} -- dict
+ ks {list} -- list of hierarchical keys
+ v {value} -- value to assign
+ """
+
+ if len(ks) > 1:
+ recursive_key_value_assign(d[ks[0]],ks[1:],v)
+ elif len(ks) == 1:
+ d[ks[0]] = v
+
+def load_config(checkpoint_dir, batch_size=None, max_epoch=None, data_path=None, arg_configs=[], save=False):
+ """
+ Loads yaml config file and overwrites parameters with function arguments and --arg_config parameters
+
+ Arguments:
+ checkpoint_dir {str} -- Checkpoint directory where config file was copied to
+
+ Keyword Arguments:
+ batch_size {int} -- [description] (default: {None})
+ max_epoch {int} -- "epochs" (number of scenes) to train (default: {None})
+ data_path {str} -- path to scenes with contact grasp data (default: {None})
+ arg_configs {list} -- Overwrite config parameters by hierarchical command line arguments (default: {[]})
+ save {bool} -- Save overwritten config file (default: {False})
+
+ Returns:
+ [dict] -- Config
+ """
+
+ config_path = os.path.join(checkpoint_dir, 'config.yaml')
+ config_path = config_path if os.path.exists(config_path) else os.path.join(os.path.dirname(__file__),'config.yaml')
+ with open(config_path,'r') as f:
+ global_config = yaml.load(f)
+
+ for conf in arg_configs:
+ k_str, v = conf.split(':')
+ try:
+ v = eval(v)
+ except:
+ pass
+ ks = [int(k) if k.isdigit() else k for k in k_str.split('.')]
+
+ recursive_key_value_assign(global_config, ks, v)
+
+ if batch_size is not None:
+ global_config['OPTIMIZER']['batch_size'] = int(batch_size)
+ if max_epoch is not None:
+ global_config['OPTIMIZER']['max_epoch'] = int(max_epoch)
+ if data_path is not None:
+ global_config['DATA']['data_path'] = data_path
+
+ global_config['DATA']['classes'] = None
+
+ if save:
+ with open(os.path.join(checkpoint_dir, 'config.yaml'),'w') as f:
+ yaml.dump(global_config, f)
+
+ return global_config
+