FS_Video_IMU_Fusion

This is the PyTorch implementation of 3-D human motion tracking from videos and/or IMUs.

Sample result

1. Installation

conda create -n vifusion python=3.9
conda activate vifusion
# Please install PyTorch according to your CUDA version.
pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt

Dataset

We used AMASS dataset for training the networks and TotalCapture dataset for the evaluation. To reproduce our networks, please do the following steps.

mkdir dataset

1. Download AMASS motion file from AMASS official website. You can skip for TotalCapture dataset as we use it for the validation. Then unzip all sub-datasets under the folder of dataset/AMASS.
2. Download Total Capture dataset from Total Capture official website. You need to download videos (for 2D keypoints detection) and IMU data.
3. Run off-the-shelf 2D keypoints detection models (e.g., ViTPose, CPN]) and store detection under the Total Capture dataset folder.
4. Download SMPL body models. You can find those from SMPLify official website.

After following above steps, your folder structure should look like:

$(REPO ROOT)/
|-- dataset
|   |-- AMASS/
|       |-- CMU/
|       |-- ...
|
|   |-- TotalCapture/
|       |-- calibration.cal
|       |-- S1
|            |-- acting1_Xsens_AuxFileds.sensors
|            |-- s1_acting1_calib_imu_ref.txt
|            |-- TC_S1_acting1_cam1.mp4
|            |-- ...
|       |-- S2
|       |-- S3
|       |-- S4
|       |-- S5
|       |-- detection
|            |-- S1_acting1_cam1.npy
|            |-- S1_acting1_cam2.npy
|            |-- ...
|
|   |-- body_models/
|       |-- smpl/
|           |-- SMPL_FEMALE.pkl
|           |-- ...

5. Process data:

python -m lib.data.gendb_amass          # Process AMASS dataset
python -m lib.data.gendb_totalcapture   # Process Total Capture dataset

Train the model

To train our model from the scratch:

1. IMUNet: python -m lib.core.train -c configs/imunet.yaml
2. VideoNet: python -m lib.core.train -c configs/videonet.yaml
3. FusionNet: python -m lib.core.train -c configs/fusionnet.yaml

Work in progress

We Plan to update this repository with following functions:

• Upload pretrained model checkpoints
• Write demo code with demo video & IMU data
• Update citation