mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensors

Sizhe An, Yin Li, Umit Y. Ogras
University of Wisconsin-Madison
Paper arXiv Video Code Data
We need to process the camera-related data due to privacy-preserving protocol, which might delay the release. We have released dataset without camera-related modalities as well as the keypoints and actions label now. The whole dataset including camera-related will be open-sourced soon.

mRI is a multi-modal human pose estimation dataset focusing on rehab movements.



Visualization of mRI from different camera poses.

Abstract

The ability to estimate 3D human body pose and movement, also known as human pose estimation~(HPE), enables many applications for home-based health monitoring, such as remote rehabilitation training. Several possible solutions have emerged using sensors ranging from RGB cameras, depth sensors, millimeter-Wave (mmWave) radars, and wearable inertial sensors. Despite previous efforts on datasets and benchmarks for HPE, few dataset exploits multiple modalities and focuses on home-based health monitoring.

To bridge this gap, we present mRI, a multi-modal 3D human pose estimation dataset with mmWave, RGB-D, and Inertial Sensors. Our dataset consists of over 5 million frames from 20 subjects performing rehabilitation exercises and supports the benchmarks of HPE and action detection. We perform extensive experiments using our dataset and delineate the strength of each modality.

We hope that the release of mRI can catalyze the research in pose estimation, multi-modal learning, and action understanding, and more importantly facilitate the applications of home-based health monitoring.

Multiple Sensing Modalities

BibTeX

@inproceedings{
    an2022mri,
    title={m{RI}: Multi-modal 3D Human Pose Estimation Dataset using mmWave, {RGB}-D, and Inertial Sensors},
    author={Sizhe An and Yin Li and Umit Ogras},
    booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
    year={2022},
    url={https://openreview.net/forum?id=Oa2-cdfBxun}
    }

@misc{https://doi.org/10.48550/arxiv.2210.08394,
    doi = {10.48550/ARXIV.2210.08394},
    url = {https://arxiv.org/abs/2210.08394},
    author = {An, Sizhe and Li, Yin and Ogras, Umit},
    title = {mRI: Multi-modal 3D Human Pose Estimation Dataset using mmWave, RGB-D, and Inertial Sensors},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution Non Commercial Share Alike 4.0 International}
    }