REveL Dataset: RGB-Event-LiDAR Dataset for Assistive Robotics

Adam Scicluna1, Cedric Le Gentil1, Sheila Sutjipto1, Gavin Paul1,
1University of Technology Sydney Robotics Institute
Paper arXiv Data

Pedestrian tracking and localisation on REveL event camera and LiDAR data utilising RVTs and SORT

Sensor suite

Sensor suite, events (polarity-coloured in red or blue) overlaid on the RGB frame, and LiDAR scan with object motion-captured ground truth poses.

Abstract

The increasing adoption of human-robot interaction presents opportunities for technology to positively impact lives, particularly those with visual impairments, through applications such as guide-dog-like assistive robotics. To fill a gap in the current landscape of publicly available datasets and provide a means to assist in the development of safer and more robust algorithms in the future, this website hosts and outlines the dataset introduced in our IEEE CASE 2024 paper named: Towards Robust Perception for Assistive Robotics: An RGB-Event-LiDAR Dataset and Multi-Modal Detection Pipeline.

The dataset introduced, named 'REveL', contains RGB, event, point cloud and Inertial Measurement Unit (IMU) data along with ground truth poses of persons in the scene. It is 14.1 minutes in length split over four ROSBags. To complement existing datasets and aid in enhancing detection model generalisation to different scenes, it was collected in an indoor scenario with a handheld sensor suite moving in the field of view of a Vicon motion-capture system. Two people, also tracked by the motion-capture system, are moving in and out of the sensor suite field of view.

Dataset Information

To collect the RGB, event, and point cloud data, the sensor suite utilised consisted of:

  • Inivation DAVIS346 event camera: Stream of event data (up to 1MHz) with each event being a tuple of x and y positions in the image space, t the timestamp, and p the polarity of the corresponding illumination change; RGB images at 23Hz; 6-DoF IMU at 1kHz (3-axis gyroscope and 3-axis accelerometer).
  • Blickfeld Cube1 LiDAR: 3D point clouds at 7.9Hz with point-wise timestamps.

The sensor’s measurements and output of the motion-capture system were recorded with ROS. We use the rpg dvs ros driver for the DVS camera and the Blickfeld ROS driver for the LiDAR. The sensor suite and helmets worn by persons in the scene were equipped with a set of reflective markers tracked by the Vicon system, subsequently providing the 6-DoF pose of the 2 persons and sensor suite in an arbitrarily fixed reference frame. For convenience and utility, the RGB portion of the dataset is labelled with the class identifier corresponding to the colour helmet worn by each person. In total, the data collected consisted of approximately:

  • 774 million events (~25000 variable length event-array messages)
  • 22000 RGB images
  • 6700 point clouds
  • 70000 ground truth poses each for two persons in the scene

Dataset Structure

  • REveL/
    • session_1/
      • calibration_data/
        • cam_lidar_calib.bag
        • vicon_cam_calib.bag
        • INFO.txt
      • dynamic/
        • images.zip
        • labels.zip
        • classes.txt
        • notes.json
        • dynamic.bag
      • calibration.yaml
    • session_2/
      • calibration_data/
        • ...
      • dog_like/
        • ...
        • dog_like.bag
      • non_square/
        • ...
        • non_square.bag
      • static/
        • ...
        • static.bag
      • calibration.yaml
    • INFO.txt

Experimentation

Comparison of Event and RGB Detection

We compared the reliability of pedestrian detection in the Event space with detection in the RGB space using YOLOv4.

Interpolation end reference image.

Event space

Interpolation end reference image.

RGB space


Pose Localisation with Motion Capture

Using the Vicon motion capture system, the poses of two persons in the scene are obtained to be used as a groundtruth for localisation. The below video shows a played back section of 'dynamic.bag' in RViz with the acquired poses overlaid on the LiDAR pointcloud data. Note that when the Vicon system can not accurately track the location of a person, the pose defaults to the origin.

BibTeX

@misc{scicluna2024robustperceptionassistiverobotics,
      title={Towards Robust Perception for Assistive Robotics: An RGB-Event-LiDAR Dataset and Multi-Modal Detection Pipeline}, 
      author={Adam Scicluna and Cedric Le Gentil and Sheila Sutjipto and Gavin Paul},
      year={2024},
      eprint={2408.13394},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2408.13394}, 
}