Fejemis Vision ROS2

Back to Fejemis ROS2 Software

fejemis_vision package

The fejemis_vision package is the computer vision subsystem used in the Fejemis robot platform. It is responsible for environmental perception and object detection using RGB and depth camera data.

Main functionalities include:

• Human detection
• Cable detection
• Net detection
• Depth-based segmentation
• Pose estimation
• Object tracking

The package is located in: src/fejemis/fejemis_vision

Directory Structure

launch/

Contains ROS 2 launch files used to start the vision system and individual detection pipelines.

Main launch files:

• main_launch.py - Launches the complete perception system.

Testing launch files:

• cable_test.launch.py - Launches the cable detection pipeline.
• net_test.launch.py - Launches the net detection pipeline.
• people_test.launch.py - Launches the people detection pipeline.

These launch files initialize the required ROS 2 nodes, camera subscriptions, and RViz visualization.

pyth/

Contains the main Python source code for the vision and detection system.

Main scripts:

• main.py - Main ROS 2 vision node coordinating the perception system

Detection scripts:

• cable_detection.py - Detects cables and linear structures in the environment.
• net_detection.py - Detects net or mesh-like structures using computer vision methods.
• people_detection_pose_segmentation.py - Human detection using pose estimation and segmentation.
• people_detection_depth_segmentation.py - Human detection using depth-based segmentation.
• people_detection_simple_segmentation.py - Simpler segmentation-based human detection approach.

Tracking:

• people_detection_deep_sort.py - Multi-person tracking using the DeepSORT tracking algorithm.

Testing scripts:

• cable_test.py
• net_test.py
• people_test_basic.py

These scripts are used for testing and validating the detection pipelines independently.

Test Image Directories

The package contains several directories with offline test images used during development and debugging:

• cable_test_images/
• net_test_images/
• people_test_images/
• other_imgs/

These images are used to evaluate the detection models without requiring a live camera stream.

deep_learning/

Contains neural network models and configuration files used by the detection system.

Files include:

• yolov4-tiny.cfg - YOLOv4-tiny network configuration.
• yolov4-tiny.weights - Pretrained YOLOv4-tiny weights.
• classes.txt - Object class labels used by the detector.
• mars-small128.pb - Feature extraction model used by DeepSORT tracking.

The package uses YOLOv4-tiny to achieve lightweight real-time object detection suitable for embedded systems such as Raspberry Pi devices.

config/

Contains configuration files used to tune the perception system.

Typical configuration includes:

• Detection thresholds
• Camera parameters
• Segmentation parameters
• ROS topic settings

These configuration files allow tuning the system without modifying the source code.

Setup

Install Dependencies

Install OpenCV:

pip install opencv-python

Install the required Python packages:

pip install -r requirements.txt

Make Python Scripts Executable

Before launching the ROS 2 nodes, ensure the Python scripts have executable permissions:

chmod +x main.py
chmod +x net_test.py
chmod +x people_test_basic.py
chmod +x cable_test.py 

Launch files

To launch the full perception pipeline:

ros2 launch fejemis_vision main.launch.py

To run individual test pipelines:

ros2 launch fejemis_vision cable_test.launch.py
ros2 launch fejemis_vision people_test.launch.py
ros2 launch fejemis_vision net.launch.py

Progress and Current State

A major part of the work involved converting the original package from ROS 1 to ROS 2 compatibility.

Due to limited project time, only partial testing and validation were completed. Additional development and tuning are still required.

Current Results

• All detection pipelines were successfully tested offline using stored test images.
• Live camera stream testing produced mixed results.
• People detection and net detection showed the most stable and reliable performance.
• RViz marker visualization worked correctly for several detection pipelines.
• Cable detection was significantly more unstable and noisy. The cables were sometimes correctly detected, however false detections and random line generation frequently occurred.

Areas for Future Improvement

• Detection parameters and thresholds were not fully tuned. Additional tuning may significantly reduce detection noise, especially for cable detection.
• Only people_detection_pose_segmentation.py was verified to work reliably during live testing. The remaining human detection methods were not fully investigated due to time limitations.
• Additional filtering and depth processing could improve robustness.
• Performance optimization may still be required for Raspberry Pi deployment.

RViz Visualization

The original ROS 1 implementation used the jsk-visualization package for RViz markers.

At the time of this project, jsk-visualization was not fully compatible with ROS 2. Therefore, the system was modified to use built-in ROS 2 MarkerArray visualization messages instead.

This allowed detected objects and positions to still be visualized in RViz.

Original ROS 1 Repository

The original ROS 1 implementation can be found here: MP-EL/fejemis_vision