Integrating OpenCV with ROS2: A Comprehensive Guide to Computer Vision in Robotics
Introduction
In the world of robotics, vision is a crucial sense that enables machines to perceive and interact with their environment. This post will guide you through the process of integrating OpenCV, a powerful computer vision library, with ROS2. We’ll explore the architecture, set up the environment, and dive into some exciting projects.
ROS2 Architecture and Computer Vision Pipeline
ROS2, the next-generation robotics middleware, provides a flexible framework for developing robotic applications. Its publish-subscribe architecture allows for modular design and easy integration of various components, including computer vision systems.
In a typical ROS2 computer vision pipeline:
- A camera node publishes raw image data to a topic.
- A vision processing node subscribes to this topic, processes the images using OpenCV, and publishes the results.
- Other nodes (e.g., control systems) can subscribe to the processed data for decision-making.
Required Libraries
To get started with OpenCV in ROS2, you’ll need the following libraries:
sudo apt-get update
sudo apt install ros-<distro>-cv-bridge
sudo apt-get install python3-opencv
pip install opencv-pythonReplace <distro> with your ROS2 distribution (e.g., foxy, galactic, humble).
Creating a ROS2 Package
To create a new ROS2 package for your computer vision project:
ros2 pkg create --build-type ament_python my_cv_package
cd my_cv_packageBasic Computer Vision Projects
Let’s explore some fundamental computer vision projects you can implement in ROS2:
a) Image Subscriber
This basic node subscribes to a camera topic and displays the received images:
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from cv_bridge import CvBridge
import cv2
class ImageSubscriber(Node):
def __init__(self):
super().__init__('image_subscriber')
self.subscription = self.create_subscription(
Image,
'/camera/image_raw',
self.listener_callback,
10)
self.br = CvBridge()
def listener_callback(self, data):
self.get_logger().info('Receiving video frame')
current_frame = self.br.imgmsg_to_cv2(data)
cv2.imshow("camera", current_frame)
cv2.waitKey(1)
def main(args=None):
rclpy.init(args=args)
image_subscriber = ImageSubscriber()
rclpy.spin(image_subscriber)
image_subscriber.destroy_node()
rclpy.shutdown()
if __name__ == '__main__':
main()
b) Color Detection
This project detects a specific color in the image and make a centroid:
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import Image
from geometry_msgs.msg import Twist
from cv_bridge import CvBridge
import cv2
import numpy as np
class LineFollower(Node):
def __init__(self):
super().__init__('line_follower')
self.subscription = self.create_subscription(Image, '/camera/image_raw', self.listener_callback, 10)
self.subscription
self.br = CvBridge()
self.publisher_ = self.create_publisher(Twist, '/cmd_vel', 10)
# Set the desired linear and angular velocity gains
self.linear_velocity_gain = 0.2
self.angular_velocity_gain = 0.3
def listener_callback(self, data):
self.get_logger().info('Receiving video frame')
img = self.br.imgmsg_to_cv2(data, 'bgr8')
img = cv2.resize(img, None, 1, 0.25, 0.25, cv2.INTER_CUBIC)
imgHSV = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
lower_yellow = np.array([0, 35, 0])
upper_yellow = np.array([57, 255, 255])
mask = cv2.inRange(imgHSV, lower_yellow, upper_yellow)
roi_height = mask.shape[0] // 4
roi = mask[mask.shape[0] - roi_height:, :]
M = cv2.moments(roi)
if M['m00'] > 0:
cx = int(M['m10'] / M['m00'])
cy = int(M['m01'] / M['m00']) + mask.shape[0] - roi_height
centroid = (cx, cy)
radius = 8
img_center = (mask.shape[1] // 2, mask.shape[0] // 2)
distance_from_center = np.sqrt((cx - img_center[0])**2 + (cy - img_center[1])**2)
max_distance = np.sqrt(img_center[0]**2 + img_center[1]**2)
color_intensity = int(255 * (1 - distance_from_center / max_distance))
color = (color_intensity, color_intensity, 0)
cv2.circle(img, centroid, radius, color, -1)
# Calculate linear and angular velocities based on centroid position
error = img_center[0] - cx
linear_velocity = self.linear_velocity_gain
angular_velocity = self.angular_velocity_gain * error
# Keep the centroid in the center of the ROI
if abs(error) > 20:
angular_velocity = np.clip(angular_velocity, -0.3, 0.3)
# Publish the velocity commands
twist = Twist()
twist.linear.x = linear_velocity
twist.angular.z = angular_velocity
self.publisher_.publish(twist)
cv2.imshow("Line Threshold", mask)
cv2.imshow("Region of Interest", roi)
cv2.imshow("Centroid Indicator", img)
cv2.waitKey(1)
def main(args=None):
rclpy.init(args=args)
line_follower = LineFollower()
rclpy.spin(line_follower)
line_follower.destroy_node()
rclpy.shutdown()
if __name__ == '__main__':
main()
Advanced Projects
As you become more comfortable with OpenCV and ROS2, you can tackle more advanced projects:
- Object Detection: Use pre-trained models like YOLO or SSD for real-time object detection.
- Face Recognition: Implement face detection and recognition algorithms.
- Visual SLAM: Implement Simultaneous Localization and Mapping using visual data.
Conclusion
Integrating OpenCV with ROS2 opens up a world of possibilities in robotics and computer vision. From basic image processing to advanced perception tasks, the combination of these powerful tools enables the development of sophisticated robotic systems capable of understanding and interacting with their environment.
Remember, the key to mastering these technologies is practice and experimentation. Start with the basic projects, then gradually move to more complex applications.
