Lesson 16:vision-face detection

Machine Vision is a very complex subject that requires a more professional knowledge of computer graphics, it provides a good introductory example in the ROS By Example Volume 1 book, so we’ll begin Diego’s machine vision tour, then we will adding more complex content by gradually.

We first come to realize  the face detection, Feature point acquisition, face tracking, robot follow, because the code used in this book depends on opencv2.4, but in Diego we installed the ROS Kinetic, Opencv3 is installed by default, so I have ported its source code so that it can be compiled in the Opencv3 environment. detail please see the diego_vision directory under github Diego1 plus.

1.About image format

ROS has its own image format, but the more popular are using opencv to image video processing, because it encapsulates a large number of advanced image algorithm, and it’s  open source, so the first need to ROS image format conversion to opencv image Format, no surprise ROS has provided cv_bridge, this package can do the conversion task, and support opencv3. If the installation is ros-kinetic-desktop-full version, the package has been installed, if there is no ros-kinetic-desktop-full need to be installed separately, it is recommended that we git clone source code installed, because if apt- Get installed, it will also install opencv2.4 version, which will cause some conflict. how to use  cv_bridge method, we can refer to the official example code.

2.ros2opencv3.py

The original ros2opencv2.py is not support opencv3.0, so I  transplanted it to support opencv3, and the file is renamed to ros2opencv3.py, class name is also changed to ROS2OpenCV3, the source code see GitHub, the following is the main function Function Description:

  • __init__:ROS2OpenCV3 initialization function, set the relevant parameters, initialize the ROS node, subscribe to RGB and RGBD message topics
  • on_mouse_click:Mouse events, the user can draw the ROI through the mouse in the video window
  • image_callback:The callback function of RGB image
  • depth_callback:The callback function of the depth image
  • convert_image:Converts RGB images from ROS format to opencv format
  • convert_depth_image:Convert the depth image from the ROS format to the opencv format
  • publish_roi:Publish the ROI (Region Of Interesting) message, which describes the area  we are interested in
  • process_image:RGB image processing function, there do not do any processing
  • process_depth_image:Depth image processing function, there do not do any processing
  • display_selection:Use the rectangle to display the area selected by the user on the video. Use a small dot to display the area where the user clicks on the image
  • is_rect_nonzero:Determine whether the rectangular area is valid
  • cvBox2D_to_cvRect:Convert the cvBox2D data structure to cvRect
  • cvRect_to_cvBox2D:Convert the cvRect data structure to cvBox2D

3.Face detection

3.1 the way for Face detection

The general step in face detection is

 

 

 

分类器暨Haar特征分类器,人脸的Haar特征分类器就是一个XML文件,该文件中会描述人脸的Haar特征值,opencv算法针对图片中的数据匹配分类器,从而识别出人脸,我们这里使用到三个分类器

the face of the Haar feature classifier is an XML file, the file will describe the face Haar eigenvalue, opencv will identify the face by matching classifier by it algorithm for the data in the picture, we will use three classifiers:

  • haarcascade_frontalface_alt2.xml
  • haarcascade_frontalface_alt.xml
  • haarcascade_profileface.xml

After reading the picture from the camera video, we first need to convert the image into a gray image, and then get the feature, last we call opencv detectMultiScale function to  detect the face

3.2 code

The code has been ported to opencv3, please see github

3.2.1face_detector.py

#!/usr/bin/env python

import rospy
import cv2
from ros2opencv3 import ROS2OpenCV3

class FaceDetector(ROS2OpenCV3):
    def __init__(self, node_name):
        super(FaceDetector, self).__init__(node_name)
                  
        # Get the paths to the cascade XML files for the Haar detectors.
        # These are set in the launch file.
        cascade_1 = rospy.get_param("~cascade_1", "")
        cascade_2 = rospy.get_param("~cascade_2", "")
        cascade_3 = rospy.get_param("~cascade_3", "")
        
        # Initialize the Haar detectors using the cascade files
        self.cascade_1 = cv2.CascadeClassifier(cascade_1)
        self.cascade_2 = cv2.CascadeClassifier(cascade_2)
        self.cascade_3 = cv2.CascadeClassifier(cascade_3)
        
        # Set cascade parameters that tend to work well for faces.
        # Can be overridden in launch file
        self.haar_scaleFactor = rospy.get_param("~haar_scaleFactor", 1.3)
        self.haar_minNeighbors = rospy.get_param("~haar_minNeighbors", 3)
        self.haar_minSize = rospy.get_param("~haar_minSize", 30)
        self.haar_maxSize = rospy.get_param("~haar_maxSize", 150)
        
        # Store all parameters together for passing to the detector
        self.haar_params = dict(scaleFactor = self.haar_scaleFactor,
                                minNeighbors = self.haar_minNeighbors,
                                flags = cv2.CASCADE_DO_CANNY_PRUNING,
                                minSize = (self.haar_minSize, self.haar_minSize),
                                maxSize = (self.haar_maxSize, self.haar_maxSize)
                                )
                        
        # Do we should text on the display?
        self.show_text = rospy.get_param("~show_text", True)
        
        # Intialize the detection box
        self.detect_box = None
        
        # Track the number of hits and misses
        self.hits = 0
        self.misses = 0
        self.hit_rate = 0

    def process_image(self, cv_image):
        # Create a greyscale version of the image
        grey = cv2.cvtColor(cv_image, cv2.COLOR_BGR2GRAY)
        
        # Equalize the histogram to reduce lighting effects
        grey = cv2.equalizeHist(grey)
            
        # Attempt to detect a face
        self.detect_box = self.detect_face(grey)
        
        # Did we find one?
        if self.detect_box is not None:
            self.hits += 1
        else:
            self.misses += 1
        
        # Keep tabs on the hit rate so far
        self.hit_rate = float(self.hits) / (self.hits + self.misses)
                    
        return cv_image


    def detect_face(self, input_image):
        # First check one of the frontal templates
        if self.cascade_1:
            faces = self.cascade_1.detectMultiScale(input_image, **self.haar_params)
                                         
        # If that fails, check the profile template
        if len(faces) == 0 and self.cascade_3:
            faces = self.cascade_3.detectMultiScale(input_image, **self.haar_params)

        # If that also fails, check a the other frontal template
        if len(faces) == 0 and self.cascade_2:
            faces = self.cascade_2.detectMultiScale(input_image, **self.haar_params)

        # The faces variable holds a list of face boxes.
        # If one or more faces are detected, return the first one.  
        if len(faces) > 0:
            face_box = faces[0]
        else:
            # If no faces were detected, print the "LOST FACE" message on the screen
            if self.show_text:
                font_face = cv2.FONT_HERSHEY_SIMPLEX
                font_scale = 0.5
                cv2.putText(self.marker_image, "LOST FACE!", 
                            (int(self.frame_size[0] * 0.65), int(self.frame_size[1] * 0.9)), 
                            font_face, font_scale, (255, 50, 50))
            face_box = None

        # Display the hit rate so far
        if self.show_text:
            font_face = cv2.FONT_HERSHEY_SIMPLEX
            font_scale = 0.5
            cv2.putText(self.marker_image, "Hit Rate: " + 
                        str(trunc(self.hit_rate, 2)), 
                        (20, int(self.frame_size[1] * 0.9)), 
                        font_face, font_scale, (255, 255, 0))
        
        return face_box
        
def trunc(f, n):
    '''Truncates/pads a float f to n decimal places without rounding'''
    slen = len('%.*f' % (n, f))
    return float(str(f)[:slen])
    
if __name__ == '__main__':
    try:
        node_name = "face_detector"
        FaceDetector(node_name)
        rospy.spin()
    except KeyboardInterrupt:
        print "Shutting down face detector node."
        cv2.destroyAllWindows()

3.2.2Code explanation

FaceDetector class inherited from ROS2OpenCV3, and rewrite the process_image to convert the the image to gray-scale, and do the face detection by detect_face function

load the classifier, cascade_1, cascade_2, cascade_3 ,defines the location of the three classifiers

        
        # Get the paths to the cascade XML files for the Haar detectors.
        # These are set in the launch file.
        cascade_1 = rospy.get_param("~cascade_1", "")
        cascade_2 = rospy.get_param("~cascade_2", "")
        cascade_3 = rospy.get_param("~cascade_3", "")
        
        # Initialize the Haar detectors using the cascade files
        self.cascade_1 = cv2.CascadeClassifier(cascade_1)
        self.cascade_2 = cv2.CascadeClassifier(cascade_2)
        self.cascade_3 = cv2.CascadeClassifier(cascade_3)

Image gray conversion in the process_image function

        # Create a greyscale version of the image
        grey = cv2.cvtColor(cv_image, cv2.COLOR_BGR2GRAY)
        
        # Equalize the histogram to reduce lighting effects
        grey = cv2.equalizeHist(grey)

Face detection in the detect_face function

        if self.cascade_1:
            faces = self.cascade_1.detectMultiScale(input_image, **self.haar_params)
                                         
        # If that fails, check the profile template
        if len(faces) == 0 and self.cascade_3:
            faces = self.cascade_3.detectMultiScale(input_image, **self.haar_params)

        # If that also fails, check a the other frontal template
        if len(faces) == 0 and self.cascade_2:
            faces = self.cascade_2.detectMultiScale(input_image, **self.haar_params)

Display information in the detect_face function

        # The faces variable holds a list of face boxes.
        # If one or more faces are detected, return the first one.  
        if len(faces) > 0:
            face_box = faces[0]
        else:
            # If no faces were detected, print the "LOST FACE" message on the screen
            if self.show_text:
                font_face = cv2.FONT_HERSHEY_SIMPLEX
                font_scale = 0.5
                cv2.putText(self.marker_image, "LOST FACE!", 
                            (int(self.frame_size[0] * 0.65), int(self.frame_size[1] * 0.9)), 
                            font_face, font_scale, (255, 50, 50))
            face_box = None

        # Display the hit rate so far
        if self.show_text:
            font_face = cv2.FONT_HERSHEY_SIMPLEX
            font_scale = 0.5
            cv2.putText(self.marker_image, "Hit Rate: " + 
                        str(trunc(self.hit_rate, 2)), 
                        (20, int(self.frame_size[1] * 0.9)), 
                        font_face, font_scale, (255, 255, 0))
        
        return face_box

The image_callback function of the parent class ROS2OpenCV3 will draw the face position  by a rectangular wireframe

4. start the face detection node

First make sure xtion was started, you can start with the following code:

roslaunch diego_vision openni_node.launch

please check face_detector.launch from github

<launch>
   <node pkg="diego_vision" name="face_detector" type="face_detector.py" output="screen">

   <remap from="input_rgb_image" to="/camera/rgb/image_color" />

   <rosparam>
       haar_scaleFactor: 1.3
       haar_minNeighbors: 3
       haar_minSize: 30
       haar_maxSize: 150
   </rosparam>

   <param name="cascade_1" value="$(find diego_vision)/data/haar_detectors/haarcascade_frontalface_alt2.xml" />
   <param name="cascade_2" value="$(find diego_vision)/data/haar_detectors/haarcascade_frontalface_alt.xml" />
   <param name="cascade_3" value="$(find diego_vision)/data/haar_detectors/haarcascade_profileface.xml" />

</node>
</launch>

start the face detector node

roslaunch diego_vision face_detector.launch

You will see a video form after the start the node, you can detect the camera to capture the face, and identified by a rectangle

Leave a Reply

Scroll to top
%d bloggers like this: