A video of Lisa Miller's implementation of this assignment from Spring Semester 2009.
Assigned: Sept 18, 12:01am, 2010
Due: Sept 26, 11:59pm, 2010
Building on your Enclosure Escape assignment, you will build a controller in ROS to perform "object seeking". In this seeking task, your robot will perceive and drive to objects that are visually recognizable by a solid color appearance or labeled with an AR tag from the robot's visual sensing (i.e., camera). For this assignment, you will be working primarily with the Create platform and a Sony PlayStation Eye USB video camera. For object recognition in ROS, you will use the cmvision package for color blobfinding and the ar_recog package for AR tag recognition.
Assuming perception of objects salient by color or pattern, you will develop an object seeking package for this assignment that enables a robot to continually drive between these (non-occluded) objects in a sequence given at run-time. Your controller's decision making should take the form a finite state machine (FSM). This FSM should use one state variable to specify the currently sought object. For motion control, you should use proportional-derivative (PD) servoing to center objects in the robot's field of view. As a whole, your controller should put the current object in the center of view, drive as close as possible to an object without hitting it, increment the state variable, and continue the process for the next object.
Your tasks for this assignment are as follows.
The following sections provide information to guide you through completing these tasks.
> sudo sh run_ps3cam.sh 04
Note: this command only needs to be run after you plug in the camera, not every time you want to run something that uses the camera.
In this class, we will use mode 04 for the PS3 camera to ensure image resolution and frame rate sufficient for object recognition. More information about installing the PS3 cam driver and other camera modes is available from Trevor Jay's rough PS3/Ubuntu guide.
The course staff has already installed PS3 cam support on our robots. run_ps3cam.sh is a simple bash file that communicates video settings to the camera:
#!/bin/bash arg=$1 if [ -z "$arg" ] ; then echo "usage: sh run_ps3cam.sh arg where argument is the videomode (00-04 and 10-16) check comments in the file for details" exit fi modprobe -r gspca-ov534 modprobe gspca-ov534 videomode=$arg echo 'ps3 camera driver started' #00:640x480@15 #01:640x480@30 #02:640x480@40 #03:640x480@50 #04:640x480@60 #10:320x240@30 #11:320x240@40 #12:320x240@50 #13:320x240@60 #14:320x240@75 #15:320x240@100 #16:320x240@125
> guvcview -d /dev/video1
guvcview should produce a graphical interface, as shown below. Make sure "whitebalance" and "autogain" settings are turned off. If your camera is mounted upside down, guvcview can set the camera driver to flip the image feed vertically. Close guvcview once you have changed the camera settings, and avoid conflicts with gscam.
> roscd gscam/bin > rosrun gscam gscam
> rosrun image_view image_view image:=/gscam/image_raw
If successful, you should see a new window emerge displaying the image stream from the robot's camera, example below. Stop image_view with the ctrl-c command in the terminal before proceeding
For color blobfinding, ROS uses the CMVision library to perform color segmentation of an image and find relatively solid colored regions (or "blobs"), as illustrated below. The cmvision package in ROS consists of two nodes: colorgui to specify (or "calibrate") colors to recognize and cmvision to find color blobs at run-time. Both of these nodes receive input from the camera by subscribing to an image topic.
The blobfinder provides a bounding box around each image region containing pixels within a specified color range. These color ranges are specified in a color calibration file, or colorfile, such as in the "colors.txt" example below. cmvision colorfiles contains two sections with the following headers:
The following example "colors.txt" illustrates the format of the colorfile for colors "Red", "Green", and "Blue":
[Colors] (255, 0, 0) 0.000000 10 Red ( 0,255, 0) 0.000000 10 Green ( 0, 0,255) 0.000000 10 Blue [Thresholds] ( 25:164, 80:120,150:240) ( 20:220, 50:120, 40:115) ( 15:190,145:255, 40:120)
In this colorfile, the color "Red" has the integer identifier "(255,0,0)" or, in hexidecimal, "0x00FF0000" and YUV thresholds "(25:164,80:120,150:240)". These thresholds are specified as a range in the the Wikipedia YUV color space. Specifically, any pixel with YUV values within this range will be labeled with the given blob color. Note: that YUV and RGB color coordinates are vastly different representations, you can refer to the Wikipedia YUV entry and the Appendix for details.
To calibrate the blobfinder, you will use colorgui to estimate YUV color ranges for objects viewed in the camera's image stream. These color ranges will then be entered into your own colorfile for use by the cmvision node. Start by running colorgui, assuming gscam is publishing images:
> rosrun cmvision colorgui image:=/gscam/image_raw
The result should pop up a window displaying the current camera image stream, just like image_view did.
The colorgui image window can now be used to find the YUV range for a single color of interest.
Using colorgui image window, you can calibrate for the color of specific objects by sampling their pixel colors. Put objects of interest in the robot's view. Mouse click on a pixel in the image window. This action should put the RGB value of the pixel into the left textbox and YUV value in the right textbox. Clicking on another pixel will update the output of the terminal to show the pixel's RGB value and the YUV range encompassing both clicked pixels. Clicking on additional pixels will expand the YUV range to span the color region of interest. Assuming your clicks represent a consistent color, you should see bounding boxes in the colorgui window represented color blobs found with the current YUV range.
As an example, cjenkins calibrated himself as illustrated in the screen capture sequence below. The sequence shows (top row) colorgui when it first starts, after selecting 4 pixels with mouse clicks, and 8 pixels. After 12 mouse clicks (bottom row), he considered the calibration sufficient (although he probably over sampled) and had the blobfinder track him as he moved side to side.
Once you have a sufficient calibration for a color, copy the YUV range shown in the colorgui textbox (or output to the terminal) to a separate text buffer temporarily or directly enter this information into your colorfile. Save this file as /home/obot/ros/colors.txt on the course netbooks You can restart this process to calibrate for another color by selecting "File->Reset" in the colorgui menu bar.
Once you have an appropriately calibrated colorfile, the cmvision blobfinder will be able to detect color blobs. This process can be used to color calibrate a variety of cameras both in real and simulated environments. However, your colorfile will likely work only for cameras and lighting conditions similar to those used at the time of calibration.
> roscd cmvision > roslaunch cmvision.launch
cmvision.launch essentially sets related ROS parameters and launches cmvision to use images from gscam and (for course netbooks) a colorfile in /home/obot/ros/colors.txt. The code for cmvision.launch is listed below:
<launch> <param name="cmvision/color_file" type="string" value="/home/obot/ros/colors.txt" /> <param name="cmvision/debug_on" type="bool" value="true"/> <param name="cmvision/color_cal_on" type="bool" value="false"/> <param name="cmvision/mean_shift_on" type="bool" value="false"/> <param name="cmvision/spatial_radius_pix" type="double" value="2.0"/> <param name="cmvision/color_radius_pix" type="double" value="40.0"/> <node name="cmvision" pkg="cmvision" type="cmvision" args="image:=/gscam/image_raw" output="screen" /> </launch>
> rosrun irobot_create_2_1 driver.py > rosrun teleop_twist_keyboard teleop_twist_keyboard.py
> roscd ar_recog/bin > rosrun ar_recog ar_recog image:=/gscam/image_raw
> rosrun image_view image_view image:=/ar/image
If successful, you should see a window with drawn green boxes overlaid on AR tags in the camera image stream:
> cd $ROS_HOME/ar_recog/src/ARToolKit/bin > ./mk_patt camera parameter: camera_para.dat # show camera tag of interest, tag is highlight, click window to choose, # save pattern as "patt.patternname" (or patt.X) > cp patt.patternname $ROS_HOME/ar_recog/bin > vi $ROS_HOME/ar_recog/bin/object_data # add pattern entry (patternname, patternfilename, width of tag in mm, center of tag usually "0.0 0.0")
mk_patt will likely use the laptop's onboard camera instead of the PS3 cam. It is usually necessary to change the configuration string of mk_patt and remaking mk_patt to use a non-default camera, which is why we do not recommend cs148 students training new tags.
> roscreate-pkg object_seeking rospy std_msgs irobot_create_2_1 cmvision ar_recog
Given appropriate color calibration, recognizing single solid color and AR tag objects should be straightforward. However, fiducials used in robot soccer to indicate specific locations on the field may have multiple solid colors. For example, the camera image in Figure XX has two solid colors stacked in a vertical order with similar shape dimensions. In such cases, your controller will need to specifically include perception routines to process the output of the blobfinder for multicolor fiducials.
The order of visitation should not be hard coded, but rather should be easily changeable (input arguments, input file, a list of object identifiers). Given a specific ordering, your client should drive the robot to visit each of the given objects continuously in this order. For example, the given ordering [3 1 2 4] should direct the robot to visit the green/orange fiducial, orange/green fiducial, yellow ball, pink fiducial, green/orange fiducial, etc. A finite state machine is a good choice for controlling this decision making. A proportional-derivative feedback controller with a form of wandering is a good choice for motion control.
You are expected to conduct least 3 trials for 3 different object sequences with 3 different initial conditions (27 trials total). For each trial, measure total time taken to visit each object and number of collisions with objects, and estimate average distance the robot approaches objects. All of your trials must use the same controller without modification.
Document your controller and experimental results in a written report based on the structure described in the course missive. You are welcome to experiment with additional enclosure escape algorithms and evaluate the relative performance of each. When completed, your report should be committed to the object_seeking/docs/username directory of your repository.
Your grade for this assignment will be determined by equal weighting of your group's implementation (50%) and your individual written report (50%). The weighted breakdown of grading factors for this assignment are as follows:
The color conversion routines used by CMVision for blobfinding are below:
#define YUV2RGB(y, u, v, r, g, b)\ r = y + ((v*1436) >>10);\ g = y - ((u*352 + v*731) >> 10);\ b = y + ((u*1814) >> 10);\ r = r < 0 ? 0 : r;\ g = g < 0 ? 0 : g;\ b = b < 0 ? 0 : b;\ r = r > 255 ? 255 : r;\ g = g > 255 ? 255 : g;\ b = b > 255 ? 255 : b #define RGB2YUV(r, g, b, y, u, v)\ y = (306*r + 601*g + 117*b) >> 10;\ u = ((-172*r - 340*g + 512*b) >> 10) + 128;\ v = ((512*r - 429*g - 83*b) >> 10) + 128;\ y = y < 0 ? 0 : y;\ u = u < 0 ? 0 : u;\ v = v < 0 ? 0 : v;\ y = y > 255 ? 255 : y;\ u = u > 255 ? 255 : u;\ v = v > 255 ? 255 : v