CS148 Assignment Object Seeking

From Brown University Robotics
(Difference between revisions)
Jump to: navigation, search
(Color Calibration)
(Experiments, Reporting, and Submission)
 
(29 intermediate revisions not shown)
Line 3: Line 3:
[[File:619px-Robot_Arm_Over_Earth_with_Sunburst.jpg|250px|right|Reach out and touch something.]]
[[File:619px-Robot_Arm_Over_Earth_with_Sunburst.jpg|250px|right|Reach out and touch something.]]
-
 
In this assignment, you will develop a ROS package for perceiving objects labeled with a solid color or an [http://en.wikipedia.org/wiki/Augmented_reality augmented reality] [http://www.esquire.com/cm/esquire/images/02-esquire-augmented-reality-covers-110909-lg-49479114.jpg (AR) tag] and controlling a robot to move between these objects.
In this assignment, you will develop a ROS package for perceiving objects labeled with a solid color or an [http://en.wikipedia.org/wiki/Augmented_reality augmented reality] [http://www.esquire.com/cm/esquire/images/02-esquire-augmented-reality-covers-110909-lg-49479114.jpg (AR) tag] and controlling a robot to move between these objects.
Line 11: Line 10:
==Important Dates==
==Important Dates==
-
Assigned: Sept 17, 1:50pm, 2010
+
Assigned: Sept 18, 12:01am, 2010
Due: Sept 26, 11:59pm, 2010
Due: Sept 26, 11:59pm, 2010
Line 39: Line 38:
# View image topics from the camera using gscam and image_view
# View image topics from the camera using gscam and image_view
-
# Color calibration: Generate a color calibration file for cmvision to recognize the solid color objects (balls and multi-color fiducials)
+
# Color calibration: generate a color calibration file for cmvision to recognize the solid color objects (balls and multi-color fiducials)
-
# AR pattern training: train new augmented reality tags that can be recognized by ar_recog
+
# AR pattern recognition: recognize augmented reality tags using ar_recog
# Drive to a single object: write a controller ROS to seek out, find, and drive towards a single object.  
# Drive to a single object: write a controller ROS to seek out, find, and drive towards a single object.  
# Drive to sequence of objects: extend the controller to continually drive between a set of objects in a sequential order
# Drive to sequence of objects: extend the controller to continually drive between a set of objects in a sequential order
Line 52: Line 51:
  > roscore
  > roscore
-
* Configure the PS3 camera video setting to mode 04 by running run_ps3cam.sh:
+
* Configure the PS3 camera video setting to mode 04 by running run_ps3cam.sh (located in your home directory):
  > sudo sh run_ps3cam.sh 04
  > sudo sh run_ps3cam.sh 04
Line 85: Line 84:
-
* Run [http://guvcview.berlios.de/ guvcview] to ensure the camera's "whitebalance" and "autogain" settings are turned off from the graphical interface. 
+
* Run [http://guvcview.berlios.de/ guvcview] to ensure the camera settings are appropriate:
-
[[File:guvcview.png|350px|center|guvcview.]]
+
> guvcview -d /dev/video1
-
If your camera is mounted upside down, guvcview can set the camera driver to flip the image feed vertically.
+
guvcview should produce a graphical interface, as shown below.  Make sure "whitebalance" and "autogain" settings are turned off.  If your camera is mounted upside down, guvcview can set the camera driver to flip the image feed vertically.  Close guvcview once you have changed the camera settings, and avoid conflicts with gscam.
-
> guvcview -d /dev/video1
+
[[File:guvcview.png|350px|center|guvcview.]]
* Run gscam to start camera driver
* Run gscam to start camera driver
Line 108: Line 107:
===Color Calibration===
===Color Calibration===
-
For color blobfinding, ROS uses the [http://www.cs.cmu.edu/~jbruce/cmvision/ CMVision] library to perform color segmentation of an image and find relatively solid colored regions (or "blobs"), as illustrated below.  The [http://www.ros.org/wiki/cmvision cmvision package] in ROS consists of two nodes: <tt>colorgui</tt> to specify (or "calibrate") colors to recognize and <tt>cmvision</tt> to find color blobs at run-time.  Both of these nodes receive input from the camera by subscribing to an image topic.  
+
For color blobfinding, ROS uses the [http://www.cs.cmu.edu/~jbruce/cmvision/ CMVision] library to perform color segmentation of an image and find relatively solid colored regions (or "blobs"), as illustrated below.  The [http://www.ros.org/wiki/cmvision cmvision package] in ROS consists of two nodes: <tt>colorgui</tt> to specify (or "calibrate") colors to recognize and <tt>cmvision</tt> to find color blobs at run-time.  Both of these nodes receive input from the camera by subscribing to an image topic.
[[File:cmvision.png|550px|center|An example of CMVision performing color segmentation on a robot soccer field (from the CMVision website).]]
[[File:cmvision.png|550px|center|An example of CMVision performing color segmentation on a robot soccer field (from the CMVision website).]]
Line 138: Line 137:
To calibrate the blobfinder, you will use colorgui to estimate YUV color ranges for objects viewed in the camera's image stream.  These color ranges will then be entered into your own colorfile for use by the cmvision node.  Start by running colorgui, assuming gscam is publishing images:
To calibrate the blobfinder, you will use colorgui to estimate YUV color ranges for objects viewed in the camera's image stream.  These color ranges will then be entered into your own colorfile for use by the cmvision node.  Start by running colorgui, assuming gscam is publishing images:
-
  > rosrun cmvision color_gui image:=/gscam/image_raw
+
  > rosrun cmvision colorgui image:=/gscam/image_raw
The result should pop up a window displaying the current camera image stream, just like image_view did.
The result should pop up a window displaying the current camera image stream, just like image_view did.
Line 145: Line 144:
Using colorgui image window, you can calibrate for the color of specific objects by sampling their pixel colors.  Put objects of interest in the robot's view.  Mouse click on a pixel in the image window.  This action should put the RGB value of the pixel into the left textbox and YUV value in the right textbox.  Clicking on another pixel will update the output of the terminal  to show the pixel's RGB value and the YUV range encompassing '''both''' clicked pixels.  Clicking on additional pixels will expand the YUV range to span the color region of interest. Assuming your clicks represent a consistent color, you should see bounding boxes in the colorgui window represented color blobs found with the current YUV range.   
Using colorgui image window, you can calibrate for the color of specific objects by sampling their pixel colors.  Put objects of interest in the robot's view.  Mouse click on a pixel in the image window.  This action should put the RGB value of the pixel into the left textbox and YUV value in the right textbox.  Clicking on another pixel will update the output of the terminal  to show the pixel's RGB value and the YUV range encompassing '''both''' clicked pixels.  Clicking on additional pixels will expand the YUV range to span the color region of interest. Assuming your clicks represent a consistent color, you should see bounding boxes in the colorgui window represented color blobs found with the current YUV range.   
 +
 +
[[File:ColorguiOutput.png|700px|center|Color calibration of output and display via colorgui]]
As an example, cjenkins calibrated himself as illustrated in the screen capture sequence below.  The sequence shows (top row) colorgui when it first starts, after selecting 4 pixels with mouse clicks, and 8 pixels.  After 12 mouse clicks (bottom row), he considered the calibration sufficient (although he probably over sampled) and had the blobfinder track him as he moved side to side.
As an example, cjenkins calibrated himself as illustrated in the screen capture sequence below.  The sequence shows (top row) colorgui when it first starts, after selecting 4 pixels with mouse clicks, and 8 pixels.  After 12 mouse clicks (bottom row), he considered the calibration sufficient (although he probably over sampled) and had the blobfinder track him as he moved side to side.
Line 154: Line 155:
<!-- From these clicks, playercam will produce a list of YUV values.  Examine this list to find appropriate minimum and maximum values for each range.  Note: if clicked pixels are chosen carefully, the straightforward min and max in each range can be used.  Otherwise, you may want to use some judgment in eliminating noisy outliers.  -->
<!-- From these clicks, playercam will produce a list of YUV values.  Examine this list to find appropriate minimum and maximum values for each range.  Note: if clicked pixels are chosen carefully, the straightforward min and max in each range can be used.  Otherwise, you may want to use some judgment in eliminating noisy outliers.  -->
-
Once you have a sufficient calibration for a color, copy the YUV range shown in the colorgui textbox (or output to the terminal) to a separate text buffer temporarily or directly enter this information into your colorfile.  You can restart this process to calibrate for another color by selecting "File->Reset" in the colorgui menu bar.
+
Once you have a sufficient calibration for a color, copy the YUV range shown in the colorgui textbox (or output to the terminal) to a separate text buffer temporarily or directly enter this information into your colorfile.  Save this file as /home/obot/ros/colors.txt on the course netbooks You can restart this process to calibrate for another color by selecting "File->Reset" in the colorgui menu bar.
Once you have an appropriately calibrated colorfile, the cmvision blobfinder will be able to detect color blobs.  This process can be used to color calibrate a variety of cameras both in real and simulated environments.  However, your colorfile will likely work only for cameras and lighting conditions similar to those used at the time of calibration.
Once you have an appropriately calibrated colorfile, the cmvision blobfinder will be able to detect color blobs.  This process can be used to color calibrate a variety of cameras both in real and simulated environments.  However, your colorfile will likely work only for cameras and lighting conditions similar to those used at the time of calibration.
 +
* Stop colorgui and use [http://www.ros.org/wiki/roslaunch roslaunch] to start cmvision and see the image stream with recognized blobs:
 +
 +
> roscd cmvision
 +
> roslaunch cmvision.launch
 +
 +
cmvision.launch essentially sets related ROS parameters and launches cmvision to use images from gscam and (for course netbooks) a colorfile in /home/obot/ros/colors.txt. The code for cmvision.launch is listed below:
 +
 +
<launch>
 +
 +
  <!-- Location of the cmvision color file -->
 +
  <param name="cmvision/color_file" type="string"
 +
          value="/home/obot/ros/colors.txt" />
 +
 +
  <!-- Turn debug output on or off -->
 +
  <param name="cmvision/debug_on" type="bool" value="true"/>
 +
 +
  <!-- Turn color calibration on or off -->
 +
  <param name="cmvision/color_cal_on" type="bool" value="false"/>
 +
 +
  <!-- Enable Mean shift filtering -->
 +
  <param name="cmvision/mean_shift_on" type="bool" value="false"/>
 +
 +
  <!-- Spatial bandwidth: Bigger = smoother image -->
 +
  <param name="cmvision/spatial_radius_pix" type="double" value="2.0"/>
 +
 +
  <!-- Color bandwidth: Bigger = smoother image-->
 +
  <param name="cmvision/color_radius_pix" type="double" value="40.0"/>
 +
 +
  <node name="cmvision" pkg="cmvision" type="cmvision" args="image:=/gscam/image_raw"
 +
        output="screen" />
 +
</launch>
 +
 +
 +
<!-- add rosrun instructions
* Stop colorgui and start cmvision:
* Stop colorgui and start cmvision:
Line 165: Line 200:
  > rosrun image_view image_view image:=/blobs
  > rosrun image_view image_view image:=/blobs
-
 
+
-->
<!-- Update for cmvision and teleop_twist_keyboard to assess blob performance
<!-- Update for cmvision and teleop_twist_keyboard to assess blob performance
playercam automatically overlays extracted blobs from the blobfinder using the colorfile in the configuration file. You can use the playerjoy or playerv utilities to then move the robot around the room and perform blobfinding of the objects from different viewpoints.  You may find small changes in camera perspective vastly change the performance of the blobfinder in such cases, you can sample pixel color values from these perspectives and adjust your YUV thresholds.  Also, make sure to properly order the [Colors] and [Thresholds] sections such that the blob color entries are aligned.   
playercam automatically overlays extracted blobs from the blobfinder using the colorfile in the configuration file. You can use the playerjoy or playerv utilities to then move the robot around the room and perform blobfinding of the objects from different viewpoints.  You may find small changes in camera perspective vastly change the performance of the blobfinder in such cases, you can sample pixel color values from these perspectives and adjust your YUV thresholds.  Also, make sure to properly order the [Colors] and [Thresholds] sections such that the blob color entries are aligned.   
-->
-->
-
 
+
* Determine if your robot can recognize blobs while moving by running the Create driver and teleop_keyboard_twist:
-
* Determine if you robot can recognize blobs while moving by running the Create driver and teleop_keyboard_twist:
+
  > rosrun irobot_create_2_1 driver.py
  > rosrun irobot_create_2_1 driver.py
Line 178: Line 212:
:* '''Disclaimer''': The calibration process is not always easy and may take several iterations to get a working calibration.  Remember, the real world can be particular and unforgiving.  Small variations make a huge difference. So, be consistent and thorough.
:* '''Disclaimer''': The calibration process is not always easy and may take several iterations to get a working calibration.  Remember, the real world can be particular and unforgiving.  Small variations make a huge difference. So, be consistent and thorough.
-
=== AR Tag Training ===
+
=== AR Tag Recognition ===
-
*[http://brown-robotics.org/index.php?title=File:Ar_patts.zip zip archive of AR tag pdfs]
+
* Assuming gscam is still running, start ar_recog:
-
*tjay's new AR training
+
> roscd ar_recog/bin
-
http://brown-robotics.org/index.php?title=Making_New_ARTags
+
> rosrun ar_recog ar_recog image:=/gscam/image_raw
-
  %ar training (need to debug, trevor has something better?)
+
* Place a print-out of one of our [http://brown-robotics.org/index.php?title=File:Ar_patts.zip trained AR tags] (alpha-kappa) in front of the camera. Make sure the tag is flat and viewed in its entirety by the camera.
-
  cd $ROS_HOME/ar_recog/src/ARToolKit/bin
+
 
-
  ./mk_patt
+
* Run image_view, using the /ar/image topic, to see tags recognized by ar_recog
-
  * camera parameter: camera_para.dat % param file for ps3 cam, pointers
+
 
-
  for other cameras
+
> rosrun image_view image_view image:=/ar/image
-
* show camera tag of interest, tag is highlight, click window to choose,
+
 
-
  save patter as "patt.patternname" (or patt.X)
+
If successful, you should see a window with drawn green boxes overlaid on AR tags in the camera image stream:
-
  cp patt.patternname $ROS_HOME/ar_recog/bin
+
 
-
  edit $ROS_HOME/ar_recog/bin/object_data
+
[[File:Arrecog_spucci.png|600px|center]]
-
  * add patter entry (patternname, patternfilename, width of tag in mm,
+
 
 +
 
 +
 
 +
* '''Optional:''' If you want to train new AR tag pattern, refer to [http://brown-robotics.org/index.php?title=Making_New_ARTags Trevor Jay's AR training instructions] or '''(not advised)''' use the mk_patt utility in ARToolKit:
 +
 
 +
  > cd $ROS_HOME/ar_recog/src/ARToolKit/bin
 +
  > ./mk_patt
 +
  camera parameter: camera_para.dat  
 +
  # show camera tag of interest, tag is highlight, click window to choose,
 +
  # save pattern as "patt.patternname" (or patt.X)
 +
  > cp patt.patternname $ROS_HOME/ar_recog/bin
 +
  > vi $ROS_HOME/ar_recog/bin/object_data
 +
  # add pattern entry (patternname, patternfilename, width of tag in mm,
  center of tag usually "0.0 0.0")  
  center of tag usually "0.0 0.0")  
 +
mk_patt will likely use the laptop's onboard camera instead of the PS3 cam.  It is usually necessary to change the configuration string of mk_patt and remaking mk_patt to use a non-default camera, which is why we '''do not recommend''' cs148 students training new tags.
<!--%!!!Given the world /course/cs148/asgn/lab3/lab3.world, augment your Lab 2 client to alternately seek the pink-over-yellow and yellow-over-pink fiducials.  Your client should actively avoid the solid pink and solid yellow fiducials.  There is one problem for your current client, it does not know what pink or yellow looks like.  Consequently, none of the fiducials in lab3.world will be properly recognized by the blobfinder device.  -->
<!--%!!!Given the world /course/cs148/asgn/lab3/lab3.world, augment your Lab 2 client to alternately seek the pink-over-yellow and yellow-over-pink fiducials.  Your client should actively avoid the solid pink and solid yellow fiducials.  There is one problem for your current client, it does not know what pink or yellow looks like.  Consequently, none of the fiducials in lab3.world will be properly recognized by the blobfinder device.  -->
-
===Seeking a Individual Objects===
+
===Seeking Individual Objects===
-
* Create "object_seeking" package with dependencies:
+
* Create a ROS package for object seeking.  Object seeking has more dependencies than enclosure escape, for cmvision is required for color blogging, and ar_recog is required for AR tag recognition.
-
rospy
+
-
cmvision
+
-
ar_recog
+
-
irobot_create_2_1
+
 +
> roscreate-pkg object_seeking rospy std_msgs irobot_create_2_1 cmvision ar_recog
-
* Within this package, create "nodes/object_seeking.py" (or similar file in the client library of your choice)
+
* Within this package, create the <tt>nodes</tt> directory and create a file called <tt>object_seeking.py</tt>, or similar file in the client library of your choice).
-
* object_seeking.py should subscribe to XX and publish XX topics
+
* Object seeking should subscribe the following topics:
-
* write code for object_seeking.py to distinguish and move to the following objects with the following integer order identifiers (Note, these do not necessarily need to be identifiers in the colorfile):
+
:* <tt>sensorPacket</tt> topic of message type <tt>SensorPacket<.tt> from <tt>irobot_create_2_1</tt> package
 +
:* <tt>blobs</tt> topic of message type <tt>Blobs</tt> from <tt>cmvision</tt> package
 +
:* <tt>tags</tt> topic of message type <tt>Tags</tt> from <tt>ar_recog</tt> package. 
 +
 
 +
* Object seeking should publish the following topics:
 +
 
 +
:* <tt>cmd_vel</tt> the topic of message type <tt>Twist</tt> from <tt>geometry_msgs</tt> package. 
 +
 
 +
:Note: The appropriate message types must be imported/included in your source code.  To see what topics are available in a given package, or to check the message type of a given topic, it is helpful to look at the <tt><topic>.msg</tt> files of a given package.
 +
 
 +
* Write an event loop in object_seeking.py that can distinguish the following objects (ordered by integer identifier):
# Yellow ball  
# Yellow ball  
 +
# Pink fiducial
# Green over orange fiducial  
# Green over orange fiducial  
# Orange over green fiducial  
# Orange over green fiducial  
-
# Pink fiducial
 
# Alpha AR tag
# Alpha AR tag
-
# ...
 
-
NOTE: Describe the .msg files, and what their controller should be subscribing to
 
-
Given appropriate color calibration, recognizing single solid color and AR tag objects should be straightforward.  However, fiducials used in robot soccer to indicate specific locations on the field may have multiple solid colors.  For example, the camera image in Figure XX has two solid colors stacked in a vertical order with similar shape dimensions.  In such cases, your controller will need to specifically include perception routines to process the output of the blobfinder for multicolor fiducials.
+
and move the robot to any one of these objects (specified as <tt>seek_visit_order</tt> by [http://www.ros.org/wiki/rosparam rosparam]).  For example, the following command will set the "Green over orange" object as the target:
 +
 
 +
> rosparam set seek_visit_order 3
 +
 
 +
From your code, you can access the parameter using the [http://www.ros.org/doc/api/rospy/html/rospy-module.html#get_param get_param] function.
 +
 
 +
:Note: these identifiers do not necessarily need to match identifiers in the colorfile.
 +
 
 +
Given appropriate color calibration, recognizing single solid color and AR tag objects should be straightforward.  However, fiducials used in robot soccer to indicate specific locations on the field may have multiple solid colors.  For example, the camera image below has two solid colors stacked in a vertical order with similar shape dimensions.  In such cases, your controller will need to specifically include perception routines to process the output of the blobfinder for multicolor fiducials.
 +
 
 +
'''Remember:''' to consider the [http://en.wikipedia.org/wiki/PID_controller proportional-derivative servo] when coding the seeking behavior of your robot.
 +
 
 +
[[File:OOGFiducial.png|center]]
===Seeking a Sequence of Objects===
===Seeking a Sequence of Objects===
-
Given a specific ordering (via file or command line; rosparam??), your client should drive the robot to visit each of the given objects continuously in this order.  For example, the given ordering [3 1 2 4] should direct the robot to visit the green/orange fiducial, orange/green fiducial, yellow ball, pink fiducial, green/orange fiducial, etc.  A finite state machine is a good choice for controlling this decision making.  A proportional-derivative feedback controller with a form of wandering is a good choice for motion control.
+
Given a specific ordering, your client should drive the robot to visit each of the given objects continuously in this order.  The order of visitation should not be hard coded, but rather should be easily changeable.  As such, this order is specified as a string using rosparam.  For example, the given ordering "[3, 1, 2, 1, 5]" should direct the robot to visit the green/orange fiducial, yellow ball, pink fiducial, yellow ball, alpha AR TagThis order would be specified by rosparam as:
 +
 
 +
> rosparam set \seekorder "[3, 1, 2, 1, 5]" 
 +
 
 +
A [http://en.wikipedia.org/wiki/Finite-state_machine finite state machine (FSM)] is a good choice for controlling the decision making process for object seekingAn FSM for seeking should have transition conditions that determine when the robot seek the next object in the input sequence.
=== Experiments, Reporting, and Submission ===
=== Experiments, Reporting, and Submission ===
Line 234: Line 302:
You are expected to conduct least 3 trials for 3 different object sequences with 3 different initial conditions (27 trials total).  For each trial, measure total time taken to visit each object and number of collisions with objects, and estimate average distance the robot approaches objects.  All of your trials must use the same controller without modification.
You are expected to conduct least 3 trials for 3 different object sequences with 3 different initial conditions (27 trials total).  For each trial, measure total time taken to visit each object and number of collisions with objects, and estimate average distance the robot approaches objects.  All of your trials must use the same controller without modification.
-
Document your controller and experimental results in a written report based on the structure described in the [http://brown-robotics.org/index.php?title=CS148_Missive#Project_Report_Format course missive]. You are welcome to experiment with additional enclosure escape algorithms and evaluate the relative performance of each. When completed, your report should be committed to the object_seeking/docs/username directory of your repository.
+
Document your controller and experimental results in a written report based on the structure described in the [http://brown-robotics.org/index.php?title=CS148_Missive#Project_Report_Format course missive]. You are welcome to experiment with additional servoing techinques and evaluate the relative performance of each. When completed, your report should be committed to the object_seeking/docs/username directory of your repository.
==Grading==
==Grading==

Current revision as of 18:07, 25 September 2010

Contents

Introduction

Reach out and touch something.

In this assignment, you will develop a ROS package for perceiving objects labeled with a solid color or an augmented reality (AR) tag and controlling a robot to move between these objects.

A video of Lisa Miller's implementation of this assignment from Spring Semester 2009.

Important Dates

Assigned: Sept 18, 12:01am, 2010

Due: Sept 26, 11:59pm, 2010

Description

Building on your Enclosure Escape assignment, you will build a controller in ROS to perform "object seeking". In this seeking task, your robot will perceive and drive to objects that are visually recognizable by a solid color appearance or labeled with an AR tag from the robot's visual sensing (i.e., camera). For this assignment, you will be working primarily with the Create platform and a Sony PlayStation Eye USB video camera. For object recognition in ROS, you will use the cmvision package for color blobfinding and the ar_recog package for AR tag recognition.

Assuming perception of objects salient by color or pattern, you will develop an object seeking package for this assignment that enables a robot to continually drive between these (non-occluded) objects in a sequence given at run-time. Your controller's decision making should take the form a finite state machine (FSM). This FSM should use one state variable to specify the currently sought object. For motion control, you should use proportional-derivative (PD) servoing to center objects in the robot's field of view. As a whole, your controller should put the current object in the center of view, drive as close as possible to an object without hitting it, increment the state variable, and continue the process for the next object.

ROS support packages

gscam (brown-ros-pkg)
Based on the GStreamer multimedia framework, gscam package provides direct access to the robot's camera image and related parameters. support for a variety of cameras, although not as intuitive as other ROS camera interfaces. While laser rangefinders provide more accurate information, cameras are often more practical in terms of their cost, weight, size, and sampling frequency. Cameras have the additional benefit of sensing color, which you will leverage in this assignment.
image_view (ros.org)
A simple viewer for subscribing to and displaying ROS image topics. Includes a specialized viewer for stereo and disparity images.
cmvision (ros.org)
Based on the CMU CMVision library, the cmvision package performs segmentation of solid colored regions (or "blobs") in an image, reported as bounding boxes. cmvision proxy thresholds and groups pixels in an images based on given YUV color ranges to estimate blobs. To calibrate color ranges, the colorgui node is include within cmvision to build color ranges from selected pixels in published image topics.
ar_recog (brown-ros-pkg)
Based on the ARToolkit augmented reality library, ar_recog recognizes augmented reality tags in an image. ar_recog publishes various information about recognized tags, such as its corners in image space and relative 6DOF pose in camera space.

Assignment Instructions

Your tasks for this assignment are as follows.

  1. View image topics from the camera using gscam and image_view
  2. Color calibration: generate a color calibration file for cmvision to recognize the solid color objects (balls and multi-color fiducials)
  3. AR pattern recognition: recognize augmented reality tags using ar_recog
  4. Drive to a single object: write a controller ROS to seek out, find, and drive towards a single object.
  5. Drive to sequence of objects: extend the controller to continually drive between a set of objects in a sequential order
  6. Experiments, Reporting, and Submission

The following sections provide information to guide you through completing these tasks.

Running gscam and image_view

  • Run roscore
> roscore
  • Configure the PS3 camera video setting to mode 04 by running run_ps3cam.sh (located in your home directory):
> sudo sh run_ps3cam.sh 04

Note: this command only needs to be run after you plug in the camera, not every time you want to run something that uses the camera.

In this class, we will use mode 04 for the PS3 camera to ensure image resolution and frame rate sufficient for object recognition. More information about installing the PS3 cam driver and other camera modes is available from Trevor Jay's rough PS3/Ubuntu guide.

The course staff has already installed PS3 cam support on our robots. run_ps3cam.sh is a simple bash file that communicates video settings to the camera:

#!/bin/bash
arg=$1
if [ -z "$arg" ] ; then
    echo "usage: sh run_ps3cam.sh arg where argument is the videomode (00-04 and 10-16) check comments in the file for details"
    exit
fi
modprobe -r gspca-ov534
modprobe gspca-ov534 videomode=$arg
echo 'ps3 camera driver started'
#00:640x480@15
#01:640x480@30
#02:640x480@40
#03:640x480@50
#04:640x480@60
#10:320x240@30
#11:320x240@40
#12:320x240@50
#13:320x240@60
#14:320x240@75
#15:320x240@100
#16:320x240@125


  • Run guvcview to ensure the camera settings are appropriate:
> guvcview -d /dev/video1

guvcview should produce a graphical interface, as shown below. Make sure "whitebalance" and "autogain" settings are turned off. If your camera is mounted upside down, guvcview can set the camera driver to flip the image feed vertically. Close guvcview once you have changed the camera settings, and avoid conflicts with gscam.

guvcview.
  • Run gscam to start camera driver
> roscd gscam/bin
> rosrun gscam gscam
  • To view camera's image stream, start the image_view node with this command:
> rosrun image_view image_view image:=/gscam/image_raw

If successful, you should see a new window emerge displaying the image stream from the robot's camera, example below. Stop image_view with the ctrl-c command in the terminal before proceeding

An example of ros image_view.

Color Calibration

For color blobfinding, ROS uses the CMVision library to perform color segmentation of an image and find relatively solid colored regions (or "blobs"), as illustrated below. The cmvision package in ROS consists of two nodes: colorgui to specify (or "calibrate") colors to recognize and cmvision to find color blobs at run-time. Both of these nodes receive input from the camera by subscribing to an image topic.

An example of CMVision performing color segmentation on a robot soccer field (from the CMVision website).

The blobfinder provides a bounding box around each image region containing pixels within a specified color range. These color ranges are specified in a color calibration file, or colorfile, such as in the "colors.txt" example below. cmvision colorfiles contains two sections with the following headers:

  • "[Colors]" section: a list identifiers for each blob color, as strings and RGB triplets, in sequential order
  • "[Thresholds]" section: a list of color range thresholds (in YUV space) sequence to match each blob color in the "[Colors]" section

The following example "colors.txt" illustrates the format of the colorfile for colors "Red", "Green", and "Blue":

[Colors]
(255,  0,  0) 0.000000 10 Red
(  0,255,  0) 0.000000 10 Green
(  0,  0,255) 0.000000 10 Blue

[Thresholds]
( 25:164, 80:120,150:240)
( 20:220, 50:120, 40:115)
( 15:190,145:255, 40:120)


In this colorfile, the color "Red" has the integer identifier "(255,0,0)" or, in hexidecimal, "0x00FF0000" and YUV thresholds "(25:164,80:120,150:240)". These thresholds are specified as a range in the the Wikipedia YUV color space. Specifically, any pixel with YUV values within this range will be labeled with the given blob color. Note: that YUV and RGB color coordinates are vastly different representations, you can refer to the Wikipedia YUV entry and the Appendix for details.

An example of using CMVision with playercam utility to color calibrate and perform blobfinding.  (Top left) 3 objects are placed in the robot's field of view and playercam is used to extract YUV values and thresholds for each object color.  (Other 5 images) Results from blobfinding with the resulting color calibration file from different robot locations and pushing the yellow ball.


To calibrate the blobfinder, you will use colorgui to estimate YUV color ranges for objects viewed in the camera's image stream. These color ranges will then be entered into your own colorfile for use by the cmvision node. Start by running colorgui, assuming gscam is publishing images:

> rosrun cmvision colorgui image:=/gscam/image_raw

The result should pop up a window displaying the current camera image stream, just like image_view did.

The colorgui image window can now be used to find the YUV range for a single color of interest.

Using colorgui image window, you can calibrate for the color of specific objects by sampling their pixel colors. Put objects of interest in the robot's view. Mouse click on a pixel in the image window. This action should put the RGB value of the pixel into the left textbox and YUV value in the right textbox. Clicking on another pixel will update the output of the terminal to show the pixel's RGB value and the YUV range encompassing both clicked pixels. Clicking on additional pixels will expand the YUV range to span the color region of interest. Assuming your clicks represent a consistent color, you should see bounding boxes in the colorgui window represented color blobs found with the current YUV range.

Color calibration of output and display via colorgui

As an example, cjenkins calibrated himself as illustrated in the screen capture sequence below. The sequence shows (top row) colorgui when it first starts, after selecting 4 pixels with mouse clicks, and 8 pixels. After 12 mouse clicks (bottom row), he considered the calibration sufficient (although he probably over sampled) and had the blobfinder track him as he moved side to side.

Color calibration of cjenkins using colorgui
  • Note: you may not want to click on all pixels of an object due to shadowing and specular ("shiny") artifacts.


Once you have a sufficient calibration for a color, copy the YUV range shown in the colorgui textbox (or output to the terminal) to a separate text buffer temporarily or directly enter this information into your colorfile. Save this file as /home/obot/ros/colors.txt on the course netbooks You can restart this process to calibrate for another color by selecting "File->Reset" in the colorgui menu bar.

Once you have an appropriately calibrated colorfile, the cmvision blobfinder will be able to detect color blobs. This process can be used to color calibrate a variety of cameras both in real and simulated environments. However, your colorfile will likely work only for cameras and lighting conditions similar to those used at the time of calibration.

  • Stop colorgui and use roslaunch to start cmvision and see the image stream with recognized blobs:
> roscd cmvision
> roslaunch cmvision.launch

cmvision.launch essentially sets related ROS parameters and launches cmvision to use images from gscam and (for course netbooks) a colorfile in /home/obot/ros/colors.txt. The code for cmvision.launch is listed below:

<launch>

  <param name="cmvision/color_file" type="string" 
         value="/home/obot/ros/colors.txt" />

  <param name="cmvision/debug_on" type="bool" value="true"/>

  <param name="cmvision/color_cal_on" type="bool" value="false"/>

  <param name="cmvision/mean_shift_on" type="bool" value="false"/>

  <param name="cmvision/spatial_radius_pix" type="double" value="2.0"/>

  <param name="cmvision/color_radius_pix" type="double" value="40.0"/>

  <node name="cmvision" pkg="cmvision" type="cmvision" args="image:=/gscam/image_raw" 
        output="screen" />
</launch>


  • Determine if your robot can recognize blobs while moving by running the Create driver and teleop_keyboard_twist:
> rosrun irobot_create_2_1 driver.py
> rosrun teleop_twist_keyboard teleop_twist_keyboard.py
  • Disclaimer: The calibration process is not always easy and may take several iterations to get a working calibration. Remember, the real world can be particular and unforgiving. Small variations make a huge difference. So, be consistent and thorough.

AR Tag Recognition

  • Assuming gscam is still running, start ar_recog:
> roscd ar_recog/bin
> rosrun ar_recog ar_recog image:=/gscam/image_raw
  • Place a print-out of one of our trained AR tags (alpha-kappa) in front of the camera. Make sure the tag is flat and viewed in its entirety by the camera.
  • Run image_view, using the /ar/image topic, to see tags recognized by ar_recog
> rosrun image_view image_view image:=/ar/image

If successful, you should see a window with drawn green boxes overlaid on AR tags in the camera image stream:


> cd $ROS_HOME/ar_recog/src/ARToolKit/bin
> ./mk_patt
camera parameter: camera_para.dat 
# show camera tag of interest, tag is highlight, click window to choose,
# save pattern as "patt.patternname" (or patt.X)
> cp patt.patternname $ROS_HOME/ar_recog/bin
> vi $ROS_HOME/ar_recog/bin/object_data
# add pattern entry (patternname, patternfilename, width of tag in mm,
center of tag usually "0.0 0.0") 

mk_patt will likely use the laptop's onboard camera instead of the PS3 cam. It is usually necessary to change the configuration string of mk_patt and remaking mk_patt to use a non-default camera, which is why we do not recommend cs148 students training new tags.


Seeking Individual Objects

  • Create a ROS package for object seeking. Object seeking has more dependencies than enclosure escape, for cmvision is required for color blogging, and ar_recog is required for AR tag recognition.
> roscreate-pkg object_seeking rospy std_msgs irobot_create_2_1 cmvision ar_recog
  • Within this package, create the nodes directory and create a file called object_seeking.py, or similar file in the client library of your choice).
  • Object seeking should subscribe the following topics:
  • sensorPacket topic of message type SensorPacket<.tt> from <tt>irobot_create_2_1 package
  • blobs topic of message type Blobs from cmvision package
  • tags topic of message type Tags from ar_recog package.
  • Object seeking should publish the following topics:
  • cmd_vel the topic of message type Twist from geometry_msgs package.
Note: The appropriate message types must be imported/included in your source code. To see what topics are available in a given package, or to check the message type of a given topic, it is helpful to look at the <topic>.msg files of a given package.
  • Write an event loop in object_seeking.py that can distinguish the following objects (ordered by integer identifier):
  1. Yellow ball
  2. Pink fiducial
  3. Green over orange fiducial
  4. Orange over green fiducial
  5. Alpha AR tag


and move the robot to any one of these objects (specified as seek_visit_order by rosparam). For example, the following command will set the "Green over orange" object as the target:

> rosparam set seek_visit_order 3

From your code, you can access the parameter using the get_param function.

Note: these identifiers do not necessarily need to match identifiers in the colorfile.

Given appropriate color calibration, recognizing single solid color and AR tag objects should be straightforward. However, fiducials used in robot soccer to indicate specific locations on the field may have multiple solid colors. For example, the camera image below has two solid colors stacked in a vertical order with similar shape dimensions. In such cases, your controller will need to specifically include perception routines to process the output of the blobfinder for multicolor fiducials.

Remember: to consider the proportional-derivative servo when coding the seeking behavior of your robot.

Seeking a Sequence of Objects

Given a specific ordering, your client should drive the robot to visit each of the given objects continuously in this order. The order of visitation should not be hard coded, but rather should be easily changeable. As such, this order is specified as a string using rosparam. For example, the given ordering "[3, 1, 2, 1, 5]" should direct the robot to visit the green/orange fiducial, yellow ball, pink fiducial, yellow ball, alpha AR Tag. This order would be specified by rosparam as:

> rosparam set \seekorder "[3, 1, 2, 1, 5]"  

A finite state machine (FSM) is a good choice for controlling the decision making process for object seeking. An FSM for seeking should have transition conditions that determine when the robot seek the next object in the input sequence.

Experiments, Reporting, and Submission

You are expected to conduct least 3 trials for 3 different object sequences with 3 different initial conditions (27 trials total). For each trial, measure total time taken to visit each object and number of collisions with objects, and estimate average distance the robot approaches objects. All of your trials must use the same controller without modification.

Document your controller and experimental results in a written report based on the structure described in the course missive. You are welcome to experiment with additional servoing techinques and evaluate the relative performance of each. When completed, your report should be committed to the object_seeking/docs/username directory of your repository.

Grading

Your grade for this assignment will be determined by equal weighting of your group's implementation (50%) and your individual written report (50%). The weighted breakdown of grading factors for this assignment are as follows:

Project Implementation

Color calibration 20%
Is your color calibration file suitable for finding objects in the Roomba Lab?
What are the restrictions (camera pose, lighting, etc.) on your color blobfinding?
Seeking a single object 15%
Does your robot find, center, drive to non-occluded objects?
How close does your robot get to sought objects?
Transitioning between objects 8%
Does your robot properly switch to other objects after visiting an object?
Controller Robustness 7%
Does your controller run without interruption?

Written Report

Introduction and Problem Statement 7%
What is your problem?
Why is it interesting?
Approach and Methods 15%
What is your approach to the problem?
How did you implement your approach and algorithms?
Could someone reproduce your algorithms?
Experiments and Results 20%
How did you validate your methods?
Describe your variables, controls, and specific tests.
Could someone reproduce your results?
Conclusion and Discussion 8%
What conclusions can be reached about your problem and approach?
What are the strengths of your approach?
What are the shortcomings of your approach?

Appendix: RGB-YUV Conversions

The color conversion routines used by CMVision for blobfinding are below:

#define YUV2RGB(y, u, v, r, g, b)\
  r = y + ((v*1436) >>10);\
  g = y - ((u*352 + v*731) >> 10);\
  b = y + ((u*1814) >> 10);\
  r = r < 0 ? 0 : r;\
  g = g < 0 ? 0 : g;\
  b = b < 0 ? 0 : b;\
  r = r > 255 ? 255 : r;\
  g = g > 255 ? 255 : g;\
  b = b > 255 ? 255 : b

#define RGB2YUV(r, g, b, y, u, v)\
  y = (306*r + 601*g + 117*b)  >> 10;\
  u = ((-172*r - 340*g + 512*b) >> 10)  + 128;\
  v = ((512*r - 429*g - 83*b) >> 10) + 128;\
  y = y < 0 ? 0 : y;\
  u = u < 0 ? 0 : u;\
  v = v < 0 ? 0 : v;\
  y = y > 255 ? 255 : y;\
  u = u > 255 ? 255 : u;\
  v = v > 255 ? 255 : v
Personal tools