Object Following Using RoboRealm

 

A simple object following robot can be built from standard off the shelf parts. Following can be accomplished via vision processing. The advantage to a vision processing follower is that little electronics knowledge is needed to build a complex robot. The following guide will walk you through the construction of a robot that will follow a red object around a room. The goal here was to create a robot "dog" that would follow me around.

 

Hardware

To start off, you will need the following parts. Keep in mind, this is just meant as a guide. Parts can be substituted if necessary:

1. 4 Channel radio control transmitter and receiver.
2. Wireless video camera
3. Video capture card/USB dongle
4. PC
5. PCTx interface
6. RoboRealm software (get it at www.roborealm.com)
7. 9v battery for camera
8. Robot chassis (I used a Tamiya Hummer R/C chassis)
9. 2 extra servos for the pan & tilt base
10. Battery pack for R/C car

Before we get started assembling anything, now is a good time to check the functionality of the radio system and PC link. Start by connecting your servos or ESC's to the receiver and then the battery. Next, power on the transmitter and verify that the servos/motors are working as expected. Connect the PCTx interface to the PC and then turn on the transmitter and start the example software. Flip the trainer switch on and then proceed to move the slider bars on the example software. The servos/motors should spin as if you were controlling the transmitter with your hands. Once verified, disconnect the power from the transmitter and receiver.

Now it is time to assemble the robot platform. To make my project go quicker, I used an R/C car that I already had lying around. One disadvantage to an R/C car is that you need a wider turning radius. An object may be lost if the car cannot turn in time to keep it within the cameras view. A zero turning radius base would be best. It also does not require a pan servo for tracking. Since I used an R/C car, I mounted the camera on a pan and tilt base. The steering servo is synchronized with the pan movement. When the camera pans left, the wheels turn in that direction. This allows the robot to steer towards the object it is tracking. A pan & tilt base is not required but it does help with tracking. The best thing to do is experiment.

At this point the robot hardware should be completed. Turn everything on and take if for an initial run to verify everything is still working correctly. If controlling the robot with the transmitter seems awkward don't worry. You can easily change the servos to operate on different channels. Although, this won't matter much as we will be controlling it via the PC next.

 

RoboRealm

Before I get started with the tutorial I just want to take a quick break to explain RoboRealm. RoboRealm is a piece of software designed to make the difficult task of vision processing easy. RoboRealm works by allowing the user to setup modules that each have a specific function and add them to a pipeline. Each module will perform its function on the output of the previous module in the pipeline in order to achieve the end result. For example we can add a module to filter out specific colors from an image. Another module can then be used to see how big the object is. As the processing continues more and more difficult functions can be added and performed one step at at time.

 

Object Tracking/Following

Now comes the fun part; software. RoboRealm makes this part very easy. Start RoboRealm just to make sure the camera is functioning correctly.

In this example we will be tracking a red ball. The first thing we need to do is add the Colors->RGB_Filter to the pipeline. This will filter out any non-red object within a certain threshold value allowing the software to see only red objects. Depending on your lighting you may have to test what threshold value is right for you. One of the common problems is how the software will respond in different environments. For the sake of the tutorial, we will just assume that the current lighting is what we will always use. To setup the threshold double click for the settings dialog and choose red as the filter color. Adjust the threshold accordingly. A good setting for me was 54.

Now the software should be able to see only red objects. The next steps is to tell the software where in its field of view the object is located. RoboRealm makes locating position very easy with the Center of gravity module. By using this module we are able to tell and approximate location of the red object.

To add the COG add Analysis->Center of Gravity to the pipeline. Again, double click for the module settings. Here choose Show COG as a cross hair. Although this is not really needed, it will give a nice visual representation of the objects position to aide in debugging. If you wish, choose Show COG coordinates as well to display the COG position.

Not only will the COG tell us the position but it can also help estimate distance from the camera. By using the Bounding box option we can estimate distance by looking at the size of the box. To do this choose Show Bounding Box Around "%XX". You will need to experiment with this setting as well. I chose 40%; it gave me about a 2ft distance from the camera.

Like mentioned earlier lighting may also have an effect on this as well. Depending on the lighting the box may be larger/smaller than expected so be sure to experiment.

At this point all the data needed to operate the robot should be ready. Now we get to do something with the processed data. In this example I am using a pan and tilt base which uses 2 channels on the R/C radio. There is a motor attached to an electronic speed control as well as a steering servo for a total of 4 channels.

So what should be done with the data to achieve the objective? Fist of all we need to know if the red object is to the left, right, or center of the field of view. By using the GetVariable parameter we are able to get the current values from the modules we previously setup. If the position is left we want to pan the camera to the left and vice versa. We can then use the SetVariable command to send a new position value to our servos. The same can be said for the rest of the functions.

The following code is written in VB Script. Please use it as a starting point for your own script.

'320x240 resolution camera

' initialize starting servo values
pan = GetVariable("PAN_SERVO")
tilt = GetVariable("TILT_SERVO")
steering = GetVariable("STR_SERVO")
throttle = GetVariable("THR_SERVO")

' get the size (width or height) of the current bounding box
size = GetVariable("COG_BOX_SIZE")

' if it is equal to "" then no object was detected
if size <> "" then

' get the horizontal center of gravity
cogX = GetVariable("COG_X")

' pan left
if cogX < 140 then
pan = pan - 2
end if

' pan right
if cogX > 180 then
pan = pan + 2
end if

' get the vertical center of gravity
cogY = GetVariable("COG_Y")

' tilt down
if cogY < 100 then
tilt = tilt - 2
' tilt up
end if

if cogY > 140 then
tilt = tilt + 2
end if

'steer where it looks
'adjust 300 accordingly, 300 is due to the steering servo being offset
steering = 300 - pan

'get the distance variable to control the movement
cogSize = GetVariable("COG_BOX_SIZE")

' forward
if cogSize < 30 then
throttle = 130
' reverse
elseif cogSize > 45 then
throttle = 160
else
throttle = 150
end if

SetVariable "STR_SERVO", steering
SetVariable "THR_SERVO", throttle
SetVariable "PAN_SERVO", pan
SetVariable "TILT_SERVO", tilt

end if

To use the above code in RoboRealm paste the code into a text file. Name it with a .vbs extension. To add the script to RoboRealm the VBScript module must be added. Go to Add Extensions->VBScript Program to add it to the pipeline. Go to the settings section and click browse and located the file you just saved to import it. At this point you can click Reload and Run to see the current image variables. I have found that the data from this module can help greatly in debugging.

The only thing left is now to send the data to a device. In the code the SetVariable command will sent the data back to RoboRealm but we need to setup a variable for it that can be mapped to the specific channel that it needs to control. For my project I used an Endurance R/C PCTx to control an R/C car. Add this module or the module for the controller you wish and open the module settings. Under the settings, you will see several slider bars that represent the position of the servo. Move the bars back and forth to verify control of the car. You can also set min and max values here to limit servo/esc travel. This will prevent the car from going too fast or over steering and damaging the model.

The final step is to associate the variable names specified in the VBScript to the correct control channel. In the text boxes to the left of the channel enter the name. Once the name is entered the channel will immediately go under software control locking the user out. Be ready to grab the car because until you fully understand how the script you wrote is working the car may go crazy.

In the event that something goes wrong be prepared to hit the stop button. Make sure that the neutral value sent is actually the neutral value required by your gear. Again, test to determine this value.

Your robot now should be able to track and follow a red object! Now that it is functional you may want to go back and adjust the variables and settings better. There are many more options available in RoboRealm to play with.