Time to complete: 5 minutes

When piloting your robot using Freedom, you can click on the video feed to send the coordinates of that point in the image to your robot. By specifying an appropriate topic, this point-and-click interface can be used to:

• Pan your robot's camera to point in a new direction
• Send a navigation waypoint to your robot
• Command an arm to a pick location
• Label an item to capture data for a machine learning pipeline (see this tutorial for more information)

Coordinate definition

Clicking on the video feed sends a message of type geometry_msgs/PointStamped to your robot on a topic you specify.

The x- and y- coordinates for the video feed image correspond to normalized image coordinates between 0 and 1. For example, point.x=0, point.y=0 is the top left corner of the image, and point.x=1, point.y=1 is the bottom right corner of the image. The z-coordinate is always 0.

An example message corresponding to a click in the center of the screen would look like this:

{
    "header": {
        "seq": 1424,
        "stamp": {
            "secs": 1252031,
            "nsecs": 378790015
        },
        "frame_id": "/head_camera/rgb/image_raw"
    },
    "point": {
        "x": 0.5, 
        "y": 0.5,
        "z": 0.0
    }
}

Specify a topic

Let's configure the video click topic for your robot. Select your robot, then navigate to SETTINGS → PILOT → NAVIGATION.

In the VIDEO CLICK INTERACTIONS section, add the topic that the coordinates should be sent to (e.g. /video/click).

Click to send coordinates

Select the PILOT tab for your robot and click TAKE OVER. When you click on the video feed, you'll see a blue circle appear, indicating that the click coordinates have been sent to your robot.