Calibrate a Camera for Motion Planning

A camera captures 2D images, but your robot operates in 3D space. To convert pixel coordinates into real-world positions (for example, to tell the arm where an object is) you need to know the camera’s intrinsic parameters. These parameters describe how the camera projects 3D space onto its 2D sensor: the focal length, the principal point (optical center), and the lens distortion characteristics.

Without accurate intrinsics, every 2D-to-3D conversion will be wrong. Detected objects will appear shifted, depth estimates will be inaccurate, and the arm will miss its targets.

Concepts

Camera intrinsic parameters

Distortion parameters

Radial distortion causes barrel or pincushion effects. Tangential distortion occurs when the lens is not perfectly parallel to the sensor.

Eye-in-hand vs eye-to-hand

• Eye-in-hand: camera mounted on the robot arm. Frame parent is the arm. Moves with the arm.
• Eye-to-hand: camera on a fixed mount. Frame parent is the world frame. Stays stationary.

The calibration process is the same for both. Only the frame configuration differs.

Steps

1. Print a calibration target

Print a standard chessboard calibration pattern (at least 8x6 inner corners). Mount it on a flat, rigid surface. Measure the square size with a ruler.

2. Capture calibration images

Take 10-15 images of the chessboard from various positions and angles using the CONTROL tab in the Viam app.

Guidelines:

• Cover the entire field of view (center, corners, edges)
• Vary the distance from closest to farthest working range
• Tilt the chessboard 15-30 degrees in different directions
• The full chessboard must be visible in every image
• Avoid shadows, glare, and motion blur

3. Run the calibration script

Download cameraCalib.py from the camera-calibration repository, then run it:

pip3 install numpy opencv-python
python3 cameraCalib.py YOUR_PICTURES_DIRECTORY

A successful calibration produces output like:

{
  "intrinsic_parameters": {
    "fx": 939.27,
    "fy": 940.29,
    "ppx": 320.61,
    "ppy": 239.14,
    "width_px": 640,
    "height_px": 480
  },
  "distortion_parameters": {
    "rk1": 0.0465,
    "rk2": 0.8003,
    "rk3": -5.408,
    "tp1": -0.000009,
    "tp2": -0.002829
  }
}

The reprojection error should be less than 1.0 pixel. Above 2.0 indicates poor calibration. Retake images.

4. Add parameters to camera config

{
  "name": "my-camera",
  "api": "rdk:component:camera",
  "model": "webcam",
  "attributes": {
    "video_path": "video0",
    "width_px": 640,
    "height_px": 480,
    "intrinsic_parameters": {
      "fx": 939.27,
      "fy": 940.29,
      "ppx": 320.61,
      "ppy": 239.14,
      "width_px": 640,
      "height_px": 480
    },
    "distortion_parameters": {
      "rk1": 0.0465,
      "rk2": 0.8003,
      "rk3": -5.408,
      "tp1": -0.000009,
      "tp2": -0.002829
    }
  }
}

5. Configure the camera frame

Eye-in-hand (camera mounted on the arm):

{
  "parent": "my-arm",
  "translation": { "x": 50, "y": 0, "z": 80 },
  "orientation": {
    "type": "ov_degrees",
    "value": { "x": 0, "y": 1, "z": 0, "th": -30 }
  }
}

Eye-to-hand (camera on a fixed mount):

{
  "parent": "world",
  "translation": { "x": 500, "y": 300, "z": 800 },
  "orientation": {
    "type": "ov_degrees",
    "value": { "x": 0, "y": 0, "z": 1, "th": 180 }
  }
}

6. Verify calibration accuracy

Place an object at a measured position. Use TransformPose to convert the detected position from camera frame to world frame and compare.

from viam.proto.common import PoseInFrame, Pose

detected_in_camera = PoseInFrame(
    reference_frame="my-camera",
    pose=Pose(x=50, y=30, z=400)
)

detected_in_world = await machine.transform_pose(detected_in_camera, "world")
print("Detected position in world frame:")
print(f"  x={detected_in_world.pose.x:.1f} mm")
print(f"  y={detected_in_world.pose.y:.1f} mm")
print(f"  z={detected_in_world.pose.z:.1f} mm")

detectedInCamera := referenceframe.NewPoseInFrame("my-camera",
    spatialmath.NewPoseFromPoint(r3.Vector{X: 50, Y: 30, Z: 400}))

detectedInWorld, err := machine.TransformPose(ctx, detectedInCamera, "world", nil)
if err != nil {
    logger.Fatal(err)
}

pt := detectedInWorld.Pose().Point()
fmt.Printf("Detected position in world frame:\n")
fmt.Printf("  x=%.1f mm\n", pt.X)
fmt.Printf("  y=%.1f mm\n", pt.Y)
fmt.Printf("  z=%.1f mm\n", pt.Z)

If the computed position is within 10-20 mm of the measured position at a working distance of 500-1000 mm, your calibration is good.

Troubleshooting

What’s Next

• Define Your Frame System: configure component frames for spatial reasoning.
• Define Obstacles: add collision geometry using calibrated camera data.
• Move an Arm to a Target Pose: use calibrated positions to plan arm movements.