VehicleMapping HD AV Spatial Data

Calibrating LiDAR-to-Camera Extrinsics with Python

Projecting LiDAR points onto a camera image — for colouring a cloud, or validating detections — requires the rigid transform between the two sensors, and a degree of error there smears points metres off at range. This task recovers that transform with a planar target: detect the board as a 3D plane in the LiDAR and as 2D corners in the camera across several poses, solve the transform that aligns them, and gate on reprojection error. It is the concrete LiDAR-camera case of the extrinsic calibration workflows, and its output is applied at runtime via tf2 frame transforms.

Board planes detected in LiDAR and corners in camera across poses constrain the LiDAR-to-camera transform:

LiDAR-to-Camera Extrinsic Calibration Multiple board-pose captures feed two detectors: LiDAR plane fit and camera corner detection. Both feed a transform solver. The solver output goes to a reprojection gate checking under half a pixel. Board poses ≥3 orientations LiDAR plane fit 3D normal + offset Camera corners 2D pixels + pose Transform solve align planes ↔ poses Reproj gate ≤ 0.5 px

Prerequisites #

  • Python 3.10+, OpenCV 4.8+ (findChessboardCorners, solvePnP), NumPy 1.24+, SciPy 1.11+ (optimize), open3d for plane segmentation.
  • Input: synchronized LiDAR clouds and camera images of a planar checkerboard at several orientations, plus the camera intrinsics.
  • Upstream stage: camera intrinsics already calibrated; timestamps aligned as in aligning LiDAR and camera timestamps in ROS.
  • Output: the 4×4 LiDAR→camera transform with a per-pose reprojection error report.

Step-by-Step #

1. Detect the board pose in the camera #

Find the checkerboard corners and solve the board's pose in the camera frame with PnP.

python
import cv2
import numpy as np

def camera_board_pose(image, pattern=(9, 6), square=0.05, K=None, dist=None):
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    ok, corners = cv2.findChessboardCorners(gray, pattern)
    if not ok:
        return None
    obj = np.zeros((pattern[0] * pattern[1], 3), np.float32)
    obj[:, :2] = np.mgrid[0:pattern[0], 0:pattern[1]].T.reshape(-1, 2) * square
    ok, rvec, tvec = cv2.solvePnP(obj, corners, K, dist)
    return rvec, tvec

Key parameters: square is the checker size in metres, setting metric scale; solvePnP returns the board's rotation and translation in the camera frame.

2. Fit the board plane in the LiDAR #

Segment the board points and fit a plane, giving the board's normal and offset in the LiDAR frame.

python
import open3d as o3d

def lidar_board_plane(cloud: o3d.geometry.PointCloud):
    model, inliers = cloud.segment_plane(distance_threshold=0.02,
                                          ransac_n=3, num_iterations=1000)
    a, b, c, d = model                       # plane: ax+by+cz+d = 0
    normal = np.array([a, b, c]); normal /= np.linalg.norm(normal)
    return normal, d, inliers

Key parameter: distance_threshold (0.02 m) is the RANSAC inlier band for the board surface; fit only after cropping to a region of interest around the board.

3. Solve the transform across all poses #

Optimize the rigid transform so each LiDAR board plane maps onto the camera-derived board pose.

python
from scipy.optimize import least_squares
from scipy.spatial.transform import Rotation

def solve_extrinsics(lidar_planes, cam_poses, x0):
    def resid(x):
        R = Rotation.from_rotvec(x[:3]).as_matrix(); t = x[3:6]
        r = []
        for (n_l, d_l), (n_c, d_c) in zip(lidar_planes, cam_poses):
            n_pred = R @ n_l
            r += list(n_pred - n_c)          # normal alignment
            r.append((R @ (-d_l * n_l) + t) @ n_c + d_c)   # offset alignment
        return r
    return least_squares(resid, x0, method="lm").x

Key parameter: the residual couples normal-direction and plane-offset error across every pose, so the solve needs the multiple orientations from step 1 to be well-constrained.

Verification & Acceptance Criteria #

Validate by projecting LiDAR board points into the camera image.

python
def reprojection_error(T, lidar_pts, board_pixels, K):
    cam_pts = (T[:3, :3] @ lidar_pts.T).T + T[:3, 3]
    uv = (K @ cam_pts.T).T
    uv = uv[:, :2] / uv[:, 2:3]
    return float(np.linalg.norm(uv - board_pixels, axis=1).mean())

def assert_calibrated(err_px):
    assert err_px <= 0.5, f"reprojection {err_px:.2f} px > 0.5"

Acceptance gate: mean reprojection error ≤0.5 px across all poses; the transform consistent with the calibration graph if other sensors are present; and the solve stable across board subsets. Errors above a pixel point to too few poses or a plane-fit problem.

Common Errors & Fixes #

The solve is ambiguous or unstable. Only one or two board orientations were used, under-constraining the normal direction. Capture at least three well-separated orientations.

Reprojection is good on training poses but bad on held-out ones. The fit overfit to a narrow set of orientations. Add poses spanning the full field of view and depth range, then re-solve.

LiDAR plane fit grabs the wall behind the board. The region of interest was too large. Crop the cloud tightly around the board before segment_plane, or raise the inlier count required.

Everything is offset by a constant. A time-sync error under motion looks like a spatial bias. Calibrate from static captures, and confirm timestamps align first.

FAQ #

Why do I need several board poses? #

A single planar board only constrains the transform in the directions its plane spans; the component along the plane normal is under-determined, so one pose leaves the solution ambiguous. Capturing the board at several orientations spanning different normals fully constrains all six degrees of freedom. As a rule, use at least three well-separated orientations, more for a robust fit, so the optimization has enough independent geometric constraints.

How is the board detected differently in LiDAR and camera? #

In the camera it is detected as a 2D pattern — checkerboard corners found by an image detector, giving pixel coordinates with known 3D positions on the board. In the LiDAR it is detected as a 3D plane — the points on the board are segmented and a plane is fitted, giving the board's normal and offset in the LiDAR frame. The calibration ties these together: the transform must map the LiDAR plane onto the camera-derived board pose for every capture.

What reprojection error should I accept? #

Aim for a mean reprojection error at or below half a pixel, evaluated by projecting LiDAR board points into the camera and comparing to the detected board region across all poses. Half a pixel corresponds to a few centimetres at typical perception range, which keeps fused LiDAR-camera products within their accuracy budget. Errors well above a pixel usually indicate too few board poses, a bad plane fit, or an unresolved time-sync offset.

Up one level: Extrinsic Calibration Workflows for LiDAR, Camera & IMU — the parent workflow this LiDAR-camera solve is one method within.