lid.project_onto_image return zero arrays for uv and colors.

[behroozbc]

I tested this code in two part of the datasets boreas-2021-01-26-10-59 and boreas-2021-10-05-15-35, which return a zero arrays for uv and colors.

pyboreas/pyboreas/tutorials/lidar_camera_projection.ipynb

Line 114 in eeb2b1b

"uv, colors, _ = lid.project_onto_image(seq.calib.P0)\n",

my code

for idx in range(len(seq.lidar_frames)):
            try:
                cam = seq.get_camera(idx)
                lid = seq.get_lidar(idx)
                lid.remove_motion(lid.body_rate)
                T_enu_camera = cam.pose
                T_enu_lidar = lid.pose
                T_camera_lidar = np.matmul(get_inverse_tf(T_enu_camera), T_enu_lidar)
                lid.transform(T_camera_lidar)
                lid.passthrough([-75, 75, -20, 10, 0, 40])  # xmin, xmax, ymin, ymax, zmin, zmax
                # Project lidar points onto the camera image, using the projection matrix, P0.
                uv, colors, _ = lid.project_onto_image(seq.calib.P0)
                fig = plt.figure(figsize=(24.48, 20.48), dpi=100)
                ax = fig.add_subplot()
                ax.imshow(np.zeros(cam.img.shape, dtype=np.uint8)) # you can change it to this `np.zeros(cam.img.shape, dtype=np.uint8)` for black background
                ax.set_xlim(0, 2448)
                ax.set_ylim(2048, 0)
                ax.scatter(uv[:, 0], uv[:, 1], c=colors, marker=',', s=3, edgecolors='none', alpha=0.7, cmap='jet')
                ax.set_axis_off()

I downloaded the full dataset for this two parts.
If any more information is require, please tell me.

[behroozbc]

I found the same problem in boreas-2021-11-28-09-18 part too

[keenan-burnett]

Hey, what you're trying to do will not work for those sequences because those sequences are in the test set, where no ground truth poses are provided. So, when you do cam.pose or lid.pose, they're just returning an identity matrix.

You can try this again with a sequence that is a part of the training set. You can find out which sequences are a part of the training / test splits here:

pyboreas/pyboreas/data/splits.py

Lines 42 to 56 in eeb2b1b

	odom_test = [
	["boreas-2020-12-04-14-00"],
	["boreas-2021-01-26-10-59"],
	["boreas-2021-02-09-12-55"],
	["boreas-2021-03-09-14-23"],
	["boreas-2021-04-22-15-00"],
	["boreas-2021-06-29-18-53"],
	["boreas-2021-06-29-20-43"],
	["boreas-2021-09-08-21-00"],
	["boreas-2021-09-09-15-28"],
	["boreas-2021-10-05-15-35"],
	["boreas-2021-10-26-12-35"],
	["boreas-2021-11-06-18-55"],
	["boreas-2021-11-28-09-18"],
	]