Get depth from ARIA glasses

top left: rgb video for reference \ top right: rectified slam left video input; right video input is omitted \ bottom left: normalized depth for visualization with a mask where approximates the overlapping regions of left and right camera FOV \ bottom right: normalized predicted disparity from RAFT-stereo

 

1. Preprocess SLAM left-right camera images

https://github.com/hongsukchoi/generic_tools/blob/master/aria_stereo.py

Undistort and rotate images

raw slam right camera image -> undistorted and rotated (clockwise 90)

def get_undistorted_rotated_image(stream_id, frame_idx, src_calib, dst_calib):
    raw_image = provider.get_image_data_by_index(stream_id, frame_idx)[0].to_numpy_array()
    undistorted_image = calibration.distort_by_calibration(raw_image, dst_calib, src_calib)

    # Rotated image by CW90 degrees
    rotated_image = np.rot90(undistorted_image, k=3)

    # Get rotated image calibration
    pinhole_cw90 = calibration.rotate_camera_calib_cw90deg(dst_calib)

    return rotated_image, pinhole_cw90

To understand why and how I do this:

https://facebookresearch.github.io/projectaria_tools/docs/tech_spec/hardware_spec

Hardware Specifications | Project Aria Docs

https://facebookresearch.github.io/projectaria_tools/docs/data_utilities/advanced_code_snippets/image_utilities#image-undistortion

Image Utilities (Python and C++) | Project Aria Docs

 

Rectify the SLAM right-left camera images using the new calibration data

I just saved and hard coded the camera info (printed pinhole_cw90 in cli) like this:

Note that the rotations are the matrix of camera axes in world frame; In other words, for example, R_left is the left_cam2world rotation or R_{world left_cam} that transforms a 3D point in the left camera frame ot the world frame when multiplied from the left.

 

Rectify with OpenCV functions

rectified left and right slam camera images

Becareful about how OpenCV function assumes about the inputs:

https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html#ga617b1685d4059c6040827800e72ad2b6

https://docs.opencv.org/3.4/d9/d0c/group__calib3d.html#ga91018d80e2a93ade37539f01e6f07de5

OpenCV: Camera Calibration and 3D Reconstruction

Look how StereoCalibrate assumes about its input R and T (output from stereoRectify)

(Optional, not correct) Get overlapping regions

    # get mask from the original image  / black area is the invalid area
    mask_left = (rectified_left != 0).astype(np.uint8)
    mask_right = (rectified_right != 0).astype(np.uint8)

    # Find overlapping region by logical AND
    overlap_mask = mask_left & mask_right

    new_mask = overlap_mask
    
    # convert the new_mask to 0 and 1, and save it as numpy
    new_mask =  new_mask.astype(np.uint8)
    np.save(os.path.join(save_dir, f'valid_mask.npy'), new_mask)
    # visualize the new_mask
    cv2.imwrite(os.path.join(save_dir, f'valid_mask.png'), new_mask*255)

 

2. Estimate disparity using RAFT-Stereo

Install raft-stereo from my version, modified from Justin's: pip install git+https://github.com/hongsukchoi/RAFT-Stereo

Before running, you need to save Q from the above setup to get depth, btw

Run: https://github.com/hongsukchoi/generic_tools/blob/master/raft_stereo.py

Vis: https://github.com/hongsukchoi/generic_tools/blob/master/vis_depth.py

It's tricky to normalize a video for good visualization. 

---

Now working on rgb-slam left camera, but somehow it's not working well :( - Oct. 3, 2024 쯤에 하다 말음