Question

每个人。我正在尝试对涉及两个摄像机的设置中位于飞机上的一些点进行三角测量（密集重建）。 [参考图像]：https://imgur.com/gOps4vP和[另一个图像]：https://imgur.com/VIiH9Rv

首先，我在基本矩阵估计的未失真点上使用5pts算法解决了相对姿势问题，我恢复了姿势。我正在使用RANSAC。

然后，我以通常的方式纠正立体声对。

R1, R2, Pn1, Pn2, Q, _, _ = cv2.stereoRectify(K1, dcoeffs1, K2, dcoeffs2, 
                                                  img1.shape[::-1], R, t, 
                                                  flags=cv2.CALIB_ZERO_DISPARITY, 
                                                  alpha=-1)

    # Compute the rigid transform that OpenCV apply to world points (USEFUL LATER)
    # in order for the rectified reference camera to be K_new[I|0]
    tn_1 = np.zeros((3,1)) # Cameras are never translated in the rectification


    G1_rect = np.block([[R1, tn_1], [np.zeros((1,3)), 1.0]])
    maps1   =   cv2.initUndistortRectifyMap(K1, dcoeffs1, R1, Pn1, (1920,1080), cv2.CV_32FC1)
    maps2   =   cv2.initUndistortRectifyMap(K2, dcoeffs2, R2, Pn2, (1920,1080), cv2.CV_32FC1)
    img1_remap = cv2.remap(img1, maps1[0], maps1[1], cv2.INTER_LANCZOS4)
    img2_remap = cv2.remap(img2, maps2[0], maps2[1], cv2.INTER_LANCZOS4)

整改结果： [修正的参考图片] https://drive.google.com/open?id=10VfgXrXFO3_lYqtO9qJXr17Dc6F1PuXU [另外一个已纠正] https://drive.google.com/open?id=13ZkeMiF5xEovGmX13LSQVaJ237hoJLX0

现在，我调用一个函数来识别图像中的已知对象（目标）。

#Now call a function that recognize a known object in the images (target)
# Find target
target_corners, _ = dt.detectTarget(img_scene1, img_target, 0.5) # return 4 corners of the detected polygon
target_corners = target_corners[:,0,:]
# Compute mask for the target cutout:
target_mask = mp.maskPolygon(target_corners, img_scene1.shape[::-1]) # Output: mask of same dimension of the image

找到目标（请注意突出显示的角）： [找到目标] https://imgur.com/QjYV8tp

然后，我使用StereoSGBM计算视差图。我只对目标视差的计算感兴趣（我将掩盖所有其他点）。在获得视差图的情况下，并使用stereoRectify提供的4x4投影矩阵Q，我将对视差图进行3d重投影。

    # Compute disparity map
    # https://docs.opencv.org/3.3.1/d2/d85/classcv_1_1StereoSGBM.html
    window_size = 5
    min_disp = 16
    max_disp = 1024
    num_disp = max_disp-min_disp # Deve essere divisibile per 16!

    stereo = cv2.StereoSGBM_create(minDisparity = min_disp,
        numDisparities = num_disp,
        blockSize = window_size,
        P1 = 8*3*window_size**2,
        P2 = 32*3*window_size**2,
        disp12MaxDiff = 1,
        uniquenessRatio = 10,
        speckleWindowSize = 150,
        speckleRange = 2
    )

    print('Calcolo SGBM della disparità...')
    disp = stereo.compute(img_scene1, img_scene2).astype(np.float32) / 16.0

    target_disparity = target_mask*disp

    points = cv2.reprojectImageTo3D(target_disparity, Q)


    # DEBUG:
    cv2.namedWindow('scene1', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('scene1', 800,450)
    cv2.imshow('scene1', img_scene1)
    cv2.namedWindow('disparity', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('disparity', 800,450)
    cv2.imshow('disparity', (disp-min_disp)/num_disp)
    cv2.namedWindow('target_disparity', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('target_disparity', 800,450)
    cv2.imshow('target_disparity', target_mask*(disp-min_disp)/num_disp)
    cv2.waitKey()
    cv2.destroyAllWindows()


    # Obtain matrix of the target 3D points starting from disparity image obtained from reprojectImageTo3D()
    mask_disp = disp > disp.min()
    mask_inf = ~(np.isinf(points[:,:,0]) | np.isinf(points[:,:,1]) | np.isinf(points[:,:,2]))
    mask_nan = ~(np.isnan(points[:,:,0]) | np.isnan(points[:,:,1]) | np.isnan(points[:,:,2]))

    mask = mask_disp & mask_inf & mask_nan

    pts3D = points[mask]

现在，我已经3d重建了与目标相对应的图像区域。我注意到，OpenCv在摄像机校正期间会对世界点进行刚性变换，以使参考原始摄像机和新的（校正后的）参考摄像机具有相同的外部特性（R = eye（3）和t = [0,0,0 ]'）。实际上，在矫正期间，两个摄像机都必须旋转，而且我认为OpenCV只是将新摄像机带回新的参考位置，以便参考矫正后的摄像机具有与原始摄像机相同的外部特性。 但这意味着重建的3d点将以世界参考表示，而不是原始相机的世界参考！

因此，将逆刚性变换应用于pts3D，我们在原始参考摄像机框架中获得了重建。（请参见代码）。

target3Dpts_hom = cv2.convertPointsToHomogeneous(target3Dpts)[:,0,:].T
    target3Dpts_hom = G.T @ target3Dpts_hom
    new_target3Dpts = cv2.convertPointsFromHomogeneous(target3Dpts_hom.T[:,np.newaxis,:])[:,0,:]

请注意，如果我不执行此操作，则通过其投影矩阵在原始摄像机上重新投影的pt3D将不会与目标点相对应！

通过重投影检查重建；现在，我可以重新投影new_target3Dpts了：让我介绍一下我所说的投影函数：

    def proj_dist(P, dcoeffs, M):

    import numpy as np
    import cv2

    K, R, t,_,_,_,_ = cv2.decomposeProjectionMatrix(P)
    rotv, _ = cv2.Rodrigues(R)

    # Projection. Returns a (N,2) shaped array
    m,_ = cv2.projectPoints(M,rotv,t[0:-1],K,dcoeffs)
    m = m.squeeze()

    return m

最后，重新投影：

    #P_kin = K_kin[eye(3),0] # Originals MPPs of two cameras
    #P_rpi = K_rpi[R,t]

    m0 = proj.proj_dist(P_kin,dcoeffs_kin,new_points).astype('int32')

    for (x, y) in m0:

        x = int(x)
        y= int(y)

        cv2.circle(img_kin, (x, y), 2, (255, 255, 0), 4)

    cv2.namedWindow('frame1', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('frame1', 800,450)
    cv2.imshow('frame1',img_kin)
    cv2.waitKey(0)

    m1 = proj.proj_dist(P_rpi,dcoeffs_rpi,new_points).astype('int32')
    img_rpi1 = img_rpi.copy()
    for (x, y) in m1:
        x = int(x)
        y = int(y)

    cv2.circle(img_rpi1, (x, y), 2, (255, 255, 0), 4)

    cv2.namedWindow('frame2', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('frame2', 800,450)
    cv2.imshow('frame2',img_rpi1)
    cv2.waitKey(0)

但是，虽然原始参考摄像机上的重新投影的点是正确的，但对于第二个是不正确的。...这些点只是翻译出来的，但是我无法解释原因。

结果：[第一帧重复] https://imgur.com/S4lo9Wz [第二帧表示。错误] https://imgur.com/y4igaEI

有什么想法吗？我现在将包括所有代码。谢谢。

SM

Answer 1

我解决了这个问题，该问题与reprojectImageto3D无关（可以正常工作），但是通过我编写的这段代码，我曾经将这些点重新投影到原始帧上：

def proj_dist(P, dcoeffs, M):

import numpy as np
import cv2

K, R, t,_,_,_,_ = cv2.decomposeProjectionMatrix(P)
rotv, _ = cv2.Rodrigues(R)

# Projection. Returns a (N,2) shaped array
m,_ = cv2.projectPoints(M,rotv,t[0:-1],K,dcoeffs)
m = m.squeeze()

return m

我已经为点投影编写了自己的函数：

def proj(P, M, hom=0):
# proj(): Esegue la proiezione prospettica dei punti 3D M secondo la MPP P,
# sul piano immagine 2D di una camera pinhole.

import numpy as np

n = M.shape[1]
M = np.concatenate((M, np.ones((1,n))))

# Proiezione
m = P @ M

m = m/m[2,:]

if hom !=1 :
    # Passo a cartesiane
    m = m[0:2,:]

return m

问题解决了！我的功能未考虑镜头失真。我将进一步研究与projectPoints（）OpenCV函数有关的问题。

reprojectImageto3d

1 个答案: