同形异像技术无法正常工作

Question

我正在尝试进行多比例模板匹配以检测模板，然后使用alpha混合并使用单应性将粘贴到检测到的区域中的png。变换图像。我是在实时网络摄像头捕获中执行此操作的，使用单应性后没有得到预期的结果。我将按照我的描述部分地提及我的代码。

1）多尺度模板匹配

import cv2 as cv2
import numpy as np
import imutils


def main():
    template1 = cv2.imread("C:\\Users\\Manthika\\Desktop\\opencvtest\\templates\\template1.jpg")
    template2 = cv2.imread("C:\\Users\\Manthika\\Desktop\\opencvtest\\templates\\temp.jpg")
    templates = [template1, template2]

    for i in range(len(templates)):
        templates[i] = cv2.cvtColor(templates[i], cv2.COLOR_BGR2GRAY)
        templates[i] = cv2.Canny(templates[i], 50, 140)
        templates[i] = cv2.GaussianBlur(templates[i],(5,5),0)
        templates[i] = imutils.resize(templates[i], width=50)

    (tH, tW) = templates[0].shape[:2]
    # print(tH)
    # print(tW)
    # cv2.imshow("Template", template)

    cap = cv2.VideoCapture(0)

    if cap.isOpened():
        ret, frame = cap.read()
    else:
        ret = False

    # loop over the frames to find the template
    while ret:
        # load the image, convert it to grayscale, and initialize the
        # bookkeeping variable to keep track of the matched region
        ret, frame = cap.read()
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        found = None

        # loop over the scales of the image
        for scale in np.linspace(0.2, 1.0, 20)[::-1]:
            # resize the image according to the scale, and keep track
            # of the ratio of the resizing
            resized = imutils.resize(gray, width=int(gray.shape[1] * scale))
            r = gray.shape[1] / float(resized.shape[1])

            # if the resized image is smaller than the template, then break
            # from the loop
            if resized.shape[0] < tH or resized.shape[1] < tW:
                print("frame is smaller than the template")
                break

            # detect edges in the resized, grayscale image and apply template
            # matching to find the template in the image
            edged = cv2.Canny(resized, 50, 160)
            blurred = cv2.GaussianBlur(edged,(5,5),0)

            curr_max = 0
            index = 0
            result = None

            # find the best match
            for i in range(len(templates)):
                # perform matchtemplate
                res = cv2.matchTemplate(blurred, templates[i], cv2.TM_CCOEFF)
                # get the highest correlation value of the result
                maxVal = res.max()
                # if the correlation is highest thus far, store the value and index of template
                if maxVal > curr_max:
                    curr_max = maxVal
                    index = i
                    result = res

            print(index)
            # result = cv2.matchTemplate(edged, templates[index], cv2.TM_CCOEFF)
            (_, maxVal, _, maxLoc) = cv2.minMaxLoc(result)

            # if we have found a new maximum correlation value, then update
            # the bookkeeping variable
            if found is None or maxVal > found[0]:
                found = (maxVal, maxLoc, r)

            # unpack the bookkeeping variable and compute the (x, y) coordinates
            # of the bounding box based on the resized ratio
        # print(found)
        (_, maxLoc, r) = found
        (startX, startY) = (int(maxLoc[0] * r), int(maxLoc[1] * r))
        (endX, endY) = (int((maxLoc[0] + tW) * r), int((maxLoc[1] + tH) * r))

这正常工作，并且符合我的预期，并且没有错误。我可以获取(startX, startY)和(endX, endY)的值来在检测到的区域周围绘制一个边界框。

2）使用alpha混合将png粘贴到检测到的区域上

        cropped = frame[startY:endY, startX:endX]
        cv2.imshow("cropped", cropped)

        # Read the foreground image with alpha channel
        foreGroundImage = cv2.imread("C:\\Users\\Manthika\\Desktop\\opencvtest\\tattoo2.png", -1)
        # Read background image
        background = cropped
        dim = (background.shape[1], background.shape[0])
        foreGroundImage = cv2.resize(foreGroundImage, dim)

        # Split png foreground image
        b, g, r, a = cv2.split(foreGroundImage)

        # Save the foregroung RGB content into a single object
        foreground = cv2.merge((b, g, r))

        # Save the alpha information into a single Mat
        alpha = cv2.merge((a, a, a))

        # background = cv2.resize(background, dim, interpolation = cv2.INTER_AREA)

        # Convert uint8 to float
        foreground = foreground.astype(float)
        background = background.astype(float)
        alpha = alpha.astype(float) / 255

        # Perform alpha blending
        foreground = cv2.multiply(alpha, foreground)
        beta = 1.0 - alpha
        background = cv2.multiply(beta, background)
        outImage = cv2.add(foreground, background)
        outImage = outImage/255
        cv2.imshow("outImage", outImage)
        print(outImage.shape)

在这里，我裁剪了检测到并粘贴png的框架部分。 outImage是该过程的输出。而且我也得到了我所期望的。

3）使用单应性变换图像

        # Read source image.
        im_src = outImage.copy()
        size = im_src.shape

        # Create a vector of source points.
        pts_src = np.array(
            [
                [0, 0],
                [size[1] - 1, 0],
                [size[1] - 1, size[0] - 1],
                [0, size[0] - 1]
            ], dtype=float
        )

        # Read destination image
        im_dst = frame.copy()
        cv2.imshow("im_dst", im_dst)

        # Create a vector of destination points.
        pts_dst = np.array(
            [
                [startX, startY],
                [endX, startY],
                [endX, endY],
                [startX, endY]
            ]
        )

        # Calculate Homography between source and destination points
        h, status = cv2.findHomography(pts_src, pts_dst)

        # Warp source image
        im_temp = cv2.warpPerspective(im_src, h, (im_dst.shape[1], im_dst.shape[0]))

        # Black out polygonal area in destination image.
        cv2.fillConvexPoly(im_dst, pts_dst.astype(int), 0, 16)

        # Add warped source image to destination image.
        im_dst = im_dst + im_temp


        cv2.imshow("Final", im_dst)
        cv2.imshow("frame2222", frame)


        if cv2.waitKey(1) == 27:
            break

    cv2.destroyAllWindows()
    cap.release()


if __name__ == "__main__":
    main()

在这里，我想将Alpha混合的outImage粘贴到框架的给定点上。当我从im_src = outImage.copy()替换im_src = cv2.imread("someimage.png")并运行时，它工作正常。 我可以读取图像并将其粘贴到框架上，但是我不能拿outImage来做 。如果您能帮助我，那将是很棒的。请告诉我您是否需要我使用的图像或输出。

编辑：

使用im_src = cv2.imread("someimage.png")输出 someimage.png显示在模板上

使用im_src = outImage.copy()输出 框架的其他部分为白色

Answer 1

问题

在最坏的情况下，即使用im_src = outImage.copy()，您拥有dtype= float64，并且如果您看一下此行：

outImage = outImage/255

然后，您会发现您的值从0-1开始。然后，您将拥有：

im_dst = frame.copy()

和

im_temp = cv2.warpPerspective(im_src, h, (im_dst.shape[1], im_dst.shape[0]))
...
im_dst = im_dst + im_temp

这意味着im_dst的类型为CV_8UC3（或numpy numpy.uint8），因为它是从相机框架中复制的。此值介于0-255之间。然后，添加两种类型不同且值范围不同的图像，这些图像最后给出类型为浮动图像，但是背景的值介于0-255之间，如果值为>，则imshow中的背景值将显示为白色浮点型图片= 1。

在好的情况下，类型相同，并且不会发生此问题。

---

解决方案：

一个要做：

im_src = np.uint8(outImage.copy() * 255)

但是如果您不需要outImage作为float进行其他操作，只需替换：

outImage = cv2.add(foreground, background)
outImage = outImage/255

针对：

outImage = cv2.add(foreground, background, dtype=np.uint8)

---

建议：

我看到可以更快（更少的操作）完成的几件事，这只是我建议您做的几处更改：

1）这个：

    # Split png foreground image
    b, g, r, a = cv2.split(foreGroundImage)

    # Save the foregroung RGB content into a single object
    foreground = cv2.merge((b, g, r))

与以下相同：

    # Split png foreground image
    a = foreGroundImage[:,:,3]

    # Save the foregroung RGB content into a single object
    foreground = foreGroundImage[:,:,0:3]

2）最后一部分完全可以通过调整大小和复制来完成。除非您打算对图像进行旋转或其他操作，否则单应性是过大的。

类似：

im_dst[startY:endY, startX,endX] = cv2.resize(im_src, (endX-startX, endY-startY))