视觉框架为捕获的视频输出中的帧中的矩形提供了不正确的坐标

Question

我需要从捕获的视频中识别出帧中的矩形。我使用以下方法在观察到的图像上方显示一个矩形。

func displayRect(for observation: VNRectangleObservation) {
    DispatchQueue.main.async { [weak self] in
        guard let size = self?.imageView.frame.size else { return }
        guard let origin = self?.imageView.frame.origin else { return }

        let transform = CGAffineTransform(scaleX: size.width, y: size.height)

        let rect = observation.boundingBox.applying(transform)
            .applying(CGAffineTransform(scaleX: 1.0, y: -1.0))
            .applying(CGAffineTransform(translationX: 0.0, y: size.height))
            .applying(CGAffineTransform(translationX: -origin.x, y: -origin.y))

        let path = UIBezierPath(rect: rect)

        let layer = CAShapeLayer()
        layer.path = path.cgPath
        layer.fillRule = kCAFillRuleEvenOdd
        layer.fillColor = UIColor.red.withAlphaComponent(0.2).cgColor

        self?.overlay.sublayers = nil
        self?.overlay.addSublayer(layer)
    }
}

这对于从相机拍摄的图像效果很好，但是对于来自捕获视频的帧，该矩形是关闭的。实际上，如果偏离90度，则看起来像是它（因此是图像的整个坐标系）。请查看下面的屏幕截图。

我是否缺少有关视频帧的信息，这些信息可能导致观测值的boundingBox属性位于完全不同的坐标系中？

下面是我实现的captureOutput委托方法。

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard let buffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

    // Also tried converting to CGImage, creating handler from that, but made no difference
    let handler = VNImageRequestHandler(cvPixelBuffer: buffer, options: [:])

    let request = VNDetectRectanglesRequest()
    request.minimumAspectRatio = VNAspectRatio(0.2)
    request.maximumAspectRatio = VNAspectRatio(1.0)
    request.minimumSize = Float(0.3)

    try? handler.perform([request])

    // Note: Only ever captures one rectangle, so calling `first` not the issue.
    guard let observations = request.results as? [VNRectangleObservation],
        let observation = observations.first else {
            return removeShapeLayer()
    }

    displayRect(for: observation, buffer: buffer)
}

Answer 1

此问题是您没有将缓冲区的方向传递给VNImageRequestHandler，因此它会将视频作为横向交易。然后，当它返回该矩形时，将其放置在以纵向显示的视频上方。

您要么需要将方向传递到VNImageRequestHandler，要么修改（旋转）返回的矩形以将其考虑在内。