Question

我正在尝试使用一个简单的矩形跟踪控制器，我可以很好地进行矩形检测，但跟踪请求总是因为我找不到的原因而失败。

有时跟踪请求会在失败前触发几次回调，有时会在单次回调发生之前立即失败。我觉得这与我提交请求的方式有关，但我无法深究它。

以下是视图控制器的代码

class TestController: UIViewController, AVCaptureVideoDataOutputSampleBufferDelegate {

    // Video capture
    private var videoSession = AVCaptureSession()
    private var videoLayer: AVCaptureVideoPreviewLayer!

    // Detection
    private var detectionRequest: VNDetectRectanglesRequest?
    private let sequenceHandler = VNSequenceRequestHandler()

    // Tracking
    private var trackingRequest: VNTrackRectangleRequest?
    private var shape: Detection?
    private var pixelBuffer: CVImageBuffer?




    // MARK: Setup
    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)

        startVideoFeed()
    }

    override func viewDidLayoutSubviews() {
        videoLayer.frame = view.layer.bounds
    }

    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        startDetectingRectangles()
    }



    private func startDetectingRectangles() {
        let request = VNDetectRectanglesRequest(completionHandler: didDetectRectangle)
        request.maximumObservations = 1
        request.minimumSize = 0.07
        request.minimumConfidence = 0.9
        request.minimumAspectRatio = 0.5
        request.maximumAspectRatio = 2
        request.quadratureTolerance = 10
        detectionRequest = request
    }

    private func didDetectRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observations = request.results, observations.count > 0 else { return }
        let results = observations.map { $0 as? VNRectangleObservation }

        for case let rectangle? in results {
            detectionRequest = nil

            let request = VNTrackRectangleRequest(rectangleObservation: rectangle, completionHandler: didTrackRectangle)
            trackingRequest = request
        }
    }


    private func didTrackRectangle(request: VNRequest, error: Error?) {

        // Fetch results of the correct type
        guard let observation = request.results?.first as? VNRectangleObservation else { return }

        // Create or update UI
    }




    // Start capturing video frames
    private func startVideoFeed() {

        // Session config
        videoSession.sessionPreset = .photo

        // Create device and input to device
        guard
            let captureDevice = AVCaptureDevice.default(for: .video),
            let deviceInput = try? AVCaptureDeviceInput(device: captureDevice)
            else {
                fatalError("Error setting up capture device.")
        }

        // Setup device output
        let deviceOutput = AVCaptureVideoDataOutput()
        deviceOutput.setSampleBufferDelegate(self, queue: DispatchQueue.global(qos: .default))

        // Set input and output
        videoSession.addInput(deviceInput)
        videoSession.addOutput(deviceOutput)

        // Setup video display layer
        videoLayer = AVCaptureVideoPreviewLayer(session: videoSession)
        videoLayer.frame = view.bounds
        videoLayer.videoGravity = .resizeAspectFill
        view.layer.addSublayer(videoLayer)

        videoSession.startRunning()
    }

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

        pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)

        guard let pixelBuffer = pixelBuffer else { return }
        do {

            var requests: [VNRequest] = []

            if let detectionRequest = detectionRequest {
                requests.append(detectionRequest) }
            if let trackingRequest = trackingRequest {
                requests.append(trackingRequest) }

            try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right)
        } catch {
            print(error)
        }
    }
}

错误是打印：

错误Domain = com.apple.vis Code = 9“内部错误：跟踪其中一个角落失败，置信度= 0.000000;阈值= 0.650000“ UserInfo = {NSLocalizedDescription =内部错误：跟踪其中一个角落失败，置信度= 0.000000;阈值= 0.650000}

跟踪请求首次失败后每帧输出一次，请求永远不会恢复。这是抛出错误的try sequenceHandler.perform(requests, on: pixelBuffer, orientation: .right)行（然后我会打印出来）。

我查看了网络上点缀的对象观察示例，他们似乎要求每个帧都有一个新的跟踪请求来跟踪对象。我尝试在VNTrackRectangleRequest函数中创建一个新的didTrackRectangle，因此每个框架都有一个新请求，但我遇到了同样的问题。

对此有任何帮助，我们非常感谢。

Answer 1

至少在跟踪清晰可见的矩形时，我能够毫无问题地运行您的代码。

跟踪请求失败时，您可能应该创建一个新的VNSequenceRequestHandler。似乎当跟踪请求失去对对象的跟踪时间太长时，它将无法恢复并重新开始对其进行跟踪。这也意味着它们永远不会被释放，并且会不断抛出错误。如果您继续向请求处理程序中添加新的跟踪请求，您将很快超过同时跟踪请求的限制。

如果您可以预期跟踪序列的结束，可以将跟踪请求的isLastFrame属性设置为true，并在出现以下情况时将其释放到可用跟踪器池中当前帧完成处理。

Answer 2

我每帧都看到此错误，并且要解决此问题，我发现我需要使用正确的performRequests:onCVPixelBuffer:orientation:error:来调用orientation。

（对于使用后置摄像头的肖像方向，我相信方向是kCGImagePropertyOrientationRight）。

来自documentation：

对于每个此类请求，请调用顺序请求处理程序的 performRequests：onCVPixelBuffer：orientation：error：方法，确保传递视频阅读器的方向，以确保垂直跟踪。

VNTrackRectangleRequest内部错误

2 个答案: