Question

我正在使用实时摄像机输出来更新MTKView上的CIImage。我的主要问题是，我遇到了很大的负面性能差异，尽管我遇到的所有设置都是相同的，但较旧的iPhone的CPU性能要比较新的iPhone好。

这是篇冗长的文章，但是我决定包括这些细节，因为它们对于导致此问题的发生可能很重要。请让我知道我还可以包括什么。

下面，我有我的captureOutput函数和两个调试布尔，可以在运行时打开和关闭它们。我用它来确定问题的原因。

applyLiveFilter -设置是否使用CIFilter操纵CIImage。

updateMetalView -设置是否更新MTKView的CIImage。

// live output from camera
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection){

    /*              
     Create CIImage from camera.
     Here I save a few percent of CPU by using a function 
     to convert a sampleBuffer to a Metal texture, but
     whether I use this or the commented out code 
     (without captureOutputMTLOptions) does not have 
     significant impact. 
    */

    guard let texture:MTLTexture = convertToMTLTexture(sampleBuffer: sampleBuffer) else{
        return
    }

    var cameraImage:CIImage = CIImage(mtlTexture: texture, options: captureOutputMTLOptions)!

    var transform: CGAffineTransform = .identity

    transform = transform.scaledBy(x: 1, y: -1)

    transform = transform.translatedBy(x: 0, y: -cameraImage.extent.height)

    cameraImage = cameraImage.transformed(by: transform)

    /*
    // old non-Metal way of getting the ciimage from the cvPixelBuffer
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else
    {
        return
    }

    var cameraImage:CIImage = CIImage(cvPixelBuffer: pixelBuffer)
    */

    var orientation = UIImage.Orientation.right

    if(isFrontCamera){
        orientation = UIImage.Orientation.leftMirrored
    }

    // apply filter to camera image
    if debug_applyLiveFilter {
        cameraImage = self.applyFilterAndReturnImage(ciImage: cameraImage, orientation: orientation, currentCameraRes:currentCameraRes!)
    }

    DispatchQueue.main.async(){
        if debug_updateMetalView {
            self.MTLCaptureView!.image = cameraImage
        }
    }

}

下面是两部手机切换上述讨论的不同布尔值的结果图表：

即使没有更新“金属”视图的CIIMage，也没有应用滤镜，iPhone XS的CPU仍比iPhone 6S Plus的CPU大2％，但这并不是很大的开销，但我怀疑相机的拍摄方式有所不同在设备之间。

我的AVCaptureSession的预设在两部手机之间设置相同（AVCaptureSession.Preset.hd1280x720）
从captureOutput创建的CIImage大小相同（范围）在两个电话之间。

在这两部手机之间，是否需要手动设置AVCaptureDevice的设置（包括activeFormat属性）以使它们在设备之间相同？

我现在的设置是：

if let captureDevice = AVCaptureDevice.default(for:AVMediaType.video) {
    do {
        try captureDevice.lockForConfiguration()
            captureDevice.isSubjectAreaChangeMonitoringEnabled = true
            captureDevice.focusMode = AVCaptureDevice.FocusMode.continuousAutoFocus
            captureDevice.exposureMode = AVCaptureDevice.ExposureMode.continuousAutoExposure
        captureDevice.unlockForConfiguration()

    } catch {
        // Handle errors here
        print("There was an error focusing the device's camera")
    }
}

我的MTKView基于西蒙·格拉德曼（Simon Gladman）编写的代码，在使用Apple建议的Core Animation将渲染放大到屏幕宽度之前，对性能进行了一些编辑和缩放。

class MetalImageView: MTKView
{
    let colorSpace = CGColorSpaceCreateDeviceRGB()

    var textureCache: CVMetalTextureCache?

    var sourceTexture: MTLTexture!

    lazy var commandQueue: MTLCommandQueue =
        {
            [unowned self] in

            return self.device!.makeCommandQueue()
            }()!

    lazy var ciContext: CIContext =
        {
            [unowned self] in

            return CIContext(mtlDevice: self.device!)
            }()


    override init(frame frameRect: CGRect, device: MTLDevice?)
    {
        super.init(frame: frameRect,
                   device: device ?? MTLCreateSystemDefaultDevice())



        if super.device == nil
        {
            fatalError("Device doesn't support Metal")
        }

        CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, self.device!, nil, &textureCache)

        framebufferOnly = false

        enableSetNeedsDisplay = true

        isPaused = true

        preferredFramesPerSecond = 30
    }

    required init(coder: NSCoder)
    {
        fatalError("init(coder:) has not been implemented")
    }

    // The image to display
    var image: CIImage?
    {
        didSet
        {
            setNeedsDisplay()
        }
    }

    override func draw(_ rect: CGRect)
    {
        guard var
            image = image,
            let targetTexture:MTLTexture = currentDrawable?.texture else
        {
            return
        }

        let commandBuffer = commandQueue.makeCommandBuffer()

        let customDrawableSize:CGSize = drawableSize

        let bounds = CGRect(origin: CGPoint.zero, size: customDrawableSize)

        let originX = image.extent.origin.x
        let originY = image.extent.origin.y

        let scaleX = customDrawableSize.width / image.extent.width
        let scaleY = customDrawableSize.height / image.extent.height

        let scale = min(scaleX*IVScaleFactor, scaleY*IVScaleFactor)
        image = image
            .transformed(by: CGAffineTransform(translationX: -originX, y: -originY))
            .transformed(by: CGAffineTransform(scaleX: scale, y: scale))


        ciContext.render(image,
                         to: targetTexture,
                         commandBuffer: commandBuffer,
                         bounds: bounds,
                         colorSpace: colorSpace)


        commandBuffer?.present(currentDrawable!)

        commandBuffer?.commit()

    }

}

我的AVCaptureSession（captureSession）和AVCaptureVideoDataOutput（videoOutput）设置如下：

func setupCameraAndMic(){
    let backCamera = AVCaptureDevice.default(for:AVMediaType.video)

    var error: NSError?
    var videoInput: AVCaptureDeviceInput!
    do {
        videoInput = try AVCaptureDeviceInput(device: backCamera!)
    } catch let error1 as NSError {
        error = error1
        videoInput = nil
        print(error!.localizedDescription)
    }

    if error == nil &&
        captureSession!.canAddInput(videoInput) {

        guard CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, MetalDevice, nil, &textureCache) == kCVReturnSuccess else {
            print("Error: could not create a texture cache")
            return
        }

        captureSession!.addInput(videoInput)            

        setDeviceFrameRateForCurrentFilter(device:backCamera)

        stillImageOutput = AVCapturePhotoOutput()

        if captureSession!.canAddOutput(stillImageOutput!) {
            captureSession!.addOutput(stillImageOutput!)

            let q = DispatchQueue(label: "sample buffer delegate", qos: .default)
            videoOutput.setSampleBufferDelegate(self, queue: q)
            videoOutput.videoSettings = [
                kCVPixelBufferPixelFormatTypeKey as AnyHashable as! String: NSNumber(value: kCVPixelFormatType_32BGRA),
                kCVPixelBufferMetalCompatibilityKey as String: true
            ]
            videoOutput.alwaysDiscardsLateVideoFrames = true

            if captureSession!.canAddOutput(videoOutput){
                captureSession!.addOutput(videoOutput)
            }

            captureSession!.startRunning()

        }

    }

    setDefaultFocusAndExposure()
}

视频和麦克风分别记录在两个流中。麦克风和录制视频的详细信息被忽略了，因为我的重点是实时摄像机输出的性能。

更新-我在GitHub上有一个简化的测试项目，可以更轻松地测试我遇到的问题：https://github.com/PunchyBass/Live-Filter-test-project

Answer 1

从我的最高思想来看，即使您使用的是A12的2.49 GHz和A9的1.85 GHz，也没有将梨与梨进行比较，即使将它们用于相同的参数，XS摄像机的多项功能需要更多的CPU资源（双摄像机，稳定器，智能HDR等）。

很抱歉，我试图找到这些功能的CPU成本的指标，但不幸的是，由于您的需要，我无法找到这些信息，因为当他们以最好的价格出售这些信息时，这些信息与营销无关相机用于智能手机。

他们也将其作为最好的处理器出售，我们不知道将XS相机与A9处理器一起使用会发生什么，它可能会崩溃，我们永远不会知道...

PS ....您的指标适用于整个处理器还是所用的内核？对于整个处理器，您还需要考虑设备可执行的其他任务，对于单核，这是200％的21％，而600％的是39％

为什么使用实时摄影机时，iPhone XS的CPU性能要比iPhone 6S Plus差？

1 个答案: