Question

我正在寻找一种在前后相机之间翻转时保持无缝音轨的方法。市场上的许多应用都可以做到这一点，其中一个例子就是SnapChat ......

解决方案应使用AVCaptureSession和AVAssetWriter。此外，它应该明确不使用AVMutableComposition，因为AVMutableComposition和AVCaptureSession ATM之间存在bug。另外，我负担不起后处理时间。

目前，当我更改视频输入时，录音会跳过并变得不同步。

我包含了可能相关的代码。

翻转相机

-(void) updateCameraDirection:(CamDirection)vCameraDirection {
    if(session) {
        AVCaptureDeviceInput* currentInput;
        AVCaptureDeviceInput* newInput;
        BOOL videoMirrored = NO;
        switch (vCameraDirection) {
            case CamDirection_Front:
                currentInput = input_Back;
                newInput = input_Front;
                videoMirrored = NO;
                break;
            case CamDirection_Back:
                currentInput = input_Front;
                newInput = input_Back;
                videoMirrored = YES;
                break;
            default:
                break;
        }

        [session beginConfiguration];
        //disconnect old input
        [session removeInput:currentInput];
        //connect new input
        [session addInput:newInput];
        //get new data connection and config
        dataOutputVideoConnection = [dataOutputVideo connectionWithMediaType:AVMediaTypeVideo];
        dataOutputVideoConnection.videoOrientation = AVCaptureVideoOrientationPortrait;
        dataOutputVideoConnection.videoMirrored = videoMirrored;
        //finish
        [session commitConfiguration];
    }
}

样本缓冲区

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    //not active
    if(!recordingVideo)
        return;

    //start session if not started
    if(!startedSession) {
        startedSession = YES;
        [assetWriter startSessionAtSourceTime:CMSampleBufferGetPresentationTimeStamp(sampleBuffer)];
    }

    //Process sample buffers
    if (connection == dataOutputAudioConnection) {
        if([assetWriterInputAudio isReadyForMoreMediaData]) {
            BOOL success = [assetWriterInputAudio appendSampleBuffer:sampleBuffer];
            //…
        }

    } else if (connection == dataOutputVideoConnection) {
        if([assetWriterInputVideo isReadyForMoreMediaData]) {        
            BOOL success = [assetWriterInputVideo appendSampleBuffer:sampleBuffer];
            //…
        }
    }
}

也许调整音频采样timeStamp？

Answer 1

嘿，我遇到了同样的问题，并发现在切换相机后，下一帧被推离了很远的地方。之后，这似乎会移动每一帧，从而导致视频和音频不同步。我的解决方案是在切换相机后将所有错位的帧移动到正确的位置。

很抱歉，我的答案将是Swift 4.2

您必须使用AVAssetWriterInputPixelBufferAdaptor才能在指定的演示时间戳记上附加示例缓冲区。

previousPresentationTimeStamp是上一帧的显示时间戳，而currentPresentationTimestamp是当前帧的显示时间戳。 maxFrameDistance在测试时效果很好，但是您可以根据自己的喜好进行更改。

let currentFramePosition = (Double(self.frameRate) * Double(currentPresentationTimestamp.value)) / Double(currentPresentationTimestamp.timescale)
let previousFramePosition = (Double(self.frameRate) * Double(previousPresentationTimeStamp.value)) / Double(previousPresentationTimeStamp.timescale)
var presentationTimeStamp = currentPresentationTimestamp
let maxFrameDistance = 1.1
let frameDistance = currentFramePosition - previousFramePosition
if frameDistance > maxFrameDistance {
    let expectedFramePosition = previousFramePosition + 1.0
    //print("[mwCamera]: Frame at incorrect position moving from \(currentFramePosition) to \(expectedFramePosition)")

    let newFramePosition = ((expectedFramePosition) * Double(currentPresentationTimestamp.timescale)) / Double(self.frameRate)

    let newPresentationTimeStamp = CMTime.init(value: CMTimeValue(newFramePosition), timescale: currentPresentationTimestamp.timescale)

    presentationTimeStamp = newPresentationTimeStamp
}

let success = assetWriterInputPixelBufferAdator.append(pixelBuffer, withPresentationTime: presentationTimeStamp)
if !success, let error = assetWriter.error {
    fatalError(error.localizedDescription)
}

也请注意-之所以有效，是因为我保持帧速率一致，因此请确保在此过程中您完全控制了捕获设备的帧速率。

I have a repo using this logic here

Answer 2

解决此问题的最“稳定方法” - 是在切换源时“暂停”录制。

但您也可以使用空白视频和无声音频帧“填补空白”。这是我在我的项目中实现的。

因此，创建布尔值来阻止在切换摄像头/麦克风时附加新的 CMSampleBuffer 的能力，并在一些延迟后重置它：

let idleTime = 1.0
self.recordingPaused = true
DispatchQueue.main.asyncAfter(deadline: .now() + idleTime) {
  self.recordingPaused = false
}
writeAllIdleFrames()

在writeAllIdleFrames方法中你需要计算你需要写多少帧：

func writeAllIdleFrames() {
    let framesPerSecond = 1.0 / self.videoConfig.fps
    let samplesPerSecond = 1024 / self.audioConfig.sampleRate
    
    let videoFramesCount = Int(ceil(self.switchInputDelay / framesPerSecond))
    let audioFramesCount = Int(ceil(self.switchInputDelay / samplesPerSecond))
    
    for index in 0..<max(videoFramesCount, audioFramesCount) {
        // creation synthetic buffers
        
        recordingQueue.async {
            if index < videoFramesCount {
                let pts = self.nextVideoPTS()
                self.writeBlankVideo(pts: pts)
            }
            
            if index < audioFramesCount {
                let pts = self.nextAudioPTS()
                self.writeSilentAudio(pts: pts)
            }
        }
    }
}

如何计算下一个 PTS？

func nextVideoPTS() -> CMTime {
    guard var pts = self.lastVideoRawPTS else { return CMTime.invalid }
    
    let framesPerSecond = 1.0 / self.videoConfig.fps
    let delta = CMTime(value: Int64(framesPerSecond * Double(pts.timescale)),
                       timescale: pts.timescale, flags: pts.flags, epoch: pts.epoch)
    pts = CMTimeAdd(pts, delta)
    return pts
}

告诉我，如果您还需要创建空白/静音视频/音频缓冲区的代码:)

Answer 3

我确实设法找到了一个同步解决方案的中间解决方案，该解决方案是使用is repo在 Woody Jean-louis 解决方案中找到的。

结果与instagram相似，但效果更好。基本上，我要做的是防止 assetWriterAudioInput 在切换相机时附加新样本。无法确切知道何时发生这种情况，因此我发现在 captureOutput 方法切换之前和之后，每0.02秒+-（最多0.04秒）发送一次视频样本。

知道这一点后，我创建了一个 self.lastVideoSampleDate ，每次将视频样本附加到 assetWriterInputPixelBufferAdator 时都将对其进行更新，而我只允许将音频样本附加到 assetWriterAudioInput 是该日期小于0.05。

 if let assetWriterAudioInput = self.assetWriterAudioInput,
            output == self.audioOutput, assetWriterAudioInput.isReadyForMoreMediaData {

            let since = Date().timeIntervalSince(self.lastVideoSampleDate)
            if since < 0.05 {
                let success = assetWriterAudioInput.append(sampleBuffer)
                if !success, let error = assetWriter.error {
                    print(error)
                    fatalError(error.localizedDescription)
                }
            }
        }

  let success = assetWriterInputPixelBufferAdator.append(pixelBuffer, withPresentationTime: presentationTimeStamp)
            if !success, let error = assetWriter.error {
                print(error)
                fatalError(error.localizedDescription)
            }
            self.lastVideoSampleDate = Date()

使用AVCaptureSession＆amp; amp;放大相机时无缝录音AVAssetWriter

3 个答案: