如何直接在IOS 4中旋转CVImageBuffer图像而不转换为UIImage？

Question

我在iPhone上使用OpenCV 2.2来检测面部。我正在使用IOS 4的AVCaptureSession来访问摄像机流，如下面的代码所示。

我的挑战是视频帧以CVBufferRef（指向CVImageBuffer）对象的形式出现，并且它们以480px宽，300px高的方式呈现为风景。如果您将手机侧向握住，这很好，但是当手机保持在垂直位置时，我想将这些框架顺时针旋转90度，以便OpenCV可以正确找到面部。

我可以将CVBufferRef转换为CGImage，然后转换为UIImage，然后旋转，就像这个人正在做的那样：Rotate CGImage taken from video frame

但是这浪费了很多CPU。我正在寻找一种更快的方式来旋转进入的图像，如果可能的话，最好使用GPU进行处理。

有什么想法吗？

伊恩

代码示例：

 -(void) startCameraCapture {
  // Start up the face detector

  faceDetector = [[FaceDetector alloc] initWithCascade:@"haarcascade_frontalface_alt2" withFileExtension:@"xml"];

  // Create the AVCapture Session
  session = [[AVCaptureSession alloc] init];

  // create a preview layer to show the output from the camera
  AVCaptureVideoPreviewLayer *previewLayer = [AVCaptureVideoPreviewLayer layerWithSession:session];
  previewLayer.frame = previewView.frame;
  previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;

  [previewView.layer addSublayer:previewLayer];

  // Get the default camera device
  AVCaptureDevice* camera = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];

  // Create a AVCaptureInput with the camera device
  NSError *error=nil;
  AVCaptureInput* cameraInput = [[AVCaptureDeviceInput alloc] initWithDevice:camera error:&error];
  if (cameraInput == nil) {
   NSLog(@"Error to create camera capture:%@",error);
  }

  // Set the output
  AVCaptureVideoDataOutput* videoOutput = [[AVCaptureVideoDataOutput alloc] init];
  videoOutput.alwaysDiscardsLateVideoFrames = YES;

  // create a queue besides the main thread queue to run the capture on
  dispatch_queue_t captureQueue = dispatch_queue_create("catpureQueue", NULL);

  // setup our delegate
  [videoOutput setSampleBufferDelegate:self queue:captureQueue];

  // release the queue.  I still don't entirely understand why we're releasing it here,
  // but the code examples I've found indicate this is the right thing.  Hmm...
  dispatch_release(captureQueue);

  // configure the pixel format
  videoOutput.videoSettings = [NSDictionary dictionaryWithObjectsAndKeys:
          [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], 
          (id)kCVPixelBufferPixelFormatTypeKey,
          nil];

  // and the size of the frames we want
  // try AVCaptureSessionPresetLow if this is too slow...
  [session setSessionPreset:AVCaptureSessionPresetMedium];

  // If you wish to cap the frame rate to a known value, such as 10 fps, set 
  // minFrameDuration.
  videoOutput.minFrameDuration = CMTimeMake(1, 10);

  // Add the input and output
  [session addInput:cameraInput];
  [session addOutput:videoOutput];

  // Start the session
  [session startRunning];  
 }

 - (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
  // only run if we're not already processing an image
  if (!faceDetector.imageNeedsProcessing) {

   // Get CVImage from sample buffer
   CVImageBufferRef cvImage = CMSampleBufferGetImageBuffer(sampleBuffer);

   // Send the CVImage to the FaceDetector for later processing
   [faceDetector setImageFromCVPixelBufferRef:cvImage];

   // Trigger the image processing on the main thread
   [self performSelectorOnMainThread:@selector(processImage) withObject:nil waitUntilDone:NO];
  }
 }

Answer 1

vImage是一种非常快速的方法。但需要ios5。呼叫说ARGB，但它适用于从缓冲区获得的BGRA。

这也有一个优点，你可以切出缓冲区的一部分并旋转它。 See my answer here

- (unsigned char*) rotateBuffer: (CMSampleBufferRef) sampleBuffer
{
 CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
 CVPixelBufferLockBaseAddress(imageBuffer,0);

 size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
 size_t width = CVPixelBufferGetWidth(imageBuffer);
 size_t height = CVPixelBufferGetHeight(imageBuffer);
 size_t currSize = bytesPerRow*height*sizeof(unsigned char); 
 size_t bytesPerRowOut = 4*height*sizeof(unsigned char); 

 void *srcBuff = CVPixelBufferGetBaseAddress(imageBuffer); 
 unsigned char *outBuff = (unsigned char*)malloc(currSize);  

 vImage_Buffer ibuff = { srcBuff, height, width, bytesPerRow};
 vImage_Buffer ubuff = { outBuff, width, height, bytesPerRowOut};

 uint8_t rotConst = 1;   // 0, 1, 2, 3 is equal to 0, 90, 180, 270 degrees rotation

 vImage_Error err= vImageRotate90_ARGB8888 (&ibuff, &ubuff, NULL, rotConst, NULL,0);
 if (err != kvImageNoError) NSLog(@"%ld", err);

 return outBuff;
}

Answer 2

如果您以90度的速度旋转，那么您可以在内存中进行旋转。以下是仅将数据复制到新像素缓冲区的示例代码。进行蛮力旋转应该是直截了当的。

- (CVPixelBufferRef) rotateBuffer: (CMSampleBufferRef) sampleBuffer
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer,0);

    size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);

    void *src_buff = CVPixelBufferGetBaseAddress(imageBuffer);

    NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGImageCompatibilityKey,
                             [NSNumber numberWithBool:YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
                             nil];

    CVPixelBufferRef pxbuffer = NULL;
    //CVReturn status = CVPixelBufferPoolCreatePixelBuffer (NULL, _pixelWriter.pixelBufferPool, &pxbuffer);
    CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, width,
                                          height, kCVPixelFormatType_32BGRA, (CFDictionaryRef) options, 
                                          &pxbuffer);

    NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);

    CVPixelBufferLockBaseAddress(pxbuffer, 0);
    void *dest_buff = CVPixelBufferGetBaseAddress(pxbuffer);
    NSParameterAssert(dest_buff != NULL);

    int *src = (int*) src_buff ;
    int *dest= (int*) dest_buff ;
    size_t count = (bytesPerRow * height) / 4 ;
    while (count--) {
        *dest++ = *src++;
    }

    //Test straight copy.
    //memcpy(pxdata, baseAddress, width * height * 4) ;
    CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
    return pxbuffer;
}

如果要将其写回AVAssetWriterInput，则可以使用AVAssetWriterInputPixelBufferAdaptor。

以上未经优化。您可能希望寻找更有效的复制算法。一个好的开始是In-place Matrix Transpose。您还需要使用像素缓冲池，而不是每次都创建一个新的。

编辑。您可以使用GPU来执行此操作。这听起来像是在推动大量数据。在CVPixelBufferRef中有密钥kCVPixelBufferOpenGLCompatibilityKey。我假设你可以从CVImageBufferRef（它只是一个像素缓冲区引用）创建一个OpenGL兼容的图像，并通过着色器推送它。再次，过度使用IMO。您可能会看到BLAS或LAPACK是否具有“不合适”的转置方法。如果他们这样做，那么你可以放心，他们是高度优化的。

90 CW其中new_width = width ...这将为您提供纵向图像。

for (int i = 1; i <= new_height; i++) {
    for (int j = new_width - 1; j > -1; j--) {
        *dest++ = *(src + (j * width) + i) ;
    }
}

Answer 3

可能更容易按照您想要的方式设置视频方向：

connection.videoOrientation = AVCaptureVideoOrientationPortrait

这样你根本不需要做那个旋转噱头

Answer 4

我知道这是一个很老的问题，但我最近一直在解决类似的问题，也许有人会发现我的解决方案很有用。

我需要从iPhone相机提供的YCbCr格式的图像缓冲区中提取原始图像数据（从[AVCaptureVideoDataOutput.availableVideoCVPixelFormatTypes firstObject]获取），丢弃标题，元信息等信息，以便将其传递给进一步处理。

另外，我只需要在捕获的视频帧的中心提取小区域，因此需要进行一些裁剪。

我的条件允许仅以横向方向捕获视频，但是当设备以横向左方向定位时，图像被上下颠倒传递，因此我需要在两个轴上翻转它。 如果图像被翻转，我的想法是以相反的顺序从源图像缓冲区复制数据，并且在每行读取数据中复制reverse bytes以在两个轴上翻转图像。这个想法确实有效，而且因为我需要从源缓冲区复制数据，如果从开头或结尾读取，似乎没有太多的性能损失（当然，更大的图像=更长的处理，但我处理的是非常小的数字）

我想知道其他人对此解决方案的看法，当然还有一些提示如何改进代码：

/// Lock pixel buffer
CVPixelBufferLockBaseAddress(imageBuffer, 0);

/// Address where image buffer starts
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);

/// Read image parameters
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);

/// See whether image is flipped upside down
BOOL isFlipped = (_previewLayer.connection.videoOrientation == AVCaptureVideoOrientationLandscapeLeft);

/// Calculate cropping frame. Crop to scanAreaSize (defined as CGSize constant elsewhere) from the center of an image
CGRect cropFrame = CGRectZero;
cropFrame.size = scanAreaSize;
cropFrame.origin.x = (width / 2.0f) - (scanAreaSize.width / 2.0f);
cropFrame.origin.y = (height / 2.0f) - (scanAreaSize.height / 2.0f);

/// Update proportions to cropped size
width = (size_t)cropFrame.size.width;
height = (size_t)cropFrame.size.height;

/// Allocate memory for output image data. W*H for Y component, W*H/2 for CbCr component
size_t bytes = width * height + (width * height / 2);

uint8_t *outputDataBaseAddress = (uint8_t *)malloc(bytes);

if(outputDataBaseAddress == NULL) {

    /// Memory allocation failed, unlock buffer and give up
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

    return NULL;
}

/// Get parameters of YCbCr pixel format
CVPlanarPixelBufferInfo_YCbCrBiPlanar *bufferInfo = (CVPlanarPixelBufferInfo_YCbCrBiPlanar *)baseAddress;

NSUInteger bytesPerRowY = EndianU32_BtoN(bufferInfo->componentInfoY.rowBytes);
NSUInteger offsetY = EndianU32_BtoN(bufferInfo->componentInfoY.offset);

NSUInteger bytesPerRowCbCr = EndianU32_BtoN(bufferInfo->componentInfoCbCr.rowBytes);
NSUInteger offsetCbCr = EndianU32_BtoN(bufferInfo->componentInfoCbCr.offset);

/// Copy image data only, skipping headers and metadata. Create single buffer which will contain Y component data
/// followed by CbCr component data.

/// Process Y component
/// Pointer to the source buffer
uint8_t *src;

/// Pointer to the destination buffer
uint8_t *destAddress;

/// Calculate crop rect offset. Crop offset is number of rows (y * bytesPerRow) + x offset.
/// If image is flipped, then read buffer from the end to flip image vertically. End address is height-1!
int flipOffset = (isFlipped) ? (int)((height - 1) * bytesPerRowY) : 0;

int cropOffset = (int)((cropFrame.origin.y * bytesPerRowY) + flipOffset + cropFrame.origin.x);

/// Set source pointer to Y component buffer start address plus crop rect offset
src = baseAddress + offsetY + cropOffset;

for(int y = 0; y < height; y++) {

    /// Copy one row of pixel data from source into the output buffer.
    destAddress = (outputDataBaseAddress + y * width);

    memcpy(destAddress, src, width);

    if(isFlipped) {

        /// Reverse bytes in row to flip image horizontally
        [self reverseBytes:destAddress bytesSize:(int)width];

        /// Move one row up
        src -= bytesPerRowY;
    }
    else {

        /// Move to the next row
        src += bytesPerRowY;
    }
}

/// Calculate crop offset for CbCr component
flipOffset = (isFlipped) ? (int)(((height - 1) / 2) * bytesPerRowCbCr) : 0;
cropOffset = (int)((cropFrame.origin.y * bytesPerRowCbCr) + flipOffset + cropFrame.origin.x);

/// Set source pointer to the CbCr component offset + crop offset
src = (baseAddress + offsetCbCr + cropOffset);

for(int y = 0; y < (height / 2); y++) {

    /// Copy one row of pixel data from source into the output buffer.
    destAddress = (outputDataBaseAddress + (width * height) + y * width);

    memcpy(destAddress, src, width);

    if(isFlipped) {

        /// Reverse bytes in row to flip image horizontally
        [self reverseBytes:destAddress bytesSize:(int)width];

        /// Move one row up
        src -= bytesPerRowCbCr;
    }
    else {

        src += bytesPerRowCbCr;
    }
}

/// Unlock pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

/// Continue with image data in outputDataBaseAddress;