使用GL_TEXTURE_2D的iOS YUV 420v在OpenGL着色器中显示错误的颜色

时间：2016-05-07 16:43:17

标签： ios opengl-es shader yuv

目标：使用GL_TEXTURE_2D代替CVOpenGLESTextureRef将YUV数据（格式为' 420v ' kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange ）推送到着色器（为什么？因为我需要使用glTexSubImage2d来操作像素，而我不能在目标为CVOpenGLESTextureGetTarget(<name>)的情况下使用它，它没有效果。我必须使用GL_TEXTURE_2D）

问题： 我正在使用自定义视频合成器来操纵AVPlayer视频。当我在Apple的 AVCustomEdit 示例代码中使用CVOpenGLESTextureRef时，它使用2个单独的着色器，一个用于Y（亮度），一个用于UV（色度），视频看起来像这样正常：

enter image description here

但是尝试使用GL_TEXTURE_2D会使视频显示为绿色和粉红色，如下所示：

enter image description here

如果我将GL_TEXTURE_2D与结合了Y和UV纹理的片段着色器一起使用，它看起来就更糟了：enter image description here

我的代码：

首先创建跟踪缓冲区和目标缓冲区：

CVPixelBufferRef foregroundSourceBuffer = [request sourceFrameByTrackID:currentInstruction.foregroundTrackID];
CVPixelBufferRef dstBuffer = [_renderContext newPixelBuffer];

然后它们被传递给render函数，该函数包含以下相关代码：

CVOpenGLESTextureRef foregroundLumaTexture  = [self lumaTextureForPixelBuffer:foregroundPixelBuffer];
CVOpenGLESTextureRef foregroundChromaTexture = [self chromaTextureForPixelBuffer:foregroundPixelBuffer];
CVOpenGLESTextureRef destLumaTexture = [self lumaTextureForPixelBuffer:destinationPixelBuffer];       
CVOpenGLESTextureRef destChromaTexture = [self chromaTextureForPixelBuffer:destinationPixelBuffer];

亮度纹理函数返回：

CVOpenGLESTextureRef luma = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                   _videoTextureCache,
                                                   pixelBuffer,
                                                   NULL,
                                                   GL_TEXTURE_2D,
                                                   GL_RED_EXT,
                                                   (int)CVPixelBufferGetWidth(pixelBuffer),
                                                   (int)CVPixelBufferGetHeight(pixelBuffer),
                                                   GL_RED_EXT,
                                                   GL_UNSIGNED_BYTE,
                                                   0,
                                                   &lumaTexture);

色度纹理函数返回：

CVOpenGLESTextureRef chroma = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                   _videoTextureCache,
                                                   pixelBuffer,
                                                   NULL,
                                                   GL_TEXTURE_2D,
                                                   GL_RG_EXT,
                                                   (int)CVPixelBufferGetWidthOfPlane(pixelBuffer, 1),
                                                   (int)CVPixelBufferGetHeightOfPlane(pixelBuffer, 1),
                                                   GL_RG_EXT,
                                                   GL_UNSIGNED_BYTE,
                                                   1,
                                                   &chromaTexture);

现在渲染函数的相关主体：

    glBindFramebuffer(GL_FRAMEBUFFER, self.offscreenBufferHandle);

    glViewport(0, 0, (int)CVPixelBufferGetWidthOfPlane(destinationPixelBuffer, 0), (int)CVPixelBufferGetHeightOfPlane(destinationPixelBuffer, 0));


#ifdef USE_GL_TEXTURE_2D

    int bufferWidth = CVPixelBufferGetWidth(foregroundPixelBuffer);
    int bufferHeight = CVPixelBufferGetHeight(foregroundPixelBuffer);


    GLuint frameTextureY;
    GLuint frameTextureUV;

    glGenTextures(1, &frameTextureY);
    glGenTextures(1, &frameTextureUV);


    if(CVPixelBufferLockBaseAddress(foregroundPixelBuffer, 0) == kCVReturnSuccess){

        glBindTexture(GL_TEXTURE_2D, frameTextureY);
        glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE, bufferWidth, bufferHeight, 0, GL_LUMINANCE, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddressOfPlane(foregroundPixelBuffer, 0));

        glBindTexture(GL_TEXTURE_2D, frameTextureUV);
        glTexImage2D(GL_TEXTURE_2D, 0, GL_LUMINANCE_ALPHA, bufferWidth/2, bufferHeight/2, 0, GL_LUMINANCE_ALPHA, GL_UNSIGNED_BYTE, CVPixelBufferGetBaseAddressOfPlane(foregroundPixelBuffer, 1));

        CVPixelBufferUnlockBaseAddress(foregroundPixelBuffer, 0);
    }
#endif

    glActiveTexture(GL_TEXTURE0);
#ifdef USE_GL_TEXTURE_2D
    glUseProgram(self.programYUV_2);
    glBindTexture(GL_TEXTURE_2D, frameTextureY);
    glUniformMatrix4fv(uniforms[UNIFORM_RENDER_TRANSFORM_YUV_2], 1, GL_FALSE, preferredRenderTransform);
#else
    glUseProgram(self.programY);
    glBindTexture(CVOpenGLESTextureGetTarget(foregroundLumaTexture), CVOpenGLESTextureGetName(foregroundLumaTexture));
    glUniformMatrix4fv(uniforms[UNIFORM_RENDER_TRANSFORM_Y], 1, GL_FALSE, preferredRenderTransform);
#endif
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);


    // Attach the destination texture as a color attachment to the off screen frame buffer
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, CVOpenGLESTextureGetTarget(destLumaTexture), CVOpenGLESTextureGetName(destLumaTexture), 0);

    if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
        NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));
        goto bail;
    }

    glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT);

#ifdef USE_GL_TEXTURE_2D
    glUniform1i(uniforms[UNIFORM_TEXTURE_YUV_2_Y], 0);
    glVertexAttribPointer(ATTRIB_VERTEX_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadVertexData1);
    glEnableVertexAttribArray(ATTRIB_VERTEX_Y_UV_INONESHADER);
    glVertexAttribPointer(ATTRIB_TEXCOORD_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadTextureData1);
    glEnableVertexAttribArray(ATTRIB_TEXCOORD_Y_UV_INONESHADER);
#else
    glUniform1i(uniforms[UNIFORM_TEXTURE_Y], 0);
    glVertexAttribPointer(ATTRIB_VERTEX_Y, 2, GL_FLOAT, 0, 0, quadVertexData1);
    glEnableVertexAttribArray(ATTRIB_VERTEX_Y);
    glVertexAttribPointer(ATTRIB_TEXCOORD_Y, 2, GL_FLOAT, 0, 0, quadTextureData1);
    glEnableVertexAttribArray(ATTRIB_TEXCOORD_Y);
#endif
    glDrawArrays(GL_TRIANGLE_STRIP, 0, 5);


    glActiveTexture(GL_TEXTURE1);
#ifdef USE_GL_TEXTURE_2D
    //no need to use different program
    glBindTexture(GL_TEXTURE_2D, frameTextureUV);
    glUniformMatrix4fv(uniforms[UNIFORM_RENDER_TRANSFORM_YUV_2], 1, GL_FALSE, preferredRenderTransform);
#else
    glUseProgram(self.programUV);
glBindTexture(CVOpenGLESTextureGetTarget(foregroundChromaTexture), CVOpenGLESTextureGetName(foregroundChromaTexture));
    glUniformMatrix4fv(uniforms[UNIFORM_RENDER_TRANSFORM_UV], 1, GL_FALSE, preferredRenderTransform);
#endif
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);


    glViewport(0, 0, (int)CVPixelBufferGetWidthOfPlane(destinationPixelBuffer, 1), (int)CVPixelBufferGetHeightOfPlane(destinationPixelBuffer, 1));

    // Attach the destination texture as a color attachment to the off screen frame buffer
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, CVOpenGLESTextureGetTarget(destChromaTexture), CVOpenGLESTextureGetName(destChromaTexture), 0);

    if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
        NSLog(@"Failed to make complete framebuffer object %x", glCheckFramebufferStatus(GL_FRAMEBUFFER));
        goto bail;
    }

    glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT);

#ifdef USE_GL_TEXTURE_2D
    glUniform1i(uniforms[UNIFORM_TEXTURE_YUV_2_UV], 1);
    glVertexAttribPointer(ATTRIB_VERTEX_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadVertexData1);
    glEnableVertexAttribArray(ATTRIB_VERTEX_Y_UV_INONESHADER);
    glVertexAttribPointer(ATTRIB_TEXCOORD_Y_UV_INONESHADER, 2, GL_FLOAT, 0, 0, quadTextureData1);
    glEnableVertexAttribArray(ATTRIB_TEXCOORD_Y_UV_INONESHADER);#else
    glUniform1i(uniforms[UNIFORM_TEXTURE_UV], 1);
    glVertexAttribPointer(ATTRIB_VERTEX_UV, 2, GL_FLOAT, 0, 0, quadVertexData1);
    glEnableVertexAttribArray(ATTRIB_VERTEX_UV);
    glVertexAttribPointer(ATTRIB_TEXCOORD_UV, 2, GL_FLOAT, 0, 0, quadTextureData1);
    glEnableVertexAttribArray(ATTRIB_TEXCOORD_UV);
#endif

    glDrawArrays(GL_TRIANGLE_STRIP, 0, 5);

    glFlush();

bail:
#ifdef USE_GL_TEXTURE_2D
    glDeleteTextures(1, &frameTextureY);
    glDeleteTextures(1, &frameTextureUV);
#endif
    CFRelease(foregroundLumaTexture);
    CFRelease(foregroundChromaTexture);
    CFRelease(destLumaTexture);
    CFRelease(destChromaTexture);

    // Periodic texture cache flush every frame
    CVOpenGLESTextureCacheFlush(self.videoTextureCache, 0);

以下是我的片段着色器，我根据不同的测试用例使用（无论是单独绘制Y还是UV，还是一起绘制）：

static const char kFragmentShaderY[] = {
"varying highp vec2 texCoordVarying; \n \
 uniform sampler2D s_texture_y; \n \
 void main() \n \
 { \n \
    gl_FragColor.r = texture2D(s_texture_y, texCoordVarying).r; \n \
 }"

}; 

static const char kFragmentShaderUV[] = {
"varying highp vec2 texCoordVarying; \n \
uniform sampler2D s_texture_uv; \n \
void main() \n \
{ \n \
    gl_FragColor.rg = texture2D(s_texture_uv, texCoordVarying).rg; \n \
}"

}; 

static const char kFragmentShaderYUV_2Textures[] = {
"varying highp vec2 texCoordVarying; \n \
uniform sampler2D s_texture_y; \n \
uniform sampler2D s_texture_uv; \n \
\n \
void main() \n \
{ \n \
    mediump vec3 yuv;// = vec3(1.1643 * (texture2D(s_texture_y, texCoordVarying).r - 0.0625), \n \
    lowp vec3 rgb; \n \
    yuv.x = texture2D(s_texture_y, texCoordVarying).r; \n \
    yuv.yz = texture2D(s_texture_uv, texCoordVarying).rg - vec2(0.5, 0.5); \n \
    \n \
    rgb = mat3(      1,       1,       1, \n \
    0, -.21482, 2.12798, \n \
    1.28033, -.38059,       0) * yuv;   \n \
    gl_FragColor = vec4(rgb, 1.0); \n \
}"

};

使用GL_TEXTURE_2D，如果我使用包含Y和UV纹理的片段着色器，视频看起来像上面的＃3。如果我使用两个单独的片段着色器（一个用于Y，一个用于UV），图片在上面＃2（ALMOST正确，但色度颜色只是绿色和粉红色）*（请注意，我会注释掉上面的一些代码）能够使用2个单独的片段着色器，当然我glBind到GL_TEXTURE_2D而不是CV，依此类推，等等。

同样，我的问题是我需要使用GL_TEXTURE_2D而不是CVOpenGLESTextureGetTarget，但如果我这样做，它就不会显示正确的色度颜色。我不知道我做错了什么。 YUV格式与kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange而不是kCVPixelFormatType_420YpCbCr8BiPlanarFullRange有什么关系呢？我还尝试了制作3 GL_LUMINANCE纹理方法，以及其他许多没有运气的排列。

1 个答案:

答案 0 :(得分：1)

事实证明问题在于使用GL_LUMINANCE和GL_LUMINANCE_ALPHA，这些显然已被弃用。当我将它们切换到GL_RED_EXT和GL_RG_EXT时，它工作，色度颜色最终是正确的。我希望这个问题和答案可以节省其他人的时间。

相关问题
最新问题