FFMPEG and JNI - pass AVFrame data to Java and Back

Question

I have some C code that decodes a video frame by frame. I get to a point where i have an AVFrame in BGR32 and would like to send it back to Java for editing.

I have a ByteBuffer object in my C code that was created in Java using AllocateDirect but i struggle to write the content of the AVFrame->data[0] (of uint8_t type) to it and read it back. I have tried memcpy with no luck. Does anyone have an idea how to achieve this?

UPDATE Followed Will's comment below and wrote this in C

char *buf = (*pEnv)->GetDirectBufferAddress(pEnv, byteBuffer);
memcpy(buf, rgb_frame->data[0], output_width*output_height*4);

The buffer does contain some data in Java but doing the following returns a null bitmap

BufferedImage frame = ImageIO.read(bitmapStream);

Where bitmapStream is a ByteBufferInputStream defined here: https://code.google.com/p/kryo/source/browse/trunk/src/com/esotericsoftware/kryo/io/ByteBufferInputStream.java?r=205

Not sure if I am not writing things correctly in this buffer

UPDATE 2

Got pretty close now thanks to the latest snippet. I am using BGR32 format in my C code ie 4 bytes per pixel. So I modified things a bit in Java:

final byte[] dataArray = new byte[width*height*4];
imageData.get(dataArray);
final BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_BGR);
final DataBuffer buffer = new DataBufferByte(dataArray, dataArray.length);
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);

I get the image correctly but there seems to be an issue with color channels

Tried different formats with no luck

Answer 1

来自Oracle的JNI职能文档 https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html#GetDirectBufferAddress

  

GetDirectBufferAddress

void* GetDirectBufferAddress(JNIEnv* env, jobject buf);

  

获取并返回内存区域的起始地址   由给定的直接java.nio.Buffer引用。

     

此函数允许本机代码访问相同的内存区域   可以通过缓冲区对象访问Java代码。 LINKAGE：

     

JNIEnv接口函数表中的索引230。参数：

     

env：JNIEnv接口指针

     

buf：一个直接的java.nio.Buffer对象（不能为NULL）RETURNS：

     

返回由引用的内存区域的起始地址   缓冲。如果内存区域未定义，则返回NULL，如果给定   对象不是直接的java.nio.Buffer，或者如果JNI直接访问   此虚拟机不支持缓冲区。时间：

     

JDK / JRE 1.4

我测试了这个C ++代码：

static char framebuf[100];

JNIEXPORT void JNICALL Java_javaapplication45_UseByteBuffer_readBuf
  (JNIEnv *env, jobject usebb, jobject bb) {
    void *addr = env->GetDirectBufferAddress(bb);
    framebuf[0] = 77;
    memcpy(addr,framebuf,100);
}

和这个Java代码：

public class UseByteBuffer {
    public native void readBuf(ByteBuffer bb);
}

...

public static void main(String[] args) {
    System.load("/home/shackle/NetBeansProjects/usebb/dist/Debug/GNU-Linux-x86/libusebb.so");
    ByteBuffer bb = ByteBuffer.allocateDirect(100);
    new UseByteBuffer().readBuf(bb);
    byte first_byte = bb.get(0);
    System.out.println("first_byte = " + first_byte);
}

它打印first_byte = 77表示正确复制了数据。

<强>更新

ImageIO.read（）不会接受任何字节集，它必须采用已安装的ImageReader之一可以识别的格式，例如JPEG或PNG。

这是一个将（3字节r，g，b）字节生成为图像

的示例

int width = 256;
int height = 256;
ByteBuffer bb = ByteBuffer.allocateDirect(height*width*3);

byte[] raw = new byte[width * height * 3];
bb.get(raw);
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_3BYTE_BGR);
DataBuffer buffer = new DataBufferByte(raw, raw.length);
SampleModel sampleModel = new ComponentSampleModel(DataBuffer.TYPE_BYTE, width, height, 3, width * 3, new int[]{0,1,2});
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);

更新2

对于BGR32，我相信这会更接近：

ByteBuffer imageData = ByteBuffer.allocateDirect(height * width * 4);
byte[] raw = new byte[width * height * 4];
imageData.get(raw);
BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_BGR);
DataBuffer buffer = new DataBufferByte(raw, raw.length);
SampleModel sampleModel = new ComponentSampleModel(
        DataBuffer.TYPE_BYTE, width, height, 4, width * 4,
        new int[]{2,1,0} // Try {1,2,3}, {3,2,1}, {0,1,2}
);
Raster raster = Raster.createRaster(sampleModel, buffer, null);
image.setData(raster);

请注意我评论过的地方，我怀疑你可能需要在ComponentSampleModel构造函数的第三个参数中试验bandOffsets数组来修复颜色模型。

更新3

可以重用sampleModel，通过使用BufferedImage.copyData（）到WritableRaster而不是使用getRaster（）来从图像中获取数据。

SampleModel sampleModel = new ComponentSampleModel(
        DataBuffer.TYPE_BYTE, width, height, 4, width * 4,
        new int[]{2, 1, 0} 
);

...

BufferedImage newImage = ImageIO.read(new File("test.png"));
byte newRaw[] = new byte[height*width*4];
DataBuffer newBuffer = new DataBufferByte(newRaw, newRaw.length);
WritableRaster newRaster = Raster.createWritableRaster(sampleModel, newBuffer, null);
newImage.copyData(newRaster);