Question

是否可以从ExoPlayer中正在运行的视频中获取像素数据（例如，作为RGB字节数组）？理想情况下，作为真实的视频分辨率而不是显示的视图的大小。我想将这些数据转发到OpenCV用于ImageProcessing目的。

或者，我正在寻找一个强大的（基于ffmpeg的）Android框架，将视频输入OpenCV，输入可能是IP摄像机，本地文件，在线文件和在线流。

感谢任何帮助。

Answer 1

警告此解决方案目前仅适用于我的Android 8模拟器，因为this。

好的，这是我的解决方案。 我正在使用SimpleExoPlayer自定义MediaCodecVideoRenderer。我没有将播放器绑定到SimpleExoPlayerView，因为如果我想手动抓取，我不被允许图像数据。 在我的自定义MediaCodecVideoRenderer中，我覆盖了processOutputBuffer并使用getOutputImage来获得一个不错的标准Android图片。然后我通过我的OpenCV代码发送它然后转换它每当我需要时，使用OpenCV Utils返回到Android位图。小心此代码需要API＆gt; = 21.此外，此代码不会输出任何音频。

private void startPlayer(Context context, Uri uri) {

    BandwidthMeter bandwidthMeter = new DefaultBandwidthMeter();
    TrackSelection.Factory videoTrackSelectionFactory = new AdaptiveTrackSelection.Factory(bandwidthMeter);
    TrackSelector trackSelector = new DefaultTrackSelector(videoTrackSelectionFactory);

    SimpleExoPlayer player = ExoPlayerFactory.newSimpleInstance((eventHandler, videoRendererEventListener, audioRendererEventListener, textRendererOutput, metadataRendererOutput) -> {
        return new Renderer[]{new CustomMediaCodecVideoRenderer(context, MediaCodecSelector.DEFAULT, 1000, eventHandler, videoRendererEventListener, 100)};
    }, trackSelector);

    DataSource.Factory dataSourceFactory = new DefaultDataSourceFactory(context, Util.getUserAgent(context, context.getPackageName()));
    ExtractorMediaSource videoSource = new ExtractorMediaSource.Factory(dataSourceFactory).createMediaSource(uri);

    LoopingMediaSource loopingMediaSource = new LoopingMediaSource(videoSource);

    player.setPlayWhenReady(true);
    player.prepare(loopingMediaSource);

}

class CustomMediaCodecVideoRenderer extends MediaCodecVideoRenderer {
    CustomMediaCodecVideoRenderer(Context context, MediaCodecSelector mediaCodecSelector, long allowedJoiningTimeMs, @Nullable Handler eventHandler, @Nullable VideoRendererEventListener eventListener, int maxDroppedFrameCountToNotify) {
        super(context, mediaCodecSelector, allowedJoiningTimeMs, eventHandler, eventListener, maxDroppedFrameCountToNotify);
    }

    @Override
    protected boolean processOutputBuffer(long positionUs, long elapsedRealtimeUs, MediaCodec codec, ByteBuffer buffer, int bufferIndex, int bufferFlags, long bufferPresentationTimeUs, boolean shouldSkip) throws ExoPlaybackException {


        Image image = codec.getOutputImage(bufferIndex);

        if (image != null) {
            Log.d(MainActivity.class.getName(), "test");

            Mat mat = convertYuv420888ToMat(image, false);
            // TODO do your thing with the OpenCV Mat
        }

        return super.processOutputBuffer(positionUs, elapsedRealtimeUs, codec, buffer, bufferIndex, bufferFlags, bufferPresentationTimeUs, shouldSkip);
    }
}

我找到了Image to Mat转换here。

private Mat convertYuv420888ToMat(Image image, boolean isGreyOnly) {
    int width = image.getWidth();
    int height = image.getHeight();

    Image.Plane yPlane = image.getPlanes()[0];
    int ySize = yPlane.getBuffer().remaining();

    if (isGreyOnly) {
        byte[] data = new byte[ySize];
        yPlane.getBuffer().get(data, 0, ySize);

        Mat greyMat = new Mat(height, width, CvType.CV_8UC1);
        greyMat.put(0, 0, data);

        return greyMat;
    }

    Image.Plane uPlane = image.getPlanes()[1];
    Image.Plane vPlane = image.getPlanes()[2];

    // be aware that this size does not include the padding at the end, if there is any
    // (e.g. if pixel stride is 2 the size is ySize / 2 - 1)
    int uSize = uPlane.getBuffer().remaining();
    int vSize = vPlane.getBuffer().remaining();

    byte[] data = new byte[ySize + (ySize/2)];

    yPlane.getBuffer().get(data, 0, ySize);

    ByteBuffer ub = uPlane.getBuffer();
    ByteBuffer vb = vPlane.getBuffer();

    int uvPixelStride = uPlane.getPixelStride(); //stride guaranteed to be the same for u and v planes
    if (uvPixelStride == 1) {
        uPlane.getBuffer().get(data, ySize, uSize);
        vPlane.getBuffer().get(data, ySize + uSize, vSize);

        Mat yuvMat = new Mat(height + (height / 2), width, CvType.CV_8UC1);
        yuvMat.put(0, 0, data);
        Mat rgbMat = new Mat(height, width, CvType.CV_8UC3);
        Imgproc.cvtColor(yuvMat, rgbMat, Imgproc.COLOR_YUV2RGB_I420, 3);
        yuvMat.release();
        return rgbMat;
    }

    // if pixel stride is 2 there is padding between each pixel
    // converting it to NV21 by filling the gaps of the v plane with the u values
    vb.get(data, ySize, vSize);
    for (int i = 0; i < uSize; i += 2) {
        data[ySize + i + 1] = ub.get(i);
    }

    Mat yuvMat = new Mat(height + (height / 2), width, CvType.CV_8UC1);
    yuvMat.put(0, 0, data);
    Mat rgbMat = new Mat(height, width, CvType.CV_8UC3);
    Imgproc.cvtColor(yuvMat, rgbMat, Imgproc.COLOR_YUV2RGB_NV21, 3);
    yuvMat.release();
    return rgbMat;
}

从Android中的Exoplayer获取视频像素数据

1 个答案: