Question

我一直在使用Android MediaCodec来实现H.264来自Live 555 Galaxy S4直播（实时）的RTSP帧（硬件）解码流。将Live 555服务器配置更改为使用ffmpeg（使用x264）对帧进行编码，严格使用x264对帧进行编码后，使用MediaCodec解码帧的时间需要更久，更长。基本上，MediaCodec无法跟上流，并以慢动作显示视频，随着时间的推移变得越来越慢。回到ffmpeg对我来说不是一个解决方案，因为我需要能够编码为离散的NAL单位，而不是像ffmpeg这样的整个框架。

我想知道这是否是： A）我的服务器编码NAL单位或 B）的问题我的问题Android客户端，特别是它接收和解码NAL单位的方式。

我使用x264的编码配置为：

x264_param_default_preset(&param,"ultrafast", "zerolatency:fastdecode");
param.i_threads = 1;
param.i_bframe = 1;
param.i_width = image_width;
param.i_height = image_height;
param.i_fps_num = 60;
param.i_fps_den = 1;
param.i_keyint_max = 10;

param.rc.i_rc_method = X264_RC_ABR;
param.rc.i.bitrate = 6000;
param.i_sps_id = 7;
param.b_repeat_headers = 1;
param.b_annexb = 0;

我的Android MediaCodec客户端设置如下：

我在单独的Live 555 NAL客户端主题上收到每个RTSP个单元。每个NAL与其大小和呈现时间一起被放入队列中。一个单独的解码器线程从这个队列中抓取NALs，如果没有，则等待直到有。{/ p>

有些说明：

我通常看到的是队列开始填充NALs，而不是接近空。所以，我知道解码器线程的工作速度不够快。我不认为这是在Android手机上解码的固有问题（例如，处理限制），因为它对非常低的比特率做同样的事情 - 同样，当我使用ffmpeg时DID工作编码。如果我省略某些NAL单位，解码器就可以开始跟上。由于我使用的是Cyanogenmod 10.1，因此提高最低CPU频率也有帮助。

修改

这是Android客户端的日志，以及专门突出显示垃圾收集器的日志 -

整个Logcat：

10-15 16:40:03.955: D/DecodeActivity(18859): INFO_OUTPUT_BUFFERS_CHANGED
10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Sync frame received
10-15 16:40:03.995: E/OMX-VDEC-1080P(288):  No color conversion required
10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Get_parameter: OMX_IndexParamPortDefinition: nPortIndex (1), nFrameWidth (1280), nFrameHeight (720), nStride (1280), nSliceHeight (736), nBitrate (-1073741824), xFramerate (0x1e), nBufferSize (1433600), nBufferCountMin (4), nBufferCountActual (8), bBuffersContiguous (1918394328), nBufferAlignment (1075643347), bEnabled (1), bPopulated (1), eCompressionFormat (0x0), eColorFormat (0x7fa30c03)
10-15 16:40:03.995: D/DecodeActivity(18859): New format {height=720, what=1869968451, color-format=2141391875, slice-height=736, crop-left=0, width=1280, crop-bottom=719, crop-top=0, mime=video/raw, stride=1280, crop-right=1279}
10-15 16:40:04.005: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4966) fps(201.369308)
10-15 16:40:04.015: W/IInputConnectionWrapper(1069): showStatusIcon on inactive InputConnection
10-15 16:40:04.025: I/ActivityManager(698): Displayed com.mathieu.alloclient.javadecoder/.MainActivity: +617ms
10-15 16:40:04.025: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4766) fps(209.819550)
10-15 16:40:04.125: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4733) fps(211.282486)
10-15 16:40:04.445: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4711) fps(212.269150)
10-15 16:40:04.495: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4700) fps(212.765961)
10-15 16:40:05.676: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4688) fps(213.310577)
10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K, paused 7ms+9ms, total 117ms
10-15 16:40:06.537: D/ALSADevice(288): standby: handle 0x40024450 h 0x0
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _verb value Inactive
10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for HiFi Lowlatency enable 0
10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: SLIMBUS_0_RX Audio Mixer MultiMedia5, value: 0
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Line
10-15 16:40:06.577: D/alsa_ucm(288): disdev: device Line not enabled, no need to disable
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Speaker
10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for Speaker enable 0
10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP1, value: ZERO
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP2, value: ZERO
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT2 Volume, value: 0
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT4 Volume, value: 0
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 Digital Volume, value: 0
10-15 16:40:06.587: D/AudioUsbALSA(288): exitPlaybackThread, mproxypfdPlayback: -1
10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
10-15 16:40:17.638: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4677) fps(213.812271)
10-15 16:40:17.698: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4644) fps(215.331604)
10-15 16:40:20.681: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4633) fps(215.842865)
10-15 16:40:21.111: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4611) fps(216.872696)
10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
10-15 16:40:28.448: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 74
10-15 16:40:28.448: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
10-15 16:40:30.841: W/SystemClock(698): time going backwards: prev 16555345563411(ioctl) vs now 16555345441341(ioctl), tid=764
10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
10-15 16:40:57.467: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4577) fps(218.483719)
10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
10-15 16:41:14.403: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.500000 hotplug_avg_load_dw: 89
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:922.000000
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 48
10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
10-15 16:41:27.466: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
10-15 16:41:27.466: E/kickstart(862): EVENT: sahara_mode                         = 2
10-15 16:41:27.466: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
10-15 16:41:27.466: E/kickstart(862): EVENT: helloRx.mode                        = 2
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 64
10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 116
10-15 16:41:27.466: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs1", ""
10-15 16:41:27.466: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
10-15 16:41:27.466: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs1"
10-15 16:41:27.526: E/kickstart(862): Total bytes received so far: 786548
10-15 16:41:27.526: E/kickstart(862): EVENT: Received: 786432 bytes
10-15 16:41:27.526: E/kickstart(862): EVENT: Writing to disk
10-15 16:41:27.526: E/kickstart(862): EVENT: Successfully wrote to disk
10-15 16:41:27.526: E/kickstart(862): Received file "m9kefs1"
10-15 16:41:27.576: E/kickstart(862): Sync finish Received file "m9kefs1"
10-15 16:41:27.576: E/kickstart(862): 786432 bytes transferred in 0.106s (7.10 MBps)
10-15 16:41:27.576: E/kickstart(862): EVENT: num_debug_entries not >=0
10-15 16:41:27.576: E/kickstart(862): Successfully downloaded files from target 
10-15 16:41:27.576: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
10-15 16:41:27.576: E/kickstart(862): Total bytes received so far: 786556
10-15 16:41:27.576: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
10-15 16:41:27.576: E/kickstart(862): Sahara protocol completed
10-15 16:41:27.576: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
10-15 16:41:27.746: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.100000 hotplug_avg_load_dw: 79
10-15 16:41:27.746: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
10-15 16:42:17.225: I/ActivityManager(698): Start proc com.cyanogenmod.lockclock for service com.cyanogenmod.lockclock/.weather.WeatherUpdateService: pid=18954 uid=10028 gids={50028, 3003, 1028}
10-15 16:42:17.865: D/WeatherXmlParser(18954): Weather updated: WeatherInfo for Santa Barbara@ Tue Oct 15 16:42:17 PDT 2013: Fair(34), temperature 29°C, low 11°, high 27°, humidity 14%, wind 11km/h at W
10-15 16:42:17.945: I/ActivityManager(698): No longer want com.google.android.apps.uploader (pid 13565): empty #17
10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
10-15 16:42:35.983: W/ThrottleService(698): unable to find stats for iface rmnet0
10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 48
10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
10-15 16:43:02.098: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
10-15 16:43:02.098: E/kickstart(862): EVENT: sahara_mode                         = 2
10-15 16:43:02.098: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
10-15 16:43:02.098: E/kickstart(862): EVENT: helloRx.mode                        = 2
10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 64
10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
10-15 16:43:02.108: E/kickstart(862): Total bytes received so far: 116
10-15 16:43:02.108: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs2", ""
10-15 16:43:02.108: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
10-15 16:43:02.108: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs2"
10-15 16:43:02.158: E/kickstart(862): Total bytes received so far: 786548
10-15 16:43:02.158: E/kickstart(862): EVENT: Received: 786432 bytes
10-15 16:43:02.158: E/kickstart(862): EVENT: Writing to disk
10-15 16:43:02.168: E/kickstart(862): EVENT: Successfully wrote to disk
10-15 16:43:02.168: E/kickstart(862): Received file "m9kefs2"
10-15 16:43:02.218: E/kickstart(862): Sync finish Received file "m9kefs2"
10-15 16:43:02.218: E/kickstart(862): 786432 bytes transferred in 0.113s (6.65 MBps)
10-15 16:43:02.218: E/kickstart(862): EVENT: num_debug_entries not >=0
10-15 16:43:02.218: E/kickstart(862): Successfully downloaded files from target 
10-15 16:43:02.218: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
10-15 16:43:02.218: E/kickstart(862): Total bytes received so far: 786556
10-15 16:43:02.218: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
10-15 16:43:02.218: E/kickstart(862): Sahara protocol completed
10-15 16:43:02.218: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
10-15 16:43:17.563: W/SystemClock(698): time going backwards: prev 16722067121029(ioctl) vs now 16722066968441(ioctl), tid=764
10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
10-15 16:43:36.452: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 100
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:194.000000
10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
10-15 16:44:10.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 133
10-15 16:44:10.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
10-15 16:44:12.707: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.100000 hotplug_avg_load_dw: 116
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
10-15 16:44:14.008: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 140
10-15 16:44:14.008: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:240.000000
10-15 16:44:14.759: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.900000 hotplug_avg_load_dw: 132
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
10-15 16:44:15.360: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.300000 hotplug_avg_load_dw: 139
10-15 16:44:15.360: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
10-15 16:44:15.560: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 133
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:95.000000
10-15 16:44:16.361: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.500000 hotplug_avg_load_dw: 132
10-15 16:44:16.361: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:242.000000
10-15 16:44:16.661: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.700000 hotplug_avg_load_dw: 127
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
10-15 16:44:20.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 122
10-15 16:44:20.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19038
10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19041

Logcat中的垃圾收集器条目：

10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K,   paused 7ms+9ms, total 117ms
10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
10-15 16:44:24.389: D/dalvikvm(698): GC_EXPLICIT freed 3618K, 41% free 23768K/39664K, paused 12ms+11ms, total 123ms

Answer 1

这很容易就是你的Android客户端代码的问题（它是所有的Java，不是吗？），特别是如果你的logcat显示垃圾收集器经常被激活的话。

Answer 2

我猜服务器配置的更改会改变rtp流，从而产生更高的数据量。它反过来揭示了客户端上的问题：例如，您可以为数据保留静态池，而不是每次都分配新的数据。

使用不同的服务器配置，MediaCodec硬件解码器要慢得多？

2 个答案: