Gstreamer rtp视频混音器，发现了一条工作流水线，但需要改进

Question

我正在尝试将多个rtp h264有效负载视频流混合到15FPS的单个视频流中。

通过15FPS的视频资源模式混合两个视频流的工作管道

VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264"

gst-launch -vvvve videomixer2 name=mix ! ffmpegcolorspace ! xvimagesink  

udpsrc caps=$VIDEO_CAPS port=3030 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox top=0 left=0 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix.  

udpsrc caps=$VIDEO_CAPS port=6666 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox top=0 left=-178 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix. 

videotestsrc ! video/x-raw-yuv, framerate=15/1, width=640, height=360 ! mix.

上面的管道有一个奇怪的问题，当试图混合时，第一个视频源变为空白，（我确定第一个视频源仍在进行流媒体播放，但它没有转过来。显示第二个视频流。详细说明如下。

上述管道的详细信息

/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstVideoMixer2Pad:sink_0: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpSession:rtpsession0.GstPad:recv_rtp_sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0.GstGhostPad:recv_rtp_sink_0: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0.GstGhostPad:recv_rtp_sink_0.GstProxyPad:proxypad0: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpSession:rtpsession0.GstPad:recv_rtp_src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpSsrcDemux:rtpssrcdemux0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpJitterBuffer:rtpjitterbuffer0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpJitterBuffer:rtpjitterbuffer0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpSession:rtpsession1.GstPad:recv_rtp_sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1.GstGhostPad:recv_rtp_sink_0: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1.GstGhostPad:recv_rtp_sink_0.GstProxyPad:proxypad1: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpSession:rtpsession1.GstPad:recv_rtp_src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpSsrcDemux:rtpssrcdemux1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpJitterBuffer:rtpjitterbuffer1.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpJitterBuffer:rtpjitterbuffer1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpPtDemux:rtpptdemux0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/GstRtpBin:rtpbin0.GstGhostPad:recv_rtp_src_0_4186542290_99.GstProxyPad:proxypad2: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2640.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpPtDemux:rtpptdemux1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/GstRtpBin:rtpbin1.GstGhostPad:recv_rtp_src_0_4186622237_99.GstProxyPad:proxypad3: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2641.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2640.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox0.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp1.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp1.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstVideoMixer2Pad:sink_1: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2641.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale1.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale1.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox1.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox1.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter4.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter4.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp2.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp2.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstVideoMixer2Pad:sink_2: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)YUY2, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)YUY2, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2875): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.

我非常怀疑它与视频框有什么关系。

Answer 1

（在问题编辑中回答。转换为社区维基回答。请参阅What is the appropriate action when the answer to a question is added to the question itself?）

OP写道：

  

正如我所说，这个问题与视频文件有关，我们必须添加视频框边界-α= 0或alpha通道。

     

功能齐全的管道：接受3个RTP h264视频，并通过视频测试模式进行混音。

gst-launch -vvvve videomixer2 name=mix ! ffmpegcolorspace ! xvimagesink sync=false  udpsrc caps=$VIDEO_CAPS port=3030 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox border-alpha=0 top=0 left=0 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix.  udpsrc caps=$VIDEO_CAPS port=6666 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox border-alpha=0 top=0 left=-176 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix. udpsrc caps=$VIDEO_CAPS port=2222 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox border-alpha=0 top=-144 left=0 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! mix. videotestsrc ! video/x-raw-yuv, framerate=15/1, width=352, height=288 ! mix.