Question

目标：

我正在构建FPV机器人，我想通过Web浏览器通过本地Wi-Fi连接对其进行控制。

我正在使用带有Raspbian Stretch的树莓派3B +。我制造了自己的电动机控制和功率调节器帽子。

经过大量研究测试，我决定将node.js用作http服务器和socket.io，以与我的机器人进行低延迟的双向通信。该堆栈可实现大约7毫秒的延迟。

Picture of the robot

问题：

我需要将低延迟视频从连接到RPI的USB摄像机流式传输到浏览器。我的目标是在10FPS时达到至少640x480分辨率，并具有50ms或更长时间的延迟。我很乐意牺牲视觉素养，以便机器人更快地做出响应。

如果可能的话，我想在UDP中进行流传输以提高流的可靠性。

如果可能的话，我想流播一个现代网络浏览器可以本地解码的视频。我想使用H264编解码器和HTML5视频标签。如果没有其他选择，我可以退一步使用JavaScript播放器。

我尝试过的事情

我进行了广泛的研究并尝试了许多工具。

除其他外，我尝试了VLC，mjpg流光，gstreamer和raspivid。我有几次去浏览器可以观看的视频流，但是在320x240的情况下，充其量我只有700毫秒的延迟。距离我的目标非常远。

目前，我正在研究WebRTC解决方案。

问题：

我希望获得NODE.JS软件包或其他解决方案的建议，以提供UDP H264视频流，该视频流可由HTML5视频标签解码，目标延迟为50ms。

谢谢

更新：

感谢您的回答！我将继续更新此问题，并在解决方案可行后将其发布。

按一下个性框架

我尝试了一种不同的方法，将200KB 640x480 jpg单个帧通过websocket推送，我得到了大约190ms的延迟。通过重用对象，我可能可以做得更好，但现在暂时搁置这种尝试。

UPDATE2：

在研究WebRTC时，我发现了一个看起来很简单的堆栈。服务器端使用V4L2作为驱动程序，FFMPEG将代码本地转换为具有TS封装的MPEG1 http流，节点js将其转换为websocket。客户端有一个JavaScript，可以解码MPEG1 TS流并将画布对象绘制到HTML页面中。

它可实现640x480 @ 20FPS，延迟为240mS。对于MVP来说已经足够好了，但是我会继续努力下去。答案中的代码。

Answer 1

我希望获得NODE.JS软件包或其他解决方案的建议，以提供UDP H264视频流，该视频流可由HTML5视频标签解码，目标延迟为50ms。

在这种配置下，几乎可以肯定这是不可能的。

如果您放弃视频标签要求，而仅在浏览器中使用直接的WebRTC，则可能可以缩短到150ms。

Answer 2

我从这里改编了代码，并将其与http服务器和socket.io控件集成在一起： https://github.com/phoboslab/jsmpeg

服务器：

V4L2-> FFMPEG（MPEG1 TS）-> NODE HTTP服务器-> NODE Websocket广播

客户：

Websocket-> Javascript（解码MPEG1 TS并将其绘制为html canvas）-> HTML画布

此堆栈可实现640x480 @ 20FPS，延迟为240ms。距离我的目标还很远，但作为MVP还是足够的。双向控件的延迟为7ms，这是非常好的。

此堆栈在转码和解码阶段受阻，并且RPI变得非常热。通过websocket传输原始数据看起来不错，我将在以后分析每个步骤的延迟。

执行：

pi@MazeRunner:~ $ node node.js &
pi@MazeRunner:~ $ ffmpeg -f v4l2 -framerate 20 -video_size 640x480 -i /dev/video0 -f mpegts -codec:v mpeg1video -s 640x480 -b:v 600k -bf 0 http://localhost:8080/mystream

服务器端NODE.JS

//operating system library. Used to get local IP address
var os = require("os");
//file system library. Used to load file stored inside back end server (https://nodejs.org/api/fs.html)
var fs = require("fs");
//http system library. Handles basic html requests
var http = require("http").createServer(http_handler);
//url library. Used to process html url requests
var url = require("url");
//Websocket
var io = require("socket.io")(http);
//Websocket used to stream video
var websocket = require("ws");

//-----------------------------------------------------------------------------------
//	CONFIGURATION
//-----------------------------------------------------------------------------------

//Port the server will listen to
var server_port = 8080;
var websocket_stream_port = 8082;
//Path of the http and css files for the http server
var file_index_name = "index.html";
var file_css_name = "style.css";
var file_jsplayer_name = "jsmpeg.min.js";
//Http and css files loaded into memory for fast access
var file_index;
var file_css;
var file_jsplayer;
//Name of the local video stream
var stream_name = "mystream";

//-----------------------------------------------------------------------------------
//	DETECT SERVER OWN IP
//-----------------------------------------------------------------------------------

//If just one interface, store the server IP Here
var server_ip;
//Get local IP address of the server
//https://stackoverflow.com/questions/3653065/get-local-ip-address-in-node-js
var ifaces = os.networkInterfaces();

Object.keys(ifaces).forEach
(
	function (ifname)
	{
		var alias = 0;

		ifaces[ifname].forEach
		(
			function (iface)
			{
				if ('IPv4' !== iface.family || iface.internal !== false)
				{
				  // skip over internal (i.e. 127.0.0.1) and non-ipv4 addresses
				  return;
				}

				if (alias >= 1)
				{
					// this single interface has multiple ipv4 addresses
					console.log('INFO: Server interface ' +alias +' - ' + ifname + ':' + alias, iface.address);
				}
				else
				{
					server_ip = iface.address;
					// this interface has only one ipv4 adress
					console.log('INFO: Server interface - ' +ifname, iface.address);
				}
				++alias;
			}
		);
	}
);

//-----------------------------------------------------------------------------------
//	HTTP SERVER
//-----------------------------------------------------------------------------------
//	Fetch and serves local files to client

//Create http server and listen to the given port
http.listen
(
	server_port,
	function( )
	{
		console.log('INFO: ' +server_ip +' listening to html requests on port ' +server_port);
		//Pre-load http, css and js files into memory to improve http request latency
		file_index = load_file( file_index_name );
		file_css = load_file( file_css_name );
		file_jsplayer = load_file( file_jsplayer_name );
	}
);

//-----------------------------------------------------------------------------------
//	HTTP REQUESTS HANDLER
//-----------------------------------------------------------------------------------
//	Answer to client http requests. Serve http, css and js files

function http_handler(req, res)
{
	//If client asks for root
	if (req.url == '/')
	{
		//Request main page
		res.writeHead( 200, {"Content-Type": detect_content(file_index_name),"Content-Length":file_index.length} );
		res.write(file_index);
		res.end();

		console.log("INFO: Serving file: " +req.url);
	}
	//If client asks for css file
	else if (req.url == ("/" +file_css_name))
	{
		//Request main page
		res.writeHead( 200, {"Content-Type": detect_content(file_css_name),"Content-Length" :file_css.length} );
		res.write(file_css);
		res.end();

		console.log("INFO: Serving file: " +req.url);
	}
	//If client asks for css file
	else if (req.url == ("/" +file_jsplayer_name))
	{
		//Request main page
		res.writeHead( 200, {"Content-Type": detect_content(file_jsplayer_name),"Content-Length" :file_jsplayer.length} );
		res.write(file_jsplayer);
		res.end();

		console.log("INFO: Serving file: " +req.url);
	}
	//Listening to the port the stream from ffmpeg will flow into
	else if (req.url = "/mystream")
	{
		res.connection.setTimeout(0);

		console.log( "Stream Connected: " +req.socket.remoteAddress + ":" +req.socket.remotePort );

		req.on
		(
			"data",
			function(data)
			{
				streaming_websocket.broadcast(data);
				/*
				if (req.socket.recording)
				{
					req.socket.recording.write(data);
				}
				*/
				//console.log("broadcast: ", data.length);
			}
		);

		req.on
		(
			"end",
			function()
			{
				console.log("local stream has ended");
				if (req.socket.recording)
				{
					req.socket.recording.close();
				}
			}
		);

	}
	//If client asks for an unhandled path
	else
	{
		res.end();
		console.log("ERR: Invalid file request" +req.url);
	}
}

//-----------------------------------------------------------------------------------
//	WEBSOCKET SERVER: CONTROL/FEEDBACK REQUESTS
//-----------------------------------------------------------------------------------
//	Handle websocket connection to the client

io.on
(
	"connection",
	function (socket)
	{
		console.log("connecting...");

		socket.emit("welcome", { payload: "Server says hello" });

		//Periodically send the current server time to the client in string form
		setInterval
		(
			function()
			{
				socket.emit("server_time", { server_time: get_server_time() });
			},
			//Send every 333ms
			333
		);

		socket.on
		(
			"myclick",
			function (data)
			{
				timestamp_ms = get_timestamp_ms();
				socket.emit("profile_ping", { timestamp: timestamp_ms });
				console.log("button event: " +" client says: " +data.payload);
			}
		);

		//"ArrowLeft"
		socket.on
		(
			"keyboard",
			function (data)
			{
				timestamp_ms = get_timestamp_ms();
				socket.emit("profile_ping", { timestamp: timestamp_ms });
				console.log("keyboard event: " +" client says: " +data.payload);
			}
		);

		//profile packets from the client are answer that allows to compute roundway trip time
		socket.on
		(
			"profile_pong",
			function (data)
			{
				timestamp_ms_pong = get_timestamp_ms();
				timestamp_ms_ping = data.timestamp;
				console.log("Pong received. Round trip time[ms]: " +(timestamp_ms_pong -timestamp_ms_ping));
			}
		);
	}
);

//-----------------------------------------------------------------------------------
//	WEBSOCKET SERVER: STREAMING VIDEO
//-----------------------------------------------------------------------------------

// Websocket Server
var streaming_websocket = new websocket.Server({port: websocket_stream_port, perMessageDeflate: false});

streaming_websocket.connectionCount = 0;

streaming_websocket.on
(
	"connection",
	function(socket, upgradeReq)
	{
		streaming_websocket.connectionCount++;
		console.log
		(
			'New websocket Connection: ',
			(upgradeReq || socket.upgradeReq).socket.remoteAddress,
			(upgradeReq || socket.upgradeReq).headers['user-agent'],
			'('+streaming_websocket.connectionCount+" total)"
		);

		socket.on
		(
			'close',
			function(code, message)
			{
				streaming_websocket.connectionCount--;
				console.log('Disconnected websocket ('+streaming_websocket.connectionCount+' total)');
			}
		);
	}
);

streaming_websocket.broadcast = function(data)
{
	streaming_websocket.clients.forEach
	(
		function each(client)
		{
			if (client.readyState === websocket.OPEN)
			{
				client.send(data);
			}
		}
	);
};


//-----------------------------------------------------------------------------------
//	FUNCTIONS
//-----------------------------------------------------------------------------------

//-----------------------------------------------------------------------------------
//	SERVER DATE&TIME
//-----------------------------------------------------------------------------------
//	Get server time in string form

function get_server_time()
{
	my_date = new Date();

	return my_date.toUTCString();
}

//-----------------------------------------------------------------------------------
//	TIMESTAMP
//-----------------------------------------------------------------------------------
//	Profile performance in ms

function get_timestamp_ms()
{
	my_date = new Date();
	return 1000.0* my_date.getSeconds() +my_date.getMilliseconds()
}

//-----------------------------------------------------------------------------------
//	FILE LOADER
//-----------------------------------------------------------------------------------
//	Load files into memory for improved latency

function load_file( file_name )
{
	var file_tmp;
	var file_path =  __dirname +"/" +file_name;

	//HTML index file
	try
	{
		file_tmp = fs.readFileSync( file_path );
	}
	catch (err)
	{
		console.log("ERR: " +err.code +" failed to load: " +file_path);
		throw err;
	}

	console.log("INFO: " +file_path +" has been loaded into memory");

	return file_tmp;
}

//-----------------------------------------------------------------------------------
//	CONTENT TYPE DETECTOR
//-----------------------------------------------------------------------------------
//	Return the right content type to give correct information to the client browser

function detect_content( file_name )
{
	if (file_name.includes(".html"))
	{
        return "text/html";
	}
	else if (file_name.includes(".css"))
	{
		return "text/css";
	}
	else if (file_name.includes(".js"))
	{
		return "application/javascript";
	}
	else
	{
		throw "invalid extension";

	}
}

客户端html

<!DOCTYPE html>
<meta charset="utf-8"/>
<html>
	<head>
		<title>Maze Runner</title>
		<link rel="stylesheet" href="style.css">
		<script type="text/javascript" src="/socket.io/socket.io.js"></script>
		<script type="text/javascript">
			var host_ip = document.location.hostname;
			console.log("connecting to host: ", host_ip);

			//Get references to the html controls
			textbox_input1 = window.document.getElementById("my_text_box")

			//Connect to the server via websocket
			var mysocket = io("http://" +host_ip +":8080");
			//Long lived frame object
			var last_frame;

			//-----------------------------------------
			//	CONNESSION ACKNOWLEDGE
			//-----------------------------------------
			//	Link is initiated by the client
			//	Server sends a welcome message when link is estabilished
			//	Server could send an auth token to keep track of individual clients and login data

			mysocket.on
			(
				"welcome",
				(message) =>
				{
					console.log("Server websocket connession acknoweldged... " +message.payload);
				}
			)

			//-----------------------------------------
			//	SERVER->CLIENT CONTROLS
			//-----------------------------------------
			//	Server can send an async message to dinamically update the page without reloading
			//	This is an example message with the server local date and time in string form

			mysocket.on
			(
				"server_time",
				(message) =>
				{
					fill_label( message.server_time );
					console.log("Server sent his local time... " +message.server_time);
				}
			)

			function fill_label( payload )
			{
				textbox_input1.value=payload;
			}

			//-----------------------------------------
			//	CLIENT->SERVER CONTROLS
			//-----------------------------------------
			//	Controls inside the webpage can emit async events to the server
			//	In this example I have a push button and I catch keyboard strokes

			//Handler for a pushbutton
			function socket_button_handler()
			{
				mysocket.emit("myclick", { payload: "button was clicked" });
				console.log("Button was clicked...");
			}

			//Listen for keystrokes
			window.document.addEventListener
			(
				"keypress",
				function onEvent(event)
				{
					//Inform the server that a key has been pressed
					mysocket.emit("keyboard", { payload: event.key });
					console.log("Key press...");
				}
			);

			//-----------------------------------------
			//	PING-PONG
			//-----------------------------------------
			//	Server sends ping messages with a timestamp
			//	Client answers with pongs to allow server to profile latency of the channel

			//profile messages means the server wants to compute roundway trip
			mysocket.on
			(
				"profile_ping",
				(message) =>
				{
					//Answer back with the received timestamp so that server can compute roundway trip
					mysocket.emit("profile_pong", { timestamp: message.timestamp });
					console.log( "server wants a pong. server absolute timestamp[ms]: " +message.timestamp );
				}
			);

		</script>
	</head>
	<body>

		<h1>Html+Css Server +low latency Websocket server</h1>
		<!-- button control with socket emitter as handler -->
		<p> This button will emit a websocket event. The server will be informed in real time of the event. </p>
		<button id="my_button" type="button" onclick="socket_button_handler()">Websocket Button!</button>

		<!-- input text control -->
		<p> This input can be filled through websockets directly by the server in real time </p>
		<input id="my_text_box" type="text" value="" size="40">

		<!-- canvas object, it's painted by the javascript video decoder -->
		<p> This canvas is painted by the javascript player and shows the live stream.'</p>
		<canvas id="video-canvas" width=640 height=480></canvas>

		<!-- Javascript video decoder, take in a data stream from a websocket and paint on a canvas -->
		<script type="text/javascript" src="jsmpeg.min.js"></script>
		<script type="text/javascript">
		var mycanvas = document.getElementById("video-canvas");
		var url = "ws://" + host_ip +":8082/";
		var player = new JSMpeg.Player(url, {canvas: mycanvas});
		</script>
	</body>
</html>

JavaScript播放器

您可以从这里获得我使用的JavaScript播放器： https://github.com/phoboslab/jsmpeg/blob/master/jsmpeg.min.js

使用NODE.JS和html5的低延迟（50ms）视频流

目标：

问题：

我尝试过的事情

问题：

更新：

UPDATE2：

2 个答案: