我发现与go实现(https://tus.io)相比,nodejs(https://github.com/tus/tus-node-server)的tus(https://github.com/tus/tusd)的服务器实现确实很慢。
在这里您可能会发现不同实现之间的比较(本地运行,同一台机器,相同的输入)
nodejs:
[2019-01-31 16:22:45,578] INFO Uploading 52428800 bytes chunk from offset: 104857600
[2019-01-31 16:22:47,329] INFO Total bytes sent: 157286400 (kb/s: 29930)
开始:
[2019-01-31 16:26:31,894] INFO Uploading 52428800 bytes chunk from offset: 104857600
[2019-01-31 16:26:32,348] INFO Total bytes sent: 209715200 (kb/s: 115639)
我已经研究了tus-node-server代码库,然后构建了一个非常简化的服务器实现(我试图减少可能的开销)。
这是代码:
const fs = require('fs');
const express = require('express');
const app = express();
let offset = 0;
let len = Math.pow(2,30);
app.post('/files',(req,res) => {
console.log("post received");
res.set({
'Location': 'http://localhost:8888/files/test',
'Tus-Resumable': '1.0.0',
});
res.status(201).end();
});
app.options('/files',(req,res) => {
console.log("options received");
res.set({
'Location': 'http://localhost:8888/files/test',
'Tus-Resumable': '1.0.0',
'Tus-Version': '1.0.0,0.2.2,0.2.1'
});
res.status(200).end();
});
app.head('/files/test',(req,res) => {
console.log("options received");
res.set({
'Upload-Offset': offset,
'Upload-Length': len
});
res.status(200).end();
});
app.patch('/files/test',(req, res) => {
let localOffset = parseInt(req.get('Upload-Offset'), 10);
// the file is pre-created
const path = `./file.tmp`;
const options = {
flags: 'r+',
start: localOffset
};
const stream = fs.createWriteStream(path, options);
let new_offset = 0;
req.on('data', (buffer) => {
new_offset += buffer.length;
});
return req.pipe(stream).on('finish', () => {
localOffset += new_offset;
offset = localOffset;
res.set({
'Upload-Offset': offset,
'Upload-Length': len
});
res.status(204).end();
});
});
const host = 'localhost';
const port = 8888;
app.listen(port, host, (err, resp) => {
if(err) {
console.error(err);
return
}
console.log('listening')
});
我认为性能不佳是由于以下代码块引起的:
const stream = fs.createWriteStream(path, options);
req.pipe(stream)
我还使用管道检查了文件副本,并且表现良好(类似于go的实现)
const fs = require('fs');
const path = require('path');
const from = path.normalize(process.argv[2]);
const to = path.normalize(process.argv[3]);
const readOpts = {}; // {highWaterMark: Math.pow(2,16)};
const writeOpts ={}; // {highWaterMark: Math.pow(2,16)};
const startTs = Date.now();
const source = fs.createReadStream(from, readOpts);
const dest = fs.createWriteStream(to, writeOpts);
let offset = 0;
source.on('data', (buffer) => {
offset += buffer.length;
});
dest.on('error', (e) => {
console.log('[FileStore] write: Error', e);
});
source.pipe(dest).on('finish',() => {
const endTs = Date.now();
const kbs = (offset / (endTs - startTs)) / 1000;
console.log("SPEED: ", kbs, offset);
});
因此瓶颈似乎正在处理请求和管道。
请帮助我了解发生的事情以及为什么与go版本相比如此慢
答案 0 :(得分:1)
我认为您在这里遇到highWaterMark
问题。
您的测试之间的差异是由于:
req
的highWaterMark为16 kb createReadStream
有一个highWaterMark of 64 kb 您可以看到附加值:
console.log('readableHighWaterMark', req.readableHighWaterMark);
相反,假设您的网络延迟可以忽略不计(因为您是本地主机),则您
可以尝试创建带有较大水印的writeStream
:
const options = {
flags: 'w',
start: localOffset,
highWaterMark: 1048576
};
const stream = fs.createWriteStream(path, options);
这应该可以加快写入速度,但会占用更多RAM。