在unix中使用gnuparallel时的并发最大进程限制

Question

我正在使用gnu parallel来启动一些并发进程。以下是我正在使用的命令。

parallel 'node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES' ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ::: 1 2 3 4 5 6 | grep "Query Executed"|wc -l

按照预期，它应同时触发120个并行进程。但是当我执行ps命令时，我看到只有90个进程正在运行。以下是输出。

[prod PTEST1] parunkarthick@dev118:~> ps -ef|grep node
2951      66965 196290 15 07:50 pts/25   00:00:00 perl /usr/bin/parallel node  /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ::: 1 2 3 4 5 6
2951      67082  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 1
2951      67087  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 2
2951      67088  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 3
2951      67093  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 4
2951      67098  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 5
2951      67103  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 1 6
2951      67108  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 1
2951      67113  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 2
2951      67118  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 3
2951      67123  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 4
2951      67128  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 5
2951      67133  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 2 6
2951      67138  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 1
2951      67143  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 2
2951      67148  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 3
2951      67153  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 4
2951      67158  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 5
2951      67163  66965 25 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 3 6
2951      67168  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 1
2951      67173  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 2
2951      67178  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 3
2951      67183  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 4
2951      67188  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 5
2951      67193  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 4 6
2951      67198  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 1
2951      67203  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 2
2951      67208  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 3
2951      67213  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 4
2951      67218  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 5
2951      67223  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 5 6
2951      67228  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 1
2951      67233  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 2
2951      67238  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 3
2951      67243  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 4
2951      67248  66965 25 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 5
2951      67253  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 6 6
2951      67258  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 1
2951      67263  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 2
2951      67268  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 3
2951      67273  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 4
2951      67278  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 5
2951      67283  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 7 6
2951      67288  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 1
2951      67293  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 2
2951      67298  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 3
2951      67303  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 4
2951      67308  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 5
2951      67313  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 8 6
2951      67318  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 1
2951      67323  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 2
2951      67328  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 3
2951      67333  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 4
2951      67338  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 5
2951      67343  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 9 6
2951      67348  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 1
2951      67353  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 2
2951      67358  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 3
2951      67363  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 4
2951      67368  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 5
2951      67373  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 10 6
2951      67378  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 1
2951      67383  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 2
2951      67388  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 3
2951      67393  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 4
2951      67398  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 5
2951      67403  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 11 6
2951      67404  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 1
2951      67409  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 2
2951      67418  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 3
2951      67423  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 4
2951      67428  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 5
2951      67433  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 12 6
2951      67438  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 1
2951      67443  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 2
2951      67448  66965 24 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 3
2951      67453  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 4
2951      67458  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 5
2951      67463  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 13 6
2951      67468  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 1
2951      67473  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 2
2951      67478  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 3
2951      67483  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 4
2951      67488  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 5
2951      67493  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 14 6
2951      67498  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 1
2951      67503  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 2
2951      67508  66965 23 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 3
2951      67513  66965 22 07:50 pts/25   00:00:00 node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES 15 4
2951      67523  66905  0 07:50 pts/0    00:00:00 grep node
[prod PTEST1] parunkarthick@dev118:~> ps -ef|grep node|wc -l
90

为什么只有90个进程同时被触发？

当我检查ulimt -u时，它的限制为2000，所以真的无法理解为什么它只产生了90个并行进程。

[prod PTEST1] parunkarthick@dev118:~> ulimit -u
2000

任何帮助都会非常感激!!!

Answer 1

默认情况下， GNU Parallel 将使用一定程度的并行度，该并行度与您可以测试的CPU核心数相匹配：

parallel --number-of-cores

如果要运行不同数量的并行作业，请使用-j参数，例如。对于200个并行作业，请使用：

parallel -j 200 ...

或者，如果您希望并行工作比核心多50％，请使用百分比：

parallel -j 150% ...

顺便说一句，如果您使用bash，则使用以下语法可能会使您的命令变得不那么笨拙：

parallel  'node  /home/XYZ.js F_CHARGES' ::: {1..20} ::: {1..6}