我尝试使用以下命令并行发送4个spark作业,并在开始新步骤之前等待所有这些作业完成。但是我注意到$cmd_trainSparkModelx
命令在'xargs'中是空的。如何将它们传递到xargs
?
eval $cmd_prepare_step
xargs -P 4 -I {} sh -c 'eval "$1"' - {} <<'EOF'
#eval "$cmd_trainSparkModel1"
#eval "$cmd_trainSparkModel2"
#eval "$cmd_trainSparkModel3"
#eval "$cmd_trainSparkModel4"
echo "$cmd_trainSparkModel1"
echo "$cmd_trainSparkModel2"
echo "$cmd_trainSparkModel3"
echo "$cmd_trainSparkModel4"
EOF
echo "finished training"
eval $cmd_postTraining_step
答案 0 :(得分:0)
以下命令有效。但是仍然想看看如何使xargs起作用,因为它还可以控制同时运行多少个作业。
echo "$cmd_trainSparkModel1" &
p1=&!
echo "$cmd_trainSparkModel2" &
p2=&!
echo "$cmd_trainSparkModel3" &
p3=&!
echo "$cmd_trainSparkModel4" &
p4=&!
wait $p1 $p2 $p3 $p4