如何并行处理函数,循环

时间:2018-05-30 21:53:55

标签: bash loops parallel-processing sh gnu-parallel

所以我有这个函数,我想让这个函数同时运行它包含的所有东西。到目前为止,它不起作用,根据其他消息来源,这就是你如何做到这一点。如果它不是并行的,函数本身就可以工作。

#!/bin/bash
foo () {
    cd ${HOME}/sh/path/to/script/execute
    for f in *.sh; do  #goes to "execute" directory and executes all 
    #scripts the current directory "execute" basically run-parts without cron

        cd ~/sh/path/to/script
        while IFS= read -r l1 #Line 1 in master.txt
              IFS= read -r l2 #Line 2 in master.txt
              IFS= read -r l3 #Line 3 in master.txt
        do
        cd /dev/shm/arb
        echo ${l1} > arg.txt & echo ${l2} > arg2.txt & echo ${l3} > arg3.txt 
        cd ${HOME}/sh/path/to/script/execute 
        bash -H ${f} #executes all scripts inside "execute" folder
        cd ~/sh/path/to/script/here
        ./here.sh &
        cd ~/sh/path/to/script &
        done <master.txt
    done

}

export -f foo
parallel ::: foo

结果

#No result at all....., just buffers. htop doesn't acknowledge any 
#processes, and when this runs its pretty taxing on the cores.

master.txt内容

如果这是相关的:

apple_fruit
apple_veggie
veggie_fruit
#apple changes
pear_fruit
pear_veggie
veggie_fruit
#pear changes
cucumber_fruit
...

我很擅长使用并行,并且不知道它在高级(和基本)情况下是如何工作的,所以循环会干扰吗?如果确实存在干扰,是否有解决方法?

1 个答案:

答案 0 :(得分:1)

结果可能类似于:

inner() {
  script="$1"
  parallel -N3 "'$script' {}; here.sh  {}"  :::: master.txt
}
export -f inner
parallel inner ::: ${HOME}/sh/path/to/script/execute/*.sh

这将调用${HOME}/sh/path/to/script/execute/(和here.sh)中的每个脚本,其中包含来自master.txt的3个参数,如下所示:

${HOME}/sh/path/to/script/execute/script1.sh apple_fruit apple_veggie veggie_fruit

您需要更改脚本,以便:

  • 他们从命令行获取参数(不是来自arg.txtarg2.txtarg3.txt)。
  • 他们将输出发送到stdout