在bash中并行执行作业

时间：2017-11-07 03:18:08

标签： bash unix parallel-processing gnu-parallel

我正在尝试使用gnu-parallel

并行执行某些命令 
cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("aomenc $x.y4m --i444 --enable-qm=1 --qm-min=$i --profile=3 -w $width -h $height -b 10 --end-usage=q --cq-level=$q -o ${x}_${q}_${i}.ivf")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("aomdec ${x}_${q}_${i}.ivf --output-bit-depth=10 -o ${x}_${q}_${i}.y4m")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("ffmpeg -nostats -loglevel 0 -y -i ${x}_${q}_${i}.y4m ${x}_${q}_${i}.png")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("convert ${x}_${q}_${i}.png PNG24:${x}_${q}_${i}_ssimulacra.png")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"

当我执行上面的代码时，转换似乎失败，说它无法找到要转换的png。

执行应按此顺序执行 aomenc aomdec ffmpeg convert

它连续完美地完成

for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
        aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
        ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
        convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
done

这是输出，我只对一些文件有问题，这些文件让我相信这个bug与并行有关

convert: unable to open image `bench.png_20_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_20_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_20_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_10.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_10.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_10_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_6.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_6.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_6_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_26_6.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_26_6.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_26_6_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_26_10.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_26_10.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_26_10_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.

y4m文件存在，我通过手动为受影响的文件运行命令来确认它。为什么不是ffmpeg将所有y4m转换为png？

1 个答案:

答案 0 :(得分：2)

如果您的序列代码有效，那么这也应该是：

#!/bin/bash

doit() {
    x="$1"
    q="$2"
    i="$3"
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
    aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
    ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
    convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
}
export -f doit

parallel doit {1} {2} {3} ::: "$x" ::: {10..40..2} ::: {4..12..2}

在使用GNU Parallel之前，您可以手动测试单个组合：

doit "$x" 10 12

通常不需要构建命令数组。

如果您在函数中有所需的本地（非导出）变量，则可以使用env_parallel代替parallel：

#!/bin/bash

. `which env_parallel.bash`

# Do this in a clean environment without any variables set
# It only needs to be run once, so it does not have to be part of the script
env_parallel --record-env

# Set the local variables after doing --record-env
myvar=...

doit() {
    x="$1"
    q="$2"
    i="$3"
    echo "$myvar"
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
    aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
    ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
    convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
}

# --env _ = ignore variables defined before --record-env
env_parallel --env _ doit {1} {2} {3} ::: "$x" ::: {10..40..2} ::: {4..12..2}

考虑花一个小时走过https://www.gnu.org/software/parallel/parallel_tutorial.html你的命令行会爱你。

相关问题
最新问题