在unix shell脚本中并行处理untar / remove

Question

问题：

我想解压一个tarfile，里面有很多tar文件并删除所有tar文件中的文件，我希望所有这些进程在Unix bash脚本中并行运行。

条件：

1. 如果任何untar / remove进程有任何错误，脚本应该返回错误。
2. 只有在所有N（解压和删除）流程成功完成后才能返回成功。

提议的解决方案：

 mkdir a
 tar -C a -xvf b.tar
 cd a
 for i in *
 do
 rm -r $i &
 done

Answer 1

如果你安装了GNU Parallel http://www.gnu.org/software/parallel/，你可以这样做：

tar xvf foo.tgz | perl -ne 'print $l;$l=$_;END{print $l}' | parallel rm

如果你没有足够的空间来提取完整​​的tar.gz文件，那么它很有用，但你需要在解压缩时处理文件：

tar xvf foo.tgz | perl -ne 'print $l;$l=$_;END{print $l}' | parallel do_stuff {}\; rm {}

您可以通过以下方式安装GNU Parallel：

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

观看GNU Parallel的介绍视频了解详情： https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Answer 2

mkdir a
tar -C a -xvf b.tar
 cd a
 success=$(for i in *
 do
 rm -r $i || echo failed & # if a job fails false will be echoed
 done
 wait)
 # if any of the jobs failed, success will be set to a value other than ""
 [[ -z "$success" ]] && exit 0 || exit 1

Answer 3

答案tar xvf a.tar | tac | xargs -P 4 rm -rv的灵感来自Burton Samograd关于xargs -P的评论

$ mkdir -p a/b/c/d
mkdir: created directory `a'
mkdir: created directory `a/b'
mkdir: created directory `a/b/c'
mkdir: created directory `a/b/c/d'

$ touch a/1 a/2 a/3 a/b/4 a/b/5

$ tar cf a.tar a

$ rm -rfv a
removed directory: `a/b/c/d'
removed directory: `a/b/c'
removed `a/b/4'
removed `a/b/5'
removed directory: `a/b'
removed `a/3'
removed `a/1'
removed `a/2'
removed directory: `a'

$ tar xvf a.tar | tac | xargs -P 4 rm -rv
removed `a/2'
removed `a/1'
removed `a/3'
removed `a/b/5'
removed `a/b/4'
removed directory: `a/b/c/d'
removed directory: `a/b/c'
removed directory: `a/b'
removed directory: `a'