Question

I'm working with FIND command with directory exclusion on an image compression tool jpeg-recompress and used xargs to speed up the process.

The command line is:

find /path/dir -type f -name '*.jpg' -not -path '*/imdb/*' | xargs -P 50 -I {} jpeg-recompress --quality high --min 60 --method smallfry --strip \{} \{} \;

Error occured after some minutes:

xargs: jpeg-recompress: terminated by signal 11

It's only happened when working on large directory which consists of hundreds of thousands images. I have searched that signal 11 related with segmentation faults, but I don't understand them quite well. The only thing I know is that presumably I am trying to access some memory I shouldn't.

Please help, I don't really understand the code and don't understand what I am doing wrong.

Thanks for your help.

Answer 1

错误来自jpeg-recompress而非xargs：jpeg-recompress因SIGSEGV死亡。分段错误通常仅在程序中存在错误时才会发生-通常是错误的指针。

我相信这是由于jpeg-recompress中的错误所致，该错误是由单个略微损坏的jpeg文件触发的（或者恰好具有jpeg-recompress所不希望的格式）。

这也解释了为什么您只能在包含数十万个图像的大型目录中看到此图像：其中一个略有损坏。

一种识别文件的方法是使用GNU Parallel的--joblog和--halt：

find /path/dir -type f -name '*.jpg' -not -path '*/imdb/*' |
  parallel --joblog my.log --halt now,fail=1 jpeg-recompress --quality high --min 60 --method smallfry --strip \{} \{}

然后，您应该在my.log的最后一行找到损坏文件的名称。

如果您认为可能还有更多损坏的文件：

find /path/dir -type f -name '*.jpg' -not -path '*/imdb/*' |
  parallel --joblog my.log jpeg-recompress --quality high --min 60 --method smallfry --strip \{} \{}
grep -P '\t11\t' my.log

Shell Error xargs: terminated by signal 11

1 个答案: