并行化awk脚本

时间:2015-08-30 03:59:29

标签: bash awk parallel-processing gnu-parallel

我正在尝试并行化以下脚本:

$ awk -F , '$3 > 25 && $3 < 26' data_temp.csv | head

...我正在获得所需的输出。 (cat data_temp.csv | awk -F , '$3 > 25 && $3 < 26' | head相同。)到目前为止我的尝试:

$ parallel "awk -F , '$3 > 25 && $3 < 26' data_temp.csv" | head
parallel: Warning: Input is read from the terminal.
parallel: Warning: Only experts do this on purpose. Press CTRL-D to exit.

$ cat data_temp.csv | parallel --pipe awk -F , \'$3 > 25 && $3 < 26\' | awk -F , '$3 > 25 && $3 < 26' | head
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
# repeated for what looks like every line

1 个答案:

答案 0 :(得分:3)

未测试:

cat data_temp.csv |
  parallel -k -q --block 100M --pipe awk -F , '$3 > 25 && $3 < 26' |
  head
parallel -k -q --block 100M --pipepart -a data_temp.csv awk -F , '$3 > 25 && $3 < 26' |
  head