Question

例如，假设我想计算10个BIG文件的行数并打印总数。

for f in files
do
    #this creates a background process for each file
    wc -l $f | awk '{print $1}' &
done

我正在尝试类似的事情：

for f in files
do
    #this does not work :/
    n=$( expr $(wc -l $f | awk '{print $1}') + $n ) &
done

echo $n

Answer 1

我终于找到了一个使用匿名管道和bash的工作解决方案：

#!/bin/bash

# this executes a separate shell and opens a new pipe, where the 
# reading endpoint is fd 3 in our shell and the writing endpoint
# stdout of the other process. Note that you don't need the 
# background operator (&) as exec starts a completely independent process.
exec 3< <(./a.sh 2&1)


# ... do other stuff


# write the contents of the pipe to a variable. If the other process
# hasn't already terminated, cat will block.
output=$(cat <&3)

Answer 2

你应该使用gnu parallel：

find . -maxdepth 1 -type f | parallel --gnu 'wc -l' | awk 'BEGIN {n=0} {n += $1} END {print n}'

或者以并行模式运行xargs：

find . -maxdepth 1 -type f | xargs -n1 -P4 wc -l | awk 'BEGIN {n=0} {n += $1} END {print n}'

如果这不符合您的需求，另一个选择是写入临时文件。如果您不想写入磁盘，只需写入/ dev / shm即可。这是大多数Linux系统上的ramdisk。

#!/bin/bash

declare -a temp_files

count=0
for f in *
do
  if [[ -f "$f" ]]; then
    temp_files[$count]="$(mktemp /dev/shm/${f}-XXXXXX)"
    ((count++))
  fi
done

count=0
for f in *
do
  if [[ -f "$f" ]]; then
    cat "$f" | wc -l > "${temp_files[$count]}" &
    ((count++))
  fi
done

wait

cat "${temp_files[@]}" | awk 'BEGIN {n=0} {n += $1} END {print n}'

for tf in "${temp_files[@]}"
do
  rm "$tf"
done

顺便说一句，这可以作为map-reduce使用wc进行映射并使用awk进行缩减。

Answer 3

您可以将其写入文件或更好，一旦数据到达就收听fifo。

以下是一个关于它们如何工作的小例子：

# create the fifo
mkfifo test

# listen to it
while true; do if read line <test; then echo $line; fi done

# in another shell 
echo 'hi there'

# notice 'hi there' being printed in the first shell

所以你可以

for f in files
do
    #this creates a background process for each file
    wc -l $f | awk '{print $1}' > fifo &
done

并听听fifo的尺寸。

如何从后台进程linux shell脚本获得结果？

3 个答案: