Question

我在本地目录中有这个〜/ Report：

Rep_{ReportType}_{Date}_{Seq}.csv

Rep_0001_20150102_0.csv
Rep_0001_20150102_1.csv
Rep_0102_20150102_0.csv
Rep_0503_20150102_0.csv
Rep_0503_20150102_0.csv

使用shell脚本，

如何从固定批量大小的本地目录中获取多个文件？
如何按报告类型将文件分隔/分组（0001个文件组合在一起，0102组合在一起，0503组合在一起等）

我将为每个组/报告类型生成一个序列文件（使用forqlift）。输出将是Report0001.seq，Report0102.seq，Report0503.seq（3个序列文件）。我将保存到另一个目录。

注意：在序列文件中，键是csv（Rep_0001_20150102.csv）的文件名，值是文件的内容。它存储为[String，BytesWritable]。

这是我的代码：

1  reportTypes=(0001 0102 8902)
2
3  # collect all files matching expression into an array
4  filesWithDir=(~/Report/Rep_[0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-1].csv)
5
6  # take only the first hundred
7  filesWithDir =( "${filesWithDir[@]:0:100}" )
8
9  # files="${filesWithDir[@]##*/}" #### commented out since forqlift cannot create sequence file without the path/to/file
10 # echo ${files[@]}
11
12 shopt -s nullglob
13
14 # Line 21 is commented out since it has a bug. It collects files in
15 # current directory when it should be filtering the "files array" created
16 # in line 7
17
18
19 for i in ${reportTypes[@]}; do
20   printf -v val '%04d' "$i"
21   # files=("Rep_${val}_"*.csv) 
     # solution to BUG: (filter files array)
     groupFiles=( $( for j in ${filesWithDir[@]} ; do echo $j ; done | grep ${val} ) )
22
23   # Generate sequence file for EACH Report Type
24   forqlift create --file="Report${val}.seq" "${groupFiles[@]}"
25 done

（注意：序列文件输出应该在当前目录中，而不是在〜/ Report中）

Answer 1

很容易只取一个数组的子集：

# collect all files matching expression into an array
files=( ~/Report/Rep_[0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].csv )

# take only the first hundred
files=( "${files[@]:0:100}" )

第二部分比较棘手：Bash有关联数组（＆＃34; maps＆＃34;），但是可以存储在数组中的唯一合法值是字符串 - 而不是其他数组 - 所以你可以＆＃39; t将文件名列表存储为与单个条目相关联的值（不将数组与字符串串行化 - 这是一个非常棘手的安全操作，因为UNIX中的文件路径可以包含除NUL之外的任何字符，包括换行符）

然后，根据需要生成数组会更好。

shopt -s nullglob # allow a glob to expand to zero arguments
for ((i=1; i<=1000; i++)); do
  printf -v val '%04d' "$i"     # pad digits: 12 -> 0012
  files=( "Rep_${val}_"*.csv )  # collect files that match

  ## emit NUL-separated list of files, if any were found
  #(( ${#files[@]} )) && printf '%s\0' "${files[@]}" >"Reports.$val.txt"

  # Create a sequence file with forqlift
  forqlift create --file="Reports-${val}.seq" "${files[@]}"

done

如果您真的不想这样做，那么我们可以将使用namevars进行重定向的东西放在一起：

#!/bin/bash
# This only works with bash 4.3
re='^REP_([[:digit:]]{4})_[[:digit:]]{8}.csv$'
counter=0
for f in *; do
  [[ $f =~ $re ]] || continue            # skip files not matching regex
  if ((++counter > 100)); then break; fi # stop after 100 files
  group=${BASH_REMATCH[1]}               # retrieve first regex group
  declare -g -a "array${group}"          # declare an array
  declare -n group_arr="array${group}"   # redirect group_arr to that array
  group_arr+=( "$f" )                    # append to the array
done

for varname in "${!array@}"; do
  declare -n group_arr="$varname"

  ## NUL-delimited form
  #printf '%s\0' "${group_arr[@]}" \
  #  >"collection${varname#array}"        # write to files named collection0001, etc.

  # forqlift sequence file form
  forqlift create --file="Reports-${varname#array}.seq" "${group_arr[@]}"
done

Answer 2

我会远离shell脚本并开始关注perl。

#!/usr/bin/env perl
use strict;
use warnings;

my %groups; 
while ( my $filename = glob ( "~/Reports/Rep_*.csv" ) ) {
     my ( $group, $id ) = ( $filename =~ m,/Rep_(\d{4})_(\d{8})\.csv$, ); 
     next unless $group; #undefined means it didn't match;

     #anything past 100 in a group is discarded:
     if ( @{$groups{$group}} < 100 ) { 
         push ( @{$groups{$group}}, $filename ); 
     }
}

foreach my $group ( keys %groups ) { 
   print "$group contains:\n";
   print join ("\n", @{$groups{$group});
}

Answer 3

另一种方法是使用regexp破坏一些bash命令。见下面的实施

# Explanation:
# ls -p = List all files and directories in local directory by path
# grep -v / = ignore subdirectories
# grep "^Rep_\d{4}_\d{8}\.csv$" = Look for files matching your regexp
# tail -100 = get 100 results
for file in $(ls -p | grep -v / | grep "^Rep_\d{4}_\d{8}\.csv$" | tail -100);
    do echo $file;

    # Use reg exp to extract the desired sequence
    re="^Rep_([[:digit:]]{4})_([[:digit:]]{8}).csv$";
    if [[ $name =~ $re ]]; then
        sequence = ${BASH_REMATCH[1};
        # Didn't end up using date, but in case you want it
        # date = ${BASH_REMATCH[2]};

        # Just in case the sequence file doesn't exist
        if [ ! -f "$sequence" ] ; then
            touch "$sequence"
        fi

        # Output/Concat your filename to the sequence file, which you can
        # read in later to do whatever administrative tasks you wish to do    
        # to them
        echo "$file" >> "$sequence"
    fi

done;

Shell脚本：隔离多个文件

3 个答案: