Question

我有一个工作解决方案，从客户端sftp服务器下载压缩的固定长度字段文本文件，使用密码解压缩，然后在文件上运行gnu awk将其转换为管道分隔的文本文件，然后清理自己之后。

Bash脚本代码：

#!/bin/bash
export ZipPassword=********
export SSHPASS=********
export WorkPath=/Users/administrator/Documents/Work/
export ArcPath=/Users/administrator/Documents/Work/archive/
export DownPath=/Users/administrator/Documents/Work/down/
export InPath=/Users/administrator/Documents/Work/input/
export ReadyPath=/Users/administrator/Documents/Work/preproc/
export OutPath=/Users/administrator/Documents/Work/Output/
export AwkPath=/Users/administrator/Documents/Work/scpost.awk


cd $DownPath

sshpass -e sftp -oBatchMode=no -b - ****@*****.*******.*** << !
    cd /frommbi
    get *.zip
    rm *.zip
    exit
!


for f in *.zip
do 
    cp -v "$f" "$InPath"
    cp -v "$f" "$ArcPath"
    rm *.zip
done    

shopt -s nullglob dotglob     # To include hidden files
files=($InPath*)
if [ ${#files[@]} -gt 0 ]; then


unzip -P $ZipPassword $InPath*.zip -d $ReadyPath


for f in $ReadyPath
do
    export PathName=/Users/administrator/Documents/Work/PreProc/*.TXT
    echo $PathName
    export FileName=`basename $PathName`
    echo $FileName
    echo $OutPath$FileName

awk -f $AwkPath $PathName > $OutPath$FileName

done



rm -f $InPath*
rm -f $ReadyPath*

fi

awk文件内容：

BEGIN{FIELDWIDTHS=" 3 2 2 18 5 9 10 10 10 14 16 30 30 30 30 30 30 30 30 45 45 45 45 45 45 45 45 16 28 6 1 1 3 2 6 2 4 3 2 30 3 3 3 40 6 5 6 3 3 3 40 6 5 6 3 3 3 40 6 5 6 3 3 3 40 6 5 6 3 3 3 40 6 5 6 3 3 3 40 6 5 6 3 3 3 40 6 5 6 3 3 3 40 6 5 6 20 7 20 2 6 13 6 6 6 32 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 40 2 6 20 30 11 12 3 1 14 14 1 4 4 4 4 4 4 4 12 28 30 8 2 1 8 8 8 8 8 10 12 8 130 1 7 65 3 82 512 528 1 "; 
OFS="|";
}
{
for (i=1;i<=NF;i++) gsub (/^ */,"",$i);for(i=1;i<=NF;i++) gsub("^[ \t]*|[ \t]*$","",$i);
}
{
print$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$27,$28,$29,$30,$31,$32,$33,$34,$35,$36,$37,$38,$39,$40,$41,$42,$43,$44,$45,$46,$47,$48,$49,$50,$51,$52,$53,$54,$55,$56,$57,$58,$59,$60,$61,$62,$63,$64,$65,$66,$67,$68,$69,$70,$71,$72,$73,$74,$75,$76,$77,$78,$79,$80,$81,$82,$83,$84,$85,$86,$87,$88,$89,$90,$91,$92,$93,$94,$95,$96,$97,$98,$99,$100,$101,$102,$103,$104,$105,$106,$107,$108,$109,$110,$111,$112,$113,$114,$115,$116,$117,$118,$119,$120,$121,$122,$123,$124,$125,$126,$127,$128,$129,$130,$131,$132,$133,$134,$135,$136,$137,$138,$139,$140,$141,$142,$143,$144,$145,$146,$147,$148,$149,$150,$151,$152,$153,$154,$155,$156,$157,$158,$159,$160,$161,$162,$163,$164,$165,$166,$167,$168,$169,$170,$171
}

在此处识别字段名称是数字，以便稍后在数据库中进行映射。

我已经安装了jq来处理从管道分离数据到json的转换，但我还没有能够正确理解语法。

修改后的bash脚本内容在第52-56行：

#!/bin/bash
export ZipPassword=********
export SSHPASS=********
export WorkPath=/Users/administrator/Documents/Work/
export ArcPath=/Users/administrator/Documents/Work/archive/
export DownPath=/Users/administrator/Documents/Work/down/
export InPath=/Users/administrator/Documents/Work/input/
export ReadyPath=/Users/administrator/Documents/Work/preproc/
export OutPath=/Users/administrator/Documents/Work/Output/
export AwkPath=/Users/administrator/Documents/Work/scpost.awk
export JsonPath=/Users/administrator/Documents/Work/JSON/


cd $DownPath

sshpass -e sftp -oBatchMode=no -b - ****@*****.*******.*** << !
    cd /frommbi
    get *.zip
    rm *.zip
    exit
!


for f in *.zip
do 
    cp -v "$f" "$InPath"
    cp -v "$f" "$ArcPath"
    rm *.zip
done    

shopt -s nullglob dotglob     # To include hidden files
files=($InPath*)
if [ ${#files[@]} -gt 0 ]; then


unzip -P $ZipPassword $InPath*.zip -d $ReadyPath


for f in $ReadyPath
do
    export PathName=/Users/administrator/Documents/Work/PreProc/*.TXT
    echo $PathName
    export FileName=`basename $PathName`
    echo $FileName
    echo $OutPath$FileName

awk -f $AwkPath $PathName > $OutPath$FileName

done
chmod 776 $OutPath$FileName

jq -Rn  --slurp --raw-input --raw-output \'
( input  | split("|") ) as $keys |
( inputs | split("|") ) as $vals |
[[$keys, $vals] | transpose[] | {key:.[0],value:.[1]}] | from_entries
' $OutPath$FileName > $JsonPath$FileName



rm -f $InPath*
rm -f $ReadyPath*
rm -f $JsonPath*


fi

有人可以帮忙吗？在你问之前，我正在使用这种方法来实现纯粹的转换速度。我的Mac Pro可以在大约20秒内转换100,000个2850个字符记录，并且每天都会这样做。转换为json将大大加快该过程的下一步。

Answer 1

你快到了。由于您使用的是input和inputs（这绝对是正确的方法），因此您不希望“啜饮”该文件。

jq  -nrR '
 ( input  | split("|") ) as $keys
 | ( inputs | split("|") ) as $vals
 | [[$keys, $vals] | transpose[] | {key:.[0], value:.[1]|tonumber}]
 | from_entries
'

顺便说一句，您可以轻松地将awk + jq步骤组合成一个awk或一个jq步骤。这样做可以节省大量不必要的修改。如果你选择坚持使用awk，我会专注于缩短那个可笑的长“打印1美元，2美元......”的声明。（“打印0美元”还不够吗？）

awk的FIELDWIDTHS当然很方便，因此在下一节中，将介绍一个jq过滤器，用于根据输入字符串发出数组以及有关字段宽度的信息。

使用jq

解析固定长度字段

# Given a string, emit a stream of the fields defined by the array of widths
def fixedfields(widths):
  foreach widths[] as $w ({s:.}; (.field = .s[:$w]) | (.s |= .[$w:]); .field);

如果你的jq没有foreach，这里有另一种实现方式：

def fixedfields(widths):
  def do_while(cond; f; g): def r: select(cond) | f | (g, r); r;
  {s:., w: widths}
  | do_while(.w|length > 0;
             .w[0] as $w | {s: .s[$w:], w: .w[1:], field: .s[:$w] };
             .field);

将文本从固定长度字段转换为json

1 个答案:

使用jq