Question

我需要将数据集从键值配对列表（informix dbaccess输出）转换为列式csv。我相当确定这可以通过awk或sed轻松完成。

更新解决方案必须是单行响应。我正在使用NSH（基于ZSH）。因此，某些典型的“基本”命令将不起作用。

这是我的数据样本集：

part_no            100000001
date_part          2010-10-13 12:12:12
history_code       ABCD
user_id            rsmith
other_information   note: Monday, December 10
pool_no            101011777

part_no            100000002
date_part          2010-10-21 12:12:12
history_code       GHIJ
user_id            jsmith
other_information
pool_no            101011888

part_no            100000002
date_part          2010-10-27 12:12:12
history_code       LMNO
user_id            fevers
other_information   [Mail]
pool_no            101011999

part_no            100000003
date_part          2010-11-13 12:12:12
history_code       QXRT
user_id            sjohnson
other_information   note: Tuesday, August 31
pool_no            101011111

我需要它看起来像这样：

part_no,date_part,history_code,user_id,other_information,pool_no
100000001,10/13/2010 12:12:12,ABCD,rsmith,note: Monday, December 10,101011777
100000002,10/21/2010 12:12:12,GHIJ,jsmith,,101011888
100000002,10/27/2010 12:12:12,LMNO,fevers,[Mail],101011999
100000003,11/13/2010 12:12:12,QXRT,sjohnson,note: Tuesday, August 31,101011111

Answer 1

您的问题尚不清楚，但这可能是您要查找的内容：

$ cat tst.awk
BEGIN { RS=""; FS="\n"; OFS=","; ofmt="\"%s\"%s" }
{
   for (i=1; i<=NF; i++) {
       tag = val = $i
       sub(/[[:space:]].*/,"",tag)
       sub(/[^[:space:]]+[[:space:]]+/,"",val)
       tags[i] = tag
       vals[i] = val
    }
}
NR==1 {
    for (i=1; i<=NF; i++) {
        printf ofmt, tags[i], (i<NF ? OFS : ORS)
    }
}
{
    for (i=1; i<=NF; i++) {
        printf ofmt, vals[i], (i<NF ? OFS : ORS)
    }
}

$ awk -f tst.awk file
"part_no","date_part","history_code","user_id","other_information","pool_no"
"100000001","2010-10-13 12:12:12","ABCD","rsmith","note: Monday, December 10","101011777"
"100000002","2010-10-21 12:12:12","GHIJ","jsmith","other_information","101011888"
"100000002","2010-10-27 12:12:12","LMNO","fevers","[Mail]","101011999"
"100000003","2010-11-13 12:12:12","QXRT","sjohnson","note: Tuesday, August 31","101011111"

Answer 2

我将其作为Informix问题而不是Awk问题解决。

使用标准的Informix SQL命令，您也可以创建CSV格式的external table，但是您必须知道可以使用未记录的格式"DB2"：

DROP TABLE IF EXISTS data_table;

CREATE TABLE data_table
(
        part_no            INTEGER,
        date_part          DATETIME YEAR TO SECOND,
        history_code       VARCHAR(4),
        user_id            VARCHAR(32),
        other_information  VARCHAR(64),
        pool_no            INTEGER
);

INSERT INTO data_table VALUES(100000001, "2010-10-13 12:12:12", "ABCD", "rsmith", "note: Monday, December 10", 101011777);
INSERT INTO data_table VALUES(100000002, "2010-10-21 12:12:12", "GHIJ", "jsmith", NULL, 101011888);
INSERT INTO data_table VALUES(100000002, "2010-10-27 12:12:12", "LMNO", "fevers", "[Mail]", 101011999);
INSERT INTO data_table VALUES(100000003, "2010-11-13 12:12:12", "QXRT", "sjohnson", "note: Tuesday, August 31", 101011111);

DROP TABLE IF EXISTS csv_data;
CREATE EXTERNAL TABLE csv_data
(
    part_no            INTEGER,
    date_part          DATETIME YEAR TO SECOND,
    history_code       VARCHAR(4),
    user_id            VARCHAR(32),
    other_information  VARCHAR(64),
    pool_no            INTEGER
)
USING (FORMAT "DB2", DELIMITER ",", DATAFILES("DISK:/tmp/data/csv_data.csv"));

INSERT INTO csv_data
        SELECT part_no, date_part, history_code, user_id, other_information, pool_no
          FROM data_table;

/tmp/data/csv_data.csv的内容如下：

100000001,2010-10-13 12:12:12,"ABCD","rsmith","note: Monday, December 10",101011777
100000002,2010-10-21 12:12:12,"GHIJ","jsmith",,101011888
100000002,2010-10-27 12:12:12,"LMNO","fevers","[Mail]",101011999
100000003,2010-11-13 12:12:12,"QXRT","sjohnson","note: Tuesday, August 31",101011111

UNLOAD格式已转换为CSV

实际上，DB-Access的默认输出不容易进行解析。在某些有限的情况下（例如您所示的情况），它可能是可行的，但是最好使用UNLOAD格式而不是命令行输出，然后将UNLOAD数据格式转换为CSV。

我有一个执行此操作的Perl脚本。它使用Perl Text :: CSV模块处理CSV格式。它不会假装用列名来处理第一行；那些不在UNLOAD格式文件中。

#!/usr/bin/env perl
#
# @(#)$Id: unl2csv.pl,v 1.3 2018/06/29 20:36:58 jleffler Exp $
#
# Convert Informix UNLOAD format to CSV

use strict;
use warnings;
use Text::CSV;
use IO::Wrap;

my $csv = new Text::CSV({ binary => 1 }) or die "Failed to create CSV handle ($!)";
my $dlm = defined $ENV{DBDELIMITER} ? $ENV{DBDELIMITER} : "|";
my $out = wraphandle(\*STDOUT);
my $rgx = qr/((?:[^$dlm]|(?:\\.))*)$dlm/sm;

# $csv->eol("\r\n");

while (my $line = <>)
{
    print "1: $line";
    MultiLine:
    while ($line eq "\\\n" || $line =~ m/[^\\](?:\\\\)*\\$/)
    {
        my $extra = <>;
        last MultiLine unless defined $extra;
        $line .= $extra;
    }
    my @fields = split_unload($line);
    $csv->print($out, \@fields);
}

sub split_unload
{
    my($line) = @_;
    my @fields;
    print "$line";

    while ($line =~ $rgx)
    {
        printf "%d: %s\n", scalar(@fields), $1;
        push @fields, $1;
    }
    return @fields;
}

__END__

=head1 NAME

unl2csv - Convert Informix UNLOAD to CSV format

=head1 SYNOPSIS

unl2csv [file ...]

=head1 DESCRIPTION

The unl2csv program converts a file from Informix UNLOAD file format to
the corresponding CSV (comma separated values) format.

The input delimiter is determined by the environment variable
DBDELIMITER, and defaults to the pipe symbol "|".
It is not assumed that each input line is terminated with a delimiter
(there are two variants of the UNLOAD format, one with and one without
the final delimiter).

=head1 EXAMPLES

Input:

  10|12|excessive|cost \|of, living|
  20|40|bou\\ncing tigger|grrrrrrrr|

Output:

  10,12,"excessive","cost |of, living"
  20,40,"bou\ncing tigger",grrrrrrrr

=head1 PRE-REQUISITES

Text::CSV_XS

=head1 AUTHOR

Jonathan Leffler <jonathan.leffler@hcl.com>

=cut

您将使用这样的命令（通过DB-Access）：

UNLOAD TO "datatable.unl" SELECT * FROM DataTable;

然后运行：

perl unl2csv datatable.unl > datatable.csv

SQLCMD程序

如果您有我的SQLCMD程序（可从软件仓库中的IIUG网站获得，并且与Microsoft的johnny-come-lately完全不相关，并且名称相同），则可以直接卸载为CSV格式：

sqlcmd -d database -F csv -e 'unload to "data_table.csv" select * from data_table'

Answer 3

尝试一下：

cat $file | cut -d ' ' -f 2- | sed 's/^[ \t]*//' | sed 's/$/,/' \
| xargs  | sed 's/ , /\n/g' | sed 's/.$//' | sed 's/, /,/g' \
| sed '1ipart_no,date_part,history_code,user_id,other_information,pool_no'

Answer 4

能否请您尝试以下操作，如果有帮助，请告诉我。

awk -v s1="," '/part_no/ && value{if(header){print header;flag=1;header=""};print value;value=""}  NF{if(!flag){header=(header?header s1 "":"")$1};sub(/^[^[:space:]]+[[:space:]]+/,"");value=value?value s1 $0:$0} END{if(value){print value}}'  Input_file

输出如下。

part_no,date_part,history_code,user_id,other_information,pool_no
100000001,2010-10-13 12:12:12,ABCD,rsmith,note: Monday, December 10,101011777
100000002,2010-10-21 12:12:12,GHIJ,jsmith,,101011888
100000002,2010-10-27 12:12:12,LMNO,fevers,[Mail],101011999
100000003,2010-11-13 12:12:12,QXRT,sjohnson,note: Tuesday, August 31,101011111

现在也添加一种非衬套形式的解决方案。

awk -v s1="," '
/part_no/ && value{
  if(header){
    print header;
    flag=1;
    header=""}
  print value;
  value=""
}
NF{
  if(!flag){
    header=(header?header s1 "":"")$1}
  sub(/^[^[:space:]]+[[:space:]]+/,"")
  value=value?value s1 $0:$0
}
END{
  if(value){
    print value}
}'   Input_file

Answer 5

我知道操作员说awk，但是bash只是坐在那里。

#
# line to be printed
line=""

#
# first value on a line flag
first=""

#
# read the file
while read key val; do
    #
    # if key is empty then the input line is empty.
    if [ "$key" = "" ] ; then
        #
        # skip leading blank lines in the file
        if [ "$line" = "" ] ; then
            continue
        else
            #
            # print and reset the line
            echo $line
            line=""
            first=""
        fi
    else
        #
        # place the first comma after the first value
        if [ "$first" = "" ] ; then
            line="\"$val\""
            first="1"
        else
            line="$line,\"$val\""
        fi
    fi
done < file.txt

#
# print the last line, if there is one
if [ "$line" != "" ] ; then
    echo $line
fi

如何使用AWK将键值配对列表转换为带有列的表？

5 个答案:

UNLOAD格式已转换为CSV

SQLCMD程序