我的Solaris服务器主目录中有两个样本管道分隔文件,如下所示:
FILE1.TXT:
ticker|sedol|cusip|exchnage
ibm |ibm_1|ib |london
hcl | |hcl_02|london
hp |hp_01|hpm |newyork
|lkp |lk_2 |newyork
FILE2.TXT:
exchnage|ticker|sedol|cusip
london |goo |goo_1|gm
london |hcl | |hcl_02
newyork |hp |hp_01|hpm
newyork |tyl | |ty_2
我需要一个结果文件,我们将通过交换分组股票代码,sedol,cusip 的唯一数量:
预期结果文件如下:
exchnage|ticker|sedol|cusip
london |3 |2 |3
newyork |3 |2 |2
我知道使用SQL很容易,但不幸的是数据库无法参与。每个文件大小可能高达300-400 MB。我们最好使用Perl,或者如果有困难则使用Python。主要环境是Solaris,但我们也可以在Unix服务器上试用它。现在添加的需求是“交换”列位置可以在文件中的任何位置。
答案 0 :(得分:1)
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
my %unique_ticker_count;
my %unique_sedol_count;
my %unique_cusip_count;
my @headers = split( '|', <DATA> );
print @headers;
while ( my $line = <DATA> ) {
my ( $exchange, $ticker, $sedol, $cusip ) = split( /\|/, $line );
if ( $ticker =~ m/\w/ ) { $unique_ticker_count{$exchange}{$ticker}++; }
if ( $sedol =~ m/\w/ ) { $unique_sedol_count{$exchange}{$sedol}++; }
if ( $cusip =~ m/\w/ ) { $unique_cusip_count{$exchange}{$cusip}++; }
}
print Dumper \%unique_ticker_count;
foreach my $exchange ( keys %unique_ticker_count ) {
print join( "|",
$exchange,
scalar keys %{ $unique_ticker_count{$exchange} } || 0,
scalar keys %{ $unique_sedol_count{$exchange} } || 0,
scalar keys %{ $unique_cusip_count{$exchange} } || 0,
),
"\n";
}
__DATA__
exchnage|ticker|sedol|cusip
london |ibm |ibm_1|ib
london |hcl | |hcl_02
newyork |hp |hp_01|hpm
newyork |lkp |lk_2 |
london |goo |goo_1|gm
london |hcl | |hcl_02
newyork |hp |hp_01|hpm
newyork |tyl | |ty_2
将打印:
newyork |3|2|2
london |3|2|3
我将留给您处理文件打开和处理 - 这仅仅是您可以采取的方法的说明。
答案 1 :(得分:1)
不是最优雅,但我为您的新要求制作的最快:
import glob
import os
import sys
path = "/tmp"
file_mask = "file*.txt"
results = {}
for file in glob.glob(os.path.join(path, file_mask)):
column_names = {}
exchange_col = None
with open(file, "r") as f:
for line_num, line in enumerate(f.xreadlines()):
# process header
if not line_num:
line_parsed = line.strip().split("|")
for column_num, column in enumerate(line_parsed):
if column.strip() == "exchnage":
exchange_col = column_num
else:
column_names[column_num] = column.strip()
if exchange_col is None:
print "Can't find exchnage field"
sys.exit(1)
continue
line_parsed = line.strip().split("|")
if len(line_parsed) != len(column_names) + 1:
continue
# prepare empty structure for excahnge, if not added yet
if not line_parsed[exchange_col].strip() in results:
results[line_parsed[exchange_col].strip()] = {column_name:set() for column_name in column_names.values()}
# add uniq items to exchange
for column_num, column in enumerate(line_parsed):
column_val = column.strip()
# add only non empty values
if column_val and column_num != exchange_col:
results[line_parsed[exchange_col].strip()][column_names[column_num]].add(column_val)
column_names = column_names.values()
print "exchnage|" + "|".join("%8s" %c for c in column_names)
for exchange, values in results.iteritems():
print "%8s|" % exchange + "|".join("%8s" % str(len(values[column])) for column in column_names)
程序输出(作为输入使用了具有不同列顺序的新文件):
$ python parser.py
exchnage| ticker| sedol| cusip
newyork| 2| 2| 3
london| 3| 2| 3
答案 2 :(得分:0)
我想,首先你需要从文件中创建一个字典列表,例如:
的 { 'exchnage': '伦敦', '股票': 'IBM', 'SEDOL': 'ibm_1', 'CUSIP': 'IB'} 即可。比你需要一个for-each声明将所有值添加到列表中,但只有当值不是None时才是空的&amp;值不在列表中。然后你在列表中得到了所有唯一值,你需要计算它们
您需要对文件中的所有列执行此操作。
之后,您需要将其写入文件中。