Question

我有~200个文件，想要获取每个文件中的数据，然后在一个.csv文件中显示所有数据。

例如，文件列表是

#targeted folder
a001.opd
a002.opd
a003.opd
...
..
.
a200.opd

每个文件都具有相同的数据结构，如

model <many spaces>          1 <many spaces>       0.003    0.002  # Title
mo(data,1) <many spaces>     1 <many spaces>       0.2      0.0001 # 1
mo(data,1) <many spaces>     2 <many spaces>      -0.1      0.04   # 2
mo(data,1) <many spaces>     3 <many spaces>      -0.4      0.005  # 3
....................................
................
.............                                                      # n-1
......                                                             # n

如果我想在我的grab_result.csv文件中看到一些内容，那么有人如何通过python实现这一目标。

#grab_result.csv  # order will be from left to right that is a001 to a200

a001                                      a002                                             
model      1 0.003 0.002  <empty column>  model      1  0.02   0.1   <empty column> 
mo(data,1) 1 0.2   0.0001 <empty column>  mo(data,1) 1  0.04   0.001 <empty column>
mo(data,1) 2 -0.1  0.04   <empty column>  mo(data,1) 2 -0.145  0.014 <empty column>
mo(data,1) 3 -0.2  0.003  <empty column>  mo(data,1) 3 -0.24   0.06  <empty column>

以下是我所做的代码。

import os

def openfolder(path, outputfile='grab_result.csv'):  # get .opd file from folder and open an output file
    if os.path.isdir(path): 
        fo = open(outputfile, 'wb')
        fo.write('filename') # write title here
        for filename in [os.path.abspath(path)+'\\'+each for each in os.listdir(path) if each.endswith('.opd')]:                         
            return openfile(filename)
    else:
        print "path unavailable"

    openfolder('C:\\path', 'C:\\path\\grab_result.csv')     


def openfile(filename):   # open file.opd
    if os.path.isfile(filename) and filename.endswith('.opd'): 
        return grabdata(open(filename, 'rb').read())         
    else:
        print "invalid file"
        return []       


def grabdata(string):   # start to grab data
    ret = []
    idx_data = string.find('model')
    # then I stop here....

有谁知道如何从这些文件中获取数据？

这是我的示例文件（http://goo.gl/HyT0wM）

Answer 1

如果您有许多包含大量内容的文件，我会使用生成器。这允许不将所有内容加载到内存中。我将如何解决这个问题：

def get_all_files(path):
    ## get a generator with all file names
    import os
    import glob
    return glob.iglob(os.path.join(path,'*.opd'))

def get_all_data(files):
    ## get a generator with all the data from all the files
    for fil in files:
        with open(fil, 'r') as the_file:
            for line in the_file:
                yield line

def write_lines_to_file(lines, outfile):
    with open(outfile, 'w') as the_file:
        for line in lines:
            ## add here an if statement if not all lines should be written to outfile
            the_file.write(line+'\n')

path = 'blah blah'
outfile = 'blah.csv'
files = get_all_files(path)
lines = get_all_data(files)
write_lines_to_file(lines, outfile)

Answer 2

这将是这样的：

def grabdata(filename): 
    # start to grab data 
    matches = []
    with open(filename) as f:            
        for line in f:
            # add if matches:
            if line.startswith("model"): # or: line.find("model") != -1
                matches.append(line.strip())
    return matches

Answer 3

这不是一个答案，只是一个扩展的评论。

为什么你想要从左到右（200 x 5列宽）的结果？如果您要转置数据，它是否会为以后添加其他列提供更大的灵活性？例如：

a001   model   mo(data,1)   mo(data,1)   mo(data,1)
         1          1           2            3
         0.003      0.2        -0.1         -0.2
         0.002      0.0001      0.04         0.003

a002   model   mo(data,1)   mo(data,1)   mo(data,1)
         1          1           2            3
...

200 x 5列宽的难度在于您需要填充列。如果文件缺少信息，那么它可能会抛弃整个结构。您还需要编写由所有200个文件中的单个切片组成的每一行。

从某些文件中抓取数据

3 个答案: