Question

我是python的新手，我正在试图弄清楚如何在每个时间步长的基础上加载包含数据块的数据文件，例如：

TIME:,0
Q01 : A:,-10.7436,0.000536907,-0.00963283,0.00102934
Q02 : B:,0,0.0168694,-0.000413983,0.00345921
Q03 : C:,0.0566665
Q04 : D:,0.074456
Q05 : E:,0.077456
Q06 : F:,0.0744835
Q07 : G:,0.140448
Q08 : H:,-0.123968
Q09 : I:,0
Q10 : J:,0.00204377,0.0109621,-0.0539183,0.000708574
Q11 : K:,-2.86115e-17,0.00947104,0.0145645,1.05458e-16,-1.90972e-17,-0.00947859
Q12 : L:,-0.0036781,0.00161254
Q13 : M:,-0.00941257,0.000249692,-0.0046302,-0.00162387,0.000981709,-0.0135982,-0.0223496,-0.00872062,0.00548815,0.0114075,.........,-0.00196206
Q14 : N:,3797, 66558
Q15 : O:,0.0579981
Q16 : P:,0
Q17 : Q:,625

TIME:,0.1
Q01 : A:,-10.563,0.000636907,-0.00963283,0.00102934
Q02 : B:,0,0.01665694
Q03 : C:,0.786,-0.000666,0.6555
Q04 : D:,0.87,0.96
Q05 : E:,0.077456
Q06 : F:,0.07447835
Q07 : G:,0.140448
Q08 : H:,-0.123968
Q09 : I:,0
Q10 : J:,0.00204377,0.0109621,-0.0539183,0.000708574
Q11 : K:,-2.86115e-17,0.00947104,0.0145645,1.05458e-16,-1.90972e-17,-0.00947859
Q12 : L:,-0.0036781,0.00161254
Q13 : M:,-0.00941257,0.000249692,-0.0046302,-0.00162387,0.000981709,-0.0135982,-0.0223496,-0.00872062,0.00548815,0.0114075,.........,-0.00196206
Q14 : N:,3797, 66558
Q15 : O:,0.0579981
Q16 : P:,0,2,4
Q17 : Q:,786

每个块包含许多变量，其中可能包含非常不同数量的数据列。每个变量的列数可能会在每个时间步长块中发生变化，但每个块的变量数在每个时间步长中都是相同的，并且始终知道导出了多少变量。没有关于数据文件中数据块数（时间步长）的信息。

读取数据后，应按每个时间步长的变量格式加载：

Time:  |  A:                                           |  B:
0      |  -10.7436,0.000536907,-0.00963283,0.00102934  |  ........
0.1    |  -10.563,0.000636907,-0.00963283,0.00102934   |  ........
0.2    |  ......                                       |  ........

如果每个时间步长的数据列数相同，并且每个变量的数据列相同，这将是一个非常简单的问题。

我想我需要逐行读取文件，在两个循环中，每个块一个，然后在每个块内一次，然后将输入存储在一个数组中（追加？）。由于我不熟悉python和numpy，因此每行的列数变化让我有点难过。

如果有人能指出我正确的方向，比如我应该用什么功能来相对有效地做到这一点，那就太棒了。

Answer 1

一种非常完美的方法是通过阅读文本文件并在扫描时创建dict结构。以下是可以实现目标的示例（根据您提供的输入）：

time = 0
output = {}
with open('path_to_file','r') as input_file:
    for line in input_file:
        line = line.strip('\n')
        if 'TIME' in line:
            time = line.split(',')[1]
            output[time] = {}
        else:
            col_name = line.split(':')[1].strip()
            col_value = line.split(':')[2].strip(',') 
            output[time][col_name] = col_value

这将传递一个output对象，该对象是具有以下结构的字典：

output = {
'0': {'A': '-10.7436,0.000536907,-0.00963283,0.00102934',
      'B': '0,0.0168694,-0.000413983,0.00345921',
      ...
      'Q': '625'},

'0.1': {'A': '-10.563,0.000636907,-0.00963283,0.00102934',
        'B': '0,0.01665694',
        ...
        'Q': '786'}
}

我认为我们认为符合您的要求。要访问此词典中的一个值，您应使用value = output['0.1']['A']，这将产生'-10.563,0.000636907,-0.00963283,0.00102934'

Answer 2

import pandas as pd
res = {}
TIME = None

# by default lazy line read
for line in open('file.txt'):
    parts = line.strip().split(':')
    map(str.strip, parts)
    if len(parts) and parts[0] == 'TIME':
        TIME = parts[1].strip(',')
        res[TIME] = {}
        print('New time section start {}'.format(TIME))
        # here you can stop and work with data from previou period
        continue

    if len(parts) <= 1:
        continue
    res[TIME][parts[1].lstrip()] = parts[2].strip(',').split(',')

df = pd.DataFrame.from_dict(res, 'columns')
# for example for TIME 0
dfZero = df['0']
print(dfZero)


df = pd.DataFrame.from_dict(res, 'index')

dfA = df['A']
print(dfA)

Answer 3

文件test.csv：

1,2,3
1,2,3,4
1,2,3,4,5
1,2
1,2,3,4

处理数据：

my_cols = ["A", "B", "C", "D", "E"]
pd.read_csv("test.csv", names=my_cols, engine='python')

<强>输出：

   A  B   C   D   E
0  1  2   3 NaN NaN
1  1  2   3   4 NaN
2  1  2   3   4   5
3  1  2 NaN NaN NaN
4  1  2   3   4 NaN

或者您可以使用names参数。

例如：

1,2,1
2,3,4,2,3
1,2,3,3
1,2,3,4,5,6

如果您阅读它，您将收到以下错误：

>>> pd.read_csv(r'D:/Temp/test.csv')
Traceback (most recent call last):
...
Expected 5 fields in line 4, saw 6

但如果您传递names个参数，则会得到结果：

>>> pd.read_csv(r'D:/Temp/test.csv', names=list('ABCDEF'))

输出：

   A  B  C   D   E   F
0  1  2  1 NaN NaN NaN
1  2  3  4   2   3 NaN
2  1  2  3   3 NaN NaN
3  1  2  3   4   5   6

希望它有所帮助。

Answer 4

此阅读器类似于@Lucas's - 每个块都是保存在按时间键入的元字典中的字典。它可能是一个列表。

blocks = {}
with open('stack37354745.txt') as f:
    for line in f:
        line = line.strip()
        if len(line)==0: continue  # blank line
        d = line.split(':')
        if len(d)==2 and d[0]=='TIME':  # new block
            time = float(d[1].strip(','))
            blocks[time] = data = {}
        else:
           key = d[1].strip()  # e.g. A, B, C
           value = d[2].strip(',').split(',')
           value = np.array(value, dtype=float) # assume valid numeric list
           data[key] = value

可以使用以下迭代获取，显示和重新组织值：

for time in blocks:
    b = blocks[time]
    print('TIME: %s'%time)
    for k in b:
        print('%4s: %s'%(k,b[k]))

产生

TIME: 0.0
   C: [ 0.0566665]
   G: [ 0.140448]
   A: [ -1.07436000e+01   5.36907000e-04  -9.63283000e-03   1.02934000e-03]
   ...
   K: [ -2.86115000e-17   9.47104000e-03   1.45645000e-02   1.05458000e-16
  -1.90972000e-17  -9.47859000e-03]

TIME: 0.1
   C: [  7.86000000e-01  -6.66000000e-04   6.55500000e-01]
   G: [ 0.140448]
   A: [ -1.05630000e+01   6.36907000e-04  -9.63283000e-03   1.02934000e-03]
   ...
   K: [ -2.86115000e-17   9.47104000e-03   1.45645000e-02   1.05458000e-16
  -1.90972000e-17  -9.47859000e-03]

（我从其中一条数据行中删除了....）

或以准表格式

fmt = '%10s | %s | %s | %s'
print(fmt%('Time','B','D','E'))
for time in blocks:
    b =  blocks[time]
    # print(list(b.keys()))
    print(fmt%(time, b['B'], b['D'],b['E']))

制造

      Time | B | D | E
       0.0 | [ 0.          0.0168694  -0.00041398  0.00345921] | [ 0.074456] | [ 0.077456]
       0.1 | [ 0.          0.01665694] | [ 0.87  0.96] | [ 0.077456]

由于像B这样的变量可以有不同的长度，因此很难像某种二维数组那样在一段时间内收集值。

通常，最简单的方法是首先关注将文件加载到某种Python结构中。这种行为几乎必须用Python编写，逐行迭代（除非你让pandas为你做）。

完成后，您可以通过多种不同方式重新组织日期以满足您的需求。对于这个变量，以某种方式瞄准矩形numpy数组是没有意义的。

Python导入文本文件，其中每行具有不同的列数

4 个答案: