Question

我的文件格式如下：

S1A23
0.01,0.01
0.02,0.02
0.03,0.03
S25A123
0.05,0.06
0.07,0.08
S3034A1
1000,0.04
2000,0.08
3000,0.1

我想通过每个“S_ A _”分解它，并计算下面数据的相关系数。到目前为止，我有：

import re
import pandas as pd

test = pd.read_csv("predict.csv",sep=('S\d+A\d+'))

print test

但这只能给我：

  Unnamed: 0     ,
0  0.01,0.01  None
1  0.02,0.02  None
2  0.03,0.03  None
3        NaN     ,
4  0.05,0.06  None
5  0.07,0.08  None
6        NaN     ,
7  1000,0.04  None
8  2000,0.08  None
9   3000,0.1  None

[10 rows x 2 columns]

理想情况下，我希望保留正则表达式分隔符，并且有类似的内容：

S1A23: 1.0
S2A123: 0.86
S303A1: 0.75

这可能吗？

修改
运行大文件（~250k行）时，收到以下错误。这不是数据的问题，因为当我将~250k行分成更小的块时，所有部分都运行良好。

Traceback (most recent call last):
  File "/Users/adamg/PycharmProjects/Subj_AnswerCorrCoef/GetCorrCoef.py", line 15, in <module>
    print(result)
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/base.py", line 35, in __str__
    return self.__bytes__()
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/base.py", line 47, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/series.py", line 857, in __unicode__
    result = self._tidy_repr(min(30, max_rows - 4))
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

我的确切代码是：

import numpy as np
import pandas as pd
import csv
pd.options.display.max_rows = None
fileName = 'keyStrokeFourgram/TESTING1'

df = pd.read_csv(fileName, names=['pause', 'probability'])
mask = df['pause'].str.match('^S\d+_A\d+')
df['S/A'] = (df['pause']
              .where(mask, np.nan)
              .fillna(method='ffill'))
df = df.loc[~mask]

result = df.groupby(['S/A']).apply(lambda grp: grp['pause'].corr(grp['probability']))
print(result)

Answer 1

sep参数用于指定分隔同一行上的值的模式。它不能用于将csv的行分隔为单独的数据帧。

编辑：有一种方法可以使用read_csv将csv读入数据框架。从read_csv should be faster开始，这比使用Python循环（在我的原始答案中所做的）更可取。这可能很重要 - 特别是对于大型csv文件。

import numpy as np
import pandas as pd
df = pd.read_csv("data", names=['x', 'y'])
mask = df['x'].str.match('^S\d+A\d+')         # 1
df['type'] = (df['x']
              .where(mask, np.nan)            # 2
              .fillna(method='ffill'))        # 3
df = df.loc[~mask]                            # 4

result = df.groupby(['type']).apply(lambda grp: grp['x'].corr(grp['y']))
print(result)

产量

type
S1A23      1.000000
S25A123    1.000000
S3034A1    0.981981
dtype: float64

'x'列中包含“type”的行的掩码为True。

In [139]: mask
Out[139]: 
0      True
1     False
2     False
3     False
4      True
5     False
6     False
7      True
8     False
9     False
10    False
Name: x, dtype: bool

df['x'].where(mask, np.nan)返回一个系列，等于df['x']其中掩码为True，否则为np.nan。

使用货币值

向前填充nans

In [141]: df['x'].where(mask, np.nan).fillna(method='ffill')
Out[141]: 
0       S1A23
1       S1A23
2       S1A23
3       S1A23
4     S25A123
5     S25A123
6     S25A123
7     S3034A1
8     S3034A1
9     S3034A1
10    S3034A1
Name: x, dtype: object

仅选择掩码为False

原始答案：

不幸的是，我没有看到将数据文件直接读入相应DataFrame的方法。您需要使用Python循环对行进行一些按摩以使其成为正确的形式。

import pandas as pd
import csv

def to_columns(f):
    val = None
    for row in csv.reader(f):
        if len(row) == 1:
            val = row[0]
        else:
            yield [val] + row

with open('data') as f:
    df = pd.DataFrame.from_records(to_columns(f), columns=['type', 'x', 'y'])

print(df)
result = df.groupby(['type']).apply(lambda grp: grp['x'].corr(grp['y']))
print(result)

在Python中通过Regex打破CSV

1 个答案: