Question

所以，我正在使用一个脚本来计算一个人在行中指定的日期之前的日期出现的次数，并且在第6列中出现1，并且还计算一个次数a person（第7列）出现在行中指定日期之前的日期列表中（注意它们是按时间顺序排序的。）（使用基于零的列引用）

示例数据集

02/01/2005,Data,Class xpv,4,11yo+,4,1,George Smith
02/01/2005,Data,Class xpv,4,11yo+,4,2,Ted James
02/01/2005,Data,Class xpv,4,11yo+,4,3,Emma Lilly
02/01/2005,Data,Class xpv,4,11yo+,4,5,George Smith
02/01/2005,Data,Class xpv,4,11yo+,6,4,Tom Phillips
03/01/2005,Data,Class tn2,4,10yo+,6,2,Tom Phillips
03/01/2005,Data,Class tn2,4,10yo+,6,5,George Smith
03/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips
03/01/2005,Data,Class tn2,4,10yo+,6,1,Emma Lilly
03/01/2005,Data,Class tn2,4,10yo+,6,6,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,6,Ted James
04/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips
04/01/2005,Data,Class tn2,4,10yo+,6,2,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,4,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,1,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,5,Tom Phillips
05/01/2005,Data,Class 22zn,2,10yo+,5,3,Emma Lilly
05/01/2005,Data,Class 22zn,2,10yo+,5,1,Ted James
05/01/2005,Data,Class 22zn,2,10yo+,5,2,George Smith
05/01/2005,Data,Class 22zn,2,10yo+,5,4,Emma Lilly
05/01/2005,Data,Class 22zn,2,10yo+,5,5,Tom Phillips

我正在使用的代码

import csv
import datetime
import copy
from collections import defaultdict

with open(r"C:\Temp\test.csv") as i, open(r"C:\Temp\resuls.csv", "wb") as o:
    rdr = csv.reader(i)
    wrt = csv.writer(o)

    data, currdate = defaultdict(lambda:[0, 0, 0, 0]), None
    for line in rdr:
        date, name = datetime.datetime.strptime(line[0], '%d/%m/%Y'), line[7]

        if date != currdate or not currdate:
            for v in data.itervalues(): v[:2] = v[2:]
            currdate = date

        wrt.writerow(line + data[name][:2])

        data[name][3] += 1
        if line[6] == "1": data[name][2] += 1

返回：

02/01/2005,Data,Class xpv,4,11yo+,4,1,George Smith,0,0
02/01/2005,Data,Class xpv,4,11yo+,4,2,Ted James,0,0
02/01/2005,Data,Class xpv,4,11yo+,4,3,Emma Lilly,0,0
02/01/2005,Data,Class xpv,4,11yo+,4,5,George Smith,0,0
02/01/2005,Data,Class xpv,4,11yo+,6,4,Tom Phillips,0,0
03/01/2005,Data,Class tn2,4,10yo+,6,2,Tom Phillips,0,1
03/01/2005,Data,Class tn2,4,10yo+,6,5,George Smith,1,2
03/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips,0,1
03/01/2005,Data,Class tn2,4,10yo+,6,1,Emma Lilly,0,1
03/01/2005,Data,Class tn2,4,10yo+,6,6,George Smith,1,2
04/01/2005,Data,Class tn2,4,10yo+,6,6,Ted James,0,1
04/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips,0,3
04/01/2005,Data,Class tn2,4,10yo+,6,2,George Smith,1,4
04/01/2005,Data,Class tn2,4,10yo+,6,4,George Smith,1,4
04/01/2005,Data,Class tn2,4,10yo+,6,1,George Smith,1,4
04/01/2005,Data,Class tn2,4,10yo+,6,5,Tom Phillips,0,3
05/01/2005,Data,Class 22zn,2,10yo+,5,3,Emma Lilly,1,2
05/01/2005,Data,Class 22zn,2,10yo+,5,1,Ted James,0,2
05/01/2005,Data,Class 22zn,2,10yo+,5,2,George Smith,2,7
05/01/2005,Data,Class 22zn,2,10yo+,5,4,Emma Lilly,1,2
05/01/2005,Data,Class 22zn,2,10yo+,5,5,Tom Phillips,0,5

最终我会想要对我生成的百分比数据执行chi平方。但是现在我希望能够实现的是能够计算并总结唯一数据类（第2列）中任何一个人的分数几率，并将其作为新列附加到csv。我不确定如果我使用的代码可以编辑，以实现这一个代码。任何有关如何做到最好的建设性意见或建议将不胜感激。

我想要的输出如下：

02/01/2005,Data,Class xpv,4,11yo+,5,1,George Smith,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,2,Ted James,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,3,Emma Lilly,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,5,George Smith,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,4,Tom Phillips,0,0,0
03/01/2005,Data,Class tn2,4,10yo+,5,2,Tom Phillips,0,1,0.2, He gets 0.2 because there was a 1 in 5 chance for previous occurrences on dates prior to today. 1/5
03/01/2005,Data,Class tn2,4,10yo+,5,5,George Smith,1,2,0.4, He gets 0.4 because there was a 2 in 5 chance for previous occurrences on dates prior to today. 2/5
03/01/2005,Data,Class tn2,4,10yo+,5,3,Tom Phillips,0,1,0.2
03/01/2005,Data,Class tn2,4,10yo+,5,1,Emma Lilly,0,1,0.2
03/01/2005,Data,Class tn2,4,10yo+,5,6,George Smith,1,2,0.4
04/01/2005,Data,Class tn2,4,10yo+,6,6,Ted James,0,1,0.2
04/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips,0,3,0.6
04/01/2005,Data,Class tn2,4,10yo+,6,2,George Smith,1,4,0.8
04/01/2005,Data,Class tn2,4,10yo+,6,4,George Smith,1,4,0.8
04/01/2005,Data,Class tn2,4,10yo+,6,1,George Smith,1,4,0.8
04/01/2005,Data,Class tn2,4,10yo+,6,5,Tom Phillips,0,3,0.4
05/01/2005,Data,Class 22zn,2,10yo+,5,3,Emma Lilly,1,2,0.4
05/01/2005,Data,Class 22zn,2,10yo+,5,1,Ted James,0,2,0.366666667
05/01/2005,Data,Class 22zn,2,10yo+,5,2,George Smith,2,7,1.3
05/01/2005,Data,Class 22zn,2,10yo+,5,4,Emma Lilly,1,2,0.4
05/01/2005,Data,Class 22zn,2,10yo+,5,5,Tom Phillips,0,5,0.733333333

Answer 1

这不应该是你问题的完整答案（因为你想要做的事情有点含糊不清），而只是向你展示pandas在这种计算中如何自然适合;您还可以通过名称而不是索引来调用列。

假设您有test.csv这样的文件：

date,x0,cls,x1,x2,x3,tag,name
02/01/2005,Data,Class xpv,4,11yo+,4,1,George Smith
02/01/2005,Data,Class xpv,4,11yo+,4,2,Ted James
02/01/2005,Data,Class xpv,4,11yo+,4,3,Emma Lilly
02/01/2005,Data,Class xpv,4,11yo+,4,5,George Smith
...

我已为每列指定了名称。您可以通过

将此文件读入pandas数据框

import pandas as pd
df = pd.DataFrame.from_csv( 'test.csv', index_col=None )

df将如下所示：

          date    x0         cls  x1     x2  x3  tag          name
0   02/01/2005  Data   Class xpv   4  11yo+   4    1  George Smith
1   02/01/2005  Data   Class xpv   4  11yo+   4    2     Ted James
2   02/01/2005  Data   Class xpv   4  11yo+   4    3    Emma Lilly
3   02/01/2005  Data   Class xpv   4  11yo+   4    5  George Smith
...

我删除了你没有使用的列（这只是为了演示，你不必删除这些列）

df.drop( labels=['x0', 'x1', 'x2', 'x3'], axis=1, inplace=True )

现在df如下所示：

          date         cls  tag          name
0   02/01/2005   Class xpv    1  George Smith
1   02/01/2005   Class xpv    2     Ted James
2   02/01/2005   Class xpv    3    Emma Lilly
3   02/01/2005   Class xpv    5  George Smith
...

假设您想要找到每个人每天之前日期出现的累计次数：

pv = df.pivot_table( cols='name', rows='date', values='tag', aggfunc=len ).shift( 1 ).fillna( 0 ).cumsum( )

api文档（参见here）包含每种方法正在做什么的详细说明。现在你有了一个如下所示的数据透视表pv

date Emma Lilly George Smith Ted James Tom Phillips 02/01/2005 0 0 0 0 03/01/2005 1 2 1 1 04/01/2005 2 4 1 3 05/01/2005 2 7 2 5

或者可以使用groupby：

df.groupby(['date', 'name'])['name'].aggregate(len).unstack( ).shift( 1 ).fillna( 0 ).cumsum( )

要进行相同的计算但仅适用于tag == 1，您可以

idx = df.tag == 1 pv1 = df[ idx ].pivot_table( cols='name', rows='date', values='tag', aggfunc=len ).shift( 1 ).fillna( 0 ).cumsum( )

或使用groupby语法：

df[ df.tag == 1 ].groupby(['date', 'name'])['name'].aggregate(len).unstack( ).shift( 1 ).fillna( 0 ).cumsum( )

将是：

date Emma Lilly George Smith Ted James 02/01/2005 0 0 0 03/01/2005 0 1 0 04/01/2005 1 1 0 05/01/2005 1 2 0

要填写两个新列，我们编写一个辅助函数，如果缺少值，则回退到0：

def lookup( pivot_table, col, idx, fall_back=0 ): try: return pivot_table[ col ][ idx ] except KeyError: return fall_back df[ 'cnt1' ] = [ lookup( pv1, row[ 'name' ], row[ 'date' ] ) for idx, row in df.iterrows( ) ] df[ 'cnt' ] = [ lookup( pv, row[ 'name' ], row[ 'date' ] ) for idx, row in df.iterrows( ) ]

我们得到：

date cls tag name cnt1 cnt 0 02/01/2005 Class xpv 1 George Smith 0 0 1 02/01/2005 Class xpv 2 Ted James 0 0 2 02/01/2005 Class xpv 3 Emma Lilly 0 0 3 02/01/2005 Class xpv 5 George Smith 0 0 4 02/01/2005 Class tn2 4 Tom Phillips 0 0 5 03/01/2005 Class tn2 2 Tom Phillips 0 1 6 03/01/2005 Class tn2 5 George Smith 1 2 7 03/01/2005 Class tn2 3 Tom Phillips 0 1 8 03/01/2005 Class tn2 1 Emma Lilly 0 1 9 03/01/2005 Class tn2 6 George Smith 1 2 10 04/01/2005 Class tn2 6 Ted James 0 1 11 04/01/2005 Class tn2 3 Tom Phillips 0 3 12 04/01/2005 Class tn2 2 George Smith 1 4 13 04/01/2005 Class tn2 4 George Smith 1 4 14 04/01/2005 Class tn2 1 George Smith 1 4 15 04/01/2005 Class tn2 5 Tom Phillips 0 3 16 05/01/2005 Class 22zn 3 Emma Lilly 1 2 17 05/01/2005 Class 22zn 1 Ted James 0 2 18 05/01/2005 Class 22zn 2 George Smith 2 7 19 05/01/2005 Class 22zn 4 Emma Lilly 1 2 20 05/01/2005 Class 22zn 5 Tom Phillips 0 5

如果我知道你是如何计算最后一栏的话，我可以坚持下去。例如为什么“汤姆菲利普斯”在第6排获得0.2？

修改：好的，让我们继续吧。我们需要找出每个人在每个日期出现的次数;这是另一个数据透视表：

appr = df.pivot_table( cols='name', rows='date', values='tag', aggfunc=len ).fillna( 0 )

或

df.groupby( ['date', 'name'] )['name'].aggregate(len).unstack( ).fillna( 0 )

输出：

date Emma Lilly George Smith Ted James Tom Phillips 02/01/2005 1 2 1 1 03/01/2005 1 2 0 2 04/01/2005 0 3 1 2 05/01/2005 2 1 1 1

每个日期有多少人出现：

total_appr = appr.sum( axis=1 )

输出：

date 02/01/2005 5 03/01/2005 5 04/01/2005 6 05/01/2005 5

计算累积分数你可以简单地将每一行除以总数，换一个（因为我们查找以前的日期）并计算累积总和：

frac = appr.apply( lambda x: x / total_appr ).shift( 1 ).fillna( 0 ).cumsum( ) df[ 'frac' ] = [ frac[ row[ 'name' ] ][ row[ 'date' ] ] for idx, row in df.iterrows( ) ]

现在df如下所示：

date cls tag name cnt1 cnt frac 0 02/01/2005 Class xpv 1 George Smith 0 0 0.000000 1 02/01/2005 Class xpv 2 Ted James 0 0 0.000000 2 02/01/2005 Class xpv 3 Emma Lilly 0 0 0.000000 3 02/01/2005 Class xpv 5 George Smith 0 0 0.000000 4 02/01/2005 Class tn2 4 Tom Phillips 0 0 0.000000 5 03/01/2005 Class tn2 2 Tom Phillips 0 1 0.200000 6 03/01/2005 Class tn2 5 George Smith 1 2 0.400000 7 03/01/2005 Class tn2 3 Tom Phillips 0 1 0.200000 8 03/01/2005 Class tn2 1 Emma Lilly 0 1 0.200000 9 03/01/2005 Class tn2 6 George Smith 1 2 0.400000 10 04/01/2005 Class tn2 6 Ted James 0 1 0.200000 11 04/01/2005 Class tn2 3 Tom Phillips 0 3 0.600000 12 04/01/2005 Class tn2 2 George Smith 1 4 0.800000 13 04/01/2005 Class tn2 4 George Smith 1 4 0.800000 14 04/01/2005 Class tn2 1 George Smith 1 4 0.800000 15 04/01/2005 Class tn2 5 Tom Phillips 0 3 0.600000 16 05/01/2005 Class 22zn 3 Emma Lilly 1 2 0.400000 17 05/01/2005 Class 22zn 1 Ted James 0 2 0.366667 18 05/01/2005 Class 22zn 2 George Smith 2 7 1.300000 19 05/01/2005 Class 22zn 4 Emma Lilly 1 2 0.400000 20 05/01/2005 Class 22zn 5 Tom Phillips 0 5 0.933333

我的号码与你的号码在最后一列中的两行不同。所以要么你的计算错了，要么你错误地计算了这两个数字。

Answer 2

这应该是非常简单的，除了不清楚你的意思是“任何一个人在一个独特数据类中的分数几率”。例如，您的数据以数据类xpv的5行开头，其中George Smith出现两次。乔治史密斯想要看到什么“分数机会”？你想为其他人（谁出现一次）看到什么？为什么您的示例输出仅显示xpv行旁边的零？

答案可能取决于日期类别是否会在以后的日期重复出现，以及这对您的计算是否重要;但是如果你能解释你如何计算前5个的值，那么剩下的就会变得清晰。（如果没有，请解释第二组，其中值变为非零。）

PS。也许这在评论的讨论中以某种方式解决，但TL; DNR。如果你可以改进这个问题，那么很容易给出正确的答案。

计算用于卡方检验的先前机会数字

示例数据集

我正在使用的代码

返回：

我想要的输出如下：

2 个答案: