Question

有没有人知道如何从ALL删除pandas.DataFrame重复的更好策略？

我知道df.drop_duplicates()，请参阅以下示例：

In [340]: import pandas as pd, string, random

In [341]: a = [''.join([random.choice(string.ascii_letters+string.digits) for _ in range(4)]) for _ in range(5)]

In [342]: b = [''.join([random.choice(string.digits) for _ in range(4)]) for i in range(5)]

In [343]: df1 = pd.DataFrame([a,b],index=list('ab')).T

In [344]: df1 = df1.append(df1.loc[1:3,:])

In [345]: df1.index = range(len(df1))

In [346]: df1 = df1.append(df1.loc[1:3,:])

In [347]: df1
Out[347]: 
      a     b
0  r4fb  4179
1  sv5e  8092
2  Oyeh  8788
3  fAdu  4018
4  PxKX  2818
5  sv5e  8092
6  Oyeh  8788
7  fAdu  4018
1  sv5e  8092
2  Oyeh  8788
3  fAdu  4018

In [348]: df1.drop_duplicates()
Out[348]: 
      a     b
0  r4fb  4179
1  sv5e  8092
2  Oyeh  8788
3  fAdu  4018
4  PxKX  2818

请注意，这不会删除ALL个重复项，即删除每个下一个非唯一行，但保留原始行的完整行...

我目前的策略和预期结果如下：

In [349]: same_first = df1.duplicated(subset=['a','b'])

In [350]: same_last = df1.duplicated(subset=['a','b'], take_last=True)

In [351]: rm_lst = ~(same_first|same_last)

In [352]: df1[rm_lst]
Out[352]: 
      a     b
0  r4fb  4179
4  PxKX  2818

请注意，现在只有真正的unique行不受影响。

有没有更好的方法来获得相同的结果，也许是因为我错过了？

谢谢。

Answer 1

这是一个单行，但不是那么可读，基本上它测试每列的值计数等于1，过滤结果列表并使用索引作为布尔indec：

In [260]:

df1[df1.a.isin((df1.a.value_counts()[df1.a.value_counts() == 1]).index) & (df1.b.isin((df1.b.value_counts()[df1.b.value_counts() == 1]).index))]
Out[260]:
      a     b
0  mlmv  3869
4  LPNz  4109

打破这一点将显示正在发生的声明：

In [261]:
# gengerate a series of the value counts
df1.a.value_counts()

Out[261]:
qPyr    3
ms7I    3
aOuL    3
LPNz    1
mlmv    1
dtype: int64

In [262]:
# we are only interested in the ones that have a unique value, this generates a boolean index we can use to index into the above series
df1.a.value_counts()[df1.a.value_counts() == 1]

Out[262]:
LPNz    1
mlmv    1
dtype: int64

In [264]:
# now use isin on the the result above, but we compare the values against the index of the result above
df1.a.isin((df1.a.value_counts()[df1.a.value_counts() == 1]).index)
Out[264]:
0     True
1    False
2    False
3    False
4     True
5    False
6    False
7    False
1    False
2    False
3    False
Name: a, dtype: bool

从pandas.DataFrame中删除所有重复项的更好策略？

1 个答案: