答案 0 :(得分:41)
使用drop_duplicates
pd.concat([df1,df2]).drop_duplicates(keep=False)
Update :
<强>
Above method only working for those dataframes they do not have duplicate itself, For example
强>
df1=pd.DataFrame({'A':[1,2,3,3],'B':[2,3,4,4]})
df2=pd.DataFrame({'A':[1],'B':[2]})
输出如下,这是错误的
输出错误:
pd.concat([df1, df2]).drop_duplicates(keep=False)
Out[655]:
A B
1 2 3
正确输出
Out[656]:
A B
1 2 3
2 3 4
3 3 4
如何实现?
方法1:将isin
与tuple
df1[~df1.apply(tuple,1).isin(df2.apply(tuple,1))]
Out[657]:
A B
1 2 3
2 3 4
3 3 4
方法2:merge
indicator
df1.merge(df2,indicator = True, how='left').loc[lambda x : x['_merge']!='both']
Out[421]:
A B _merge
1 2 3 left_only
2 3 4 left_only
3 3 4 left_only
答案 1 :(得分:7)
对于行,请尝试此操作,并将cols
设置为您要比较的列列表:
m = df1.merge(df2, on=cols, how='outer', suffixes=['', '_'], indicator=True)
对于列,请尝试:
set(df1.columns).symmetric_difference(df2.columns)
答案 2 :(得分:1)
import pandas as pd
# given
df1 = pd.DataFrame({'Name':['John','Mike','Smith','Wale','Marry','Tom','Menda','Bolt','Yuswa',],
'Age':[23,45,12,34,27,44,28,39,40]})
df2 = pd.DataFrame({'Name':['John','Smith','Wale','Tom','Menda','Yuswa',],
'Age':[23,12,34,44,28,40]})
# find elements in df1 that are not in df2
df_1notin2 = df1[~(df1['Name'].isin(df2['Name']) & df1['Age'].isin(df2['Age']))].reset_index(drop=True)
# output:
print('df1\n', df1)
print('df2\n', df2)
print('df_1notin2\n', df_1notin2)
# df1
# Age Name
# 0 23 John
# 1 45 Mike
# 2 12 Smith
# 3 34 Wale
# 4 27 Marry
# 5 44 Tom
# 6 28 Menda
# 7 39 Bolt
# 8 40 Yuswa
# df2
# Age Name
# 0 23 John
# 1 12 Smith
# 2 34 Wale
# 3 44 Tom
# 4 28 Menda
# 5 40 Yuswa
# df_1notin2
# Age Name
# 0 45 Mike
# 1 27 Marry
# 2 39 Bolt
答案 3 :(得分:1)
Accepted answer方法1不适用于内部具有NaN的数据帧,例如pd.np.nan != pd.np.nan
。我不确定这是否是最好的方法,但是可以通过
df1[~df1.astype(str).apply(tuple, 1).isin(df2.astype(str).apply(tuple, 1))]
答案 4 :(得分:1)
我有一个棘手的方法。首先我们将“名称”设置为问题给出的两个数据帧的索引。由于两个df中具有相同的“名称”,因此我们可以将“较小”的df索引从“更大的df。 这是代码。
df1.set_index('Name',inplace=True)
df2.set_index('Name',inplace=True)
newdf=df1.drop(df2.index)
答案 5 :(得分:0)
@liangli解决方案的一个小变化,不需要更改现有数据帧的索引:
newdf = df1.drop(df1.join(df2.set_index('Name').index))
答案 6 :(得分:0)
也许是一个简单的单行代码,具有相同或不同的列名。即使df2 ['Name2']包含重复值也可以使用。
{{1}}
答案 7 :(得分:0)
按索引查找差异。假设df1是df2的子集,并且在进行子集设置时索引会结转
df1.loc[set(df1.index).symmetric_difference(set(df2.index))].dropna()
# Example
df1 = pd.DataFrame({"gender":np.random.choice(['m','f'],size=5), "subject":np.random.choice(["bio","phy","chem"],size=5)}, index = [1,2,3,4,5])
df2 = df1.loc[[1,3,5]]
df1
gender subject
1 f bio
2 m chem
3 f phy
4 m bio
5 f bio
df2
gender subject
1 f bio
3 f phy
5 f bio
df3 = df1.loc[set(df1.index).symmetric_difference(set(df2.index))].dropna()
df3
gender subject
2 m chem
4 m bio
答案 8 :(得分:0)
除了可接受的答案外,我还想提出一个更广泛的解决方案,该解决方案可以找到两个具有任意index
/ columns
的数据帧的 2D集合差异(对于两个数据集可能并不完全相同)。同样,该方法还可以为float
元素设置公差以进行数据帧比较(它使用np.isclose
)
import numpy as np
import pandas as pd
def get_dataframe_setdiff2d(df_new: pd.DataFrame,
df_old: pd.DataFrame,
rtol=1e-03, atol=1e-05) -> pd.DataFrame:
"""Returns set difference of two pandas DataFrames"""
union_index = np.union1d(df_new.index, df_old.index)
union_columns = np.union1d(df_new.columns, df_old.columns)
new = df_new.reindex(index=union_index, columns=union_columns)
old = df_old.reindex(index=union_index, columns=union_columns)
mask_diff = ~np.isclose(new, old, rtol, atol)
df_bool = pd.DataFrame(mask_diff, union_index, union_columns)
df_diff = pd.concat([new[df_bool].stack(),
old[df_bool].stack()], axis=1)
df_diff.columns = ["New", "Old"]
return df_diff
示例:
In [1]
df1 = pd.DataFrame({'A':[2,1,2],'C':[2,1,2]})
df2 = pd.DataFrame({'A':[1,1],'B':[1,1]})
print("df1:\n", df1, "\n")
print("df2:\n", df2, "\n")
diff = get_dataframe_setdiff2d(df1, df2)
print("diff:\n", diff, "\n")
Out [1]
df1:
A C
0 2 2
1 1 1
2 2 2
df2:
A B
0 1 1
1 1 1
diff:
New Old
0 A 2.0 1.0
B NaN 1.0
C 2.0 NaN
1 B NaN 1.0
C 1.0 NaN
2 A 2.0 NaN
C 2.0 NaN
答案 9 :(得分:0)
如上所述here
df1[~df1.apply(tuple,1).isin(df2.apply(tuple,1))]
是正确的解决方案,但如果出现错误,则会产生错误的输出
df1=pd.DataFrame({'A':[1],'B':[2]})
df2=pd.DataFrame({'A':[1,2,3,3],'B':[2,3,4,4]})
在这种情况下,上述解决方案将给出
Empty DataFrame ,相反,您应在从每个datframe中删除重复项后使用concat
方法。
使用concate with drop_duplicates
df1=df1.drop_duplicates(keep="first")
df2=df2.drop_duplicates(keep="first")
pd.concat([df1,df2]).drop_duplicates(keep=False)
答案 10 :(得分:0)
使用 lambda 函数,您可以过滤具有 _merge
值 “left_only”
的行,以获取 df1
中 df2
中缺少的所有行
df3 = df1.merge(df2, how = 'outer' ,indicator=True).loc[lambda x :x['_merge']=='left_only']
df
答案 11 :(得分:0)
当一侧有重复项而另一侧至少有一个重复项时,我在处理重复项时遇到了问题,因此我使用 Counter.collections
进行了更好的区分,确保双方具有相同的计数。这不会返回重复项,但如果双方的计数相同,则不会返回任何项。
from collections import Counter
def diff(df1, df2, on=None):
"""
:param on: same as pandas.df.merge(on) (a list of columns)
"""
on = on if on else df1.columns
df1on = df1[on]
df2on = df2[on]
c1 = Counter(df1on.apply(tuple, 'columns'))
c2 = Counter(df2on.apply(tuple, 'columns'))
c1c2 = c1-c2
c2c1 = c2-c1
df1ondf2on = pd.DataFrame(list(c1c2.elements()), columns=on)
df2ondf1on = pd.DataFrame(list(c2c1.elements()), columns=on)
df1df2 = df1.merge(df1ondf2on).drop_duplicates(subset=on)
df2df1 = df2.merge(df2ondf1on).drop_duplicates(subset=on)
return pd.concat([df1df2, df2df1])
> df1 = pd.DataFrame({'a': [1, 1, 3, 4, 4]})
> df2 = pd.DataFrame({'a': [1, 2, 3, 4, 4]})
> diff(df1, df2)
a
0 1
0 2