python如何获取多列下的所有行对之间的差异

时间:2018-08-01 04:59:22

标签: python-3.x pandas

我有两个CSV文件,并且两个文件都具有多列和多行,我期待着获得这两个文件所有行之间的差异。让我们假设如果文件之间的Asset Tag Number中存在差异,则以任何形式突出显示差异(可能是粗体值或适当的东西)。此外,这里有一个Serial Number的键是唯一的这两个文件。因此,最好将行的差异放入new.csv文件中,并在删除相同行的同时突出显示差异。

仅此而已,我的两个文件都具有超过100列。

我的实际数据列在两个csv文件中均如下所示。

Columns: [Asset Tag Number_a, Serial Number_a, System Name_a, Domain_a, System manufacturer_a, Model Name_a, System Type_a, Critical Level_a, Purpose Level 1_a, Purpose2_a, ShareIndv_a, Site_a, Building_a, Room_a, Rack_a, serverCostCenter_a, User ID   BU Grp Mgr_a, OS Name_a, OS Version_a, OS Type_a, Service Pack_a, Notification Group_a, Off The Network_a, First Name_a, Last Name_a, Manager Name_a, Status_a, BU Cost Center_a, BU CC Description_a, Organization Name_a, Higher Level BU_a, Business Contact_a, Description_a, Asset Type_a, System Type SW_a, Server _a, Host ID(Unix)_a, IP Address_a, MAC Address_a, Installed RAM_a, Disk Capacity_a, Installed Disk_a, Server Status _a, High Level Status_a, Lifecycle Status_a, EndOfLifeDate_a, Last Audit_a, AltVersion_a, BIOS Vendor_a, BIOS Version_a, BIOS Release Date_a, SMBIOS Enabled_a, SMBios Version_a, Region_a, Currency_a, Acquisition Cost USD_a, Net Book Value USD_a, CPU Type_a, CPU Speed_a, Acquisition Date_a, Age_a, DateModified_a, Altiris Exception_a, Inventory Owner_a, Last Logon User_a, Inventory Owner Last Logon User_a, Client Date_a, Reporting Status_a, Contact Status_a, Comments_a, Exception Reason_a, DNR_a, Asset Tag Number_b, Serial Number_b, System Name_b, Domain_b, System manufacturer_b, Model Name_b, System Type_b, Critical Level_b, Purpose Level 1_b, Purpose2_b, ShareIndv_b, Site_b, Building_b, Room_b, Rack_b, serverCostCenter_b, User ID   BU Grp Mgr_b, OS Name_b, OS Version_b, OS Type_b, Service Pack_b, Notification Group_b, Off The Network_b, First Name_b, Last Name_b, Manager Name_b, Status_b, BU Cost Center_b, ...]
Index: []

作为熊猫的新手学习者,我使用了几种编码方法,但似乎不太适合,因此寻求慷慨的帮助和建议。

1)尝试了第一个代码。

#!/grid/common/pkgs/python/v3.6.1/bin/python3
import pandas as pd

A = pd.read_csv('a.csv', index_col=0)
B = pd.read_csv('b.csv', index_col=0)

C = pd.merge(left=A,right=B, how='outer', left_index=True, right_index=True, suffixes=['_a', '_b'])

not_in_a = C.drop( A.index )
not_in_b = C.drop( B.index )

not_in_a.to_csv('not_in_a.csv')
not_in_b.to_csv('not_in_b.csv')

2)尝试了另一段代码,但是输出的宽度是如此之大,很难阅读,而此代码段应删除重复的代码,并且仅打印不同的代码。

from __future__ import print_function
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
import csv
import pandas as pd


##### Python pandas, widen output display to see more columns. ####
pd.set_option('display.height', None)
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('expand_frame_repr', True)

a = pd.read_csv('a.csv')
b = pd.read_csv('b.csv')
c = pd.concat([a,b], axis=0)

c.drop_duplicates(keep='first', inplace=True)
c.reset_index(drop=True, inplace=True)
print(c)

我做了一些Google搜索,发现了关于该主题的一些堆栈溢出讨论。但是,线程中有一些不错的解决方案,但是我觉得没有什么可以满足我的要求,因此我在这里发布了。

3)另一个与python集配合使用的代码部分有效..

#!/grid/common/pkgs/python/v3.6.1/bin/python3
import os
orig = open('aa.csv','r')
new = open('bb.csv','r')
bigb = set(new) - set(orig)
print(bigb)
# Write to output file
with open('different.csv', 'w') as file_out:
    for line in bigb:
        file_out.write(line)
    orig.close()
    new.close()
    file_out.close()

下面有两个示例文件供您参考,它们看起来与我的数据相似,我们可以将Serial Number作为输出逻辑和代码的键。

下面是我的两个csv文件file1.csv和file2.csv

  

文件1:

wrkStaId                     Asset Tag Number  Serial Number System Name
                                                              mac-ymatsuok2
                                                              PC-ABNER-W10
                                                              PC-ADAMLIN-W10                                                                                              
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}    1234     ser123         sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}    3456     ser124         10210277      
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}    456      ser345         A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A}    4485     ser900         dgs1sj
  

样本文件2:

    wrkStaId                Asset Tag Number Serial Number  System Name
                                                            mac-ymatsuok2
                                                            PC-Karn-W10
                                                            PC-ADAMLIN-W10
                                                            PC-ADRIANA-W10
   {ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}   1234 ser123     sfreder
   {8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}   3456 ser124     10210277
   {E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}   1709 ser345     A313819
   {445EC096-A70C-47D1-91FF-0DFE747F762A}   4485 ser900     dgs1sj
  

所需结果:您希望如何表示差异,因为这些   是非数字值。您是否要同时打印两行   放入一个新文件中,如果相同则将其删除?

ANS:Yes

所需的输出。

文件1中的差异(不在文件2中)

wrkStaId                     Asset Tag Number  Serial Number System Name
                                                              PC-ABNER-W10                                                                                                
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}    456      ser345         A313819

文件2中的差异(文件1中没有)

    wrkStaId                Asset Tag Number Serial Number  System Name
                                                            PC-Karn-W10
                                                            PC-ADRIANA-W10
   {E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}   1709 ser345     A313819

非常感谢@ w-m,但是我仍然希望能散布SO专家的更多想法。

1 个答案:

答案 0 :(得分:3)

您的数据似乎包含两部分:System Name列表,然后是行表。由于结构大不相同,因此建议您将数据分成System Name和完整行的列表,然后分别处理。

首先提取System Name列表:

l1 = df1[df1.wrkStaId == ""].System_Name
l2 = df2[df2.wrkStaId == ""].System_Name

您可以使用Python设置差异代码来获得差异:

>>> set(l1).difference(set(l2))
{'PC-ABNER-W10'}
>>> set(l2).difference(set(l1))
{'PC-ADRIANA-W10', 'PC-Karn-W10'}

现在删除空的wrkStaId条目:

df1 = df1[df1.wrkStaId != ""].set_index("wrkStaId")
df2 = df2[df1.wrkStaId != ""].set_index("wrkStaId")

其余数据现在包含以wrkStaId作为索引的完整行。

df1:

                                        Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}            1234.0        ser123     sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}            3456.0        ser124    10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}             456.0        ser345     A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A}            4485.0        ser900      dgs1sj

df2:

                                        Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193}            1234.0        ser123     sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24}            3456.0        ser124    10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}            1709.0        ser345     A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A}            4485.0        ser900      dgs1sj

您现在可以像this一样对df熊猫进行设置差异:

>>> df1[~df1.isin(df2).all(1)]
                                        Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}             456.0        ser345     A313819

>>> df2[~df2.isin(df1).all(1)]
                                            Asset_Tag_Number Serial_Number System_Name
wrkStaId                                                                          
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41}            1709.0        ser345     A313819

您可能需要稍微修改一下代码才能准确获得所需的内容,但是我希望这可以使您继续前进。