Question

这是一个数据挖掘任务，我们可以自动对提取质量进行评分。有一个黄金标准csv可能包含看起来像

的字段

golden_standard.csv

| id | description             | amount  | date       |
|----|-------------------------|---------|------------|
| 1  | Some description.       | $150.54 | 12/12/2012 |
| 2  | Some other description. | $200    | 10/10/2015 |
| 3  | Other description.      | $25     | 11/11/2014 |
| 4  | My description          | $11.35  | 01/01/2015 |
| 5  | Your description.       | $20     | 03/03/2013 |

，然后有两个可能的提取结果文件：

extract1.csv

| id | description             | date       |
|----|-------------------------|------------|
| 1  | Some description.       | 12/12/2012 |
| 2  | Some other description. | 10/10/2015 |
| 3  | Other description.      | 11/11/2014 |
| 4  | 122333222233332221      | 11/11/2014 |
| 5  | Your description.       | 03/03/2013 |

extract2.csv

| id | description             | amount  | date       |
|----|-------------------------|---------|------------|
| 1  | Some description.       | $150.54 | 12/12/2012 |
| 2  | Some other description. | $200    | 10/10/2015 |
| -  | ----------------------- | -----   | ---------- |
| 5  | Your description.       | $20     | 03/03/2013 |

extract3.csv

| Garbage  | More Garbage       |
| Garbage  | More Garbage       |

我希望我的程序报告提取1缺少一列，并且第2列中的值没有正确匹配。

对于第二种情况，我缺少条目，并且有些行都不匹配。

在最后一种情况下，结果csv全部搞砸了，但我仍然希望程序检测到一些有意义的异常。

有没有人在python中有一些快速而聪明的方法来进行这种比较？

我有常规的，逐行的逐列和逐列迭代方式，我可以在这里发布，但我认为可能有更快，更优雅的Pythonic方式来做到这一点。

非常感谢任何帮助。

Answer 1

免责声明：我的方法使用pandas库。

首先，数据设置。

<强> gold_std.csv

id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
4,My description,$11.35,01/01/2015
5,Your description.,$20,03/03/2013

<强> extract1.csv

id,description,date
1,Some description.,12/12/2012
2,Some other description.,10/10/2015
3,Other description.,11/11/2014
4,122333222233332221,11/11/2014
5,Your description.,03/03/2013

<强> extract2.csv

id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
5,Your description.,$20,03/03/2013

其次，代码。

import pandas as pd

def compare_extract(extract_name, reference='gold_std.csv'):

    gold = pd.read_csv(reference)
    ext = pd.read_csv(extract_name)

    gc = set(gold.columns)
    header = ext.columns
    extc = set(header)

    if gc != extc:
        missing = ", ".join(list(gc - extc))
        print "Extract has the following missing columns: {}".format(missing)
    else:
        print "Extract has the same column as standard. Checking for abberant rows..."
        gold_list = gold.values.tolist()
        ext_list = ext.values.tolist()
        # Somewhat non-pandaic approach because possible no same IDs so we're relying
        # on set operations instead. A bit hackish, actually.
        diff = list(set(map(tuple, gold_list)) - set(map(tuple, ext_list)))
        df = pd.DataFrame(diff, columns=header)
        print "The following rows are not in the extract: "
        print df

第三，测试运行。

e1 = 'extract1.csv'
compare_extract(e1)
# Extract has the following missing columns: amount

e2 = 'extract2.csv'
compare_extract(e2)
# Extract has the same column as standard. Checking for abberant rows...
# The following rows are not in the extract: 
#    id     description  amount        date
# 0   4  My description  $11.35  01/01/2015

最后，最后一次提取有点武断。我认为对于那个你最好不要编写非pandas算法。

在Python中比较黄金标准csv文件和提取的值csv文件

1 个答案: