我正在使用dedupe python library。
对于example this，任何代码示例都可以使用。
假设我有一个受过训练的deduper，并用它来成功地对数据集进行重复数据删除。
现在，我向数据集添加一行。
我要检查此新行是否重复。
有没有办法在重复数据删除中做到这一点（无需重新分类整个数据集）？
更新：
我尝试了@libreneitor的建议，但得到了No records have been blocked together. Is the data you are trying to match like the data you trained on?，这是我的代码（csv file）：
import csv
import exampleIO
import dedupe
def canonicalImport(filename):
preProcess = exampleIO.preProcess
data_d = {}
with open(filename) as f:
reader = csv.DictReader(f)
for (i, row) in enumerate(reader):
clean_row = {k: preProcess(v) for (k, v) in
viewitems(row)}
data_d[i] = clean_row
return data_d, reader.fieldnames
raw_data = 'tests/datasets/restaurant-nophone-training.csv'
data_d, header = canonicalImport(raw_data)
training_pairs = dedupe.trainingDataDedupe(data_d, 'unique_id', 5000)
fields = [{'field': 'name', 'type': 'String'},
{'field': 'name', 'type': 'Exact'},
{'field': 'address', 'type': 'String'},
{'field': 'cuisine', 'type': 'ShortString',
'has missing': True},
{'field': 'city', 'type': 'ShortString'}
]
deduper = dedupe.Gazetteer(fields, num_cores=5)
deduper.sample(data_d, 10000)
deduper.markPairs(training_pairs)
deduper.train(index_predicates=False)
alpha = deduper.threshold(data_d, 1)
data_d_test = {}
data_d_test[0] = data_d[0]
del data_d[0];
clustered_dupes = deduper.match(data_d, threshold=alpha)
clustered_dupes2 = deduper.match(data_d_test, threshold=alpha) <- exception here