在熊猫中优化字符串匹配

时间:2018-08-23 11:23:17

标签: python python-3.x pandas performance

目前,我有以下代码行,尝试在熊猫的一列中进行字符串匹配:

input_supplier = input_supplier[input_supplier['Category Level - 3'].str.contains(category, flags=re.IGNORECASE)]

但是,此操作需要很多时间。熊猫df的大小为:(8098977, 16)

有什么方法可以优化此特定操作?

2 个答案:

答案 0 :(得分:1)

就像Josh Friedlander所说的那样,添加列然后进行过滤应该会更快一些。

len(df3)

9599904

# Creating a column then filtering
start_time = time.time()
search = ['Emma','Ryan','Gerald','Billy','Helen']
df3['search'] = df3['First'].str.contains('|'.join(search))
new_df = df3[df3['search'] == True]
end_time = time.time()
print(f'Elapsed time was {(end_time - start_time)} seconds')

Elapsed time was 6.525546073913574 seconds

只做一个str包含:

start_time = time.time()
search = ['Emma','Ryan','Gerald','Billy','Helen']
input_supplier = df3[df3['First'].str.contains('|'.join(search), flags=re.IGNORECASE)]
end_time = time.time()
print(f'Elapsed time was {(end_time - start_time)} seconds')

Elapsed time was 11.464462518692017 seconds

创建新列并对其进行过滤的速度大约是对str.contains()进行过滤的两倍

答案 1 :(得分:0)

在将搜索列值和类别列表值转换为小写之后,使用快速numpy的“ where”和“ isin”函数。如果列和/或类别列表包含非字符串,请首先转换为字符串。如果要查看索引到搜索列结果的原始数据框中的所有列,请删除最后一行的列标签。

import numpy as np
import pandas as pd
import re

names = np.array(['Walter', 'Emma', 'Gus', 'Ryan', 'Skylar', 'Gerald',
                  'Saul', 'Billy', 'Jesse', 'Helen'] * 1000000)
input_supplier = pd.DataFrame(names, columns=['Category Level - 3'])

len(input_supplier)
10000000

category = ['Emma', 'Ryan', 'Gerald', 'Billy', 'Helen']

方法1(请注意,此方法不能忽略大小写)

%%timeit
input_supplier['search'] = \
    input_supplier['Category Level - 3'].str.contains('|'.join(category))
df1 = input_supplier[input_supplier['search'] == True]

4.42 s ± 37.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

方法2

%%timeit
df2 = input_supplier[input_supplier['Category Level - 3'].str.contains(
    '|'.join(category), flags=re.IGNORECASE)]

5.45 s ± 25.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

numpy方法忽略大小写:

%%timeit
lcase_vals = [x.lower() for x in input_supplier['Category Level - 3']]
category_lcase = [x.lower() for x in category]
df3 = input_supplier.iloc[np.where(np.isin(lcase_vals, category_lcase))[0]]

2.02 s ± 31.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

大小写匹配的情况下的numpy方法:

%%timeit
col_vals = input_supplier['Category Level - 3'].values
df4 = input_supplier.iloc[np.where(np.isin(col_vals, category))[0]]

623 ms ± 1.12 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)