我希望基于另一个只有三列的较小数据框来过滤大型数据框（数百万行）：ID，Start，End。
以下是我整理的内容（有效），但似乎select t1.*,
CASE
WHEN
REGEXP_LIKE( t1.cellname, '^[^a-zA-Z]*' || REGEXP_SUBSTR( t1.tablename, '[a-zA-Z]{1,3}$' ) )
AND REGEXP_LIKE( t2.cellname, '^[^a-zA-Z]*' || REGEXP_SUBSTR( t2.tablename, '[a-zA-Z]{1,3}$' ) )
THEN 'pass'
ELSE 'fail'
END result
FROM mytable t1
INNER JOIN mtable t2 ON t1.cell_name = t2.cell_name AND t2.source = 'SC2'
WHERE t1.source='SRC1';
或groupby()可能会更快。
设置：
np.where工作代码
import pandas as pd
import io
csv = io.StringIO(u'''
time id num
2018-01-01 00:00:00 A 1
2018-01-01 01:00:00 A 2
2018-01-01 02:00:00 A 3
2018-01-01 03:00:00 A 4
2018-01-01 04:00:00 A 5
2018-01-01 05:00:00 A 6
2018-01-01 06:00:00 A 6
2018-01-03 07:00:00 B 10
2018-01-03 08:00:00 B 11
2018-01-03 09:00:00 B 12
2018-01-03 10:00:00 B 13
2018-01-03 11:00:00 B 14
2018-01-03 12:00:00 B 15
2018-01-03 13:00:00 B 16
2018-05-29 23:00:00 C 111
2018-05-30 00:00:00 C 122
2018-05-30 01:00:00 C 133
2018-05-30 02:00:00 C 144
2018-05-30 03:00:00 C 155
''')
df = pd.read_csv(csv, sep = '\t')
df['time'] = pd.to_datetime(df['time'])
csv_filter = io.StringIO(u'''
id start end
A 2018-01-01 01:00:00 2018-01-01 02:00:00
B 2018-01-03 09:00:00 2018-01-03 12:00:00
C 2018-05-30 00:00:00 2018-05-30 08:00:00
''')
df_filter = pd.read_csv(csv_filter, sep = '\t')
df_filter['start'] = pd.to_datetime(df_filter['start'])
df_filter['end'] = pd.to_datetime(df_filter['end'])
输出
df = pd.merge_asof(df, df_filter, left_on = 'time', right_on = 'start', by = 'id').dropna(subset = ['start']).drop(['start','end'], axis = 1)
df = pd.merge_asof(df, df_filter, left_on = 'time', right_on = 'end', by = 'id', direction = 'forward').dropna(subset = ['end']).drop(['start','end'], axis = 1)
对更优雅/更快的解决方案有何想法？
答案 0 :(得分：1)
为什么不在过滤器前merge。请注意，当数据集变大时，这会耗尽您的内存。
newdf=df.merge(df_filter)
newdf=newdf.loc[newdf.time.between(newdf.start,newdf.end),df.columns.tolist()]
newdf
Out[480]:
time id num
1 2018-01-01 01:00:00 A 2
2 2018-01-01 02:00:00 A 3
9 2018-01-03 09:00:00 B 12
10 2018-01-03 10:00:00 B 13
11 2018-01-03 11:00:00 B 14
12 2018-01-03 12:00:00 B 15
15 2018-05-30 00:00:00 C 122
16 2018-05-30 01:00:00 C 133
17 2018-05-30 02:00:00 C 144
18 2018-05-30 03:00:00 C 155