Question

我希望基于另一个只有三列的较小数据框来过滤大型数据框（数百万行）：ID，Start，End。

以下是我整理的内容（有效），但似乎select t1.*, CASE WHEN REGEXP_LIKE( t1.cellname, '^[^a-zA-Z]*' || REGEXP_SUBSTR( t1.tablename, '[a-zA-Z]{1,3}$' ) ) AND REGEXP_LIKE( t2.cellname, '^[^a-zA-Z]*' || REGEXP_SUBSTR( t2.tablename, '[a-zA-Z]{1,3}$' ) ) THEN 'pass' ELSE 'fail' END result FROM mytable t1 INNER JOIN mtable t2 ON t1.cell_name = t2.cell_name AND t2.source = 'SC2' WHERE t1.source='SRC1';或groupby()可能会更快。

设置：

np.where

工作代码

import pandas as pd
import io

csv = io.StringIO(u'''
time    id  num
2018-01-01 00:00:00 A   1
2018-01-01 01:00:00 A   2
2018-01-01 02:00:00 A   3
2018-01-01 03:00:00 A   4
2018-01-01 04:00:00 A   5
2018-01-01 05:00:00 A   6
2018-01-01 06:00:00 A   6
2018-01-03 07:00:00 B   10
2018-01-03 08:00:00 B   11
2018-01-03 09:00:00 B   12
2018-01-03 10:00:00 B   13
2018-01-03 11:00:00 B   14
2018-01-03 12:00:00 B   15
2018-01-03 13:00:00 B   16
2018-05-29 23:00:00 C   111
2018-05-30 00:00:00 C   122
2018-05-30 01:00:00 C   133
2018-05-30 02:00:00 C   144
2018-05-30 03:00:00 C   155
''')

df = pd.read_csv(csv, sep = '\t')
df['time'] = pd.to_datetime(df['time'])

csv_filter = io.StringIO(u'''
id  start   end
A   2018-01-01 01:00:00 2018-01-01 02:00:00
B   2018-01-03 09:00:00 2018-01-03 12:00:00
C   2018-05-30 00:00:00 2018-05-30 08:00:00
''')

df_filter = pd.read_csv(csv_filter, sep = '\t')
df_filter['start'] = pd.to_datetime(df_filter['start'])
df_filter['end'] = pd.to_datetime(df_filter['end'])

输出

df = pd.merge_asof(df, df_filter, left_on = 'time', right_on = 'start', by = 'id').dropna(subset = ['start']).drop(['start','end'], axis = 1)
df = pd.merge_asof(df, df_filter, left_on = 'time', right_on = 'end', by = 'id', direction = 'forward').dropna(subset = ['end']).drop(['start','end'], axis = 1)

对更优雅/更快的解决方案有何想法？

Answer 1

为什么不在过滤器前merge。请注意，当数据集变大时，这会耗尽您的内存。

newdf=df.merge(df_filter)
newdf=newdf.loc[newdf.time.between(newdf.start,newdf.end),df.columns.tolist()]
newdf
Out[480]: 
                  time id  num
1  2018-01-01 01:00:00  A    2
2  2018-01-01 02:00:00  A    3
9  2018-01-03 09:00:00  B   12
10 2018-01-03 10:00:00  B   13
11 2018-01-03 11:00:00  B   14
12 2018-01-03 12:00:00  B   15
15 2018-05-30 00:00:00  C  122
16 2018-05-30 01:00:00  C  133
17 2018-05-30 02:00:00  C  144
18 2018-05-30 03:00:00  C  155

熊猫：按日期范围/确切ID过滤

1 个答案: