Question

我正在尝试编写一个执行时间序列分析的Web应用程序。

我在Python中编写了一个函数来获取属于给定日期时间范围的数据集（Python dict）的子集（使用Python的datetime.datetime类）。

在我的网络应用程序中，我正在进行一次计算，它会调用此函数几百次来对大约10,000点的数据集进行选择。这大约需要25秒，这是可以的，但并不理想。

我附上了运行某些示例数据的方法示例。是否有更好的方法以更好的性能获得相同的结果？关于更好的框架使用的建议也很受欢迎（例如，用numpy数组做这个更好，还是完全放弃Python？）。

该功能输出经过的时间。

from datetime import datetime
from datetime import timedelta

data_dict = {'times':[], 'data':[]}

#Generate sample data
start_datetime = datetime(2014, 8, 23, 15, 17, 17, 392943)
for i in range(10000):
    data_dict['times'].append(start_datetime+timedelta(minutes = 5*i))
    data_dict['data'].append(i)

startTime = datetime.now()

def data_select(data_dict, time_range):
    start = 0
    end = 1
    for x in data_dict['times']:
        if x - time_range[0] < timedelta(seconds = 0):
            start += 1
        if x - time_range[1] <= timedelta(seconds = 0):
            end += 1
    data_dict['times'] = list(data_dict['times'][start:end])
    data_dict['data'] = list(data_dict['data'][start:end])
    return data_dict

#Example function call   
data_sub_dict = data_select(data_dict, [datetime(2014, 8, 30, 0, 0, 0, 0), datetime(2014, 9, 5, 0, 0, 0, 0)])

print "Time elapsed: " + str((datetime.now() - startTime))

Answer 1

由于您的数据已排序，您可以欺骗并使用非常有用的bisect模块。它不是在数据列表中进行线性搜索，而是检查中间值，然后采用左半部分或右半部分 - 批次更少的比较。如果输出数据正确，则{10}数据点的bisect速度提高约800倍。

源

import bisect
from datetime import datetime
from datetime import timedelta

data_dict = {'times':[], 'data':[]}

#Generate sample data
start_datetime = datetime(2014, 8, 23, 15, 17, 17, 392943)
for i in range(10000):
    data_dict['times'].append(start_datetime+timedelta(minutes = 5*i))
    data_dict['data'].append(i)

startTime = datetime.now()

def data_select_search(data_dict, time_range):
    start = 0
    end = 1
    times = data_dict['times']
    for x in times:
        if x - time_range[0] < timedelta(seconds = 0):
            start += 1
        if x - time_range[1] <= timedelta(seconds = 0):
            end += 1
    # print 'search:',start,end
    data_dict['times'] = list(data_dict['times'][start:end])
    data_dict['data'] = list(data_dict['data'][start:end])
    return data_dict

def data_select_bisect(data_dict, time_range):
    times = data_dict['times']
    start = bisect.bisect_left(times, time_range[0])
    end = bisect.bisect_right(times, time_range[1], lo=start) + 1
    # print 'bisect:',start,end
    return dict(
        times=data_dict['times'][start:end],
        data=data_dict['data'][start:end],
        )

drange = [
        datetime(2014, 8, 30, 0, 0, 0, 0), 
        datetime(2014, 9, 5, 0, 0, 0, 0)
    ]
data_sub_dict = data_select_search(data_dict.copy(), drange)
_dict2 = data_select_bisect(data_dict.copy(), drange)

import timeit
mysetup = "from __main__ import data_select_bisect, data_select_search, data_dict, drange"
num = 100

print('search:', timeit.timeit(
    "data_select_search(data_dict.copy(), drange)", 
    setup=mysetup,
    number=num
    ))
print('bisect:', timeit.timeit(
    "data_select_bisect(data_dict.copy(), drange)", 
    setup=mysetup,
    number=num,
    ))

输出

('search:', 1.2735650539398193)
('bisect:', 0.0015599727630615234)

从特定日期时间范围获取数据

1 个答案:

源

输出