所以我需要找到一种有效的方法来迭代python中的大列表。
给定：整数数组和数字（子列表的长度）
约束：最多100K元素的数组，范围内的元素（1,2 ** 31）
任务：对于每个子列表，找到最大值和最小值之间的差异。打印出最大的不同。
Ex: [4,6,3,4,8,1,9], number = 3
As far as I understand I have to go through every sublist:
[4,6,3] max - min = 6 - 3 = 3
[6,3,4] 3
[3,4,8] 5
[4,8,1] 7
[8,1,9] 8
final max = 8
所以我的解决方案是：
import time
def difference(arr, number):
maxDiff = 0
i = 0
while i+number != len(arr)+1:
diff = max(arr[i:i+number]) - min(arr[i:i+number])
if diff > maxDiff:
maxDiff = diff
i += 1
print maxDiff
length = 2**31
arr = random.sample(xrange(length),100000) #array wasn't given. My sample
t0 = time.clock()
difference(arr,3)
print 'It took :',time.clock() - t0
答案：
2147101251
It took : 5.174262
我也对for循环做了同样的事情，这会给你带来更糟糕的时间：
def difference(arr,d):
maxDiff = 0
if len(arr) == 0:
maxDiff = 0
elif len(arr) == 1:
maxDiff = arr[0]
else:
i = 0
while i + d != len(arr)+1:
array = []
for j in xrange(d):
array.append(arr[i + j])
diff = max(array) - min(array)
if diff > maxDiff:
maxDiff = diff
i += 1
print maxDiff
length = 2**31
arr = random.sample(xrange(length),100000) #array wasn't given. My sample
t0 = time.clock()
difference(arr,1000)
print 'It took :',time.clock() - t0
答案：
2147331163
It took : 14.104639
我的挑战是将时间缩短到2秒。
最有效的方法是什么？
基于@rchang和@gknicker的回答和评论，我得到了改进。我想知道我能做些什么吗？
def difference(arr,d):
window = arr[:d]
arrayLength = len(arr)
maxArrayDiff = max(arr) - min(arr)
maxDiff = 0
while d < arrayLength:
localMax = max(window)
if localMax > maxDiff:
diff = localMax - min(window)
if diff == maxArrayDiff:
return diff
break
elif diff > maxDiff:
maxDiff = diff
window.pop(0)
window.append(arr[d])
d += 1
return maxDiff
#arr = [3,4,6,15,7,2,14,8,1,6,1,2,3,10,1]
length = 2**31
arr = random.sample(xrange(length),100000)
t0 = time.clock()
print difference(arr,1000)
print 'It took :',time.clock() - t0
答案：
2147274599
It took : 2.54171
不错。还有其他建议吗？
答案 0 :(得分：1)
这是我尝试解决这个问题。
我已经进行了相当多的实验和测量，得出了以下结论：
subset_length对效果有重大影响。numpy min / max比函数中的构建快得多，但仅适用于下面的大型数组，比方说50，构建更快。请注意，array必须是numpy.array()且subset_length必须为3或更多。
def difference_np(array, subset_length):
assert subset_length > 2, "subset_length must be larger than 2"
length = array.size
total_diff = array.max()-array.min()
current_min = array[:subset_length].min()
current_max = array[:subset_length].max()
max_diff = current_max - current_min
max_diff_index = 0
index = subset_length
while index < length:
i_new = index
i_old = index-number
index += 1
new = array[i_new]
old = array[i_old]
# the idea here is to avoid calculating the
# min/max over the entire subset as much as possible,
# so we treat every edge case separately.
if new < current_min:
current_min = new
if old == current_max:
current_max = array[i_old+1:i_new-1].max()
elif new > current_max:
current_max = new
if old == current_min:
current_min = array[i_old+1:i_new-1].min()
elif old == current_min:
current_min = array[i_old+1:i_new].min()
elif old == current_max:
current_max = array[i_old+1:i_new].max()
else:
continue
current_diff = current_max-current_min
if current_diff > max_diff:
max_diff = current_diff
max_diff_index = i_old
# shortcut-condition
if max_diff == total_diff:
print('shortcut at', (index-1)/(length-subset_length), '%' )
break
return max_diff, max_diff_index
我不确定快捷条件是否有效，因为它很少应用，并且需要花费输入数组的两次完整迭代。
修改
如果算法使用list.pop(0)，则存在其他改进余地。由于list针对右侧操作进行了优化，list.pop(0)相对较贵。使用collections.deque，存在一种提供快速左侧弹出的替代方法：deque.popleft()。为整体速度带来了相当大的改善。
这是我的算法的非基于numpy collections.deque的版本：
def difference_deque(array, subset_length):
assert subset_length > 1, "subset_length must be larger than 1"
length = len(array)
total_diff = max(array)-min(array)
current_slice = collections.deque(array[:subset_length])
current_min = min(current_slice)
current_max = max(current_slice)
max_diff = current_max - current_min
max_diff_index = 0
index = subset_length
while index < length:
i_new = index
i_old = index-number
index += 1
new = array[i_new]
old = current_slice.popleft()
if new < current_min:
current_min = new
if old == current_max:
current_max = max(current_slice)
current_slice.append(new)
elif new > current_max:
current_max = new
if old == current_min:
current_min = min(current_slice)
current_slice.append(new)
elif old == current_min:
current_slice.append(new)
current_min = min(current_slice)
elif old == current_max:
current_slice.append(new)
current_max = max(current_slice)
else:
current_slice.append(new)
continue
current_diff = current_max-current_min
if current_diff > max_diff:
max_diff = current_diff
max_diff_index = i_old+1
# shortcut-condition
if max_diff == total_diff:
print('shortcut at', (index-1)/(length-number), '%' )
break
return max_diff, max_diff_index
它稍微扭曲了运行时排名： - 最多10个算法（带双端队列）最好 - 最多100个我的算法（带deque）是最好的 - 超过100我的算法（有numpy）是最好的
答案 1 :(得分：0)
我想出了这个优化，可能会减少你第一次实施的时间。我没有使用切片来隔离每次迭代要考虑的数字，而是使用切片一次初始化＆＃34;窗口＆＃34;。在每次迭代中，＆＃34;最右边＆＃34;元素被添加到窗口和＆＃34;最左边＆＃34;元素被驱逐。
import time
import random
def difference(arr, number):
thisSlice = arr[:number-1]
arrSize = len(arr)
maxDiff = -1000
while number < arrSize:
# Put the new element onto the window's tail
thisSlice.append(arr[number])
thisDiff = max(thisSlice) - min(thisSlice)
if thisDiff > maxDiff: maxDiff = thisDiff
number += 1
# Get rid of the "leftmost" element, we won't need it for next iteration
thisSlice.pop(0)
print maxDiff
if __name__ == '__main__':
length = 2**31
arr = random.sample(xrange(length),100000)
t0 = time.clock()
difference(arr, 1000)
print 'It took :', time.clock() - t0
至少在我的笔记本电脑上，这不会低于2秒，但与我们发布的第一个实施相比，我确实看到了一些收益。平均而言，您的第一个解决方案在我的笔记本电脑上运行4.2至4.3秒。这个零碎的窗户结构版本平均在3.5到3.6秒之间运行。
希望它有所帮助。
答案 2 :(得分：0)
我认为您可以使用numpy魔法使用各种as_strided滚动窗口函数之一 - 比如我刚从here偷来的那个：
def rolling_window(a, window):
shape = a.shape[:-1] + (a.shape[-1] - window + 1, window)
strides = a.strides + (a.strides[-1],)
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
使用您的原始difference但使用return代替print，并使用arr作为numpy数组：
>>> w = 3
>>> %timeit old_d = difference(arr, w)
1 loops, best of 3: 718 ms per loop
>>> %timeit q = rolling_window(arr, w); ma=q.max(1);mi=q.min(1); new_d=(ma-mi).max()
100 loops, best of 3: 5.68 ms per loop
和
>>> w = 1000
>>> %timeit old_d = difference(arr, w)
1 loops, best of 3: 25.1 s per loop
>>> %timeit q = rolling_window(arr, w); ma=q.max(1);mi=q.min(1); new_d=(ma-mi).max()
1 loops, best of 3: 326 ms per loop