Question

有没有办法从除了使用for循环之外的文件中获取输入？我正在使用，

data = fileinput.input()
c = [int(i) for i in data]
c.sort()

但是对于非常大量的数据，处理过程需要很长时间。输入的格式为

Answer 1

如果我创造了一个大型的＆＃39;文件：

from random import randint 

with open('/tmp/nums.txt', 'w') as fout:
    a,b=100002/10000, 100002*10000
    for i in range(100002):
        fout.write('{}\n'.format(randint(a,b)))

我可以阅读它，将其转换为整数，然后对数据进行排序：

with open('/tmp/nums.txt') as fin:    
    nums=[int(e) for e in fin]
    nums.sort()

我的计算机上此操作的总时间为50毫秒。 50毫秒很长一段时间？

有一个更正式的时间：

def f1():
    with open('/tmp/nums.txt') as fin:    
        nums=[int(e) for e in fin]
        nums.sort()
    return nums

def f2():
    with open('/tmp/nums.txt') as fin:  
        return sorted(map(int, fin))

def f3():
    with open('/tmp/nums.txt') as fin:  
        nums=list(map(int, fin))
        nums.sort()    
    return nums    

if __name__ =='__main__':
    import timeit     
    import sys
    if sys.version_info.major==2:
        from itertools import imap as map

    result=[]    
    for f in (f1, f2, f3):
        fn=f.__name__
        fs="f()"
        ft=timeit.timeit(fs, setup="from __main__ import f", number=3)
        r=eval(fs)
        result.append((ft, fn, str(r[0:5])+'...'+str(r[-6:-1]) ))         

    result.sort(key=lambda t: t[0])    

    for i, t in enumerate(result):
        ft, fn, r = t
        if i==0:
            fr='{}: {:.4f} secs is fastest\n\tf(x)={}\n========'.format(fn, ft, r)   
        else:
            t1=result[0][0]
            dp=(ft-t1)/t1
            fr='{}: {:.4f} secs - {} is {:.2%} faster\n\tf(x)={}'.format(fn, ft, result[0][1], dp, r)

        print(fr)

你可以看到它们之间的差异不是巨大的（除了PyPy，其中f3显然有优势）：

Python 2.7.8：

f3: 0.2630 secs is fastest
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
========
f2: 0.2641 secs - f3 is 0.41% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
f1: 0.2779 secs - f3 is 5.67% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]

Python 3.4.1：

f2: 0.1873 secs is fastest
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
========
f3: 0.1881 secs - f2 is 0.41% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
f1: 0.2071 secs - f2 is 10.59% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]

PyPy：

f3: 0.1300 secs is fastest
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
========
f2: 0.1428 secs - f3 is 9.81% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
f1: 0.2223 secs - f3 is 70.94% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]

PyPy3：

f3: 0.2483 secs is fastest
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
========
f2: 0.2588 secs - f3 is 4.23% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]
f1: 0.2878 secs - f3 is 15.88% faster
    f(x)=[3025, 18834, 19637, 29124, 42088]...[999964829, 999970030, 999984585, 1000005692, 1000010131]

Answer 2

使用readlines和map使用with打开文件似乎在测试200行文件时效率更高。

In [3]: %%timeit
with open("in.txt",'rb') as f:
    lines = map(int,f)
    lines.sort()
   ...: 
10000 loops, best of 3: 183 µs per loop


In [5]: %%timeit
data = fileinput.input("in.txt")
c = [int(i) for i in data]
c.sort()
   ...: 
1000 loops, best of 3: 443 µs per loop

在python中将文件输入转换为列表

2 个答案: