Question

我需要将一个函数应用于一个大的列表，例如[funz(x) for x in my_list]

每次评估都必须非常快。假设funz(x)应该在不到0.1秒内执行。

如果执行时间超过funz秒，我希望None返回0.1。

实际上，如果my_list有n个元素。代码应该在不到0.1*n秒内执行。

最好的方法是什么？

Answer 1

这是我目前的解决方案。我不知道这是否是解决问题的最有效方法：

def ip2host(ip, return_dict):                                                                                                                                                                                      
    return_dict[ip] = socket.getfqdn(ip)                                                                                                                                                                           
    return                                                                                                                                                                                                         


def ips2hosts(ips, timeout=1):                                                                                                                                                                                     
    ips = set(ips)                                                                                                                                                                                                 
    manager = multiprocessing.Manager()                                                                                                                                                                            
    return_dict = manager.dict()                                                                                                                                                                                   

    for i, ip in enumerate(ips):                                                                                                                                                                                   
        if i % 100 == 0:                                                                                                                                                                                           
            print i, ' out of ', len(ips), " N:", len(return_dict.values()), ' Time:', time.time()                                                                                                                 
        p = multiprocessing.Process(target=ip2host, args=(ip, return_dict))                                                                                                                                        
        p.start()                                                                                                                                                                                                  
        r = p.join(timeout)                                                                                                                                                                                        
        # If thread is still active                                                                                                                                                                                
        if p.is_alive():                                                                                                                                                                                           
            p.terminate()                                                                                                                                                                                          
            p.join()                                                                                                                                                                                               
    return return_dict                                                                                                                                                                                             


d = ips2hosts(df['src_ip'].unique(), timeout=0.1)

这是stdout

0  out of  12986  N: 0  Time: 1513350796.71                                                                                                                                                                        
100  out of  12986  N: 99  Time: 1513350808.22                                                                                                                                                                     
200  out of  12986  N: 190  Time: 1513350819.22                                                                                                                                                                    
300  out of  12986  N: 283  Time: 1513350830.14                                                                                                                                                                    
400  out of  12986  N: 379  Time: 1513350840.73                                                                                                                                                                    
500  out of  12986  N: 474  Time: 1513350851.43                                                                                                                                                                    
600  out of  12986  N: 569  Time: 1513350861.35                                                                                                                                                                    
700  out of  12986  N: 657  Time: 1513350874.25                                                                                                                                                                    
800  out of  12986  N: 742  Time: 1513350886.62                                                                                                                                                                    
900  out of  12986  N: 815  Time: 1513350901.21                                                                                                                                                                    
1000  out of  12986  N: 894  Time: 1513350914.35                                                                                                                                                                   
1100  out of  12986  N: 969  Time: 1513350927.44                                                                                                                                                                   
1200  out of  12986  N: 1060  Time: 1513350939.92                                                                                                                                                                  
1300  out of  12986  N: 1151  Time: 1513350952.52                                                                                                                                                                  
1400  out of  12986  N: 1243  Time: 1513350966.56                                                                                                                                                                  
1500  out of  12986  N: 1337  Time: 1513350979.73                                                                                                                                                                  
1600  out of  12986  N: 1426  Time: 1513350993.5                                                                                                                                                                   
1700  out of  12986  N: 1520  Time: 1513351006.6                                                                                                                                                                   
1800  out of  12986  N: 1613  Time: 1513351017.97                                                                                                                                                                  
1900  out of  12986  N: 1697  Time: 1513351028.07                                                                                                                                                                  
2000  out of  12986  N: 1780  Time: 1513351037.6                                                                                                                                                                   
2100  out of  12986  N: 1868  Time: 1513351046.74                                                                                                                                                                  
2200  out of  12986  N: 1961  Time: 1513351055.68                                                                                                                                                                  
2300  out of  12986  N: 2044  Time: 1513351066.37                                                                                                                                                                  
2400  out of  12986  N: 2135  Time: 1513351080.32                                                                                                                                                                  
2500  out of  12986  N: 2216  Time: 1513351095.13                                                                                                                                                                  
2600  out of  12986  N: 2312  Time: 1513351107.42                                                                                                                                                                  
2700  out of  12986  N: 2406  Time: 1513351120.16                                                                                                                                                                  
2800  out of  12986  N: 2491  Time: 1513351135.09                                                                                                                                                                  
2900  out of  12986  N: 2565  Time: 1513351149.7                                                                                                                                                                   
3000  out of  12986  N: 2634  Time: 1513351164.28                                                                                                                                                                  
3100  out of  12986  N: 2721  Time: 1513351177.83                                                                                                                                                                  
3200  out of  12986  N: 2806  Time: 1513351191.08                                                                                                                                                                  
3300  out of  12986  N: 2885  Time: 1513351205.0                                                                                                                                                                   
3400  out of  12986  N: 2965  Time: 1513351219.65                                                                                                                                                                  
3500  out of  12986  N: 3056  Time: 1513351232.24

似乎使用大约10秒来处理100 ips，即使有时它有点慢。可能是由于所有计算所需的开销

python stop函数，超时很短

1 个答案: