Question

是否可以将功能应用于pandas中DataFrame 多线程中的每个单元格？

我知道pandas.DataFrame.applymap但它似乎不允许本机多线程：

import numpy as np
import pandas as pd
np.random.seed(1)
frame = pd.DataFrame(np.random.randn(4, 3), columns=list('bde'), 
                     index=['Utah', 'Ohio', 'Texas', 'Oregon'])
print(frame)
format = lambda x: '%.2f' % x
frame = frame.applymap(format)
print(frame)

返回：

               b         d         e
Utah    1.624345 -0.611756 -0.528172
Ohio   -1.072969  0.865408 -2.301539
Texas   1.744812 -0.761207  0.319039
Oregon -0.249370  1.462108 -2.060141

            b      d      e
Utah     1.62  -0.61  -0.53
Ohio    -1.07   0.87  -2.30
Texas    1.74  -0.76   0.32
Oregon  -0.25   1.46  -2.06

相反，我想使用多个核心来执行操作，因为应用的功能可能很复杂。

Answer 1

按列拆分：

from multiprocessing import Pool

def format(col):
    return col.apply(lambda x: '%.2f' % x)

cores = 5
pool = Pool(cores)
for out_col in pool.imap(format, [frame[i] for i in frame]):
    frame[out_col.name] = out_col
pool.close()
pool.join()

或者按照评论中提到的分区大小进行拆分：

size = 10
frame_split = np.array_split(frame, size)
frame = pd.concat(pool.imap(func, frame_split))

Answer 2

请注意，在Microsoft Windows上，要避免出现问题Attempt to start a new process before the current process has finished its bootstrapping phase，必须将代码放在主函数中，例如：

<div class="home-content-1"></div>
<div class="home-content-1_5"></div>
<div class="home-content-2"></div>
<div class="home-content-3"></div>

关于import numpy as np import pandas as pd from multiprocessing import Pool def format(col): return col.apply(lambda x: '%.2f' % x) if __name__ == "__main__": np.random.seed(1) frame = pd.DataFrame(np.random.randn(4, 3), columns=list('bde'), index=['Utah', 'Ohio', 'Texas', 'Oregon']) print(frame) cores = 2 pool = Pool(cores) for out_col in pool.imap(format, [frame[i] for i in frame]): frame[out_col.name] = out_col pool.close() pool.join() print(frame)的使用，由于数据帧被转换为numpy数组，因此它仅适用于数字。例如：

np.array_split

返回：

import numpy as np
import pandas as pd

from multiprocessing import Pool

def myfunc(a, b):
    '''
    Return a-b if a>b, otherwise return a+b
    Taken from https://docs.scipy.org/doc/numpy/reference/generated/numpy.vectorize.html
    '''
    if a > b:
        return a - b
    else:
        return a + b

def format(col):
    vfunc = np.vectorize(myfunc)
    return pd.DataFrame(vfunc(col,2))

if __name__ == "__main__":
    np.random.seed(1)
    frame = pd.DataFrame(np.random.randn(4, 3), columns=list('bde'), 
                         index=['Utah', 'Ohio', 'Texas', 'Oregon'])
    print(frame)
    cores = 2
    size = 2
    pool = Pool(cores)
    frame_split = np.array_split(frame.as_matrix(), size)
    print (frame_split)

    columns = frame.columns
    frame = pd.concat(pool.imap(format, frame_split)).set_index(frame.index)    
    frame.columns = columns
    pool.close()
    pool.join()
    print(frame)

在pandas中多线程地将函数应用于DataFrame中的每个单元格

2 个答案: