Python multiprocessing: Reading a large file and updating an imported dictionary

Question

I need to read a large file and update an imported dictionary accordingly, using multiprocessing Pool and Manager. Here is my code:

from multiprocessing import Pool, Manager

manager = Manager()
d = manager.dict()
imported_dic = json.load(~/file.json) #loading a file containing a large dictionary
d.update(imported_dic)

def f(line):
    data = line.split('\t')
    uid = data[0]
    tweet = data[2].decode('utf-8')

    if #sth in tweet:
        d[uid] += 1

p = Pool(4)
with open('~/test_1k.txt') as source_file:
    p.map(f, source_file)

But it does not work properly. Any idea what am I doing wrong here?

Answer 1

试试这段代码：

d = init_dictionary( ) # some your magic here

def f(line):
    data = line.split('\t')
    uid = data[0]
    tweet = data[2].decode('utf-8')

    if uid in d: 
        for n in d[uid].keys(): 
            if n in tweet: 
                 yield uid, n, 1 
            else: 
                 yield uid, n, 0 


p = Pool(4)

with open('~/test_1k.txt') as source_file:
    for stat in p.map(f, source_file):
          uid, n, r = stat
          d[uid][n] += r

这是相同的解决方案，但没有共享字典。