Python多处理struct.error:'i'格式要求-2147483648 <=数字<= 2147483647

时间:2019-06-12 07:47:46

标签: python pandas numpy multiprocessing mmap

我正在其他字典的帮助下创建查找字典。 我有32核和64GB RAM实例在其中运行。 names是一个庞大的numpy数组,包含约28184057个元素。当我尝试在“名称”上应用pool.map时,出现以下错误:

struct.error: 'i' format requires -2147483648 <= number <= 2147483647

这是我的代码:

pool = mp.Pool(32)
uni_lookup = pool.map(self.create_lookup_dict, names)
pool.close()

create_lookup_dict包含以下代码:

def create_lookup_dict(self, word):
    lookup_word_level = {word: list(set(i[0] for i in self.groups.items() \
                                if word in i[1]))}
    final = []

    if self.tag_dict.get(word, 0) is 0:
        print("No tag")
        return (word, lookup_word_level[word])
    if self.secondary_category_tag_dict.get(word, 0) is 0:
        print("No secondary category tag")
        return (word, lookup_word_level[word])
    for i in lookup_word_level[word]:
        if self.tag_dict[i].intersection(self.tag_dict[word]) and \
        self.secondary_category_tag_dict[i].intersection(self.secondary_category_tag_dict[word]):
            final.append(i)
    return (word, final)

我想念什么?

0 个答案:

没有答案