我正在其他字典的帮助下创建查找字典。
我有32核和64GB RAM实例在其中运行。
names
是一个庞大的numpy数组,包含约28184057
个元素。当我尝试在“名称”上应用pool.map
时,出现以下错误:
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
这是我的代码:
pool = mp.Pool(32)
uni_lookup = pool.map(self.create_lookup_dict, names)
pool.close()
create_lookup_dict
包含以下代码:
def create_lookup_dict(self, word):
lookup_word_level = {word: list(set(i[0] for i in self.groups.items() \
if word in i[1]))}
final = []
if self.tag_dict.get(word, 0) is 0:
print("No tag")
return (word, lookup_word_level[word])
if self.secondary_category_tag_dict.get(word, 0) is 0:
print("No secondary category tag")
return (word, lookup_word_level[word])
for i in lookup_word_level[word]:
if self.tag_dict[i].intersection(self.tag_dict[word]) and \
self.secondary_category_tag_dict[i].intersection(self.secondary_category_tag_dict[word]):
final.append(i)
return (word, final)
我想念什么?