我尝试使用multiprocessing.Pool并行执行一些CypherQuery()。
当我运行neo4j.CypherQuery()非并行时,它工作正常。当我在multiprocessing.Pool上只运行1个neo4j.CypherQuery()时,它运行正常。一旦我启动了2个或更多neo4j.CypherQuery()进程,它就会失败,并显示以下错误消息。
from mulitprocessing import Pool
from py2neo import neo4j
pool = Pool(processes=4)
db = neo4j.GraphDatabaseService("http://localhost:7474/db/data/")
def cypher_query(db):
try:
# very simple cypher query
query_string = "MATCH (n:Label) RETURN n.name, n"
query = neo4j.CypherQuery(db, query_string)
result = query.execute()
return_dict = {}
for r in result:
return_dict[r[0]] = r[1]
return return_dict
except:
# print stack trace
print('%s' % (traceback.format_exc()))
result1 = pool.apply_async(cypher_query, [db])
result2 = pool.apply_async(cypher_query, [db])
# close pool and wait for all processes to finish
pool.close()
pool.join()
# here I would collect results, something fails before
result1.get()
result2.get()
错误讯息:
Traceback (most recent call last):
File "/path/to/my/script.py", line 237, in my_function
query = neo4j.CypherQuery(db, query_string)
File "build/bdist.linux-x86_64/egg/py2neo/neo4j.py", line 976, in __init__
self._cypher = Resource(graph_db.__metadata__["cypher"])
File "build/bdist.linux-x86_64/egg/py2neo/neo4j.py", line 320, in __metadata__
self.refresh()
File "build/bdist.linux-x86_64/egg/py2neo/neo4j.py", line 342, in refresh
self._metadata = ResourceMetadata(self._get().content)
File "build/bdist.linux-x86_64/egg/py2neo/packages/httpstream/http.py", line 532, in content
elif self.is_text:
File "build/bdist.linux-x86_64/egg/py2neo/packages/httpstream/http.py", line 513, in is_text
return self.content_type.partition("/")[0] == "text"
AttributeError: 'NoneType' object has no attribute 'partition'
我没有收到错误消息。我尝试了不同的Cypher查询以及execute()和stream(),但总是失败。所有查询都运行良好的非并行。显然,我遗漏了一些破坏我的功能并行化的东西,但我不知道如何解决它。
答案 0 :(得分:0)
我知道py2neo / httpstream不适用于多处理,GitHub中存在一个问题:
https://github.com/nigelsmall/httpstream/issues/3
但是,由于我自己并不需要多处理,并且对此模块知之甚少,因此在解决此问题方面没有取得任何进展。我很乐意让贡献者研究这个并提供补丁,但到目前为止还没有人自愿这样做。
答案 1 :(得分:0)
在版本1.6.1中的数据库连接字符串中使用http而不是https时也会出现此错误