我对下面的代码有疑问。我使用chunk函数生成列表列表。代码从这里改编
Splitting a list of into N parts of approximately equal length
在shell中检查后,这就是我作为列表和子列表的类型。
In [18]: urls = list(range(957))
In [19]: type(urls)
Out[19]: list
In [20]: chunks = chunk(urls, 10)
In [21]: type(chunks)
Out[21]: list
In [22]: type(chunks[0])
Out[22]: list
他们都是名单。但是,当我将此列表发送给芹菜任务时,它会抛出一个错误,抱怨它不是一个列表,它实际上是一个切片。
这是追溯:
[2016-10-03 12:30:57,360: ERROR/MainProcess] Task tasks.scrape_id_group[3759ff40-45f3-455f-87e7-1dadbbb201fc] raised unexpected: TypeError("unhashable type: 'slice'",)
Traceback (most recent call last):
File "/home/milan/.virtualenvs/upwork/lib/python3.4/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/milan/.virtualenvs/upwork/lib/python3.4/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/home/milan/skilled/upwork/tasks.py", line 55, in scrape_id_group
for i, sublist in enumerate(chunk(ids, count))
File "/home/milan/skilled/upwork/utils.py", line 219, in chunk
return [l[i::n] for i in range(n)]
File "/home/milan/skilled/upwork/utils.py", line 219, in <listcomp>
return [l[i::n] for i in range(n)]
TypeError: unhashable type: 'slice'
这是用于显示问题的修改后的代码:
from celery import Celery, group
app = Celery('tasks', broker='redis://localhost:6379/0')
def chunk(l, n):
return [l[i::n] for i in range(n)]
@app.task
def scrape_list_task(account, urls):
print(urls)
@app.task
def scrape_id_task(account, ids):
print(ids)
@app.task
def scrape_list_group():
accounts = list(range(10))
count = len(accounts)
urls = list(range(957))
return group(
scrape_list_task.s(accounts[i], sublist)
for i, sublist in enumerate(chunk(urls, count))
)
@app.task
def scrape_id_group(ids):
accounts = list(range(10))
count = len(accounts)
return group(
scrape_id_task.s(accounts[i], sublist)
for i, sublist in enumerate(chunk(ids, count))
)
@app.task
def scrape_task():
(scrape_list_group.s() | scrape_id_group.s())()
我在默认芹菜队列中运行此代码,并将redis作为消息代理。有什么建议?谢谢!
编辑:
在芹菜队列中运行scrape_task时会出现实际问题。当使用延迟调用scrape_list_group任务时
scrape_list_group.delay()
任务运行良好
问题似乎是任务的链接。试图将列表转换为元组,但也没有结果。感谢您的任何信息!
解决方案:
我已将块代码更改为迭代方法。感谢@ Nf4r的建议:)
def chunk(l, n):
result = [[] for _ in range(n)]
for i, el in enumerate(l):
result[i % n].append(el)
return result