Question

我最近将Celery安装升级到4.0。经过几天与升级过程的摔跤，我终于开始工作......有点儿。有些任务会返回，但最终的任务不会。

我有一个类，SFF，它接收并解析一个文件：

# Constructor with I/O file
def __init__(self, file):

    # File data that's gonna get used a lot
    sffDescriptor = file.fileno()
    fileName = abspath(file.name)

    # Get the pointer to the file
    filePtr = mmap.mmap(sffDescriptor, 0, flags=mmap.MAP_SHARED, prot=mmap.PROT_READ)

    # Get the header info
    hdr = filePtr.read(HEADER_SIZE)
    self.header = SFFHeader._make(unpack(HEADER_FMT, hdr))

    # Read in the palette maps
    print self.header.onDemandDataSize
    print self.header.onLoadDataSize
    palMapsResult = getPalettes.delay(fileName, self.header.palBankOff - HEADER_SIZE, self.header.onDemandDataSize, self.header.numPals)

    # Read the sprite list nodes
    nodesStart = self.header.sprListOff
    nodesEnd = self.header.palBankOff
    print nodesEnd - nodesStart
    sprNodesResult = getSprNodes.delay(fileName, nodesStart, nodesEnd, self.header.numSprites)

    # Get palette data
    self.palettes = palMapsResult.get()

    # Get sprite data
    spriteNodes = sprNodesResult.get()

    # TESTING
    spritesResultSet = ResultSet([])
    numSpriteNodes = len(spriteNodes)
    # Split the nodes into chunks of size 32 elements
    for x in xrange(0, numSpriteNodes, 32):
        spritesResult = getSprites.delay(spriteNodes, x, x+32, fileName, self.palettes, self.header.palBankOff, self.header.onDemandDataSizeTotal)
        spritesResultSet.add(spritesResult)
        break  # REMEMBER TO REMOVE FOR ENTIRE SFF

    self.sprites = spritesResultSet.join_native()

如果它是一个返回整个spritesResult的单个任务并不重要，或者如果我使用ResultSet拆分它，结果总是相同的：我正在使用的Python控制台挂起{{1} }或spritesResultSet.join_native()（取决于我如何格式化）。

以下是有问题的任务：

spritesResult.get()

我知道它到达了return语句，因为如果我在它上方放置一个打印件，它将在Celery窗口中打印。我也知道任务正在运行完成，因为我收到了来自worker的以下消息：

[2016-11-16 00：03：33,639：INFO / PoolWorker-4]任务framedatabase.tasks.getSprites [285ac9b1-09b4-4cf1-a251-da6212863832]成功获得0.137236133218s：'[{“width”： 120，“palNo”：30，“group”：9000，“xAxis”：0，“yAxis”：0，“data”：...'

以下是我在settings.py中的芹菜设置：

@task
def getSprites(nodes, start, end, fileName, palettes, palBankOff, onDemandDataSizeTotal):
sprites = []

with open(fileName, "rb") as file:
    sffDescriptor = file.fileno()
    sffData = mmap.mmap(sffDescriptor, 0, flags=mmap.MAP_SHARED, prot=mmap.PROT_READ)

    for node in nodes[start:end]:
        sprListNode = dict(SprListNode._make(node)._asdict())  # Need to convert it to a dict since values may change.
        #print node
        #print sprListNode

        # If it's a linked sprite, the data length is 0, so get the linked index.
        if sprListNode['dataLen'] == 0:
            sprListNodeTemp = SprListNode._make(nodes[sprListNode['index']])
            sprListNode['dataLen'] = sprListNodeTemp.dataLen
            sprListNode['dataOffset'] = sprListNodeTemp.dataOffset
            sprListNode['compression'] = sprListNodeTemp.compression

        # What does the offset need to be?
        dataOffset = sprListNode['dataOffset']
        if sprListNode['loadMode'] == 0:
            dataOffset += palBankOff #- HEADER_SIZE
        elif sprListNode['loadMode'] == 1:
            dataOffset += onDemandDataSizeTotal #- HEADER_SIZE

        #print sprListNode

        # Seek to the data location and "read" it in. First 4 bytes are just the image length
        start = dataOffset + 4
        end = dataOffset + sprListNode['dataLen']
        #sffData.seek(start)

        compressedSprite = sffData[start:end]

        # Create the sprite
        sprite = Sprite(sprListNode, palettes[sprListNode['palNo']], np.fromstring(compressedSprite, dtype=np.uint8))
        sprites.append(sprite)

return json.dumps(sprites, cls=SpriteJSONEncoder)

...和我的celery.py：

# Celery settings
BROKER_URL='redis://localhost:1717/1'
CELERY_RESULT_BACKEND='redis://localhost:1717/0'
CELERY_IGNORE_RESULT=False
CELERY_IMPORTS = ("framedatabase.tasks", )

Answer 1

发现问题。显然它导致了死锁，如Celery文档中的“避免启动同步子任务”一节所述：http://docs.celeryproject.org/en/latest/userguide/tasks.html#tips-and-best-practices

所以我摆脱了界限：

sprNodesResult.get()

并将最终结果更改为链：

self.sprites = chain(getSprNodes.s(fileName, nodesStart, nodesEnd, self.header.numSprites),
    getSprites.s(0,32,fileName,self.palettes,self.header.palBankOff,self.header.onDemandDataSizeTotal))().get()

它有效！现在我只需要找到一种方法来按照我想要的方式进行拆分！

将Celery从3.1升级到4.0后Redis没有返回结果

1 个答案: