Question

我在多进程环境中遇到以下Python / Django代码的问题。它尝试更新InnoDB表，就像它是一个队列一样，弹出最高值。它工作正常，直到负载达到一定水平，并且同时在此代码中有多个进程。当一个进程持有要删除的表时，会发生死锁，另一个进程会尝试更新该表。

我之前对InnoDB不了解的是索引与表分离，因此一个进程可以锁定表并等待锁定索引，而另一个进程可以反过来，创建僵局。这似乎发生在下面的代码中的UPDATE / DELETE死锁。

简而言之，代码首先更新表中的单行（LIMIT 1），以防止任何其他进程更新同一行。然后，它选择该行以检索其他内容，执行一些工作，然后从表中删除该行。

@classmethod
@transaction.commit_manually
def pop(cls, unique_id):
    """Retrieves the next item and removes it from the queue."""
    transaction.commit()    # This has the side effect of clearing the object cache

    if cls.objects.all().filter(unique_id__isnull=True).count() == 0:  return None

    sql = "UPDATE queue SET unique_id=%d" % unique_id
    sql += " WHERE unique_id IS NULL"
    sql += " ORDER BY id LIMIT 1"
    cursor = connection.cursor()
    try:
        cursor.execute(sql)
        row = cursor.fetchone()
    except OperationalError, oe:    # deadlock, retry later
        transaction.rollback()
        return None
    # If an item is available, it is now marked with our process_id.

    # Retrieve any items with this process id
    items = cls.objects.all().filter(process_id=process_id)
    if len(items) == 0:     # No items available
        transaction.rollback()
        return None
    top = items[0]      # there should only be one item
    # ... perform some other actions here ...
    top.delete()        # This item is deleted from the queue
    transaction.commit()
    return item

如上所述，当两个进程同时运行同一代码时会发生死锁。当另一个进程正在执行top.delete()查询时，一个进程尝试执行UPDATE。 InnoDB行级锁定不会阻止第二个进程在此处尝试更新表。以下是show engine innodb status;的输出：

------------------------
LATEST DETECTED DEADLOCK
------------------------
130625 13:30:22
*** (1) TRANSACTION:
TRANSACTION 0 821802296, ACTIVE 0 sec, process no 27565, OS thread id 139724816692992 starting index read
mysql tables in use 2, locked 2
LOCK WAIT 38 lock struct(s), heap size 14320, 14936 row lock(s)
MySQL thread id 2984294, query id 3142696474 example.com 10.0.1.1 mysql Sending data
UPDATE queue SET unique_id=100804 WHERE unique_id IS NULL ORDER BY id LIMIT 1
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1476 page no 2469 n bits 280 index `PRIMARY` of table `mysql`.`queue` trx id 0 821802296 lock_mode X locks rec but not gap waiting
Record lock, heap no 208 PHYSICAL RECORD: n_fields 9; compact format; info bits 32
 0: len 4; hex 80290a1b; asc  )  ;; 1: len 6; hex 000030fbb547; asc   0  G;; 2: len 7; hex 0000002a471282; asc    *G  ;; 3: len 4; hex 80000005; asc     ;; 4: len 4; hex 800
00051; asc    Q;; 5: len 4; hex 07a27ca6; asc   | ;; 6: len 8; hex 8000124f06c3361e; asc    O  6 ;; 7: SQL NULL; 8: SQL NULL;

*** (2) TRANSACTION:
TRANSACTION 0 821802311, ACTIVE 0 sec, process no 27565, OS thread id 139724817225472 updating or deleting, thread declared inside InnoDB 499
mysql tables in use 1, locked 1
3 lock struct(s), heap size 1216, 2 row lock(s), undo log entries 1
MySQL thread id 2984268, query id 3142696582 example.com 10.0.1.1 mysql updating
DELETE FROM `queue` WHERE `id` IN (2689563)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1476 page no 2469 n bits 280 index `PRIMARY` of table `mysql`.`queue` trx id 0 821802311 lock_mode X locks rec but not gap
Record lock, heap no 208 PHYSICAL RECORD: n_fields 9; compact format; info bits 32
 0: len 4; hex 80290a1b; asc  )  ;; 1: len 6; hex 000030fbb547; asc   0  G;; 2: len 7; hex 0000002a471282; asc    *G  ;; 3: len 4; hex 80000005; asc     ;; 4: len 4; hex 800
00051; asc    Q;; 5: len 4; hex 07a27ca6; asc   | ;; 6: len 8; hex 8000124f06c3361e; asc    O  6 ;; 7: SQL NULL; 8: SQL NULL;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1476 page no 351 n bits 1200 index `queue_queue_id` of table `mysql`.`queue` trx id 0 821802311 lock_mode X locks rec but not gap waiting
Record lock, heap no 1128 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 4; hex 80000005; asc     ;; 1: len 4; hex 80290a1b; asc  )  ;;

*** WE ROLL BACK TRANSACTION (2)

我的问题是：在Python / Django和InnoDB中实现队列和执行pop()操作的正确方法是什么？特别是，需要对此代码进行哪些更改？

Innodb流行队列

0 个答案: