Postgis UPDATE永远挂起

时间:2014-07-17 15:20:02

标签: django postgresql geospatial postgis

我正在使用Django和Postgis作为网络应用。在我的测试环境中,我运行了一个南模式迁移,将denorm_locs = MultiPointField(null=True)添加到模型中(向数据库表中添加一个新列),然后使用数据迁移来填充新字段。这是数据迁移:

# -*- coding: utf-8 -*-
from south.utils import datetime_utils as datetime
from south.db import db
from south.v2 import DataMigration
from django.db import models

from django.contrib.gis.geos import MultiPoint

import time

class Migration(DataMigration):

    def forwards(self, orm):
        "Write your forwards methods here."
        # Note: Don't use "from appname.models import ModelName". 
        # Use orm.ModelName to refer to models in this application,
        # and orm['appname.ModelName'] for models in other applications.

        i = 0
        batch_size = 10

        start = time.time()
        print 'Start: %s' % start

        cont = True

        while cont:
            id_list = orm.MyModel.objects.filter(denorm_locs__isnull=True).order_by('id').values_list('id', flat=True)[:batch_size]

            id_list = list(id_list)

            print 'Got id_list (%s ids): %s' % (len(id_list), time.time() - start,)

            if len(id_list) < batch_size:
                cont = False

            for mymodel_id in id_list:
                locs = [
                    l.loc
                    for l in orm.Location.objects.filter(mymodel3__mymodels__id=mymodel_id)
                    if l.loc
                ]

                mp = MultiPoint(locs)

                orm.MyModel.objects.filter(id=mymodel_id).update(denorm_locs=mp)

            i += 1
            print '%s batches of %s completed' % (i, batch_size,)

Location模型上只有一个名为loc的PointField。

运行orm.MyModel.objects.filter(id=mymodel_id).update(denorm_locs=mp)时,它使用以下查询:

UPDATE "core_mymodel" SET "denorm_locs" = ST_GeomFromEWKB('\x0104000020e610000001000000010100000086c954c1a81054c01d5a643bdfbf3940'::bytea) WHERE "core_mymodel"."id" = 123

我已经完成了一些迭代,但很快for循环遇到了一个在尝试UPDATE时永远挂起的行。作为一个测试,我将迁移更改为UPDATE一个不同的列,并且当for循环遇到同一行时它挂起大约30秒然后继续,但它确实完成了,而当尝试更新denorm_locs时它根本不会完成

以下是SELECT * FROM pg_stat_activity;的输出:

 datid | datname  | pid  | usesysid | usename  | application_name | client_addr | client_hostname | client_port |         backend_start         |          xact_start           |          query_start          |         state_change          | waiting | state  |                                                                                  query                                                                                   
-------+----------+------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 16384 | rdsadmin | 1668 |       10 | rdsadmin |                  |             |                 |             |                               |                               |                               |                               |         |        | <insufficient privilege>
 16388 | mydb     | 3658 |    16386 | mydb     |                  | 10.1.250.12 | 10.1.250.12     |       33658 | 2014-07-17 14:57:09.592821+00 | 2014-07-17 15:06:14.208525+00 | 2014-07-17 15:06:14.208525+00 | 2014-07-17 15:06:14.20853+00  | f       | active | SELECT * FROM pg_stat_activity;
 16388 | mydb     | 2241 |    16386 | mydb     |                  | 10.1.0.7    | 10.1.0.7        |       51805 | 2014-07-16 20:25:13.621986+00 | 2014-07-16 20:25:14.474963+00 | 2014-07-16 20:25:15.040656+00 | 2014-07-16 20:25:15.040661+00 | f       | active | UPDATE "core_mymodel" SET "denorm_locs" = ST_GeomFromEWKB('\x0104000020e61000000100000001010000009b1da9bef36c54c0815ce2c803b93c40'::bytea) WHERE "core_mymodel"."id" = 1000 
 16388 | mydb     | 2744 |    16386 | mydb     |                  | 10.1.250.12 | 10.1.250.12     |       33554 | 2014-07-17 14:42:09.569238+00 | 2014-07-17 14:50:58.94293+00  | 2014-07-17 14:50:58.960509+00 | 2014-07-17 14:50:58.960512+00 | t       | active | UPDATE "core_mymodel" SET "denorm_locs" = ST_GeomFromEWKB('\x0104000020e610000001000000010100000086c954c1a81054c01d5a643bdfbf3940'::bytea) WHERE "core_mymodel"."id" = 123 
(4 rows)

你仍然可以看到昨晚从未完成的另一个查询。

这是SELECT relation::regclass, * FROM pg_locks WHERE NOT granted;

 relation |   locktype    | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid  |   mode    | granted | fastpath 
----------+---------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+------+-----------+---------+----------
          | transactionid |          |          |      |       |            |      14771737 |         |       |          | 5/11758            | 2744 | ShareLock | f       | f
(1 row)

这是SELECT relation::regclass, * FROM pg_locks WHERE granted;

                          relation                          |   locktype    | database | relation | page  | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid  |       mode       | granted | fastpath 
------------------------------------------------------------+---------------+----------+----------+-------+-------+------------+---------------+---------+-------+----------+--------------------+------+------------------+---------+----------
 pg_locks                                                   | relation      |    16388 |    11090 |       |       |            |               |         |       |          | 3/39008            | 3658 | AccessShareLock  | t       | t
                                                            | virtualxid    |          |          |       |       | 3/39008    |               |         |       |          | 3/39008            | 3658 | ExclusiveLock    | t       | t
 core_mymodel_mymodel3s                                          | relation      |    16388 |    19314 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel3                                                 | relation      |    16388 |    19386 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_location                                              | relation      |    16388 |    19243 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_denorm_locs_id                                  | relation      |    16388 |  2725185 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_title_index                                     | relation      |    16388 |  2700114 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_detail_html_index                               | relation      |    16388 |  2700113 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 title                                                      | relation      |    16388 |  2645243 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation      |    16388 |  2576203 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_mymodels_query                                    | relation      |    16388 |  2484701 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_mymodel2_id                                     | relation      |    16388 |   333698 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_mymodel4_id                         | relation      |    16388 |   302491 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_mymodel5_id                       | relation      |    16388 |   302490 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_start_date                                      | relation      |    16388 |   302487 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_end_date                                        | relation      |    16388 |   302483 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel_pkey                                            | relation      |    16388 |   302308 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
 core_mymodel                                                 | relation      |    16388 |    19293 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | t
                                                            | virtualxid    |          |          |       |       | 4/44       |               |         |       |          | 4/44               | 2241 | ExclusiveLock    | t       | t
 core_mymodel_mymodel3s                                          | relation      |    16388 |    19314 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel3                                                 | relation      |    16388 |    19386 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_location                                              | relation      |    16388 |    19243 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_denorm_locs_id                                  | relation      |    16388 |  2725185 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_title_index                                     | relation      |    16388 |  2700114 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_detail_html_index                               | relation      |    16388 |  2700113 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 title                                                      | relation      |    16388 |  2645243 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation      |    16388 |  2576203 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_mymodels_query                                    | relation      |    16388 |  2484701 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_mymodel2_id                                     | relation      |    16388 |   333698 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_mymodel4_id                         | relation      |    16388 |   302491 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_mymodel5_id                       | relation      |    16388 |   302490 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_start_date                                      | relation      |    16388 |   302487 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_end_date                                        | relation      |    16388 |   302483 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel_pkey                                            | relation      |    16388 |   302308 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
 core_mymodel                                                 | relation      |    16388 |    19293 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | t
                                                            | virtualxid    |          |          |       |       | 5/11758    |               |         |       |          | 5/11758            | 2744 | ExclusiveLock    | t       | t
 core_mymodel_mymodel4_id                         | relation      |    16388 |   302491 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_mymodel3s_pkey                                     | relation      |    16388 |   302312 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel_mymodel3s_mymodel_id_5dcd946e263a391f_uniq           | relation      |    16388 |   302310 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel_start_date                                      | relation      |    16388 |   302487 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
                                                            | transactionid |          |          |       |       |            |      14771737 |         |       |          | 4/44               | 2241 | ExclusiveLock    | t       | f
 core_location_loc_id                                       | relation      |    16388 |   302470 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel3_pkey                                            | relation      |    16388 |   302361 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel_end_date                                        | relation      |    16388 |   302483 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_pkey                                            | relation      |    16388 |   302308 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel3_name                                            | relation      |    16388 |   333705 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel_denorm_locs_id                                  | relation      |    16388 |  2725185 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_title_index                                     | relation      |    16388 |  2700114 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel3_name_129459df841fd9de_uniq                      | relation      |    16388 |   333668 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel                                                 | tuple         |    16388 |    19293 | 93104 |     3 |            |               |         |       |          | 5/11758            | 2744 | ExclusiveLock    | t       | f
 core_mymodel3_location_id                                     | relation      |    16388 |   302517 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel3_name_index                                      | relation      |    16388 |  2700014 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel_mymodel5_id                       | relation      |    16388 |   302490 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_mymodel2_id                                     | relation      |    16388 |   333698 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation      |    16388 |  2576203 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_mymodel3s_mymodel_id                                 | relation      |    16388 |   302488 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel3_address_id                                      | relation      |    16388 |   302516 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel3_mymodel2_id                                     | relation      |    16388 |   333704 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel                                                 | relation      |    16388 |    19293 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_location_pkey                                         | relation      |    16388 |   302276 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel_mymodel3s_mymodel3_id                                 | relation      |    16388 |   302489 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 title                                                      | relation      |    16388 |  2645243 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_detail_html_index                               | relation      |    16388 |  2700113 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_mymodels_query                                    | relation      |    16388 |  2484701 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_location_pkey                                         | relation      |    16388 |   302276 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel_mymodel3s_mymodel3_id                                 | relation      |    16388 |   302489 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 title                                                      | relation      |    16388 |  2645243 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_detail_html_index                               | relation      |    16388 |  2700113 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_mymodels_query                                    | relation      |    16388 |  2484701 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_mymodel3s_mymodel_id                                 | relation      |    16388 |   302488 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel3_address_id                                      | relation      |    16388 |   302516 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel3_mymodel2_id                                     | relation      |    16388 |   333704 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel                                                 | relation      |    16388 |    19293 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel3_name_index                                      | relation      |    16388 |  2700014 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel_mymodel5_id                       | relation      |    16388 |   302490 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_mymodel2_id                                     | relation      |    16388 |   333698 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation      |    16388 |  2576203 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_denorm_locs_id                                  | relation      |    16388 |  2725185 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_title_index                                     | relation      |    16388 |  2700114 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel3_name_129459df841fd9de_uniq                      | relation      |    16388 |   333668 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel3_location_id                                     | relation      |    16388 |   302517 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel_end_date                                        | relation      |    16388 |   302483 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_pkey                                            | relation      |    16388 |   302308 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel3_name                                            | relation      |    16388 |   333705 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
                                                            | transactionid |          |          |       |       |            |      14771961 |         |       |          | 5/11758            | 2744 | ExclusiveLock    | t       | f
 core_location_loc_id                                       | relation      |    16388 |   302470 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel3_pkey                                            | relation      |    16388 |   302361 |       |       |            |               |         |       |          | 5/11758            | 2744 | AccessShareLock  | t       | f
 core_mymodel_mymodel3s_mymodel_id_5dcd946e263a391f_uniq           | relation      |    16388 |   302310 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
 core_mymodel_start_date                                      | relation      |    16388 |   302487 |       |       |            |               |         |       |          | 5/11758            | 2744 | RowExclusiveLock | t       | f
 core_mymodel_mymodel4_id                         | relation      |    16388 |   302491 |       |       |            |               |         |       |          | 4/44               | 2241 | RowExclusiveLock | t       | f
 core_mymodel_mymodel3s_pkey                                     | relation      |    16388 |   302312 |       |       |            |               |         |       |          | 4/44               | 2241 | AccessShareLock  | t       | f
(91 rows)

据我所知,尝试访问id = 123行的唯一客户端是尝试更新它的客户端。我也不确定为什么它要求所有这些列的锁,当我从DB获取的所有内容都是主键列表时,该查询已经完成。此外,当我必须在挂起时终止迁移过程时,很多这些授予的锁定都会被锁定。我尝试重新启动数据库,但迁移仍然失败。

此外,SELECT pg_terminate_backend(pid);并未停止任何后端,只有重新启动数据库才会清除它们。

1 个答案:

答案 0 :(得分:0)

我重新启动了我的数据库,删除了新字段,对其进行了读取,然后运行VACUUM ANALYZE VERBOSE。 UPDATE查询现在似乎已经完成,尽管其中一些查询将随机地比其他查询长几个数量级。