我正在使用Django和Postgis作为网络应用。在我的测试环境中,我运行了一个南模式迁移,将denorm_locs = MultiPointField(null=True)
添加到模型中(向数据库表中添加一个新列),然后使用数据迁移来填充新字段。这是数据迁移:
# -*- coding: utf-8 -*-
from south.utils import datetime_utils as datetime
from south.db import db
from south.v2 import DataMigration
from django.db import models
from django.contrib.gis.geos import MultiPoint
import time
class Migration(DataMigration):
def forwards(self, orm):
"Write your forwards methods here."
# Note: Don't use "from appname.models import ModelName".
# Use orm.ModelName to refer to models in this application,
# and orm['appname.ModelName'] for models in other applications.
i = 0
batch_size = 10
start = time.time()
print 'Start: %s' % start
cont = True
while cont:
id_list = orm.MyModel.objects.filter(denorm_locs__isnull=True).order_by('id').values_list('id', flat=True)[:batch_size]
id_list = list(id_list)
print 'Got id_list (%s ids): %s' % (len(id_list), time.time() - start,)
if len(id_list) < batch_size:
cont = False
for mymodel_id in id_list:
locs = [
l.loc
for l in orm.Location.objects.filter(mymodel3__mymodels__id=mymodel_id)
if l.loc
]
mp = MultiPoint(locs)
orm.MyModel.objects.filter(id=mymodel_id).update(denorm_locs=mp)
i += 1
print '%s batches of %s completed' % (i, batch_size,)
Location模型上只有一个名为loc
的PointField。
运行orm.MyModel.objects.filter(id=mymodel_id).update(denorm_locs=mp)
时,它使用以下查询:
UPDATE "core_mymodel" SET "denorm_locs" = ST_GeomFromEWKB('\x0104000020e610000001000000010100000086c954c1a81054c01d5a643bdfbf3940'::bytea) WHERE "core_mymodel"."id" = 123
我已经完成了一些迭代,但很快for循环遇到了一个在尝试UPDATE时永远挂起的行。作为一个测试,我将迁移更改为UPDATE一个不同的列,并且当for循环遇到同一行时它挂起大约30秒然后继续,但它确实完成了,而当尝试更新denorm_locs时它根本不会完成
以下是SELECT * FROM pg_stat_activity;
的输出:
datid | datname | pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | xact_start | query_start | state_change | waiting | state | query
-------+----------+------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+---------+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16384 | rdsadmin | 1668 | 10 | rdsadmin | | | | | | | | | | | <insufficient privilege>
16388 | mydb | 3658 | 16386 | mydb | | 10.1.250.12 | 10.1.250.12 | 33658 | 2014-07-17 14:57:09.592821+00 | 2014-07-17 15:06:14.208525+00 | 2014-07-17 15:06:14.208525+00 | 2014-07-17 15:06:14.20853+00 | f | active | SELECT * FROM pg_stat_activity;
16388 | mydb | 2241 | 16386 | mydb | | 10.1.0.7 | 10.1.0.7 | 51805 | 2014-07-16 20:25:13.621986+00 | 2014-07-16 20:25:14.474963+00 | 2014-07-16 20:25:15.040656+00 | 2014-07-16 20:25:15.040661+00 | f | active | UPDATE "core_mymodel" SET "denorm_locs" = ST_GeomFromEWKB('\x0104000020e61000000100000001010000009b1da9bef36c54c0815ce2c803b93c40'::bytea) WHERE "core_mymodel"."id" = 1000
16388 | mydb | 2744 | 16386 | mydb | | 10.1.250.12 | 10.1.250.12 | 33554 | 2014-07-17 14:42:09.569238+00 | 2014-07-17 14:50:58.94293+00 | 2014-07-17 14:50:58.960509+00 | 2014-07-17 14:50:58.960512+00 | t | active | UPDATE "core_mymodel" SET "denorm_locs" = ST_GeomFromEWKB('\x0104000020e610000001000000010100000086c954c1a81054c01d5a643bdfbf3940'::bytea) WHERE "core_mymodel"."id" = 123
(4 rows)
你仍然可以看到昨晚从未完成的另一个查询。
这是SELECT relation::regclass, * FROM pg_locks WHERE NOT granted;
:
relation | locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath
----------+---------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+------+-----------+---------+----------
| transactionid | | | | | | 14771737 | | | | 5/11758 | 2744 | ShareLock | f | f
(1 row)
这是SELECT relation::regclass, * FROM pg_locks WHERE granted;
:
relation | locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath
------------------------------------------------------------+---------------+----------+----------+-------+-------+------------+---------------+---------+-------+----------+--------------------+------+------------------+---------+----------
pg_locks | relation | 16388 | 11090 | | | | | | | | 3/39008 | 3658 | AccessShareLock | t | t
| virtualxid | | | | | 3/39008 | | | | | 3/39008 | 3658 | ExclusiveLock | t | t
core_mymodel_mymodel3s | relation | 16388 | 19314 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel3 | relation | 16388 | 19386 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_location | relation | 16388 | 19243 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_denorm_locs_id | relation | 16388 | 2725185 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_title_index | relation | 16388 | 2700114 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_detail_html_index | relation | 16388 | 2700113 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
title | relation | 16388 | 2645243 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation | 16388 | 2576203 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_mymodels_query | relation | 16388 | 2484701 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_mymodel2_id | relation | 16388 | 333698 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_mymodel4_id | relation | 16388 | 302491 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_mymodel5_id | relation | 16388 | 302490 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_start_date | relation | 16388 | 302487 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_end_date | relation | 16388 | 302483 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel_pkey | relation | 16388 | 302308 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
core_mymodel | relation | 16388 | 19293 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | t
| virtualxid | | | | | 4/44 | | | | | 4/44 | 2241 | ExclusiveLock | t | t
core_mymodel_mymodel3s | relation | 16388 | 19314 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel3 | relation | 16388 | 19386 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_location | relation | 16388 | 19243 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_denorm_locs_id | relation | 16388 | 2725185 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_title_index | relation | 16388 | 2700114 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_detail_html_index | relation | 16388 | 2700113 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
title | relation | 16388 | 2645243 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation | 16388 | 2576203 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_mymodels_query | relation | 16388 | 2484701 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_mymodel2_id | relation | 16388 | 333698 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_mymodel4_id | relation | 16388 | 302491 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_mymodel5_id | relation | 16388 | 302490 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_start_date | relation | 16388 | 302487 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_end_date | relation | 16388 | 302483 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel_pkey | relation | 16388 | 302308 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
core_mymodel | relation | 16388 | 19293 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | t
| virtualxid | | | | | 5/11758 | | | | | 5/11758 | 2744 | ExclusiveLock | t | t
core_mymodel_mymodel4_id | relation | 16388 | 302491 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_mymodel3s_pkey | relation | 16388 | 302312 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel_mymodel3s_mymodel_id_5dcd946e263a391f_uniq | relation | 16388 | 302310 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel_start_date | relation | 16388 | 302487 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
| transactionid | | | | | | 14771737 | | | | 4/44 | 2241 | ExclusiveLock | t | f
core_location_loc_id | relation | 16388 | 302470 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel3_pkey | relation | 16388 | 302361 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel_end_date | relation | 16388 | 302483 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_pkey | relation | 16388 | 302308 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel3_name | relation | 16388 | 333705 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel_denorm_locs_id | relation | 16388 | 2725185 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_title_index | relation | 16388 | 2700114 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel3_name_129459df841fd9de_uniq | relation | 16388 | 333668 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel | tuple | 16388 | 19293 | 93104 | 3 | | | | | | 5/11758 | 2744 | ExclusiveLock | t | f
core_mymodel3_location_id | relation | 16388 | 302517 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel3_name_index | relation | 16388 | 2700014 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel_mymodel5_id | relation | 16388 | 302490 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_mymodel2_id | relation | 16388 | 333698 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation | 16388 | 2576203 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_mymodel3s_mymodel_id | relation | 16388 | 302488 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel3_address_id | relation | 16388 | 302516 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel3_mymodel2_id | relation | 16388 | 333704 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel | relation | 16388 | 19293 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_location_pkey | relation | 16388 | 302276 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel_mymodel3s_mymodel3_id | relation | 16388 | 302489 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
title | relation | 16388 | 2645243 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_detail_html_index | relation | 16388 | 2700113 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_mymodels_query | relation | 16388 | 2484701 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_location_pkey | relation | 16388 | 302276 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel_mymodel3s_mymodel3_id | relation | 16388 | 302489 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
title | relation | 16388 | 2645243 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_detail_html_index | relation | 16388 | 2700113 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_mymodels_query | relation | 16388 | 2484701 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_mymodel3s_mymodel_id | relation | 16388 | 302488 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel3_address_id | relation | 16388 | 302516 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel3_mymodel2_id | relation | 16388 | 333704 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel | relation | 16388 | 19293 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel3_name_index | relation | 16388 | 2700014 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel_mymodel5_id | relation | 16388 | 302490 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_mymodel2_id | relation | 16388 | 333698 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_mymodel5_id_6dce956e264c19df_uniq | relation | 16388 | 2576203 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_denorm_locs_id | relation | 16388 | 2725185 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_title_index | relation | 16388 | 2700114 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel3_name_129459df841fd9de_uniq | relation | 16388 | 333668 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel3_location_id | relation | 16388 | 302517 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel_end_date | relation | 16388 | 302483 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_pkey | relation | 16388 | 302308 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel3_name | relation | 16388 | 333705 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
| transactionid | | | | | | 14771961 | | | | 5/11758 | 2744 | ExclusiveLock | t | f
core_location_loc_id | relation | 16388 | 302470 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel3_pkey | relation | 16388 | 302361 | | | | | | | | 5/11758 | 2744 | AccessShareLock | t | f
core_mymodel_mymodel3s_mymodel_id_5dcd946e263a391f_uniq | relation | 16388 | 302310 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
core_mymodel_start_date | relation | 16388 | 302487 | | | | | | | | 5/11758 | 2744 | RowExclusiveLock | t | f
core_mymodel_mymodel4_id | relation | 16388 | 302491 | | | | | | | | 4/44 | 2241 | RowExclusiveLock | t | f
core_mymodel_mymodel3s_pkey | relation | 16388 | 302312 | | | | | | | | 4/44 | 2241 | AccessShareLock | t | f
(91 rows)
据我所知,尝试访问id = 123行的唯一客户端是尝试更新它的客户端。我也不确定为什么它要求所有这些列的锁,当我从DB获取的所有内容都是主键列表时,该查询已经完成。此外,当我必须在挂起时终止迁移过程时,很多这些授予的锁定都会被锁定。我尝试重新启动数据库,但迁移仍然失败。
此外,SELECT pg_terminate_backend(pid);
并未停止任何后端,只有重新启动数据库才会清除它们。
答案 0 :(得分:0)
我重新启动了我的数据库,删除了新字段,对其进行了读取,然后运行VACUUM ANALYZE VERBOSE
。 UPDATE查询现在似乎已经完成,尽管其中一些查询将随机地比其他查询长几个数量级。