打开扭曲的套接字客户端的Dj​​ango命令会留下空闲的数据库连接

时间:2016-05-31 14:12:28

标签: python django postgresql twisted

我使用twisted连接到套接字服务器,并且使用django命令运行扭曲:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import json

from twisted.internet import reactor, task
from twisted.internet.protocol import ReconnectingClientFactory
from twisted.protocols.basic import LineReceiver

from django.core.management.base import BaseCommand, CommandError

from django.conf import settings


class SocketClientProtocol(LineReceiver):

    MAX_LENGTH = 64*1024*1024

    def __init__(self):
        self.setLineMode()

    def connectionMade(self):
        print "CONNECTION ESTABLISHED"

    def connectionLost(self, reason):
        print "CONNECTION LOST"
        print reason.getErrorMessage()

    def lengthLimitExceeded(self, length):
        print "EXCEED"
        print length

    def lineReceived(self, msg):
        data = json.loads(msg)
        #do sth with the data from the socket server
        handle_message(data["type"], data["data"])


class SocketClientFactory(ReconnectingClientFactory):

    def startedConnecting(self, connector):
        print 'Started to connect.'

    def buildProtocol(self, addr):
        print 'Connected'
        self.resetDelay()
        return SocketClientProtocol()

    def clientConnectionLost(self, connector, reason):
        print 'Lost connection.  Reason:', reason
        ReconnectingClientFactory.clientConnectionLost(self, connector, reason)

    def clientConnectionFailed(self, connector, reason):
        print 'Connection failed. Reason:', reason
        ReconnectingClientFactory.clientConnectionFailed(self, connector, reason)

class Command(BaseCommand):

    def handle(self, *args, **options):
        client_factory = SocketClientFactory()
        reactor.connectUNIX(settings.UNIX_OUT_SOCKET, client_factory)
        reactor.run()

我通过python manage.py connect启动此命令,然后让进程在Apache后面的wsgi webserver旁边运行。这很好用,但我遇到的问题是,在这个套接字客户端内启动的数据库连接通常不能正确关闭,只是堆积为开放和空闲连接。

select * from pg_stat_activity where datname = 'lt';的输出:

 datid | datname |  pid  | usesysid | usename | application_name | client_addr | client_hostname | client_port |         backend_start         | xact_start |          query_start          |         state_change          | waiting | state |            query                                                                                                                                        
 36649 | lt      | 11026 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       55167 | 2016-05-31 15:50:38.417288+02 |            | 2016-05-31 15:50:38.450209+02 | 2016-05-31 15:50:38.451045+02 | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'
 36649 | lt      | 11132 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       55348 | 2016-05-31 15:50:48.057967+02 |            | 2016-05-31 15:50:48.089656+02 | 2016-05-31 15:50:48.090441+02 | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'
 36649 | lt      | 10386 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       53816 | 2016-05-31 15:49:19.597695+02 |            | 2016-05-31 15:49:19.633149+02 | 2016-05-31 15:49:19.634685+02 | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'
 36649 | lt      | 11145 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       55356 | 2016-05-31 15:50:48.353581+02 |            | 2016-05-31 15:50:48.389896+02 | 2016-05-31 15:50:48.390985+02 | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'
 36649 | lt      | 11150 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       55373 | 2016-05-31 15:50:49.705282+02 |            | 2016-05-31 15:50:49.7269+02   | 2016-05-31 15:50:49.729029+02 | f       | idle  | SELECT "mediatorapp_audiodata"."id", "mediatorapp_audiodata"."index", "mediatorapp_audiodata"."source_file", "mediatorapp_audiodata"."duration", "mediatorapp_audiodata"."audiostream_id", "mediatorapp_audiodata"."start_timestamp", "mediatorapp_audiodata"."stop_timestamp" FROM "mediatorapp_audiodata" WHERE "mediatorapp_audiodata"."id" = 419871
 36649 | lt      | 10944 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       54990 | 2016-05-31 15:50:27.940832+02 |            | 2016-05-31 15:50:27.978202+02 | 2016-05-31 15:50:27.97917+02  | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'
 36649 | lt      | 11059 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       55243 | 2016-05-31 15:50:42.737117+02 |            | 2016-05-31 15:50:42.773878+02 | 2016-05-31 15:50:42.774562+02 | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'
 36649 | lt      | 11127 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       55342 | 2016-05-31 15:50:47.868249+02 |            | 2016-05-31 15:50:47.905184+02 | 2016-05-31 15:50:47.90642+02  | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'
 36649 | lt      | 10901 |    16384 | dbuser  |                  | 127.0.0.1   |                 |       54896 | 2016-05-31 15:50:23.208558+02 |            | 2016-05-31 15:50:23.246194+02 | 2016-05-31 15:50:23.247041+02 | f       | idle  | SELECT "mediatorapp_audiostream"."id", "mediatorapp_audiostream"."language_tag", "mediatorapp_audiostream"."session_id", "mediatorapp_audiostream"."out_file", "mediatorapp_audiostream"."source_file" FROM "mediatorapp_audiostream" WHERE "mediatorapp_audiostream"."id" = '1104880479774183399:de'

在某些时候,这会导致以下错误:

OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections

上面列出的查询由此函数生成:

def get_audio_stream(audiostream_id):
    try:
        return AudioStream.objects.get(id=audiostream_id)
    except AudioStream.DoesNotExist:
        return None

所以我只想通过它的id来获取一个对象。据我所知,Django之后应该关闭连接,但事实并非如此。是不是唯一不时挂起的查询,其他查询也是如此。但这个是最常见的一个。

我使用最新的django(9.6),扭曲的15.5和Postgresql 9.3。我已经尝试将CONN_MAX_AGE设置为不同的值,但没有成功。

那么这个问题的原因是什么?是因为我在这个扭曲的引擎后面运行它吗?我该如何解决?

1 个答案:

答案 0 :(得分:1)

在上面发布的代码中似乎没有代码可以创建与postgres的连接。问题出在其他地方,也就是上面连接代码的服务器代码在客户端断开连接时不会关闭数据库连接。

编辑:如果您在非WSGI应用程序中使用Django ORM调用,则数据库连接不会自动关闭。自动连接管理仅在WSGI请求的上下文中完成。它通过signals.request_finished信号完成此操作。

您可以使用以下代码手动管理连接并在不需要时关闭它们:

from django import db

# Use this to close all configured db connections after your query code.
# Note that Django ORM performs lazy queries, i.e. only executing SQL if
# you access the Model object properties retrieved from DB. So you should
# do this after you have finished ORM object tasks.
db.connections.close_all()

# or use this
db.connection.close()

# or for more control over which connection:
db.connections['default'].close()  # closes the default DB connection

如果您要手动管理连接,您还需要手动管理交易。