尝试使用docker-compose设置相互连接时pgpool2看门狗崩溃

时间：2018-10-01 18:12:05

标签： postgresql docker docker-compose pgpool

我一直试图让pgpool2 +看门狗在本地docker堆栈中运行。没有看门狗，这实际上很容易，而且我能够按照文档操作，并让pgpool在主从模式下进行负载平衡。

麻烦来自看门狗。哪个pgpool容器首先启动，最终会选择自己作为主节点，因为它首先进行了初始化。好，没问题但是，任何其他初始化并尝试连接到主节点的节点都将被完全拒绝，从而导致它们退出。

（在此为所有日志和配置文件提前道歉；我将confs限制在与看门狗相关的部分，但是如果需要，我可以添加更多内容）

pgpool-1 conf 

use_watchdog = on
trusted_servers = 'app'
ping_path = '/bin'
wd_ipc_socket_dir = '/var/run/pgpool'
wd_monitoring_interfaces_list = ''
wd_hostname = '172.31.0.7'
wd_port = 9000
wd_authkey = ''

delegate_IP = ''
ifconfig_path = '/sbin'
if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0'
if_down_cmd = 'ifconfig eth0:0 down'
arping_path = '/usr/sbin'
arping_cmd = 'arping -U $_IP_$ -w 1'
clear_memqcache_on_escalation = on
wd_escalation_command = ''
wd_monitoring_interfaces_list = 'eth0'
wd_lifecheck_method = 'heartbeat'
wd_interval = 10
wd_heartbeat_port = 9694
wd_heartbeat_keepalive = 2
wd_heartbeat_deadtime = 30
heartbeat_destination0 = 'pgpool2'
heartbeat_destination_port0 = 9694
heartbeat_device0 = 'eth0'

wd_life_point = 3
wd_lifecheck_query = 'SELECT 1'
wd_lifecheck_dbname = 'template1'
wd_lifecheck_user = 'nobody'
wd_lifecheck_password = ''

other_pgpool_hostname0 = 'pgpool2'
other_pgpool_port0 = 5432
other_wd_port0 = 9000

pgpool-2 conf 

use_watchdog = on
trusted_servers = 'srp'
ping_path = '/bin'
wd_ipc_socket_dir = '/var/run/pgpool'
wd_monitoring_interfaces_list = ''
wd_hostname = '172.31.0.9'
wd_port = 9000
wd_authkey = ''
delegate_IP = ''
ifconfig_path = '/sbin'
if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0'
if_down_cmd = 'ifconfig eth0:0 down'
arping_path = '/usr/sbin'           # arping command path
arping_cmd = 'arping -U $_IP_$ -w 1'
clear_memqcache_on_escalation = on
wd_escalation_command = ''
wd_monitoring_interfaces_list = 'eth0'
wd_lifecheck_method = 'heartbeat'
wd_interval = 10
wd_heartbeat_port = 9694
wd_heartbeat_keepalive = 2
wd_heartbeat_deadtime = 30
heartbeat_destination0 = 'pgpool1'
heartbeat_destination_port0 = 9694 
heartbeat_device0 = 'eth0'

wd_life_point = 3
wd_lifecheck_query = 'SELECT 1'
wd_lifecheck_dbname = 'template1'
wd_lifecheck_user = 'nobody'
wd_lifecheck_password = ''

other_pgpool_hostname0 = 'pgpool1'
other_pgpool_port0 = 5432
other_wd_port0 = 9000

docker-compose.yml 

services:
...

  database:
    container_name: "database"
    image: postgres:9.5-custom
    environment:
      - POSTGRES_PASSWORD=password
      - PGDATA=/var/lib/postgresql/data/pgdata
    volumes:
      - '/var/lib/postgresql/data'
    expose:
      - '5432'
    ports:
      - '5431:5432'
  database-slave:
    container_name: "database-slave"
    image: postgres:9.5-custom
    environment:
      - POSTGRES_PASSWORD=password
      - PGDATA=/var/lib/postgresql/data/pgdata
      - REPLICATE_FROM=database
    volumes:
      - '/var/lib/postgresql/data'
    expose:
     - '5432'
    ports:
      - '5430:5432'
    depends_on:
      - database
  pgpool1:
    container_name: "pgpool1"
    image: pgpool:latest
    depends_on:
      - database
    environment:
      - PGPOOL_BACKENDS=0:database:5432,1:database-slave:5432
      - PGPOOL_SR_CHECK_PASSWORD=password
      - PGPOOL_POOL_PASSWD_USER=someuser
      - PGPOOL_POOL_PASSWD_PASSWORD=password
      - PGPOOL_WD_TRUSTED_SERVERS=app
      - PGPOOL_OTHER_PGPOOL_HOSTNAMES=0:pgpool2:9000
      - PGPOOL_WD_HEARTBEAT_DESTINATIONS=0:pgpool2:eth0
      - SLEEP=10
    expose:
      - '9000'
      - '5432'
      - '9694'

  pgpool2:
    container_name: "pgpool2"
    image: pgpool:latest
    depends_on:
      - database
    environment:
      - PGPOOL_BACKENDS=0:database:5432,1:database-slave:5432
      - PGPOOL_SR_CHECK_PASSWORD=password
      - PGPOOL_POOL_PASSWD_USER=someuser
      - PGPOOL_POOL_PASSWD_PASSWORD=password
      - PGPOOL_WD_TRUSTED_SERVERS=app
      - PGPOOL_OTHER_PGPOOL_HOSTNAMES=0:pgpool1:9000
      - PGPOOL_WD_HEARTBEAT_DESTINATIONS=0:pgpool1:eth0
      - SLEEP=15
    expose:
      - '9000'
      - '5432'
      - '9694'

pgpool-1（主）输出：

2018-10-01 18:00:56: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:00:56: pid 21: DETAIL:  num_backends: 2 total_weight: 2.000000
2018-10-01 18:00:56: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:00:56: pid 21: DETAIL:  backend 0 weight: 1073741823.500000 flag: 0000
2018-10-01 18:00:56: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:00:56: pid 21: DETAIL:  backend 1 weight: 1073741823.500000 flag: 0000
2018-10-01 18:00:56: pid 21: DEBUG:  loading hba configuration
2018-10-01 18:00:56: pid 21: DETAIL:  loading file :"/etc/pgpool2/pool_hba.conf" for client authentication configuration file
2018-10-01 18:00:56: pid 21: LOG:  Backend status file /var/log/postgresql/pgpool_status does not exist
2018-10-01 18:00:56: pid 21: DEBUG:  pool_coninfo_size: num_init_children (32) * max_pool (4) * MAX_NUM_BACKENDS (128) * sizeof(ConnectionInfo) (136) = 2228224 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: DEBUG:  ProcessInfo: num_init_children (32) * sizeof(ProcessInfo) (32) = 1024 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: DEBUG:  Request info are: sizeof(POOL_REQUEST_INFO) 5304 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: DEBUG:  Recovery management area: sizeof(int) 4 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: LOG:  waiting for watchdog to initialize
2018-10-01 18:00:56: pid 22: LOG:  setting the local watchdog node name to "172.31.0.8:5432 Linux 12affe97a14d"
2018-10-01 18:00:56: pid 22: LOG:  watchdog cluster is configured with 1 remote nodes
2018-10-01 18:00:56: pid 22: LOG:  watchdog remote node:0 on pgpool2:9000
2018-10-01 18:00:56: pid 22: DEBUG:  adding monitoring interface [0] name eth0 index 1950
2018-10-01 18:00:56: pid 22: INFO:  IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
2018-10-01 18:00:56: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:00:56: pid 22: DEBUG:  network interface tunl0 having flags 128
2018-10-01 18:00:56: pid 22: DEBUG:  network interface ip6tnl0 having flags 128
2018-10-01 18:00:56: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:00:56: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:00:56: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" is up and we can continue
2018-10-01 18:00:56: pid 22: LOG:  watchdog node state changed from [DEAD] to [LOADING]
2018-10-01 18:00:56: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = LOADING
2018-10-01 18:00:56: pid 22: DEBUG:  error in outbound connection to pgpool2:9000
2018-10-01 18:00:56: pid 22: DETAIL:  Connection refused
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = TIMEOUT Current State = LOADING
2018-10-01 18:01:01: pid 22: LOG:  watchdog node state changed from [LOADING] to [JOINING]
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = JOINING
2018-10-01 18:01:01: pid 22: DEBUG:  error in outbound connection to pgpool2:9000
2018-10-01 18:01:01: pid 22: DETAIL:  Connection refused
2018-10-01 18:01:01: pid 22: LOG:  new watchdog node connection is received from "172.31.0.9:27844"
2018-10-01 18:01:01: pid 22: DEBUG:  received watchdog packet type:A
2018-10-01 18:01:01: pid 22: DEBUG:  reading packet type A of length 242
2018-10-01 18:01:01: pid 22: DEBUG:  ADD NODE MESSAGE from hostname:"172.31.0.9" port:9000 pgpool_port:5432
2018-10-01 18:01:01: pid 22: NOTICE:  add node from hostname:"172.31.0.9" port:9000 pgpool_port:5432 rejected.
2018-10-01 18:01:01: pid 22: DETAIL:  verify the other watchdog node configurations
2018-10-01 18:01:01: pid 22: DEBUG:  sending packet, watchdog node:[�
                                                                     WaVU] command id:[4] type:[REJECT] state:[JOINING]
2018-10-01 18:01:01: pid 22: DEBUG:  sending watchdog packet to socket:7, type:[R], command ID:4, data Length:0

pgpool-2（崩溃）输出：

2018-10-01 18:01:01: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:01:01: pid 21: DETAIL:  num_backends: 2 total_weight: 2.000000
2018-10-01 18:01:01: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:01:01: pid 21: DETAIL:  backend 0 weight: 1073741823.500000 flag: 0000
2018-10-01 18:01:01: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:01:01: pid 21: DETAIL:  backend 1 weight: 1073741823.500000 flag: 0000
2018-10-01 18:01:01: pid 21: DEBUG:  loading hba configuration
2018-10-01 18:01:01: pid 21: DETAIL:  loading file :"/etc/pgpool2/pool_hba.conf" for client authentication configuration file
2018-10-01 18:01:01: pid 21: LOG:  Backend status file /var/log/postgresql/pgpool_status does not exist
2018-10-01 18:01:01: pid 21: DEBUG:  pool_coninfo_size: num_init_children (32) * max_pool (4) * MAX_NUM_BACKENDS (128) * sizeof(ConnectionInfo) (136) = 2228224 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: DEBUG:  ProcessInfo: num_init_children (32) * sizeof(ProcessInfo) (32) = 1024 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: DEBUG:  Request info are: sizeof(POOL_REQUEST_INFO) 5304 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: DEBUG:  Recovery management area: sizeof(int) 4 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: LOG:  waiting for watchdog to initialize
2018-10-01 18:01:01: pid 22: LOG:  setting the local watchdog node name to "172.31.0.9:5432 Linux b251a3a36756"
2018-10-01 18:01:01: pid 22: LOG:  watchdog cluster is configured with 1 remote nodes
2018-10-01 18:01:01: pid 22: LOG:  watchdog remote node:0 on pgpool1:9000
2018-10-01 18:01:01: pid 22: DEBUG:  adding monitoring interface [0] name eth0 index 1952
2018-10-01 18:01:01: pid 22: INFO:  IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
2018-10-01 18:01:01: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 22: DEBUG:  network interface tunl0 having flags 128
2018-10-01 18:01:01: pid 22: DEBUG:  network interface ip6tnl0 having flags 128
2018-10-01 18:01:01: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" is up and we can continue
2018-10-01 18:01:01: pid 22: LOG:  watchdog node state changed from [DEAD] to [LOADING]
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = LOADING
2018-10-01 18:01:01: pid 22: LOG:  new outbound connection to pgpool1:9000
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = NEW OUTBOUND_CONNECTION Current State = LOADING
2018-10-01 18:01:01: pid 22: DEBUG:  sending packet, watchdog node:[] command id:[4] type:[ADD NODE] state:[LOADING]
2018-10-01 18:01:01: pid 22: DEBUG:  sending watchdog packet to socket:7, type:[A], command ID:4, data Length:242
2018-10-01 18:01:01: pid 22: DEBUG:  received watchdog packet type:R
2018-10-01 18:01:01: pid 22: DEBUG:  reading packet type R of length 0
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = LOADING
2018-10-01 18:01:01: pid 22: DEBUG:  received packet, watchdog node:[] command id:[4] type:[REJECT] state:[LOADING]
2018-10-01 18:01:01: pid 22: FATAL:  Add to watchdog cluster request is rejected by node "pgpool1:9000"
2018-10-01 18:01:01: pid 22: HINT:  check the watchdog configurations.
2018-10-01 18:01:01: pid 22: LOG:  Watchdog is shutting down
2018-10-01 18:01:01: pid 21: DEBUG:  reaper handler
2018-10-01 18:01:01: pid 21: DEBUG:  watchdog child process with pid: 22 exit with FATAL ERROR. pgpool-II will be shutdown
2018-10-01 18:01:01: pid 21: LOG:  watchdog child process with pid: 22 exits with status 768
2018-10-01 18:01:01: pid 21: FATAL:  watchdog child process exit with fatal error. exiting pgpool-II
2018-10-01 18:01:01: pid 23: LOG:  setting the local watchdog node name to "172.31.0.9:5432 Linux b251a3a36756"
2018-10-01 18:01:01: pid 23: LOG:  watchdog cluster is configured with 1 remote nodes
2018-10-01 18:01:01: pid 23: LOG:  watchdog remote node:0 on pgpool1:9000
2018-10-01 18:01:01: pid 23: DEBUG:  adding monitoring interface [0] name eth0 index 1952
2018-10-01 18:01:01: pid 23: INFO:  IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
2018-10-01 18:01:01: pid 23: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 23: DEBUG:  network interface tunl0 having flags 128
2018-10-01 18:01:01: pid 23: DEBUG:  network interface ip6tnl0 having flags 128
2018-10-01 18:01:01: pid 23: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 23: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 23: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" is up and we can continue

pgpool-2在最后一个日志条目处退出。

我尝试过的，已经确认的

  • 尝试以root用户身份运行pgpool，尝试以postgres身份运行pgpool
  • 尝试wd_authkey，但不尝试SSL
  • 尝试与docker privileged: true一起运行
  • 尝试显式定义ports:而不是仅仅公开它们，因此docker网络不需要将随机端口映射到指定的公开端口
  • 尝试使用通配符listen_address，并尝试将主机ip作为listen_address（此ip源自docker容器内，因此我不确定是否可以使用
  • 尝试使用和不定义wd_heartbeat设置
  • 将pgpool.conf用于两个进程，如果我将其设置为非通配符，则唯一不同的设置是other_wd_hosts，wd_hostname和listen_address

在这一点上，我已经走了好几次，脑袋清晰地重新审视，但我仍然不知道为什么我会看到这种行为。我在这里缺少的任何内容都可能导致连接被主看门狗拒绝吗？

1 个答案:

答案 0 :(得分：0)

哦，天哪，发布此消息仅几分钟后，我正在仔细检查我以前仔细检查过的输出，但没有在其中留下任何服务器名称，帐号或类似信息。我发现了问题。

tl; dr ：wd_hostname和other_pgpool_hostname的设置必须在不同的pgpool节点之间匹配。 other_pgpool_hostname基本上是白名单设置。

我注意到这一行：

2018-10-01 18:01:01: pid 23: LOG: setting the local watchdog node name to "172.31.0.9:5432 Linux b251a3a36756"

我想知道是否将wd_hostname设置为IP地址是否有问题，因为没有将other_pgpool_hostname设置设置为IP，而是将它们设置为docker-compose。 yml服务名称（正确）。

我更改了配置，将wd_hostname设置为与给定容器的docker-compose.yml服务名称匹配，并且神奇地连接了所有东西。

相关问题
最新问题