当活动主服务器失败时,Mesos副本主服务器不会继续

时间:2015-11-01 12:00:35

标签: apache-zookeeper centos7 mesos

我有以下设置 - 名为master的4个CentOS 7.0虚拟机,box01,box02,box03。

master VM有:mesos-master,mesos-slave

box01 :mesos-master,mesos-slave,zkServer

box02 :mesos-master,mesos-slave,zkServer

box03 :mesos-slave,zkServer

每当我在集群上运行一个mesos框架而没有使用zookeeper时,一切运行正常。但是,当我部署并启动zookeeper集群时,如果框架是从作为ACTIVE mesos master的SAME机器运行的,那么我运行的框架将只能完成。

E.g。我当选的主人在box01。如果我从box01运行一个框架,那就很好了。如果我从主盒运行它,我在客户端获得以下日志,它永远不会继续:

I1101 13:56:11.997733  5384 sched.cpp:164] Version: 0.24.0
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@716: Client environment:host.name=master.localdomain
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@724: Client environment:os.arch=3.10.0-229.el7.x86_64
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@725: Client environment:os.version=#1 SMP Fri Mar 6 11:36:42 UTC 2015
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@733: Client environment:user.name=root
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@log_env@753: Client environment:user.dir=/home/user/download
2015-11-01 13:56:12,011:5383(0x7f55fee16700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=box01:2181,box02:2181,box03:2181 sessionTimeout=10000 watcher=0x7f560236e6d4 sessionId=0 sessionPasswd=<null> context=0x7f5604003c50 flags=0
2015-11-01 13:56:12,018:5383(0x7f55fd613700):ZOO_INFO@check_events@1703: initiated connection to server [10.0.0.11:2181]
2015-11-01 13:56:12,025:5383(0x7f55fd613700):ZOO_INFO@check_events@1750: session establishment complete on server [10.0.0.11:2181], sessionId=0x150c2c9ffc6002d, negotiated timeout=10000
I1101 13:56:12.027992  5398 group.cpp:331] Group process (group(1)@10.0.0.10:35217) connected to ZooKeeper
I1101 13:56:12.028153  5398 group.cpp:805] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1101 13:56:12.028198  5398 group.cpp:403] Trying to create path '/mesos' in ZooKeeper
I1101 13:56:12.036267  5398 detector.cpp:156] Detected a new leader: (id='11')
I1101 13:56:12.037309  5398 group.cpp:674] Trying to get '/mesos/json.info_0000000011' in ZooKeeper
I1101 13:56:12.041631  5398 detector.cpp:481] A new leading master (UPID=master@10.0.0.11:5050) is detected
I1101 13:56:12.042068  5398 sched.cpp:262] New master detected at master@10.0.0.11:5050
I1101 13:56:12.043937  5398 sched.cpp:272] No credentials provided. Attempting to register without authentication

我们可以看到客户端成功发现10.0.0.11(box01)是代理主服务器。如果此时我杀死了代理mesos master(box01),那么将发生新的选举,并且由于2的法定人数(master和box03盒子)将选出一个新的主人。如果此主服务器是主服务器框,则框架将成功执行该任务。如果是box03,客户端会发现这是主人,并再次挂起。应该有一个简单的解释,但我现在似乎无法摆脱我的思维框。请帮忙。

我使用的是mesos-0.24.0,zookeeper-3.4.6。

动物园管理员-3.4.6 / CONF / zoo.cfg

tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=box01:2888:3888
server.2=box02:2888:3888
server.3=box03:2888:3888

/ etc / hosts文件

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.10   master master.localdomain
10.0.0.11   box01 box01.localdomain
10.0.0.12   box02 box02.localdomain
10.0.0.13   box03 box03.localdomain

在每台计算机上,防火墙设置为:

--firewall-cmd --list-ports
5051/tcp 3888/tcp 2181/tcp 2888/tcp 5050/tcp

要启动mesos-master我使用:

/home/user/download/mesos-0.24.0/build/bin/mesos-master.sh --ip=10.0.0.10 --work_dir=/home/user/download/data-mesos --zk=zk://box01:2181,box02:2181,box03:2181/mesos --quorum=2

要启动mesos-slave我使用:

/home/user/download/mesos-0.24.0/build/bin/mesos-slave.sh --master=zk://box01:2181,box02:2181,box03:2181/mesos

编辑:

事实证明,如果我在box02(10.0.0.12)上运行独立的mesos master,并且我尝试从master(10.0.0.10)框运行框架,则mesos master会收到框架运行请求作业,但是它没有被执行

box02 master log

主箱框架日志

[root@master ~]# java -Djava.library.path=/usr/local/lib -jar /home/user/download/test-framework/example-framework-1.0-SNAPSHOT-jar-with-dependencies.jar box02:5050
I1103 13:44:21.898962 20958 sched.cpp:164] Version: 0.24.0
I1103 13:44:21.910660 20972 sched.cpp:262] New master detected at master@10.0.0.12:5050
I1103 13:44:21.913422 20972 sched.cpp:272] No credentials provided. Attempting to register without authentication

因此,似乎zookeeper与问题无关,而是由于某种原因,master无法将任何内容发送回执行框架的机器(mesos调度程序)。

1 个答案:

答案 0 :(得分:0)

在您提供的主日志中,我猜测主服务器无法打开与您的框架的连接。主日志的这一部分看起来很可疑:

I1103 13:44:21.513394 11288 master.cpp:2094] Received SUBSCRIBE call for framework 'framework-example' at scheduler-a42792c3-3b5d-4bd3-a840-0e9ed4eaaab5@10.0.0.10:36455
I1103 13:44:21.513703 11288 master.cpp:2164] Subscribing framework framework-example with checkpointing disabled and capabilities [  ]
I1103 13:44:21.516088 11288 hierarchical.hpp:391] Added framework 20151103-134410-201326602-5050-11260-0000
I1103 13:44:21.517375 11288 master.cpp:4613] Sending 1 offers to framework 20151103-134410-201326602-5050-11260-0000 (framework-example) at scheduler-a42792c3-3b5d-4bd3-a840-0e9ed4eaaab5@10.0.0.10:36455
E1103 13:44:21.519042 11291 socket.hpp:174] Shutdown failed on fd=14: Transport endpoint is not connected [107]
I1103 13:44:21.520539 11288 master.cpp:1051] Framework 20151103-134410-201326602-5050-11260-0000 (framework-example) at scheduler-a42792c3-3b5d-4bd3-a840-0e9ed4eaaab5@10.0.0.10:36455 disconnected
I1103 13:44:21.520593 11288 master.cpp:2370] Disconnecting framework 20151103-134410-201326602-5050-11260-0000 (framework-example) at scheduler-a42792c3-3b5d-4bd3-a840-0e9ed4eaaab5@10.0.0.10:36455
I1103 13:44:21.520608 11288 master.cpp:2394] Deactivating framework 20151103-134410-201326602-5050-11260-0000 (framework-example) at scheduler-a42792c3-3b5d-4bd3-a840-0e9ed4eaaab5@10.0.0.10:36455
W1103 13:44:21.520922 11288 master.hpp:1409] Master attempted to send message to disconnected framework 20151103-134410-201326602-5050-11260-0000 (framework-example) at scheduler-a42792c3-3b5d-4bd3-a840-0e9ed4eaaab5@10.0.0.10:36455

请问您是否可以检查框架节点上的LIBPROCESS_IP变量是否设置正确,并且主服务器可以打开与框架节点的连接?