盐请求超时。主人没有回复

时间：2017-06-08 16:53:27

标签： salt-stack

我们突然开始收到错误：

  

Salt请求超时。主人没有回应。如果在验证主服务器启动后此错误仍然存​​在，则可能需要增加worker_threads。＆＃34;

虽然这似乎很容易解决，但将工作线程数从5增加到100并没有解决问题。类似salt "*" cmd.run 'echo "hello"'的命令成功，而同一命令放入状态文件并使用salt "*" state.apply hello调用失败。虽然今天甚至cmd.run调用都失败了。

我们尝试过各种各样的补救措施，我们在这个问题上找到了这些补救措施，而其他补救措施也没有成功，特别是：清除缓存，查杀所有作业，删除/ var / cache / salt / minion目录并重新启动minions，重新启动主处理器，删除/ var / cache / salt / master然后停止并重新启动salt-master，确保在停止时仍然没有任何杂散进程正在运行。

我们没有运行防火墙，我们已经验证了端口是否已打开并正在通信（通过成功的cmd.run调用证明）

主人的盐版本：

---> salt-master --versions-report
Salt Version:
           Salt: 2016.11.3

Dependency Versions:
           cffi: 0.8.6
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 11 2015, 17:47:16)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.2 Maipo
        machine: x86_64
        release: 3.10.0-327.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.2 Maipo

奴才的盐版：

==>salt-minion --versions-report
Salt Version:
           Salt: 2016.11.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug 29 2016, 10:12:21)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.2 Maipo
        machine: x86_64
        release: 3.10.0-327.36.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.2 Maipo

我创建了一个简单的状态文件作为示例，向您展示我们正在看到的输出。我故意在输出中输出IP值和主机名。

---> cat hello.sls
say hi:
  cmd.run:
    - name: echo "hello"

在主人身上运行时的结果：

---> salt -l debug "*" state.apply hello
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: xx.xx.xx
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: xx.xx.xx
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'xxx.xxx.xxx_master', 'tcp://xx.xx.xx.xx:4506', 'clear')
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] LazyLoaded local_cache.get_load
[DEBUG   ] Reading minion list from /var/cache/salt/master/jobs/9f/e5c817cfcbf1eedb46f97566072a7a3a102bd7505af34a3d42f889bb189e27/.minions.p
[DEBUG   ] get_iter_returns for jid 20170608110512021926 sent to set(['xxx.xxx', 'xxx.xxx']) will timeout at 11:05:17.037291
[DEBUG   ] Checking whether jid 20170608110512021926 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'xxx.xxx.xxx_master', 'tcp://xx.xx.xx.xx:4506', 'clear')
[DEBUG   ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
[DEBUG   ] Checking whether jid 20170608110512021926 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'xxx.xxx.xxx_master', 'tcp://xx.xx.xx.xx:4506', 'clear')
[DEBUG   ] Passing on saltutil error. This may be an error in saltclient. 'retcode'
[DEBUG   ] Checking whether jid 20170608110512021926 is still running
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/master', 'xxx.xxx.xxx_master', 'tcp://xx.xx.xx.xx:4506', 'clear')
[DEBUG   ] SaltReqTimeoutError, retrying. (1/3)
[DEBUG   ] SaltReqTimeoutError, retrying. (2/3)
[DEBUG   ] SaltReqTimeoutError, retrying. (3/3)
[DEBUG   ] LazyLoaded nested.output
Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.

在小兵上运行时的结果：

==>salt-call -l debug state.apply hello
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: xxx
[DEBUG   ] Configuration file path: /etc/salt/minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/_schedule.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/_schedule.conf
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'xxx.xxx', 'tcp://xx.xx.xx.xx:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (4260)
[DEBUG   ] Setting zmq_reconnect_ivl to '4260ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'xxx.xxx', 'tcp://xx.xx.xx.xx:4506', 'clear')
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_64f68ad064_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_64f68ad064_pull.ipc
[DEBUG   ] Initializing new IPCClient for path: /var/run/salt/minion/minion_event_64f68ad064_pull.ipc
[DEBUG   ] Sending event: tag = salt/auth/creds; data = {'_stamp': '2017-06-08T15:08:50.733104', 'creds': {'publish_port': 4505, 'aes': 'L/jKbh4rxM99DNrJeL6hCHTIqr4sztwZs8vqM5M9vVzOfaAe+BZrOWTFNQ87Pu7UA0T7YmGNcf0=', 'master_uri': 'tcp://xx.xx.xx.xx:4506'}, 'key': ('/etc/salt/pki/minion', 'xxx.xxx', 'tcp://xx.xx.xx.xx:4506')}
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Determining pillar cache
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'xxx.xxx', 'tcp://xx.xx.xx.xx:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'xxx.xxx', 'tcp://xx.xx.xx.xx:4506')
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'xxx.xxx', 'tcp://xx.xx.xx.xx:4506', 'aes')
[DEBUG   ] Initializing new AsyncAuth for ('/etc/salt/pki/minion', 'xxx.xxx', 'tcp://xx.xx.xx.xx:4506')
[DEBUG   ] SaltReqTimeoutError, retrying. (1/3)
[DEBUG   ] SaltReqTimeoutError, retrying. (2/3)
[DEBUG   ] SaltReqTimeoutError, retrying. (3/3)
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
SaltReqTimeoutError: Message timed out
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
     salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 379, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 543, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 1002, in sls
    mocked=kwargs.get('mock', False))
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 3526, in __init__
    BaseHighState.__init__(self, opts)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 2441, in __init__
    self.opts = self.__gen_opts(opts)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 2468, in __gen_opts
    mopts = self.client.master_opts()
  File "/usr/lib/python2.7/site-packages/salt/fileclient.py", line 1338, in master_opts
    return self.channel.send(load)
  File "/usr/lib/python2.7/site-packages/salt/utils/async.py", line 75, in wrap
    ret = self._block_future(ret)
  File "/usr/lib/python2.7/site-packages/salt/utils/async.py", line 85, in _block_future
    return future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 276, in send
    ret = yield self._crypted_transfer(load, tries=tries, timeout=timeout, raw=raw)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 244, in _crypted_transfer
    ret = yield _do_transfer()
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 228, in _do_transfer
    tries=tries,
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
SaltReqTimeoutError: Message timed out
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 379, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 134, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 197, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 543, in apply_
    return sls(mods, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 1002, in sls
    mocked=kwargs.get('mock', False))
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 3526, in __init__
    BaseHighState.__init__(self, opts)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 2441, in __init__
    self.opts = self.__gen_opts(opts)
  File "/usr/lib/python2.7/site-packages/salt/state.py", line 2468, in __gen_opts
    mopts = self.client.master_opts()
  File "/usr/lib/python2.7/site-packages/salt/fileclient.py", line 1338, in master_opts
    return self.channel.send(load)
  File "/usr/lib/python2.7/site-packages/salt/utils/async.py", line 75, in wrap
    ret = self._block_future(ret)
  File "/usr/lib/python2.7/site-packages/salt/utils/async.py", line 85, in _block_future
    return future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 276, in send
    ret = yield self._crypted_transfer(load, tries=tries, timeout=timeout, raw=raw)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 244, in _crypted_transfer
    ret = yield _do_transfer()
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/site-packages/salt/transport/zeromq.py", line 228, in _do_transfer
    tries=tries,
  File "/usr/lib64/python2.7/site-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib64/python2.7/site-packages/tornado/concurrent.py", line 214, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out

1 个答案:

答案 0 :(得分：0)

尝试从小爪中除去盐和小爪盐。之后，从目录（您用于默认python版本的目录）中删除所有扩展名为.pyc的文件：

/ usr / lib / python / site-packages / salt / *。pyc

这样做之后，重新安装盐小爪。我希望这能解决您的问题。

相关问题
最新问题