FIWARE Sanity check status errors because of request timeouts

时间：2015-11-12 11:09:44

标签： fiware-health

We use FIWARE Sanity checks for our FIWARE Lab node (OpenStack environment in Juno using Ubuntu 12.04.4) and last week we have noticed errors appearing in different tests every time. The load on our node is not heavy and has not changed dramatically the last three months. All OpenStack services seem to work in a normal way and we don't have a network traffic problem. We have noticed that errors become from timeouts in requests to our node from the machine which executes the sanity checks. We don't know where these response delays come from and we don't know how to resolve this problem. Follows part of the nova-api.log that might help.

===== nova-api.log =================

2015-11-10 12:08:12.448 121152 ERROR nova.api.openstack [req-bbcf04ce-2976-48ae-8d4a-5d057468afcd None] Caught error: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack Traceback (most recent call last):
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return req.get_response(self.application)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack application, catch_exc_info=False)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 748, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self._call_app(env, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 684, in _call_app
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack content_type, body, accept)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1001, in _process_stack
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     request, action_args)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 880, in post_process_extensions
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack **action_args)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 597, in show
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self._show(req, resp_obj)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 593, in _show
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack self._extend_servers(req, [resp_obj.obj['server']])
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 557, in _extend_servers
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack servers))
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 346, in get_instances_security_groups_bindings
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 305, in _get_ports_from_server_list
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack ports.extend(neutron.list_ports(**search_opts).get('ports'))
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 98, in with_params
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 312, in list_ports
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack **_params)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1334, in list
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1347, in _pagination
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     res = self.get(path, params=params)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1320, in get
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack headers=headers, params=params)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 748, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self._call_app(env, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 684, in _call_app
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return resp(environ, start_response)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack content_type, body, accept)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1001, in _process_stack
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     request, action_args)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 880, in post_process_extensions
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack **action_args)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 597, in show
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     return self._show(req, resp_obj)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 593, in _show
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack self._extend_servers(req, [resp_obj.obj['server']])
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/security_groups.py", line 557, in _extend_servers
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack servers))
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 346, in get_instances_security_groups_bindings
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     ports = self._get_ports_from_server_list(servers, neutron)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 305, in _get_ports_from_server_list
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack ports.extend(neutron.list_ports(**search_opts).get('ports'))
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 98, in with_params
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     ret = self.function(instance, *args, **kwargs)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 312, in list_ports
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack **_params)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1334, in list
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     for r in self._pagination(collection, path, **params):
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1347, in _pagination
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     res = self.get(path, params=params)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1320, in get
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack headers=headers, params=params)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack headers=headers, params=params)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1240, in do_request
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack content_type=self.content_type())
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 180, in do_request
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack **kwargs)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 133, in _cs_request
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack     raise exceptions.ConnectionFailed(reason=e)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='192.168.0.2', port=9696): Read timed out. (read timeout=30)
2015-11-10 12:08:12.448 121152 TRACE nova.api.openstack
2015-11-10 12:08:12.450 121152 INFO nova.api.openstack [req-bbcf04ce-2976-48ae-8d4a-5d057468afcd None] http://<controller_ip>:8774/v2/8368995edf4042da9df4fb3ddcd45d89/servers/657a22a3-a91e-4f2a-a9b6-12c0268d9b25 returned with HTTP 500
2015-11-10 12:08:12.450 121152 INFO nova.osapi_compute.wsgi.server [req-bbcf04ce-2976-48ae-8d4a-5d057468afcd None] 130.206.84.3 "GET /v2/8368995edf4042da9df4fb3ddcd45d89/servers/657a22a3-a91e-4f2a-a9b6-12c0268d9b25 HTTP/1.1" status: 500 len: 359 time: 30.0451050

1 个答案:

答案 0 :(得分：0)

通常，Sanity Check工具在定期执行这些测试的计算机上运行。这台机器的IP是130.206.84.3，你可以在日志中看到。

您报告的错误是由于此机器（完整性检查）导致我们无法检查虚拟机的状态，因为对您的nova服务的请求花费超过30秒，或者在您的情况下，您的nova正在响应GET操作有500个状态错误，而不是相应的服务器数据。

相关问题
最新问题