Docker-Celery无法连接到Redis

时间:2019-03-29 03:33:05

标签: docker redis docker-compose celery

项目结构:

client
nginx
web/
   celery_worker.py
   project
   config.py
   api/

我在docker-compose中拥有以下服务:

version: '3.6'

services:

  web:
    build:
      context: ./services/web
      dockerfile: Dockerfile-dev
    volumes:
      - './services/web:/usr/src/app'
    ports:
      - 5001:5000
    environment:
      - FLASK_ENV=development
      - APP_SETTINGS=project.config.DevelopmentConfig
      - DATABASE_URL=postgres://postgres:postgres@web-db:5432/web_dev 
      - DATABASE_TEST_URL=postgres://postgres:postgres@web-db:5432/web_test
      - SECRET_KEY=my_precious  
    depends_on:  
      - web-db
      - redis

  celery:
    image: dev3_web
    restart: always
    volumes:
      - ./services/web:/usr/src/app
      - ./services/web/logs:/usr/src/app
    command: celery worker -A celery_worker.celery --loglevel=INFO -Q cache
    environment:
      - CELERY_BROKER=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - web
      - redis
    links:
      - redis:redis


  redis:
    image: redis:5.0.3-alpine
    restart: always
    expose:
      - '6379'
    ports:
      - '6379:6379'


  monitor:
    image: dev3_web
    ports:
      - 5555:5555
    command:  flower -A celery_worker.celery --port=5555 --broker=redis://redis:6379/0
    depends_on:
      - web
      - redis


  web-db:  
    build:
      context: ./services/web/project/db
      dockerfile: Dockerfile
    ports:
      - 5435:5432
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres


  nginx:
    build:
      context: ./services/nginx
      dockerfile: Dockerfile-dev
    restart: always
    ports:
      - 80:80
      - 8888:8888
    depends_on:
      - web
      - client
      - redis


  client:
    build:
      context: ./services/client
      dockerfile: Dockerfile-dev
    volumes:
      - './services/client:/usr/src/app'
      - '/usr/src/app/node_modules'
    ports:
      - 3007:3000
    environment:
      - NODE_ENV=development
      - REACT_APP_WEB_SERVICE_URL=${REACT_APP_WEB_SERVICE_URL}
    depends_on:
      - web
      - redis
  

芹菜日志

但是,根据此日志,芹菜无法连接:

celery_1   | [2019-03-29 03:09:32,111: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379/0: Error 99 connecting to localhost:6379. Address not available..
celery_1   | Trying again in 2.00 seconds...
  

网络日志

同样,不是web服务(运行后端),通过相同的日志:

web_1      | Waiting for postgres...
web_1      | PostgreSQL started
web_1      |  * Environment: development
web_1      |  * Debug mode: on
web_1      |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
web_1      |  * Restarting with stat
web_1      |  * Debugger is active!
web_1      |  * Debugger PIN: 316-641-271
web_1      | 172.21.0.9 - - [29/Mar/2019 03:03:17] "GET /users HTTP/1.0" 200 -
web_1      | 172.21.0.9 - - [29/Mar/2019 03:03:26] "POST /auth/register HTTP/1.0" 500 -
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 492, in connect
web_1      |     sock = self._connect()
web_1      |   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 550, in _connect
web_1      |     raise err
web_1      |   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 538, in _connect
web_1      |     sock.connect(socket_address)
web_1      | OSError: [Errno 99] Address not available
web_1      | 
web_1      | During handling of the above exception, another exception occurred:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
web_1      |     yield
web_1      |   File "/usr/lib/python3.6/site-packages/celery/app/base.py", line 744, in send_task
web_1      |     self.backend.on_task_call(P, task_id)
web_1      |   File "/usr/lib/python3.6/site-packages/celery/backends/redis.py", line 265, in on_task_call
web_1      |     self.result_consumer.consume_from(task_id)
web_1      |   File "/usr/lib/python3.6/site-packages/celery/backends/redis.py", line 125, in consume_from
web_1      |     return self.start(task_id)
web_1      |   File "/usr/lib/python3.6/site-packages/celery/backends/redis.py", line 107, in start
web_1      |     self._consume_from(initial_task_id)
web_1      |   File "/usr/lib/python3.6/site-packages/celery/backends/redis.py", line 132, in _consume_from
web_1      |     self._pubsub.subscribe(key)
web_1      |   File "/usr/lib/python3.6/site-packages/redis/client.py", line 3096, in subscribe
web_1      |     ret_val = self.execute_command('SUBSCRIBE', *iterkeys(new_channels))
web_1      |   File "/usr/lib/python3.6/site-packages/redis/client.py", line 3003, in execute_command
web_1      |     self.shard_hint
web_1      |   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 994, in get_connection
web_1      |     connection.connect()
web_1      |   File "/usr/lib/python3.6/site-packages/redis/connection.py", line 497, in connect
web_1      |     raise ConnectionError(self._error_message(e))
web_1      | redis.exceptions.ConnectionError: Error 99 connecting to localhost:6379. Address not available.
web_1      | 
web_1      | During handling of the above exception, another exception occurred:
web_1      | 
web_1      | Traceback (most recent call last):
web_1      |   File "/usr/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
web_1      |     return self.wsgi_app(environ, start_response)
web_1      |   File "/usr/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
web_1      |     response = self.handle_exception(e)
web_1      |   File "/usr/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
web_1      |     return cors_after_request(app.make_response(f(*args, **kwargs)))
web_1      |   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
web_1      |     reraise(exc_type, exc_value, tb)
web_1      |   File "/usr/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
web_1      |     raise value
web_1      |   File "/usr/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
web_1      |     response = self.full_dispatch_request()
web_1      |   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
web_1      |     rv = self.handle_user_exception(e)
web_1      |   File "/usr/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
web_1      |     return cors_after_request(app.make_response(f(*args, **kwargs)))
web_1      |   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
  

REDIS日志

Redis似乎可以正常工作:

redis_1    | 1:C 29 Mar 2019 02:33:32.722 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1    | 1:C 29 Mar 2019 02:33:32.722 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1    | 1:C 29 Mar 2019 02:33:32.722 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1    | 1:M 29 Mar 2019 02:33:32.724 * Running mode=standalone, port=6379.
redis_1    | 1:M 29 Mar 2019 02:33:32.724 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1    | 1:M 29 Mar 2019 02:33:32.724 # Server initialized
redis_1    | 1:M 29 Mar 2019 02:33:32.724 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1    | 1:M 29 Mar 2019 02:33:32.725 * DB loaded from disk: 0.000 seconds
redis_1    | 1:M 29 Mar 2019 02:33:32.725 * Ready to accept connections
  

config.py

class DevelopmentConfig(BaseConfig):
    """Development configuration"""
    DEBUG_TB_ENABLED = True 
    DEBUG = True
    BCRYPT_LOG_ROUNDS = 4 
    #set key
    #sqlalchemy
    SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL')
    #SQLALCHEMY_DATABASE_URI= "sqlite:///models/data/database.db"
    # mail
    MAIL_SERVER='smtp.gmail.com'
    MAIL_PORT = 587
    MAIL_USE_TLS = True
    MAIL_DEBUG = True
    MAIL_USERNAME = 'me@gmail.com'
    MAIL_PASSWORD = 'MEfAc6w74WGx'

    SEVER_NAME = 'http://127.0.0.1:8080'
    # celery broker
    REDIS_HOST = "0.0.0.0"
    REDIS_PORT = 6379
    BROKER_URL = os.environ.get('REDIS_URL', "redis://{host}:{port}/0".format(
                                                                    host=REDIS_HOST, 
                                                                    port=str(REDIS_PORT)))
    INSTALLED_APPS = ['routes']
    # celery config
    CELERYD_CONCURRENCY = 10
    CELERY_BROKER_URL = BROKER_URL
    CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
    CELERY_IMPORTS = ('project.api.routes.background',)

我在这里想念什么?

1 个答案:

答案 0 :(得分:1)

运行docker-compose时,它将创建一个新网络,所有容器都在该网络下运行。 Docker引擎还创建了一个内部路由,该路由允许所有容器使用它们的名称相互引用。

在您的情况下,您的Web和celery容器试图通过localhost访问Redis。但是在容器中,localhost表示自己的本地主机。您需要更改配置,以将主机名指向容器的name

如果您没有使用docker,但是每个容器都有不同的机器,那么localhost就意味着他们自己的服务器。为了连接到Redis服务器,您需要传递运行Redis的机器的IP地址。在docker中,由于上面讨论了引擎的路由,因此您只能传递容器的名称而不是IP地址。

请注意,您仍然可以为每个容器分配静态IP地址,并使用这些IP地址代替container_names。有关更多详细信息,请阅读Docker文档的网络部分。