从“ CI”框通过堡垒到Redshift的SSH隧道

时间:2019-08-05 20:11:02

标签: python amazon-web-services ssh sqlalchemy ssh-tunnel

我有一个堡垒主机可以访问的redshift实例。

我正在尝试触发一个循环CI作业,该作业会更改要重命名的db模式以包含git release标签版本。我们希望在创建针对master的git发行版时触发该​​事件。

CI工作流程执行以下操作:

1)通过AWS进行身份验证

2)运行python脚本,该脚本创建一个安全组,将其附加到堡垒主机上,并配置入口规则,以允许将CI盒的IP列为cidr。

3)打开一个ssh隧道,以通过该堡垒将sqlalchemy命令转发到redshift主机。

4)运行另一个更改数据库架构的python脚本

前两个步骤没有任何问题。 ssh隧道似乎正在出现问题。

ssh隧道在后台运行并调用python脚本ssh -o StrictHostKeyChecking=no -N -L 127.0.0.1:5439:$PRODUCTION_REDSHIFT_DNS:5439 -i /home/circleci/.ssh/id_rsa_d88b40ac655a406a2c26ef75202fc144 circle_ci@$PRODUCTION_BASTION_DNS & python3 archive_schema.py

$ PROD_REDSHIFT_CONNECTION_STRING redshift+psycopg2://$USER:$PASS@localhost:5439/$DB_NAME

我有一个简单的python脚本,该脚本打印数据库中的第一个表名并退出。我使用相同的连接字符串在堡垒中成功运行了它,但是当我在后台设置ssh隧道后运行它时,事情就变了。

import os
from sqlalchemy import create_engine, inspect

engine = create_engine(os.getenv("PROD_REDSHIFT_CONNECTION_STRING"))
inspector = inspect(engine)
for table_name in inspector.get_table_names():
    print(table_name)
    break

任何帮助将不胜感激。这是从圈子CI运行脚本时遇到的错误:

Traceback (most recent call last):
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 154, in reraise
    raise value
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 451, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5439?
could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5439?
could not connect to server: Cannot assign requested address
    Is the server running on host "localhost" (::1) and accepting
    TCP/IP connections on port 5439?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "z.py", line 7, in <module>
    inspector = inspect(engine)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/inspection.py", line 63, in inspect
    ret = reg(subject)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/reflection.py", line 141, in _insp
    return Inspector.from_engine(bind)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/reflection.py", line 136, in from_engine
    return bind.dialect.inspector(bind)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/base.py", line 2209, in __init__
    reflection.Inspector.__init__(self, conn)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/reflection.py", line 111, in __init__
    bind.connect().close()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2193, in connect
    return self._connection_cls(self, **kwargs)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 103, in __init__
    else engine.raw_connection()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2293, in raw_connection
    self.pool.unique_connection, _connection
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
    e, dialect, self
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 399, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value.with_traceback(tb)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 303, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 760, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 492, in checkout
    rec = pool._do_get()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 139, in _do_get
    self._dec_overflow()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 154, in reraise
    raise value
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 136, in _do_get
    return self._create_connection()
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 308, in _create_connection
    return _ConnectionRecord(self)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 437, in __init__
    self.__connect(first_connect_check=True)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 639, in __connect
    connection = pool._invoke_creator(self)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 451, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/home/circleci/repo/venv/lib/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5439?
could not connect to server: Connection refused
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 5439?
could not connect to server: Cannot assign requested address
    Is the server running on host "localhost" (::1) and accepting
    TCP/IP connections on port 5439?

(Background on this error at: http://sqlalche.me/e/e3q8)
Exited with code 1

1 个答案:

答案 0 :(得分:1)

这可能是因为执行第二个命令时第一个命令尚未建立隧道。您应该避免在ssh tunnel命令中使用&并使用-f