Question

我尝试使用moto中的@mock_redshift来模拟与AWS Redshift的连接，我正在使用boto3创建测试集群，但是当我进行查询时遇到以下错误：

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "recon-test.cg034hpkmmjt.us-east-1.redshift.amazonaws.com" to address: Name or service not known

代码如下：

@mock_redshift
@mock_s3
@pytest.mark.vcr
def test_extract_stp_both_dates():
    client = boto3.client('redshift', region_name='us-east-1')
    response = client.create_cluster(
        DBName='recon',
        ClusterIdentifier='recon-test',
        ClusterType='single-node',
        NodeType='ds2.xlarge',
        MasterUsername='cuenca',
        MasterUserPassword='password',
    )

    host = response['Cluster']['Endpoint']['Address']
    port = response['Cluster']['Endpoint']['Port']

    rs_client = RedshiftClient(
        'recon',
        'cuenca',
        'password',
        host,
        port,
    )

    rs_client.s.execute("CREATE TABLE table_test (attr VARCHAR);") # The error is here

    conn = boto3.resource('s3', region_name='us-east-1')
    conn.create_bucket(Bucket=os.environ['S3_BUCKET'])
    random.seed(1)
    responses.add_passthru('https://')
    extract('01/11/2018', '30/11/2018', rs_client=rs_client)

这是RedshiftClient

import sqlalchemy as sa
from sqlalchemy.orm import sessionmaker


class RedshiftClient:
    def __init__(
        self, database: str, user: str, password: str, host: str, port: str
    ):
        self.connection_string = (
            f'redshift+psycopg2://{user}:{password}@{host}:{port}/{database}'
        )
        self.engine = sa.create_engine(self.connection_string)
        self.sessionmaker = sessionmaker(bind=self.engine)
        self.s = self.sessionmaker()

    def exec_query(self, query: str) -> list:
        return self.s.execute(query).fetchall()

Answer 1

看看Cluster in their Github的moto实现。

mock_redshift功能模拟了boto3的Redshift集群管理API，而不模拟数据库本身。 moto提供的URL连接（即test.cg034hpkmmjt.us-east-1.redshift.amazonaws.com）将永远无法工作。

如果您想测试真正的Redshift访问，我认为一个不错的主意是使用运行Postgres的容器进行测试，一个容易提供此功能的Python不错的lib是testcontainers（Github link）< / p>

您可以使用以下方法安装到虚拟环境中

pip install testcontainers[postgresql]

并修改您的测试以包括

from testcontainers.postgres import PostgresContainer


def test_docker_run_postgress():
    postgres_container = PostgresContainer("postgres:9.5")
    with postgres_container as postgres:
        e = sqlalchemy.create_engine(postgres.get_connection_url())
        result = e.execute("SELECT version()")

我想指出，Postgres与Redshift不同，实际上有很多重要的区别（like highlighted in this article）。例如，最近一次影响我的人是Redshift不接受DISTINCT ON (column)语法。

尝试在python测试中模拟Redshift时出错

1 个答案: