SQLAlchemy-Redshift无法反映现有视图,`reflecttable`返回`KeyError`

时间:2018-12-10 19:44:45

标签: python sqlalchemy amazon-redshift

我正在使用sqlalchemy-redshift连接到AWS中的现有Redshift数据库。数据库中的大多数表和视图都有几列,因此我想自动加载其架构并获取Table对象。即我不想在python代码中指定表/视图的架构。我的问题是,尽管我能够自动加载表,但无法自动加载数据库中的视图。

SQLAlchemy docs表明它使用名为reflection的进程来加载视图/表的架构。虽然我能够注意到表的反射是成功的,但是反射视图却失败了。这是一个代码段。

In [1]: from sqlalchemy import inspect, create_engine, Table
In [2]: from sqlalchemy.ext.declarative import declarative_base
In [3]: Base=declarative_base()
In [4]: connection_string = 'redshift+psycopg2://user:password@host:5439/dbname'
In [5]: engine=create_engine(connection_string,
                             connect_args={'sslmode': 'verify-ca'},
                             echo=True)  # using echo=True only for debugging.
In [6]: insp = inspect(engine)
2018-12-10 11:02:56,544 INFO sqlalchemy.engine.base.Engine select version()
2018-12-10 11:02:56,545 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:02:56,620 INFO sqlalchemy.engine.base.Engine select current_schema()
2018-12-10 11:02:56,621 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:02:56,693 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
2018-12-10 11:02:56,694 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:02:56,732 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
2018-12-10 11:02:56,732 INFO sqlalchemy.engine.base.Engine {}

我可以成功反映表(并自动加载它们)。例如,

In [7]: insp.get_table_names()
2018-12-10 11:03:10,475 INFO sqlalchemy.engine.base.Engine 
        SELECT
          c.relkind,
          n.oid as "schema_oid",
          n.nspname as "schema",
          c.oid as "rel_oid",
          c.relname,
          CASE c.reldiststyle
            WHEN 0 THEN 'EVEN' WHEN 1 THEN 'KEY' WHEN 8 THEN 'ALL' END
            AS "diststyle",
          c.relowner AS "owner_id",
          u.usename AS "owner_name",
          TRIM(TRAILING ';' FROM pg_catalog.pg_get_viewdef(c.oid, true))
            AS "view_definition",
          pg_catalog.array_to_string(c.relacl, '
') AS "privileges"
        FROM pg_catalog.pg_class c
             LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
             JOIN pg_catalog.pg_user u ON u.usesysid = c.relowner
        WHERE c.relkind IN ('r', 'v', 'm', 'S', 'f')
          AND n.nspname !~ '^pg_'
        ORDER BY c.relkind, n.oid, n.nspname;

2018-12-10 11:03:10,475 INFO sqlalchemy.engine.base.Engine {}
Out[7]: 
['tab_1',
 'tab_2',
 'tab_3']
In [8]: load_tab=Table('tab_1',Base.metadata)
    ...: insp.reflecttable(load_tab, None)
2018-12-10 11:15:00,537 INFO sqlalchemy.engine.base.Engine 
            SELECT t.typname as "name",
               pg_catalog.format_type(t.typbasetype, t.typtypmod) as "attype",
               not t.typnotnull as "nullable",
               t.typdefault as "default",
               pg_catalog.pg_type_is_visible(t.oid) as "visible",
               n.nspname as "schema"
            FROM pg_catalog.pg_type t
               LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace
            WHERE t.typtype = 'd'

2018-12-10 11:15:00,538 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:15:00,669 INFO sqlalchemy.engine.base.Engine 
        SELECT
          n.nspname as "schema",
          c.relname as "table_name",
          t.contype,
          t.conname,
          t.conkey,
          a.attnum,
          a.attname,
          pg_catalog.pg_get_constraintdef(t.oid, true) as condef,
          n.oid as "schema_oid",
          c.oid as "rel_oid"
        FROM pg_catalog.pg_class c
        LEFT JOIN pg_catalog.pg_namespace n
          ON n.oid = c.relnamespace
        JOIN pg_catalog.pg_constraint t
          ON t.conrelid = c.oid
        JOIN pg_catalog.pg_attribute a
          ON t.conrelid = a.attrelid AND a.attnum = ANY(t.conkey)
        WHERE n.nspname !~ '^pg_'
        ORDER BY n.nspname, c.relname

2018-12-10 11:15:00,671 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:15:00,882 INFO sqlalchemy.engine.base.Engine 
            SELECT c.oid
            FROM pg_catalog.pg_class c
            LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
            WHERE (pg_catalog.pg_table_is_visible(c.oid))
            AND c.relname = %(table_name)s AND c.relkind in ('r', 'v', 'm', 'f')

2018-12-10 11:15:00,883 INFO sqlalchemy.engine.base.Engine {'table_name': 'tab_1'}
2018-12-10 11:15:01,003 INFO sqlalchemy.engine.base.Engine 
            SELECT
                cons.conname as name,
                cons.consrc as src
            FROM
                pg_catalog.pg_constraint cons
            WHERE
                cons.conrelid = %(table_oid)s AND
                cons.contype = 'c'

2018-12-10 11:15:01,005 INFO sqlalchemy.engine.base.Engine {'table_oid': 123456}
2018-12-10 11:15:01,114 INFO sqlalchemy.engine.base.Engine 
            SELECT
                pgd.description as table_comment
            FROM
                pg_catalog.pg_description pgd
            WHERE
                pgd.objsubid = 0 AND
                pgd.objoid = %(table_oid)s

2018-12-10 11:15:01,116 INFO sqlalchemy.engine.base.Engine {'table_oid': 123456}

但是,如果我尝试反映观点,则会得到KeyError

In [9]: insp.get_view_names()
Out[9]: 
['view_0',
 'view_1',
 'view_2',
 'view_3']
In [8]: view_info=Table('view_1', Base.metadata)
...: insp.reflecttable(view_info, None)
2018-12-10 11:08:41,429 INFO sqlalchemy.engine.base.Engine SHOW search_path
2018-12-10 11:08:41,430 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:08:41,505 INFO sqlalchemy.engine.base.Engine 
            SELECT nspname AS "name"
            FROM pg_catalog.pg_namespace n
            WHERE nspname !~ '^pg_'
                AND nspname <> 'information_schema'
                AND n.oid NOT IN
                  (SELECT esoid FROM pg_catalog.pg_external_schema)
            ORDER BY 1

2018-12-10 11:08:41,507 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:08:41,546 INFO sqlalchemy.engine.base.Engine SET LOCAL search_path TO "public"
2018-12-10 11:08:41,548 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:08:41,585 INFO sqlalchemy.engine.base.Engine 
            SELECT
              n.nspname as "schema",
              c.relname as "table_name",
              d.column as "name",
              encoding as "encode",
              type, distkey, sortkey, "notnull", adsrc, attnum,
              pg_catalog.format_type(att.atttypid, att.atttypmod),
              pg_catalog.pg_get_expr(ad.adbin, ad.adrelid) AS DEFAULT,
              n.oid as "schema_oid",
              c.oid as "table_oid"
            FROM pg_catalog.pg_class c
            LEFT JOIN pg_catalog.pg_namespace n
              ON n.oid = c.relnamespace
            JOIN pg_catalog.pg_table_def d
              ON (d.schemaname, d.tablename) = (n.nspname, c.relname)
            JOIN pg_catalog.pg_attribute att
              ON (att.attrelid, att.attname) = (c.oid, d.column)
            LEFT JOIN pg_catalog.pg_attrdef ad
              ON (att.attrelid, att.attnum) = (ad.adrelid, ad.adnum)
            WHERE n.nspname !~ '^pg_'
            ORDER BY n.nspname, c.relname, att.attnum

2018-12-10 11:08:41,586 INFO sqlalchemy.engine.base.Engine {}
2018-12-10 11:08:42,169 INFO sqlalchemy.engine.base.Engine SET LOCAL search_path TO "$user", public
2018-12-10 11:08:42,170 INFO sqlalchemy.engine.base.Engine {}
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-15-fb159573ea93> in <module>
      1 view_info=Table('view_1',
      2              Base.metadata)
----> 3 insp.reflecttable(view_info, None)

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py in reflecttable(self, table, include_columns, exclude_columns, _extend_on)
    600         # reflect table options, like mysql_engine
    601         tbl_opts = self.get_table_options(
--> 602             table_name, schema, **table.dialect_kwargs)
    603         if tbl_opts:
    604             # add additional kwargs to the Table if the dialect

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py in get_table_options(self, table_name, schema, **kw)
    310             return self.dialect.get_table_options(
    311                 self.bind, table_name, schema,
--> 312                 info_cache=self.info_cache, **kw)
    313         return {}
    314 

<string> in get_table_options(self, connection, table_name, schema, **kw)

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/sqlalchemy/engine/reflection.py in cache(fn, self, con, *args, **kw)
     52     ret = info_cache.get(key)
     53     if ret is None:
---> 54         ret = fn(self, con, *args, **kw)
     55         info_cache[key] = ret
     56     return ret

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/sqlalchemy_redshift/dialect.py in get_table_options(self, connection, table_name, schema, **kw)
    551                                             schema, **kw)
    552         columns = self._get_redshift_columns(connection, table_name,
--> 553                                              schema, **kw)
    554         sortkey_cols = sorted([col for col in columns if col.sortkey],
    555                               key=keyfunc)

~/.pyenv/versions/3.7.0/lib/python3.7/site-packages/sqlalchemy_redshift/dialect.py in _get_redshift_columns(self, connection, table_name, schema, **kw)
    643         if key not in all_columns.keys():
    644             key = key.unquoted()
--> 645         return all_columns[key]
    646 
    647     def _get_redshift_constraints(self, connection, table_name,

KeyError: 'public.view_1'

以下是正在使用的各种软件包的版本:

In [28]: import sqlalchemy_redshift; sqlalchemy_redshift.__version__
Out[28]: '0.7.1'
In [29]: import sqlalchemy; sqlalchemy.__version__
Out[29]: '1.2.14'
In [31]: import sys; sys.version
Out[31]: '3.7.0 (default, Oct 17 2018, 11:19:19) \n[Clang 10.0.0 (clang-1000.10.44.2)]'

任何人都可以分享有关如何解决此问题的见解吗?

1 个答案:

答案 0 :(得分:0)

这些视图未得到反映,因为它们是使用no schema binding选项创建的(视图 no schema binding可以正常工作)。有关此主题的更多讨论,请参见:https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/issues/122