sqlalchemy金字塔查询keyerror

时间：2014-03-14 22:36:00

标签： mysql python-2.7 orm sqlalchemy pyramid

我正在开发一个项目，使用SQLAlchemy和金字塔框架将本地数据加载到本地服务器上。以下描述了设置：

- 本地数据分为多个CSV文件

- 从每个单独的CSV文件

创建单独的SQL对象（称为DataSource）

- 所有DataSource的父级也是DataSource。因此，所有CSV数据都是子项

因此，创建了父DataSource，然后创建了所有子DataSource并将其链接到父级。单独的函数检查DataSource是否已存在。如果没有，则创建它。

所以，首先我们创建父DS：

sName = 'Parent DS'
oDS = fetch_or_create_data_source(name=sName,author="Parent_DS_author"

）

然后我们遍历所有CSV并创建子DS：

for sCSV_name in lCSVs:
    oChildDS = fetch_or_create_data_source(name=sCSV_name,parent=oDS)

问题在于：

fetch_or_create_data_source在CSV加载的第二次迭代中创建一个keyerror。因此创建父级和第一个子DataSources没有问题。下面的错误日志是在第二个子DS上创建的。应该创建一个新的DS，因为不存在具有给定父级和名称的DS。所有CSV文件名都是唯一的 任何人都可以看到为什么我在DS创建的第二次迭代中得到一个神秘的密钥错误？我在网上找不到任何东西。

（编辑）：我知道错误是在lDS = list(oDS.filter_by(name=name).filter_by(parent=parent))中创建的，但我不明白为什么，特别是在前两次运行后没有问题。我更改了代码，以便调用parent_id而不是父对象，但KeyError仍然存在。该错误是由任何filter_by()语句引起的。为了检查原始值，我添加了一些print语句来检查fetch_or_create_data_source中的参数。结果：

name = 'Parent DS' type <type 'str'>
author = 'Parent_DS_author' type <type 'str'>
parent = None
Current DS id = '352' type <type 'int'>
.
.
name = 'tab19' type <type 'str'>
author = '' type <type 'str'>
parent = <pyramidapp.models.DataSource object at 0x7fcb28066b50>
parent_id = 352 type <type 'int'>
Current DS id = '353' type <type 'int'>
.
.
name = 'tab42' type <type 'str'>
author = '' type <type 'str'>
parent = <pyramidapp.models.DataSource object at 0x7fcb28066b50>
parent_id = 352 type <type 'int'>

当我明确声明oDS为None时，键错误来自第3次迭代中的DBSession.flush（）。

为什么fetch_or_create_data_source的前两个实例运行没有问题，但第三个不是？

代码和追溯：

fetch_or_create_data_source:

def fetch_or_create_data_source(name,parent=None,author=""):
    from pyramidapp.models import DBSession,DataSource
    oDS   = DBSession.query(DataSource)

    print "name = '{0}' type {1}".format(name,type(name))
    print "author = '{0}' type {1}".format(author,type(author))
    print "parent = {0}".format(parent)
    if parent:
        print "parent_id = {0} type {1}".format(parent.id,type(parent.id))

    if parent is None:
        lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent))
    else:
        lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent.id)) <=== Key error from here

    if len(lDS)==0:
        oDS = None
    else:
        oDS = lDS[0]

    if not oDS:
        oDS = DataSource()
        oDS.name = name
        if parent:
            oDS.parent = parent
            if parent.author:
                oDS.author = parent.author
        if author:
            oDS.author = author
        DBSession.add(oDS)
        DBSession.flush()

        print "Current DS id = '{0}' type {1}".format(oDS.id,type(oDS.id))

    return oDS



Pyramidapp.models.DataSource:

    class DataSource(Base):
        """
        this represents a release of data. It may be a group of documents
        or a single spreadsheet
        """
        __tablename__   = 'data_sources'
        id              = Column(Integer,Sequence('data_src_seq'),primary_key=True)
        name            = Column(String(100))
        notes           = Column(Text)
        title           = Column(String(100))
        author          = Column(String(100))   
        parent_id       = Column(Integer,ForeignKey('data_sources.id'))
        parent          = relationship('DataSource',backref=backref("children",cascade="all, delete"),remote_side=[id,])
        .
        .
        ...




    Traceback (most recent call last):
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_debugtoolbar-2.0.2-py2.7.egg/pyramid_debugtoolbar/panels/performance.py", line 55, in resource_timer_handler
    result = handler(request)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 82, in tm_tween
    reraise(*exc_info)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid_tm-0.7-py2.7.egg/pyramid_tm/__init__.py", line 63, in tm_tween
    response = handler(request)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/router.py", line 163, in handle_request
    response = view_callable(context, request)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/config/views.py", line 355, in rendered_view
    result = view(context, request)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/pyramid-1.5b1-py2.7.egg/pyramid/config/views.py", line 477, in _class_requestonly_view
    response = getattr(inst, attr)()
  File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/views/basic_views.py", line 108, in run_ds_script
    oScript.run(dSettings)

  File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/scripts/data_source_specific/load_prescriptions_dispensed_in_the_community_data.py", line 82, in run
    oChildDS = fetch_or_create_data_source(name=sCSV_name,parent=oDS)
  File "/home/roman/Critical_ID/big_data/pyramidapp/pyramidapp/tools/data_source_script_tools.py", line 22, in fetch_or_create_data_source
    lDS = list(oDS.filter_by(name=name).filter_by(parent_id=parent.id))

  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2397, in __iter__
    self.session._autoflush()
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1184, in _autoflush
    self.flush()
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1879, in flush
    self._flush(objects)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1997, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py", line 57, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1961, in _flush
    flush_context.execute()
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 370, in execute
    rec.execute(self)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py", line 523, in execute
    uow
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 64, in save_obj
    mapper, table, insert)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py", line 594, in _emit_insert_statements
    execute(statement, params)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 717, in execute
    return meth(self, multiparams, params)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 814, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 927, in _execute_context
    context)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1079, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 920, in _execute_context
    context)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/SQLAlchemy-0.9.3-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 425, in do_execute
    cursor.execute(statement, parameters)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/cursors.py", line 93, in execute
    escaped_args = tuple(conn.escape(arg) for arg in args)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/cursors.py", line 93, in <genexpr>
    escaped_args = tuple(conn.escape(arg) for arg in args)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/connections.py", line 698, in escape
    return escape_item(obj, self.charset)
  File "/home/roman/Critical_ID/big_data/lib/python2.7/site-packages/PyMySQL-0.6.1-py2.7.egg/pymysql/converters.py", line 24, in escape_item
    encoder = encoders[type(val)]
KeyError: <class 'pyramidapp.models.DataSource'>

1 个答案:

答案 0 :(得分：0)

如果你从追溯的最后开始并走几行，你会发现PyMySQL lib中的错误是在它试图逃避该值时引发的 - 据说它有某种类型的dict被调用encoders，其中包含编码器的映射。如果pyramidapp.models.DataSource类传递了一个实例，它看起来就会失败。

当然，PyMySQL对SQLAlchemy声明性类一无所知，只能转义基本类型（整数，字符串等）。所以它甚至意外地看到了DataSource实例。

从您的代码示例中不太清楚导致问题的原因，但我最好的猜测是您要么将类实例分配给应该保存原始值的属性，要么使用类实例需要原始值的查询。像这样：

ds = DataSource(...)
other_ds.parent_id = ds  # instead of .parent

或

session.query(DataSource).filter(DataSource.parent_id==ds)

实际上，这是：

lDS = list(oDS.filter_by(name=name).filter_by(parent=parent))
相关问题
最新问题