我在SQLAlchemy中遇到的困难。假设我有两个模型模型:
TextLocator
----------------
id (primary key)
docid
span_start
span_end
UNIQUE constraint docid, span_start, span_end
tags = relationship('Tag', back_populates='text_locator')
Tag
----------------
id
locator_id (Foreign Key text_locator.id)
tag_name
text_locator = relationship('TextLocator', back_populates='tags')
如果我导入具有一百万行的csv,而每一行是:
docid, span_start, span_end, tag_name
对于我来说,在SQLA中将其插入标签表的最有效方法是什么?
我面临的问题是找到需要创建和查询的TextLocator。
如果我一次只做一排,我可以做类似的事情:
text_locator_instance = (
session.query(TextLocator)
.filter(
and_(docid == <docid>. span_start == <span_start, span_end == <span_end>)).first()
if not text_locator_instance:
text_locator_instance = TextLocator(docid=<docid>, span_start=<span_start>, span_end=<span_end>)
tag_instance = Tag(tag_name=<tag_name>)
tag_instance.text_locator = text_locator_instance
我将如何大规模处理100万行?
如果是纯SQL,则将一百万个CSV插入临时表中,将JOIN插入TextLocator以获取ID,将其插入联接失败的TextLocator中,然后通过JOIN插入具有所有ID的Tag。