Question

我需要拆分一个巨大的xml文件然后插入数据库这是做我正在做的事情最有效的方式

这是我的代码

import xml.etree.cElementTree as etree
filename = r'D:\test\Books.xml'
context = iter(etree.iterparse(filename, events=('start', 'end')))
_, root = next(context)
books = []
for event, elem in context:
    if event == 'start' and elem.tag == '{http://www.book.org/Book-19200/biblography}Book':
        etree.register_namespace("", "http://www.book.org/Book-19200/biblography")
        xml = etree.tostring(elem)
        xmls.append(xml)
        if len(xmls) == 100:
            populate_db(books)
            books = []
        root.clear()

def populate_db(books):
    c.executemany('INSERT INTO Books VALUES (?)', books)

我的样本book.xml看起来像这样

<Books>
    <Book xmlns="http://www.book.org/Book-19200/biblography"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    ISBN="519292296"
    xsi:schemaLocation="http://www.book.org/Book-19200/biblography ../../book.xsd 
    http://www.w3.org/2000/12/xmldsig# ../../xmldsig-core-schema.xsd">
        <Detail ID="67">
            <BookName>Code Complete 2</BookName>
            <Author>Steve McConnell</Author>
            <Pages>960</Pages>
            <ISBN>0735619670</ISBN>        
            <BookName>Application Architecture Guide 2</BookName>
            <Author>Microsoft Team</Author>
            <Pages>496</Pages>
            <ISBN>073562710X</ISBN>
        </Detail>
    </Book>
    <Book xmlns="http://www.book.org/Book-19200/biblography"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    ISBN="519292296"
    xsi:schemaLocation="http://www.book.org/Book-19200/biblography ../../book.xsd 
    http://www.w3.org/2000/12/xmldsig# ../../xmldsig-core-schema.xsd">
        <Detail ID="87">
            <BookName>Rocking Python</BookName>
            <Author>Guido Rossum</Author>
            <Pages>960</Pages>
            <ISBN>0735619690</ISBN>
            <BookName>Python Rocks</BookName>
            <Author>Microsoft Team</Author>
            <Pages>496</Pages>
            <ISBN>073562710X</ISBN>
        </Detail>
    </Book>
</Books>

这种方法是附加到List并使用100批次并高效插入数据库还是我需要考虑多线程

Answer 1

这取决于您的XML文件的“巨大”程度。兆字节还是太字节？

解析XML文件很可能比DB插入花费更多时间，因此不要过度设计它。只需尝试使用代码即可。

记住：过早优化是万恶之源。

打破巨大的xml文件后插入数据库的有效方法

1 个答案: