最快的方式在python中将数据（数百万数据）插入mysql

Question

将数据插入Mysql

中的CiqHistorical主表

                                sql ="""INSERT INTO CiqHistorical(CiqRefID, CoID, GVKEY, IID, GRID, CreateDID, SectorID,
                                 UserID, ClientID, MinPeriodID, MaxPeriodID, MaxPeriodDID, MinAnnualID, MaxAnnualID,
                                 MaxAnnualDID) VALUES(%s,%s,'%s','%s',%s,GetDateID(now()),%s,%s,%s,%s,%s,
                                 GetDateID('%s'),%s,%s,GetDateID('%s'));""" %(ciq_ref_id, coid, gvkey, iid,
                                                                    grid, sector_id,user_id, client_id, min_period_id,
                                                                    max_period_id, max_period_did, min_annual_id,
                                                                    max_annual_id, max_annual_did)
                                ciq_hist = self.mysql_hermes.execute(sql)

Answer 1

您可以使用一个INSERT插入许多记录，但是您应该保持查询的大小不要太大。这是一个可以满足您需求的脚本：

CHUNK_SIZE = 1000

def insert_many(data)
    index = 0
    while True:
        chunk = data[index : index + CHUNK_SIZE]
        if not chunk:
            break

        values_str = ", ".join(
            "('{0}', '{1}', '{2}', ...)".format(row['field1'], row['field2'], row['field3'], ...)
            for row in chunk
        )

        sql = "INSERT INTO `your_table` (field1, field2, field3, ...) VALUES {0}".format(values_str)

        self.mysql_hermes.execute(sql)

        index += CHUNK_SIZE