Question

我有一个3000万行的大型csv文件（1.6 gb），我正在使用pymysql将数据从csv加载到mysql表中。我删除了表架构中的所有约束，以加快加载速度，并将超时值设置为较大的值。

def setTimeOutLimit(connection):
try:
    with connection.cursor() as cursor:
        query = "SET GLOBAL innodb_lock_wait_timeout = 28800"
        cursor.execute(query)

        query2 = "SET innodb_lock_wait_timeout = 28800"
        cursor.execute(query2)

        query3 = "SET GLOBAL connect_timeout = 28800"
        cursor.execute(query3)

        query4 = "SET GLOBAL wait_timeout = 28800"
        cursor.execute(query4)

        query5 = "SET GLOBAL interactive_timeout = 28800"
        cursor.execute(query5)

        query6 = "SET GLOBAL max_allowed_packet = 1073741824"
        cursor.execute(query6)

except:
    conn.close()
    sys.exit(" Could not set timeout limit ")

数据被插入到表中，但是我需要将该列之一作为主键，因此我正在创建另一个表，通过忽略重复值使该列成为主索引。（tableName_1是旧表tableName是新表）

def createNewTableFromOld(connection, tableName):

try:
    pprint( " Creating new table from old table  with constraints" )

    with connection.cursor() as cursor:

        query = (" CREATE TABLE " + tableName + 
                 " Like " + tableName + "_1")

        cursor.execute(query)

        query2 = (" ALTER TABLE " + tableName +
                  " ADD PRIMARY KEY(TimeStamp) ")

        cursor.execute(query2)

        query3 = (" INSERT IGNORE INTO " + tableName + 
                  " SELECT * FROM " + tableName + "_1")

        cursor.execute(query3)

        query4 = ("DROP TABLE " + tableName + "_1")

        cursor.execute(query4)

        connection.commit()

except:
    conn.close()
    sys.exit(" Could not create table with Primary Key ")

在执行此方法期间，经过5-6分钟后，我收到此错误， pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([WinError 10054] An existing connection was forcibly closed by the remote host)')

当我检查服务时，MYSQL80自动崩溃并停止。我还在my.ini文件中将max_allowed_packet_size设置为1 gb，并且所有超时都手动设置为8小时。可能是什么问题？

原始表架构为：

query = ("CREATE TABLE IF NOT EXISTS " + table + " ("
                  " TimeStamp  DECIMAL(15, 3), " + 
                  " Value      DECIMAL(30, 11), " +
                  " Quality    INT, " +
                  " TagName    varchar(30) )"
                  )

Answer 1

我最终通过将my.ini文件中的innodb_buffer_pool_size设置为2GB来解决了该问题，而之前只有4M。

使用python的LOAD DATA从大型csv传输数据时，MySQL崩溃

1 个答案: