Question

以下代码无法运行。

它将遍历CSV文件并检索值，并将其格式化为元组数组（插入查询），以备后用。问题是csv的最后一列有时是字符串或什么都不是（如下面的csv示例所示）。错误随之而来。有人可以帮我吗？

def csv_to_DB(csv_input):
    with open(csv_input, newline='') as csvfile:
        csv_data = csv.reader(csvfile, delimiter=',', quotechar='"')
        to_insert = [] # will be list of tuples
        insert_str = "INSERT INTO table (ID, user, user_version, value, description) VALUES "
        template = "('%s', '%s', '%s', '%s', '%s')"

        for row in csv_data:
            to_insert.append(tuple(row)) # convert row/list to tuple and add it to list
            query = insert_str + '\n'.join(template % to_insert)
        #use query for other operations...

CSV示例：

1,aaa,1,0.0,
2,bbb,1,0.13,
3,ccc,1,0.0,
4,ddd,3,1.0,Rom
5,eee,1,0.08,

错误：

    query = insert_str + '\n'.join(template % to_insert)
TypeError: not enough arguments for format string

注意：这个问题是来自 this question

更新

为了澄清：目标是创建一个具有多个值而不是多个插入的INSERT。在这种情况下：

INSERT INTO table (ID, user, user_version, value, description) VALUES 
('1', 'aaa', '1', '0.0', ''), 
('2', 'bbb', '1', '0.13', ''), 
('3', 'ccc', '1', '0.0', ''), 
('4', 'ddd', '3', '1.0', 'Rom'), 
('5', 'eee', '1', '0.08', '')

to_insert将是：

[('1', 'aaa', '1', '0.0', ''), ('2', 'bbb', '1', '0.13', ''), ('3', 'ccc', '1', '0.0', ''), ('4', 'ddd', '3', '1.0', 'Rom'), ('5', 'eee', '1', '0.08', '')]

Answer 1

只需添加简单的字符串即可实现所需的输出，而无需字符串模板：

def xing_csv_to_crmDB2(csv_input):
    query = ''
    with open(csv_input, 'r', newline='') as csvfile:
        csv_data = csv.reader(csvfile, delimiter=',', quotechar='"')
        insert_str = "INSERT INTO table (ID, user, user_version, value, description) VALUES "

        for row in csv_data:
            query += '\n' + str(tuple(row))
        insert_str += query
# do something

这将产生以下输出：

INSERT INTO table (ID, user, user_version, value, description) VALUES
('1', 'aaa', '1', '0.0', '')
('2', 'bbb', '1', '0.13', '')
('3', 'ccc', '1', '0.0', '')
('4', 'ddd', '3', '1.0', 'Rom')
('5', 'eee', '1', '0.08', '')

更新：根据@Tomerikoo的想法，它是一个更简化的版本：

def xing_csv_to_crmDB2(csv_input):
    with open(csv_input, 'r', newline='') as csvfile:
        csv_data = csv.reader(csvfile, delimiter=',', quotechar='"')
        insert_str = "INSERT INTO table (ID, user, user_version, value, description) VALUES "

        for row in csv_data:
            insert_str += '\n' + str(tuple(row))
# do something

输出仍然相同。

Answer 2

您的问题在于此表达式：

(template % to_insert)

模板期望有5个参数，to_insert 将始终为1！，因为它是一个列表，因此被视为单个参数。

将to_insert更改为tuple(to_insert)并将查询移出循环将解决您的问题，具体取决于您要获取的内容。

尝试将循环更改为以下内容：

for row in csv_data:
    to_insert.append(tuple(row)) # convert row/list to tuple and add it to list
query = insert_str + '\n'.join(template % tuple(to_insert))

更新：根据@JonyD的更新，模板根本不是必需的，因为它会强制您进入5行。另外，您希望将列表而不是字符串传递给join()。您应该做的是：

def csv_to_DB(csv_input):
    with open(csv_input, newline='') as csvfile:
        csv_data = csv.reader(csvfile, delimiter=',', quotechar='"')
        to_insert = [] # will be list of tuples
        insert_str="INSERT INTO table (ID, user, user_version, value, description) VALUES"

        for row in csv_data:
            to_insert.append(tuple(row)) # convert row/list to tuple and add it to list
        query = insert_str + '\n'.join(to_insert)

Answer 3

这是我想要的答案。随意使用它。非常快。要在RDS mysql中插入3,800万条记录，当bloc_size=10000时需要2分钟。感谢torresmateo

def csv2mysql(csv_input, db_opts, insert_conf, block_size='1000'):
    """
    :param csv_input: the input csv file path
    :param db_opts: is a dictionary. Should be like the following Example:

    tvnow_db_opts = {
        'user': db_conn.login,
        'password': db_conn.password,
        'host': db_conn.host,
        'database': db_conn.schema
    }

    :param insert_conf: see explanation below

    insert_conf = {
        'table_name': 'my_table',
        'columns': 'ID, field1, field2, field3, field_4',
        'values_template': "('%s', '%s', '%s', '%s', '%s')"
    }

    table_name: DB table name where data will be inserted
    columns:    columns corresponding to csv; separated by comma. 
                Example: "ID, field1, field2, field3, field_4"

    values_template:  String with following format "('%s', '%s', '%s', '%s', '%s')". Nr of '%s' must be the same as the nr of fields in the csv/columns in the table

    :param block_size: nr of rows/records to be inserted per sql insert command. Default 1000
    """

    print("Inserting csv file {} to database {}".format(csv_input, db_opts['host']))
    conn = pymysql.connect(**db_opts)
    cur = conn.cursor()

    try:
        with open(csv_input, newline='') as csvfile:
            csv_data = csv.reader(csvfile, delimiter=',', quotechar='"')
            to_insert = [] # will be list of tuples
            insert_str = "INSERT INTO {} ({}) VALUES ".format(insert_conf.table_name, insert_conf.columns)
            count = 0

            for row in csv_data:
                count += 1
                to_insert.append(tuple(row))  # convert row/list to tuple and add it to list
                if count % block_size == 0:
                    query = insert_str + ',\n'.join([insert_conf.values_template % r for r in to_insert])
                    cur.execute(query)
                    to_insert = []
                    conn.commit()

            # commit/insert the remaining rows
            if len(to_insert) > 0:
                query = insert_str + ',\n'.join([insert_conf.values_template % r for r in to_insert])
                cur.execute(query)
                conn.commit()
    finally:
        conn.close()
    print('Finished inserting csv file to database')

Answer 4

template = "('%s', '%s', '%s', '%s', '%s')"-五个参数！

但是您使用：

1,aaa,1,0.0,-4个参数（错误）

2,bbb,1,0.13,-5个参数（确定）

3,ccc,1,0.0,-4个参数（错误）

4,ddd,3,1.0,Rom-5个参数（确定）

5,eee,1,0.08,-4个参数（错误）

使用百分比格式的Python字符串格式（“ TypeError：格式字符串的参数不足”）

4 个答案: