如何使用python_ columns将csv文件插入数据库

时间：2017-02-18 14:24:57

标签： python mysql sql csv

这是我的代码

import pymysql
import csv

conn=pymysql.connect("localhost", "root","****","finance")
cursor=conn.cursor()       

print "done"
csv_data = csv.reader(file('naver_loan_output.csv'))

for row in csv_data:
    cursor.execute('INSERT INTO 'daily_new' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )' 'VALUES("%s", "%s", "%s", "%s", "%s", "%s")',row)
    conn.commit()
    cursor.close()
    print "Done"

这就是错误：

  

文件“D：\ dropbox \ Dropbox \ lab \ 7_0218 \ insert_daily_new.py”，第13行   cursor.execute（'INSERT INTO'daily_new'（date，cust_bal，cust_credit，   fund_stock，fund_hyb，fund_bond）''VALUES（“％s”，“％s”，“％s”，“％s”，   “％s”，“％s”）'，行）                                                ^ SyntaxError：语法无效[在0.104s完成]

我尝试了很多，但我不确定正确的SQL插入查询语法。如何从csv获取列？我的csv文件中有6列。

使用此更新示例，代码可以工作：

import pymysql
import csv

csv_data= csv.reader(file('naver_loan_output.csv'))

conn=pymysql.connect("localhost","finance")
cursor=conn.cursor()
print "Done"
for row in csv_data:
    #cursor.execute('INSERT INTO \'daily_new\' (date, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES({}, {}, {}, {}, {}, {})'.format(row[0], row[1], row[2], row[3], row[4], row[5]))
    sql="INSERT INTO  daily_n (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES('2017-01-01','2','2','2','2','2')"
    cursor.execute(sql)
    conn.commit()

cursor.close()

所以，我认为问题是for row或%s。

3 个答案:

答案 0 :(得分：1)

主要是，你的报价是问题。

  1. 如果较大的sql字符串用单引号括起来，则需要转义单引号;或者只是将更大的字符串换成双引号，就像更新的示例一样。请注意：SyntaxError是Python编译错误（不是MySQL运行时异常）。
  2. 对于参数化查询，请不要引用占位符%s。
  3. MySQL（实际上适用于所有RDMS）不像使用'daily_new'那样使用单引号括起表名和列名引用。在MySQL中使用反引号作为名称。一些数据库和ANSI标准允许对象标识符的双引号（而不是字符串文字）。

    4. 使用with()考虑使用file()更高效的csv文件读取过程进行以下调整，而不是像strSQL = "INSERT INTO `daily_new` (`date`, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )" + \ " VALUES(%s, %s, %s, %s, %s, %s)" with open('naver_loan_output.csv', 'r') as f: csv_data = csv.reader(f) for row in csv_data: cursor.execute(strSQL, row) conn.commit() cursor.close() conn.close() print "Done" 那样使用它。如参数化所示，您可以准备一次sql语句，然后在循环中迭代地绑定值。

    reverse()

答案 1 :(得分：0)

我认为你可以尝试这样的事情：

17/02/18 19:35:35 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
17/02/18 19:35:35 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/02/18 19:35:35 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/02/18 19:35:35 INFO tool.CodeGenTool: Beginning code generation
17/02/18 19:35:36 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
17/02/18 19:35:36 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
17/02/18 19:35:36 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/adithyan/hadoop_dir/hadoop-1.2.1
Note: /tmp/sqoop-adithyan/compile/35afadf151a1dd1626a3658577cbc2dd/employee.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/02/18 19:35:41 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-adithyan/compile/35afadf151a1dd1626a3658577cbc2dd/employee.jar
17/02/18 19:35:41 INFO mapreduce.ExportJobBase: Beginning export of employee
17/02/18 19:35:45 INFO input.FileInputFormat: Total input paths to process : 1
17/02/18 19:35:45 INFO input.FileInputFormat: Total input paths to process : 1
17/02/18 19:35:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library
17/02/18 19:35:45 WARN snappy.LoadSnappy: Snappy native library not loaded
17/02/18 19:35:46 INFO mapred.JobClient: Running job: job_201702181051_0002
17/02/18 19:35:47 INFO mapred.JobClient:  map 0% reduce 0%
17/02/18 19:36:17 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000000_0, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '"firstName":"Tom"'
    at employee.__loadFromFields(employee.java:596)
    at employee.parse(employee.java:499)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
    ... 10 more
Caused by: java.lang.NumberFormatException: For input string: ""firstName":"Tom""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:569)
    at java.lang.Integer.valueOf(Integer.java:766)
    at employee.__loadFromFields(employee.java:548)
    ... 12 more

17/02/18 19:36:18 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000001_0, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '{'
    at employee.__loadFromFields(employee.java:596)
    at employee.parse(employee.java:499)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
    ... 10 more
Caused by: java.lang.NumberFormatException: For input string: "{"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.valueOf(Integer.java:766)
    at employee.__loadFromFields(employee.java:548)
    ... 12 more

17/02/18 19:36:29 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000000_1, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '"firstName":"Tom"'
    at employee.__loadFromFields(employee.java:596)
    at employee.parse(employee.java:499)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
    ... 10 more
Caused by: java.lang.NumberFormatException: For input string: ""firstName":"Tom""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:569)
    at java.lang.Integer.valueOf(Integer.java:766)
    at employee.__loadFromFields(employee.java:548)
    ... 12 more

17/02/18 19:36:29 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000001_1, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '{'
    at employee.__loadFromFields(employee.java:596)
    at employee.parse(employee.java:499)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
    ... 10 more
Caused by: java.lang.NumberFormatException: For input string: "{"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.valueOf(Integer.java:766)
    at employee.__loadFromFields(employee.java:548)
    ... 12 more

17/02/18 19:36:42 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000000_2, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '"firstName":"Tom"'
    at employee.__loadFromFields(employee.java:596)
    at employee.parse(employee.java:499)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
    ... 10 more
Caused by: java.lang.NumberFormatException: For input string: ""firstName":"Tom""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:569)
    at java.lang.Integer.valueOf(Integer.java:766)
    at employee.__loadFromFields(employee.java:548)
    ... 12 more

17/02/18 19:36:42 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000001_2, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '{'
    at employee.__loadFromFields(employee.java:596)
    at employee.parse(employee.java:499)
    at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
    ... 10 more
Caused by: java.lang.NumberFormatException: For input string: "{"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.valueOf(Integer.java:766)
    at employee.__loadFromFields(employee.java:548)

只需看看CSV File Reading and Writing doc中的代码片段：

query = """'INSERT INTO 'daily_new' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond )' 'VALUES("""+row+""")'"""
cursor.execute(query)

我希望这对你有用或者让你妨碍你。

答案 2 :(得分：0)

绊倒你有三个问题：

  1. 字符串转义字符

    2. 在您的代码中， daily_new 之前的单引号正在停止字符串，而python正在将 daily_new 解释为未定义的变量。

    要解决此问题，您应该使用转义字符＆＃34; \＆＃34;在每个单引号之前，您希望 in 字符串如下：

    cursor.execute('INSERT INTO \'daily_new\' (date,cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES("%s", "%s", "%s", "%s", "%s", "%s")',row)
    1. 列访问

      2. csv模块将行作为列表返回。要访问列表中的元素（或您的案例中的行），请使用括号表示法。对于row[0]访问行中的第一个元素（列），row[5]访问第6列。

      1. 字符串替换

        2. 您面临的第三个问题是如何正确地将值传递给字符串替换。虽然有很多方法可以做到这一点，但一个简单的方法是format()方法。例如：如果我想构建一个表示＆＃34; Count为1＆＃34;的字符串，我可以运行"Count is {}".format(1)。

        在您的情况下，您希望向字符串添加6个值，因此您可以在希望将值替换为字符串的任何位置添加一对{}，并将另一个参数添加到format()函数中。 / p>

        全部放在一起

        因此，要纠正您的循环代码，您需要这样的东西：

        csv_data = csv.reader(file('naver_loan_output.csv'))
for row in csv_data:
    cursor.execute('INSERT INTO daily_new (date, cust_bal, cust_credit, fund_stock, fund_hyb, fund_bond ) VALUES ({}, {}, {}, {}, {}, {})'.format(row[0], row[1], row[2], row[3], row[4], row[5]))
相关问题
最新问题