Question

我一直在使用WAMP来摄取一些csv日志，并希望通过编写我需要采取的一些常规操作来转移到更自动化的过程。

我在PHPmyadmin中使用直接CSV导入功能来处理CSV的方言和细节。

我在Python中编写了一个上传程序，使用解析日志文件的MySQLdb，但由于日志包含一些无用的字符，我发现我需要做很多运行清理输入，我可能不想这样做是...

例如，日志是来自目录扫描程序的一些数据，我无法控制人们使用的文件夹命名约定。我有这个文件夹： -

"C:\user\NZ Business Roundtable_Download_13Feb2013, 400 Access"

并且,字符被读作新字段标记（毕竟它是csv）。我真正想要它做的是忽略引号内的所有文本： - "......"

我发现'字符存在类似问题，我相信会有更多问题。

我发现了这一点： - http://www.tech-recipes.com/rx/2345/import_csv_file_directly_into_mysql/显示了如何编写Python脚本以像PHPmyadmin加载例程一样运行。主要使用此代码段：

load data local infile 'uniq.csv' into table tblUniq fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(uniqName, uniqCity, uniqComments)

然而，我希望保护的表格有一些深度处理和更改，我已经编写了脚本，所以想知道是否有办法“告诉”MySQL我想使用""作为文字封装。我想要保护的主要处理是在创建新表时给它一个特定的表名，并在整个剩余的处理过程中使用它。

我的表制作者脚本示例： -

def make_table(self):
    query ="DROP TABLE IF EXISTS `atl`.`{}`".format(self.table)
    self.cur.execute(query)
    query = "CREATE TABLE IF NOT EXISTS `atl`.`{}` (`PK` INT NOT NULL AUTO_INCREMENT PRIMARY KEY, `ID` varchar(10), `PARENT_ID` varchar(10), `URI` varchar(284), \
        `FILE_PATH` varchar(230), `NAME` varchar(125), `METHOD` varchar(9), `STATUS` varchar(4), `SIZE` varchar(9), \
        `TYPE` varchar(9), `EXT` varchar(11), `LAST_MODIFIED` varchar(19), `EXTENSION_MISMATCH` varchar(20), `MD5_HASH` varchar(32), \
        `FORMAT_COUNT` varchar(2), `PUID` varchar(9), `MIME_TYPE` varchar(71), `FORMAT_NAME` varchar(59), `FORMAT_VERSION` varchar(7), \
        `delete_flag` tinyint, `delete_reason` VARCHAR(80), `move_flag` TINYINT, `move_reason` VARCHAR(80), \
        `ext_change_flag` TINYINT, `ext_change_reason` VARCHAR(80), `ext_change_value` VARCHAR(4), `fname_change_flag` TINYINT, `fname_change_reason` VARCHAR(80),\
        `fname_change_value` VARCHAR(80))".format(self.table)
    self.cur.execute(query)
    self.mydb.commit()

我的摄取脚本示例： -

 def ingest_row(self, row):
    query = "insert"
    # Prepare SQL query to INSERT a record into the database.
    query = "INSERT INTO `atl`.`{0}` (`ID`, `PARENT_ID`, `URI`, `FILE_PATH`, `NAME`, `METHOD`, `STATUS`, `SIZE`, `TYPE`, `EXT`, \
        `EXTENSION_MISMATCH`, `LAST_MODIFIED`, `MD5_HASH`, `FORMAT_COUNT`, `PUID`, `MIME_TYPE`, `FORMAT_NAME`,  `FORMAT_VERSION`) \
        VALUES ('{1}','{2}','{3}','{4}','{5}','{6}','{7}','{8}','{9}','{10}','{11}','{12}','{13}','{14}','{15}','{16}','{17}','{18}')".format(self.table, row[0], row[1], row[2], row[3], row[4], \
         row[5], row[6], row[7], row[8], row[9], row[10], row[11], row[12], row[13], row[14], row[15], row[16], row[17])
    try:
        self.cur.execute(query)
        self.mydb.commit()
    except:
        print query
        quit()

日志示例： -

"ID","PARENT_ID","URI","FILE_PATH","NAME","METHOD","STATUS","SIZE","TYPE","EXT","LAST_MODIFIED","EXTENSION_MISMATCH","MD5_HASH","FORMAT_COUNT","PUID","MIME_TYPE","FORMAT_NAME","FORMAT_VERSION"
"1","","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access","NZ Business Roundtable_Download_13Feb2013, 400 Access",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""
"2","1","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/1993/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access\1993","1993",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""

Answer 1

从不使用字符串格式化，连接等来构建SQL查询！

dbapi要求所有驱动程序支持参数化查询，应将参数提供给游标的execute方法。对于MySQLdb，whch支持格式样式参数化，它看起来像：

cursor.execute('insert into sometable values (%s, %s)', ('spam', 'eggs'))

库提供的参数被正确转义，因此，如果您的字符串包含必须转义的字符，则无关紧要。

在你的特殊情况下唯一的例外是表名，因为转义会产生非法的sql。

Answer 2

您应该使用SQL prepared statements。将数据和SQL代码与format混合打开了SQL injection的大门（几乎总是1st in the top 25 software flaw / security issue）。

例如，这是您的数据：

>>> log = """\
... "ID","PARENT_ID","URI","FILE_PATH","NAME","METHOD","STATUS","SIZE","TYPE","EXT","LAST_MODIFIED","EXTENSION_MISMATCH","MD5_HASH","FORMAT_COUNT","PUID","MIME_TYPE","FORMAT_NAME","FORMAT_VERSION"
... "1","","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access","NZ Business Roundtable_Download_13Feb2013, 400 Access",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""
... "2","1","file:/C:/jay/NZ%20Business%20Roundtable_Download_13Feb2013,%20400%20Access/1993/","C:\jay\NZ Business Roundtable_Download_13Feb2013, 400 Access\1993","1993",,"Done","","Folder",,"2013-06-28T11:31:36","false",,"",,"","",""
... """

我没有该文件，所以让我假装：

>>> import StringIO
>>> logfile = StringIO.StringIO(log)

然后让我们构建查询：

>>> import csv
>>> csvreader = csv.reader(logfile)
>>> fields = csvreader.next()
>>> 
>>> table = 'mytable'
>>> 
>>> fields_fmt = ', '.join([ '`%s`' % f for f in fields ])
>>> values_fmt = ', '.join(['%s'] * len(fields))
>>> query = "INSERT INTO `atl`.`{0}` ({1}) VALUES ({2})".format(
... #        self.table, fields_fmt, values_fmt)
...         table, fields_fmt, values_fmt)
>>> query
'INSERT INTO `atl`.`mytable` (`ID`, `PARENT_ID`, `URI`, `FILE_PATH`, `NAME`, `METHOD`, `STATUS`, `SIZE`, `TYPE`, `EXT`, `LAST_MODIFIED`, `EXTENSION_MISMATCH`, `MD5_HASH`, `FORMAT_COUNT`, `PUID`, `MIME_TYPE`, `FORMAT_NAME`, `FORMAT_VERSION`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'

然后按摩ingest_row：

def ingest_row(self, row):
    try:
        self.cur.execute(query, row)
        self.mydb.commit()
    except:
        print query
        quit()

然后您可以使用以下内容导入数据：

for row in csvreader:
    ingest_row(row)

Python / MySQLdb从CSV导入数据封装

2 个答案: