Question

我知道在这里抛出代码并寻求帮助来解决问题是很糟糕的。这个问题似乎有点过头了。

代码应该循环遍历所有文件和子文件夹。我不认为这里有任何日志错误。问题是我遇到了问题，即再次处理相同的文件并导致数据库插入在主键约束上失败。

这是我的代码：

import csv
import pypyodbc
import os
import sys

extension = ".tsv"
connStr = """DSN=database_test;"""

sys.stdout = open('c:\\temp\\python.log', 'w')
print 'starting ...'

def LoadFile(path):   
    i = 0 
    for item in os.listdir(path): # loop through items in dir        
        full_path = os.path.join(path, item)
        if os.path.isfile(full_path) and full_path.endswith(extension): # check for ".tsv" extension

            if full_path.find('IM') > 0:
                table_name = 'table_a'
            else:
                table_name = 'table_b'

            if os.stat(full_path).st_size > 0:
                print "Processing file:", i, "|", full_path
                i = i + 1
                with open (full_path, 'r') as f:
                    reader = csv.reader(f, delimiter='\t')
                    columns = next(reader) 
                    query = 'insert into ' + table_name + '({0}) values ({1})'
                    crsr = cnxn.cursor()
                    for data in reader:
                        query = query.format(', '.join(columns), ', '.join('?' * len(columns)))          
                        #print(query, "with ", data)
                        if(data[1] != ''):
                            crsr.execute(query, data)
                    crsr.commit()  
                    crsr.close()                                
        elif os.path.isdir(full_path):                        
            print "Process Folder: ", full_path    
            LoadFile(full_path)                        
        else:
            print("invalid file name:", item)
    print "Process Folder total files: ", i, ":", full_path
    return         

cnxn = pypyodbc.connect(connStr)

dir_name = 'X:\\TopLevelFolder'
LoadFile(dir_name)

cnxn.close()
print("Completed")

Answer 1

我原来的问题有点误导......由于一些无法解释的硬件问题，代码无法正常工作。在我切换到python 2.7并重新安装所有库之后。代码工作正常。

将csv文件插入SQL Server数据库：listdir出错？

1 个答案: