Question

OS：macOS High Sierra，版本10.13.1 Xcode中的Python 3.6

我正在使用熊猫和常规python的csv模块读取csv文件。显然，csv模块要比熊猫快得多。我知道NaN会检查熊猫，但我有两个问题：（i）有没有办法加快熊猫速度？（ii）在大熊猫上使用python的csv模块有不利之处吗？

我的代码：

def read_pd():
    path=os.getcwd()
    filename="Sen_01_sample.csv"
    filePathname=path +"/"+filename
    #print('reading...')
    data=pd.read_csv(filePathname,nrows=11,engine="c",skiprows=0)
    #print('read')

def read_csv():
    path=os.getcwd()
    filename="Sen_01_sample.csv"
    filePathname=path+"/"+filename
    with open(filePathname,mode='r') as csvfile:
        csvread=csv.reader(csvfile)
        col_count=len(next(csvread))
        row_count = sum(1 for row in csvread)
        #print(row_count)

        #print(col_count)
        csvfile.seek(0)
        data=np.zeros((row_count-1,col_count-5),dtype=np.float32)
        row=0
        line_count=0
        for rows in csvread:
            if line_count ==0:
                dummy=0
            else:
                data[row]=np.array(rows[4:-1],dtype=np.float32)
                #print (data[row][-1])
            line_count+=1

计时结果：

Time to read a csv file from Pandas 130.997ms.
Time to read a csv file using Python's native csv module 15.448ms.
Python csv module reads file at 11.79% of the time taken by Pandas.

有没有更好的方法来加快熊猫读取csv文件的速度？

0 个答案: