Question

我正在尝试读取几个名称相同但内部结构相同的csv文件：

u'/1001/download field1.imem_LOG.csv', 
u'/1001/IRDP_01_25_2018_11_47_16.imem_LOG.csv',
u'/1001/IRDP_08_06_2014_15_07_35.imem_LOG.csv',
u'/1001/IRDP_03_17_2014_15_38_03_LOG.csv',
u'/1001/IRDP-KK-1001_09_11_2017_14_40_48.imem_LOG.csv',

我看不到第一个和最后一个：

folder = dataiku.Folder("AfbOboIX")



list_files = folder.list_paths_in_partition()
list_files
list_files_split = map(lambda x : x.split("_"),list_files)
list_files_split

list_idx = [list_files_split.index(x) for x in list_files_split if "LOG.csv" in x]list_idx
list_files_log = [list_files[idx] for idx in list_idx]
list_files_log
    output = pd.DataFrame()
    for name in list_files_log:
        with folder.get_download_stream(name) as f:
            try:
                tmp = pd.read_csv(f)
                tmp = tmp.fillna(0)
                output = output.append(tmp)
            except:
                pass

`

我该如何解决？

Answer 1

不含glob
的新代码

我认为您想要的可以使用Ipython解决：

import pandas as pd

files= !ls *.csv #IPhython stuff
data= pd.concat([pd.read_csv(f, enconding='latin1') for f in files], sort=False)

希望有帮助。

旧代码

我认为您可以使用glob库解决您想要的事情：

import glob
import pandas as pd

dir= 'dir/*.csv'

files = glob.glob(dir)
data= pd.concat([pd.read_csv(f) for f in sorted(files)], sort=False)

希望有帮助。

Answer 2

import glob

print(glob.glob(path_of_folder+"/*.csv"))

和

for index,filename in enumerate(glob.glob(path + "/*.csv")):
    dfs.append(pd.read_csv(filename))

Answer 3

您可以循环浏览文件，并将它们一个接一个地添加

import glob
import pandas as pd

dir = 'path_to_directory/*.csv'

files = glob.glob(dir)

data = pd.DataFrame()

for f in files:
    df = pd.read_csv(f)
    data = data.append(df)

data将包含目录中的所有数据框

Answer 4

将“ Pass”替换为例外。引发异常或将发生的错误写入日志。然后更容易理解问题所在。

import logging
output = pd.DataFrame()
for name in list_files_log:
    with folder.get_download_stream(name) as f:
        try:
            tmp = pd.read_csv(f)
            tmp = tmp.fillna(0)
            output = output.append(tmp)
        except Exception as err:
            logging.error(err.message)
            logging.exception(err)  # Shows the traceback.

读取CSV文件熊猫-文件名

4 个答案: