Question

I have multiple csv files with the same format (14 rows 4 columns). I tried to load all of them into a single dataFrame, and use file's name to rename the values of the first column (1-14)

    1   500 0   0
    2   350 0   1
    3   500 1   0
    .............
    13  600 0   0
    14  800 0   0

I tried the following code but I am not getting what I am expecting:

    filenames = os.listdir('Threshold/')
    Y = pd.DataFrame () #empty df
    # file name are in the following foramt "subx_ICA_thre.csv"
    # need to get x (subject number to be used later for renaming columns values)
    Sub_list=[]
    for filename in filenames:
    s= int(''.join(filter(str.isdigit, filename)))
    Sub_list.append(int(s))
    S_Sub_list= sorted(Sub_list) 

    for x in S_Sub_list: # get the file according to the subject number
    temp = pd.read_csv('sub' +str(x)+'_ICA_thre.csv' )
    df = pd.concat([Y, temp])  # concat the obtained frame with the empty frame
    df.columns = ['id', 'data', 'isEB', 'isEM']
    #  replace the column values using subject id
         for sub in range(1,15):
           df['id'].replace(sub, 'sub' +str(x)+'_ICA_'+str(sub) ,inplace=True)
    print (df)

output:

                id  data  isEB  isEM
   0    sub1_ICA_2   200     0     0
   1    sub1_ICA_3   275     0     0
   2    sub1_ICA_4   500     1     0
   ................................
   11  sub1_ICA_13   275     0     0
   12  sub1_ICA_14   300     0     0
                id  data  isEB  isEM
   0    sub2_ICA_2   275     0     0
   1    sub2_ICA_3   500     0     0
   2    sub2_ICA_4   400     0     0
   .................................
   11  sub2_ICA_13   300     0     0
   12  sub2_ICA_14   450     0     0

First, it seems that the code makes different dataFrame not a single one.Second, the first row is removed (sub1_ICA_1 is missing, may be replaced with column names). I couldn't find the problem in the loop that I am using

Answer 1

I think you need create list of DataFrames first, then concat with parameter keys for new values by range in MultiIndex, then modify column id and last remove MultiIndex by reset_index:

Also was added parameter names to read_csv for custom columns names.

Y = []
for x in S_Sub_list: 
    n = ['id', 'data', 'isEB', 'isEM']
    temp = pd.read_csv('sub' + str(x) +'_ICA_thre.csv', names = n)
    Y.append(temp)

#list comprehension alternative
#n = ['id', 'data', 'isEB', 'isEM']
#Y = [pd.read_csv('sub' + str(x) +'_ICA_thre.csv', names = n) for x in S_Sub_list]

df = pd.concat(Y, keys=range(1,len(S_Sub_list) + 1))

df['id'] = 'sub' + df.index.get_level_values(0).astype(str) +'_ICA_'+ df['id'].astype(str)
df = df.reset_index(drop=True)

load multiple csv files into Dataframe: columns names issue

1 个答案: