如何创建超过2维的pandas数据帧?

时间:2016-04-21 05:30:00

标签: arrays numpy pandas multidimensional-array dataframe

我希望能够创建n-dimensional个数据帧。我听说过panels中使用pandas的3D数据帧方法,但如果可能的话,我想通过将不同的数据集组合到超级数据框

来扩展维度超过3个dims

我尝试了这个,但我无法弄清楚如何在我的测试数据集中使用这些方法 - > Constructing 3D Pandas DataFrame

此外,这对我的情况没有帮助 - > Pandas Dataframe or Panel to 3d numpy array

我制作了一个随机测试数据集,其中任意轴数据试图模拟真实情况;有3个轴(即患者,年份和样本)。我尝试将一堆数据帧添加到列表中然后使用它创建一个数据帧但它不起作用:(我甚至尝试了panel,如上面的第二个链接,但我无法让它工作

有人知道如何使用标签创建N维pandas数据框吗?

我尝试的第一种方式:

#Reproducibility
np.random.seed(1618033)

#Set 3 axis labels/dims
axis_1 = np.arange(2000,2010) #Years
axis_2 = np.arange(0,20) #Samples
axis_3 = np.array(["patient_%d" % i for i in range(0,3)]) #Patients

#Create random 3D array to simulate data from dims above
A_3D = np.random.random((years.size, samples.size, len(patients))) #(10, 20, 3)

#Create empty list to store 2D dataframes (axis_2=rows, axis_3=columns) along axis_1
list_of_dataframes=[]

#Iterate through all of the year indices
for i in range(axis_1.size):
    #Create dataframe of (samples, patients)
    DF_slice = pd.DataFrame(A_3D[i,:,:],index=axis_2,columns=axis_3)
    list_of_dataframes.append(DF_slice)
#     print(DF_slice) #preview of the 2D dataframes "slice" of the 3D array
#           patient_0  patient_1  patient_2
#      0    0.727753   0.154701   0.205916
#      1    0.796355   0.597207   0.897153
#      2    0.603955   0.469707   0.580368
#      3    0.365432   0.852758   0.293725
#      4    0.906906   0.355509   0.994513
#      5    0.576911   0.336848   0.265967
#     ...
#     19   0.583495   0.400417   0.020099

# DF_3D = pd.DataFrame(list_of_dataframes,index=axis_2, columns=axis_1)
# Error
# Shape of passed values is (1, 10), indices imply (10, 20)

我试过的第二种方式:

DF = pd.DataFrame(axis_3,columns=axis_2) 
#Error:
#Shape of passed values is (1, 3), indices imply (20, 3)

# p={}
# for i in axis_1:
#     p[i]=DF
# panel= pd.Panel(p)

我想我可以这样做,但我真的很喜欢pandas,如果存在,我宁愿使用他们的方法之一:

#Set data for query
query_year = 2007
query_sample = 15
query_patient = "patient_1"

#Index based on query
A_3D[
     (axis_1 == query_year).argmax(),
     (axis_2 == query_sample).argmax(),
     (axis_3 == query_patient).argmax()
]
#0.1231212416981845

以这种方式访问​​数据真棒:

DF_3D[query_year][query_sample][query_patient]
#Where DF_3D[query_year] would give a list of 2D arrays (row=sample, col=patient)
# DF_3D[query_year][query_sample] would give a 1D vector/list of patient data for a particular year, of a particular sample.
# and DF_3D[query_year][query_sample][query_patient] would be a particular sample of a particular patient of a particular year

3 个答案:

答案 0 :(得分:3)

与使用n维面板不同,您可能最好使用二维数据表示,但使用MultiIndexes作为索引,列或两者。

例如:

np.random.seed(1618033)

#Set 3 axis labels/dims
years = np.arange(2000,2010) #Years
samples = np.arange(0,20) #Samples
patients = np.array(["patient_%d" % i for i in range(0,3)]) #Patients

#Create random 3D array to simulate data from dims above
A_3D = np.random.random((years.size, samples.size, len(patients))) #(10, 20, 3)

# Create the MultiIndex from years, samples and patients.
midx = pd.MultiIndex.from_product([years, samples, patients])

# Create sample data for each patient, and add the MultiIndex.
patient_data = pd.DataFrame(np.random.randn(len(midx), 3), index = midx)

>>> patient_data.head()
                         0         1         2
2000 0 patient_0 -0.128005  0.371413 -0.078591
       patient_1 -0.378728 -2.003226 -0.024424
       patient_2  1.339083  0.408708  1.724094
     1 patient_0 -0.997879 -0.251789 -0.976275
       patient_1  0.131380 -0.901092  1.456144

一旦你有了这种形式的数据,就可以相对容易地处理它。例如:

>>> patient_data.unstack(level=0).head()  # Years.
                    0                                                                                              ...            2                                                                                          
                 2000      2001      2002      2003      2004      2005      2006      2007      2008      2009    ...         2000      2001      2002      2003      2004      2005      2006      2007      2008      2009
0 patient_0 -0.128005  0.051558  1.251120  0.666061 -1.048103  0.259231  1.535370  0.156281 -0.609149  0.360219    ...    -0.078591 -2.305314 -2.253770  0.865997  0.458720  1.479144 -0.214834 -0.791904  0.800452  0.235016
  patient_1 -0.378728 -0.117470 -0.306892  0.810256  2.702960 -0.748132 -1.449984 -0.195038  1.151445  0.301487    ...    -0.024424  0.114843  0.143700  1.732072  0.602326  1.465946 -1.215020  0.648420  0.844932 -1.261558
  patient_2  1.339083 -0.915771  0.246077  0.820608 -0.935617 -0.449514 -1.105256 -0.051772 -0.671971  0.213349    ...     1.724094  0.835418  0.000819  1.149556 -0.318513 -0.450519 -0.694412 -1.535343  1.035295  0.627757
1 patient_0 -0.997879 -0.242597  1.028464  2.093807  1.380361  0.691210 -2.420800  1.593001  0.925579  0.540447    ...    -0.976275  1.928454 -0.626332 -0.049824 -0.912860  0.225834  0.277991  0.326982 -0.520260  0.788685
  patient_1  0.131380  0.398155 -1.671873 -1.329554 -0.298208 -0.525148  0.897745 -0.125233 -0.450068 -0.688240    ...     1.456144 -0.503815 -1.329334  0.475751 -0.201466  0.604806 -0.640869 -1.381123  0.524899  0.041983

要选择数据,请参阅MultiIndexing的文档。

答案 1 :(得分:0)

从输入数据结构派生的另一种方法(Alexander)是:

np.random.seed(1618033)

#Set 3 axis labels/dims
years = np.arange(2000,2010) #Years
samples = np.arange(0,20) #Samples
patients = np.array(["patient_%d" % i for i in range(0,3)]) #Patients

#Create random 3D array to simulate data from dims above
A_3D = np.random.random((years.size, samples.size, len(patients))) #(10, 20, 3)

# Reshape data to 2 dimensions
maj_dim = 1
for dim in A_3D.shape[:-1]:
    maj_dim = maj_dim*dim
new_dims = (maj_dim, A_3D.shape[-1])
A_3D = A_3D.reshape(new_dims)

# Create the MultiIndex from years, samples and patients.
midx = pd.MultiIndex.from_product([years, samples])
# Note that Cartesian product order is the same as the 
# C-order used by default in ``reshape``.

# Create sample data for each patient, and add the MultiIndex.
patient_data = pd.DataFrame(data = A_3D, 
                            index = midx,
                            columns = patients)

>>>> patient_data.head()

        patient_0  patient_1  patient_2
2000 0   0.727753   0.154701   0.205916
     1   0.796355   0.597207   0.897153
     2   0.603955   0.469707   0.580368
     3   0.365432   0.852758   0.293725
     4   0.906906   0.355509   0.994513

答案 2 :(得分:0)

您应该考虑改用 xarray。来自他们的documentation

<块引用>

Panel 是 Pandas 的 3D 数组数据结构,与 Series 和 DataFrame 相比,它始终是第二类数据结构。为了让 Pandas 开发人员能够更加专注于围绕 DataFrame 构建的核心功能,pandas 删除了 Panel,转而将使用多维数组的用户引导至 xarray