Question

我已经阅读了与此问题类似的答案，但没有找到解决我目标的方法。我有一个将近150MB的大型csv文件，格式如下：

logs.csv:

id,lat,lon,days,mode
656001,41.163172,-8.5838214,42461.0046296296,3
656001,41.163237,-8.58381,42461.0046412037,3
656001,41.1632328,-8.5838378,42461.0046527778,3
656001,41.163234,-8.5838637,42461.0046643519,3
656001,41.1632204,-8.583885,42461.0046759259,3
.....
758001,39.9966599,-8.6113725,42461.4125578704,1
758001,39.9969224,-8.6111087,42461.4125694444,1
758001,39.9972031,-8.6108471,42461.4125810185,1
....
829000,40.6022533,-7.2600605,42461.6981944444,2
829000,40.6020222,-7.2601668,42461.6982060185,2
829000,40.6017725,-7.2602641,42461.6982175926,2
829000,40.6015003,-7.2603968,42461.6982291667,2
......
863025,41.1459056,-8.6131507,42461.7629050926,0
863025,41.1459103,-8.6131553,42461.7629166667,0
863025,41.1459149,-8.6131682,42461.7629282407,0

然后我想通过id将此数据加载为数组数组，以使每个嵌套数组都具有四列：lat, lon, days, mode，格式如下：

[
  [41.163172 -8.5838214 42461.0046296296 3]
  [41.163237 -8.58381 42461.0046412037 3]
  [41.1632328 -8.5838378 42461.0046527778 3]
  ...
  [39.9966599 -8.6113725 42461.4125578704 1]
  [39.9969224 -8.6111087 42461.4125694444 1]
  .....
  .....
  [41.1459056 -8.6131507 42461.7629050926 0]
  [41.1459103 -8.6131553 42461.7629166667 0]
  [41.1459149 -8.6131682 42461.7629282407 0]
]

我首先将数据作为numpy ndarray加载，如下所示：

my_data = np.genfromtxt('logs.csv', delimiter=',', skip_header=True)
my_data.shape
(22, 5)

然后尝试将其进一步处理到所需的输出（通过id），但这会改变所需数组的形状：

#group by id
unique_id = set(my_data[:,0])
unique_id
{656001.0, 758001.0, 829000.0, 863025.0}

grouped_data = np.array([my_data[my_data[:,0]== pvalue, 1:]
                       for pvalue in unique_id])
grouped_data.shape
(503,)

但是我想要嵌套数组的形状，因为我要遍历它的元素。我期待着(X,4)

的形状

然后我尝试使用pandas dataframe，所以：

data = pd.read_csv('logs.csv')
data.head()
      id       lat         lon        days       mode
0   656001  41.163172   -8.583821   42461.004630    3
1   656001  41.163237   -8.583810   42461.004641    3
2   656001  41.163233   -8.583838   42461.004653    3
3   656001  41.163234   -8.583864   42461.004664    3
4   656001  41.163220   -8.583885   42461.004676    3

显然，熊猫不会产生预期的结果：

data.groupby('id').head()
      id       lat        lon        days        mode
0   656001  41.163172   -8.583821   42461.004630    3
1   656001  41.163237   -8.583810   42461.004641    3
2   656001  41.163233   -8.583838   42461.004653    3
3   656001  41.163234   -8.583864   42461.004664    3
.....

我的任何尝试都不会导致所需的数组数组，如开头所示。我该怎么做？

Answer 1

您可以使用列表推导对id值进行分组，并提取该id的每个矩阵。

[matrix.to_numpy() for _, matrix in df.groupby('id')]
# or, depending on intended use:
# np.array([matrix.to_numpy() for _, matrix in df.groupby('id')])

将csv文件读入多维数组

1 个答案: