Question

我正在使用pandas来读取一个大的csv文件。该文件是14G，我的机器在Linux上有60G的可用内存。文件大约45,000,000行和55列。似乎不会出现这个问题。所以是任何解决它的想法？错误是

packages/pandas/core/internals.py", line 4188, in _stack_arrays
stacked = np.empty(shape, dtype=dtype)
MemoryError

Answer 1

您可以传递参数

{"music" : [ 
{ "title" : "Mercy and Compassion",
  "album" : "Standard Deacon Responses",
  "artist" : "Cantor Ibrahim Ayad",
  "genre" : "Standard Deacon Responses",
  "source" : "http://copticchurch-library.org/media/matinsandvespers/Mercy_compassion.mp3",
  "image" : "http://copticchurch-library.org/media/matinsandvespers/image1.jpg",
  "trackNumber" : 1,
  "totalTrackCount" : 5,
  "duration" : 57,
  "site" : "http://copticchurch-library.org/media/matinsandvespers"
},
{ "title" : "Litany of the Departed",
  "album" : "Standard Deacon Responses",
  "artist" : "Cantor Ibrahim Ayad",
  "genre" : "Standard Deacon Responses",
  "source" : "http://copticchurch-library.org/media/matinsandvespers/Litany_departed.mp3",
  "image" : "http://copticchurch-library.org/media/matinsandvespers/image2.jpg",
  "trackNumber" : 2,
  "totalTrackCount" : 5,
  "duration" : 95,
  "site" : "http://copticchurch-library.org/media/matinsandvespers"
},
{ "title" : "Litany of the Sick",
  "album" : "Standard Deacon Responses",
  "artist" : "Cantor Ibrahim Ayad",
  "genre" : "Standard Deacon Responses",
  "source" : "http://copticchurch-library.org/media/matinsandvespers/Litany_sick.mp3",
  "image" : "http://copticchurch-library.org/media/matinsandvespers/image3.jpg",
  "trackNumber" : 3,
  "totalTrackCount" : 5,
  "duration" : 43,
  "site" : "http://copticchurch-library.org/media/matinsandvespers"
},
    { "title" : "Litany of the Travelers",
  "album" : "Standard Deacon Responses",
  "artist" : "Cantor Ibrahim Ayad",
  "genre" : "Standard Deacon Responses",
  "source" : "http://copticchurch-library.org/media/matinsandvespers/Litany_travlers.mp3",
  "image" : "http://copticchurch-library.org/media/matinsandvespers/image4.jpg",
  "trackNumber" : 4,
  "totalTrackCount" : 5,
  "duration" : 61,
  "site" : "http://copticchurch-library.org/media/matinsandvespers"
},

在阅读csv文件时。或者尝试分解成块然后添加这些块。

读取大数据帧时的pandas MemoryError

1 个答案: