绘制来自大数据集的热图时的python memoryError

时间:2016-06-10 19:16:17

标签: python memory matplotlib out-of-memory data-visualization

对于我的论文,我在多个纳米结构周围绘制电场。这很容易。现在,我试图使用(2x2)子图将两个不同的字段彼此相邻并且它会杀了我。

在x和y方向上以2500个间隔对场进行采样,总计6.250.000点并保存为(x,y,Efield)。结果数据集各约为240 Mb。当我加载一个文件并绘制它时一切正常但是当我并排绘制四个时它太多而且我得到了一个memoryError。

该代码适用于较小的样本(500x500)但准确性太低。我使用的代码的精简版本和错误消息如下。

另一点可能很重要,模拟是在具有以下规范的集群上运行的:

类型:E5-2650 v2
时钟:2.60 GHz
划痕:870 GB
记忆:64 GB QPI 8.00 GT / s
缓存:20 MB
核心:16
Python:2.7.9

def plotMulitpleHeamapSubplot(files, title, subtitles):

    # Create the figure
    fig, axes = plt.subplots(2,2, sharex='col', sharey='row')
    fig.set_size_inches(18.5, 10.5)
    grid = [axes[0][0], axes[1][0], axes[0][1], axes[1][1]]

    for i in range(1,5):
        filePath = files[i-1]
        ax = grid[i-1]

        # Make the heatmap in the subplot
        im = plotHeatMap(filePath, 0.001, rectangles=(0.233, 0.150, .5), ylab = "z direction", title=subtitles[i-1], fig=fig, ax=ax, zRange=(0,1.5))


    # Make it look good
    axes[0][0].set_xlabel("")
    axes[0][1].set_xlabel("")
    axes[0][1].set_ylabel("")
    axes[1][1].set_ylabel("")

    # More looking good
    fig.suptitle(title, fontsize=36, y=1.01)
    cax = fig.add_axes([.925, 0.1, 0.035, 0.8])
    cbar = fig.colorbar(im, cax=cax)
    cbar.ax.tick_params(labelsize=26) 

    # Here it crashes
    fig.subplots_adjust(wspace=.1)
    fig.savefig(path+str(title) + ".png", bbox_inches='tight')


def plotHeatmap()
    x, y, value = [], [], []

    # Load everything from a file.
    # File is formated with a point (x,y,z) on each line
    for line in f:
        a, b, c = line.split(",")
        x.append(float(a))
        y.append(float(b))
        value.append(float(c))

    # Not really important, use this to fill the grid later
    scaledX = np.array([round(xi) for xi in (np.array(x) / resolution)])
    scaledY = np.array([round(yi) for yi in (np.array(y) / resolution)])
    scaledX = scaledX - min(scaledX) 
    scaledY = scaledY - min(scaledY)

    # Create the grid and fill it with the z values
    size = int(len(x)**.5)
    grid = np.zeros((size, size))

    # The scaled arrays contain integers so I can use them for the index.
    for i in range(len(x)):
        grid[scaledX[i], scaledY[i]] = value[i]


    # Use the original x and y lists to get the correct grids.
    x, y = np.mgrid[slice(min(x), max(x) + resolution, resolution),
                slice(min(y), max(y)+ resolution, resolution)]


    cax = ax.pcolormesh(x, y, grid, vmin=zRange[0], vmax=zRange[1])


Traceback (most recent call last):
  File "pythonScripts.py", line 1452, in <module>
    plotMulitpleHeamapSubplotControler(0)
  File "pythonScripts.py", line 1211, in plotMulitpleHeamapSubplotControler
    plotMulitpleHeamapSubplot(fileList, title, titleList)
  File "pythonScripts.py", line 1242, in plotMulitpleHeamapSubplot
    fig.savefig(PATH+str(title) + ".png", bbox_inches='tight')
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/figure.py", line 1470, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 2151, in print_figure
    bbox = a.get_window_extent(renderer)
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/collections.py", line 207, in get_window_extent
    return self.get_datalim(transforms.IdentityTransform())
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/collections.py", line 180, in get_datalim
    paths = self.get_paths()
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/collections.py", line 1675, in get_paths
    self.set_paths()
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/collections.py", line 1680, in set_paths
    self._meshWidth, self._meshHeight, self._coordinates)
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/collections.py", line 1706, in convert_mesh_to_paths
    return [Path(x) for x in points]
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/path.py", line 157, in __init__
    self._update_values()
  File "/sara/sw/python-2.7.9/lib/python2.7/site-packages/matplotlib/path.py", line 205, in _update_values
    self._has_nonfinite = not np.isfinite(self._vertices).all()
MemoryError

0 个答案:

没有答案