Question

我正在复杂系统中创建扩散模拟，将任意图像作为基板，并允许任意产生扩散前沿，并允许表面反应以及在起始基板上沉积新材料。结果我到目前为止感到非常自豪，你可以查看我用它制作的电影，用于CVD和SFD沉积在粒子上。

不幸的是，由于内存不足，我无法生成超过50个帧。我在整个模拟过程中尝试尽可能地清理东西，但我认为我必须遗漏一些东西。总结一下：

我首先创建一个空列表

ims = []

然后，每次我的“模拟”运行时，如果frame number % frame "rate" == 0，它会生成一个框架：

使用plt.ion()到plt.draw()和
使用ims.append()将渲染的绘图添加到动画帧数组中。

在每个帧渲染之前，我运行plt.clf()以防止绘图只有越来越多的重叠图。

没有ims.append()步骤，代码消耗140到170MB的RAM。通过该步骤， 50帧消耗近1.4GB的RAM 。显然，这是非常有限的。 50帧是不错的，但我真的很喜欢350.这条路线可能是不可能的，但这表明内存使用纯粹是每帧大约24MB的ims数组。

解决方法是创建框架并将其呈现到循环内的.svg或.png文件并将其保存到磁盘。我发现这个渲染过程非常占用CPU，因此这样做通常会使代码变得非常慢。此外，创建350个PNG文件，然后手动将它们转换为视频是非常混乱的，所以我喜欢以某种方式将它全部放在程序本身内。

有没有人知道如何在不诉诸渲染并将每个帧写入磁盘的情况下减少此示例代码的内存使用量？

在这个玩具代码中，我只是使用随机数生成器来填充两个数据集，如评论中所述，以加快速度。

代码：

import matplotlib.pyplot as plt
import matplotlib.animation as anim
from numpy import *
from matplotlib import *
import time

# Defines the number of frames of animation to render.
outputframes = 50

# Defines the size of the canned simulation.
nx = 800
ny = 800

# Defines the number of actual simulation timesteps
nt = 100

# This gets the number of timesteps between outputframes.
framestep = 2

# For reporting.
framenum = 0

# Creates two steps, one for the stepped simulated step,
# and one for the prior state. There are two independently
# changing materials, each of which will have half the simulation
# space containing random values here, plus 10% overlap in the
# middle.

p1 = zeros((nx, ny, 2))
p1[360:800,:,0] = random.rand(440, ny)
p2 = zeros((nx, ny, 2))
p2[0:440,:,0] = random.rand(440, ny)

# Animation colormap setup
norm = colors.Normalize(vmin=0, vmax = 1)
# And sets up two corresponding colormaps, one blue and one
# red for p1 and p2 respectively (goal is overlaid).
cmap1 = cm.Blues
cmap2 = cm.Reds

# Sets up an empty array to hold animation frames.
ims = []

# Sets up and uses ion to draw the figure without blocking.
plt.ion()
fig = plt.figure()
plt.draw()

# Run the simulation.

for t in range(nt):
    # This looks to see how far we are, and if we're at a point
    # where t is an even multiple of framestep, we should render
    # a new frame.

    if (t%framestep == 0):
        print('Frame ' + str(framenum))
        framenum = framenum + 1
        plt.clf()
        # In here I did a bunch of stuff to get special colors in
        # the colormap to get substrates and surfaces and other
        # features clearly identified. I am creating a new frame1
        # and frame2 object because in reality I will be doing a
        # log plot math to convert to the graphic frame.
        frame1 = p1[:,:,0]

        # This part is necessary in my real program because
        # I manually modify the colormap after it's created
        # to include the above mentioned special colors.
        frame1_colors = cmap1(norm(frame1))

        # This is my (not quite right) attempt to do overlaid plots.
        plt.imshow(frame1_colors, alpha = 0.5)

        # Do the same for the second set of data.
        frame2 = p2[:,:,0]
        frame2_colors = cmap2(norm(frame2))

        # The goal here was to take the combined output and make
        # it into an animation frame to append to ims, the image
        # array.

        # This is where I start to run into problems. Without the
        # ims.append, the program has constant memory usage. With
        # it, I am using 1340MB by the 50th frame. This is the
        # biggest issue. Even throwing away all other simulation
        # data, this image array for animation is *enormous*.

        # With the ims.append line replaced with the plt.imshow
        # line alone, memory usage is much smaller, ranging from
        # 140-170MB depending on execution point, but relatively
        # constant.

        ims.append([plt.imshow(frame2_colors, alpha = 0.5)])
#        plt.imshow(frame2_colors, alpha = 0.5)

        # Then try to draw updating animation to show progress
        # using draw(). As best I can tell, this basically works,
        # in that the plot is displaying with all components.
        plt.draw()


    # I'll put in a timer so that this doesn't go too fast, since
    # the actual calculation is very complex.
    time.sleep(0.01)

    # Proxy for the actual calculation. Just overwrite with new
    # random data in the overlapping ranges to show some change
    # visually.
    p1[360:800,:,1] = random.rand(440, ny)
    p2[0:440,:,1] = random.rand(440, ny)

    # In this version, it is trivial, but in the real simulation
    # p1[:,:,1] does not end up equal to p1[:,:,0], so the following
    # resets the simulation for the next timestep, overwriting the
    # old values to avoid memory overflow from the p1 and p2 arrays
    # being enormous.

    # Copy new values into old values.
    p1[:,:,0] = p1[:,:,1]
    p2[:,:,0] = p2[:,:,1]

# This is just a repeat for the final frame.
plt.clf()
frame1 = p1[:,:,0]
frame1_colors = cmap1(norm(frame1))
plt.imshow(frame1_colors, alpha = 0.5)
frame2 = p2[:,:,0]
frame2_colors = cmap2(norm(frame2))

# As above, the ims.append uses tons of memory, the imshow alone works well.
ims.append([plt.imshow(frame2_colors, alpha = 0.5)])
# plt.imshow(frame2_colors, alpha = 0.5)

plt.draw()

anim = anim.ArtistAnimation(fig, ims, blit=True)
anim.save('test.mp4', fps=10, writer='avconv')

Answer 1

最后，我决定唯一合理的方法是将我需要的每个帧渲染为.png，然后使用avconv从图像生成电影。

感谢所有建议，但由于未压缩图像的RAM使用情况，这似乎只是一个限制。

matplotlib.animation的Python内存问题

1 个答案: