Question

我有一些迭代生成的numpy数组。我想将每个数组保存到文件中。然后我生成下一个数组并将其附加到文件中等等（如果我一次性完成它会使用太多内存）。我怎么做到最好？有没有办法使我们成为像numpy这样的功能。 numpy.savetxt？（我无法为该功能找到附加选项。）

我目前的代码是：

with open('paths.dat','w') as output:
    for i in range(len(hist[0])):
        amount = hist[0][i].astype(int)
        array = hist[1][i] * np.ones(amount)
        for value in array:
            output.write(str(value)+'\n')

Answer 1

您可以将打开的文件（句柄）传递给public static void main(String[] args) { Scanner input = new Scanner(System.in); System.out.print("How many numbers do you want to divide? "); int division = input.nextInt(); double[] divisionArray = new double[division]; for(int i = 0; i < division; i++) { System.out.print("Enter your " + (i + 1) + ". number: "); divisionArray[i] = input.nextDouble(); } //remember the first value and divide it trough the second, //third, fourth and so on... double result = divisionArray[0]; for(int k = 1; k < division; k ++) { result = result / divisionArray[k]; } System.out.println("Result: " + result); }

savetxt

如果给出名称，

with open('paths.dat','w') as output: for i in range(len(hist[0])): amount = hist[0][i].astype(int) myArray = hist[1][i] * np.ones(amount) np.savetxt(output, myArray, delimiter=',', fmt='%10f')会打开文件，否则会使用该文件。

然后迭代数组的行并写入它们

np.savetxt

其中for row in myArray: f.write(fmt % tuple(row))是您提供的字符串，或者是复制的字符串，以匹配数组中的列数。

Answer 2

我建议使用HDF5。它们对IO非常快。以下是您编写数据的方式：

import numpy as np
import tables

fname = 'myOutput.h5'
length = 100  # your data length
my_data_generator = xrange(length) # Your data comes here instead of the xrange

filters = tables.Filters(complib='blosc', complevel=5)  # you could change these
h5file = tables.open_file(fname, mode='w', title='yourTitle', filters=filters)
group = h5file.create_group(h5file.root, 'MyData', 'MyData')
x_atom = tables.Float32Atom()

x = h5file.create_carray(group, 'X', atom=x_atom, title='myTitle',
                         shape=(length,), filters=filters)

# this is a basic example.  It will be faster if you write it in larger chunks in your real code
# like x[start1:end1] = elements[start2:end2]
for element_i, element in enumerate(my_data_generator):
    x[element_i] = element
    h5file.flush()

h5file.close()

阅读时请使用：

h5file = tables.open_file(fname, mode='r')
x = h5file.get_node('/MyData/X')
print x[:10]

结果：

marray([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.], dtype=float32)

使用新行中的新元素将1D numpy数组附加到文件

2 个答案: