Question

我遇到mxnet的基本IO问题。我正在尝试使用mxnet.io.NDArrayIter来读取内存数据集以便在mxnet中进行培训。我有下面的代码（为了简洁而精简），它预处理代码并尝试迭代它（严重基于tutorial）：

import csv
import mxnet as mx
import numpy as np

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.pipeline import Pipeline


with open('data.csv', 'r') as data_file:
    data = list(csv.reader(data_file))

labels = np.array(map(lambda x: x[1], data)) # one-hot encoded classes
data = map(lambda x: x[0], data) # raw text in need of pre-processing

transformer = Pipeline(steps=(('count_vectorizer', CountVectorizer()),
                              ('tfidf_transformer', TfidfTransformer())))

preprocessed_data = np.array([np.array(row) for row in transformer.fit_transform(data)])

training_data = mx.io.NDArrayIter(data=preprocessed_data, label=labels, batch_size=50)

for i, batch in enumerate(training_data):
    print(batch)

执行此代码时，收到以下错误：

    Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/mxnet/io.py", line 510, in _init_data
    data[k] = array(v)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/utils.py", line 146, in array
    return _array(source_array, ctx=ctx, dtype=dtype)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 2245, in array
    arr[:] = source_array
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 437, in __setitem__
    self._set_nd_basic_indexing(key, value)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 698, in _set_nd_basic_indexing
    self._sync_copyfrom(value)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/ndarray/ndarray.py", line 856, in _sync_copyfrom
    source_array = np.ascontiguousarray(source_array, dtype=self.dtype)
  File "/usr/local/lib/python3.5/dist-packages/numpy/core/numeric.py", line 581, in ascontiguousarray
    return array(a, dtype, copy=False, order='C', ndmin=1)
TypeError: float() argument must be a string or a number, not 'csr_matrix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mxnet_test.py", line 20, in <module>
    training_data = mx.io.NDArrayIter(data=preprocessed_data, label=labels, batch_size=50)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/io.py", line 643, in __init__
    self.data = _init_data(data, allow_empty=False, default_name=data_name)
  File "/usr/local/lib/python3.5/dist-packages/mxnet/io.py", line 513, in _init_data
    "should be NDArray, numpy.ndarray or h5py.Dataset")
TypeError: Invalid type '<class 'numpy.ndarray'>' for data, should be NDArray, numpy.ndarray or h5py.Dataset

我不明白，因为我的数据在创建numpy.ndarray实例之前被转换为NDArrayIter。是否有人愿意提供有关如何在mxnet中读取数据的一些见解？

上面的代码目前使用以下版本：

mxnet-1.1.0
numpy的-1.14.2

Answer 1

在user2357112的帮助下，通过在Python 3中使用异常链来查找异常（已更新）解决了这个问题：

transformer管道正在返回numpy.array个scipy.sparse.csr_matrix矩阵，而不是2-d numpy.array。通过添加更改以下行以使用toarray方法进行转换，脚本将运行。

preprocessed_data = np.array([row.toarray() for row in transformer.fit_transform(data)])

最佳解决方案：在toarray上使用时，scipy.sparse.csr_matrix在内存消耗方面效率低下。在1.10的{{1}}版本中，可以使用mxnet更有效地存储数据：

mxnet.nd.sparse.array

唯一需要注意的是，必须使用... preprocessed_data = mx.nd.sparse.array(transformer.fit_transform(data)) training_data = mx.io.NDArrayIter(data=preprocessed_data, label=preprocessed_labels, batch_size=5, last_batch_handle='discard') for i, batch in enumerate(training_data): print(batch)中的last_batch_handle='discard'关键字参数（NDArrayIter here的功能）

MXNET - 无效类型＆＃39; <type'numpy.ndarray'=“”>＆＃39;对于数据，应该是NDArray，numpy.ndarray，

1 个答案: