我有一个非常大的csv文件,我无法在内存中完整加载。所以我想逐个阅读,将其转换为numpy数组,然后进行更多处理。
我已经检查过了: Lazy Method for Reading Big File in Python?
但问题是这是一个普通的读者,我无法在csvReader中找到任何指定大小的选项。
另外,因为我想将行转换为numpy数组,所以我不想读取任何一半的行,所以不是指定大小,我想要一些我可以在读者中指定“no of rows”的东西。
是否有任何内置功能或简单的方法。
答案 0 :(得分:2)
csv.reader
不会将整个文件读入内存。当您遍历reader
对象时,它会逐行地遍历文件。因此,您可以像往常一样使用reader
,但是在您阅读后的迭代中使用break
但是您要阅读的行数很多。您可以在C-code used to implement the reader
object。
Initializer for the reader objecT:
static PyObject *
csv_reader(PyObject *module, PyObject *args, PyObject *keyword_args)
{
PyObject * iterator, * dialect = NULL;
ReaderObj * self = PyObject_GC_New(ReaderObj, &Reader_Type);
if (!self)
return NULL;
self->dialect = NULL;
self->fields = NULL;
self->input_iter = NULL;
self->field = NULL;
// stuff we dont care about here
// ...
self->input_iter = PyObject_GetIter(iterator); // here we save the iterator (file object) we passed in
if (self->input_iter == NULL) {
PyErr_SetString(PyExc_TypeError,
"argument 1 must be an iterator");
Py_DECREF(self);
return NULL;
}
static PyObject *
Reader_iternext(ReaderObj *self) // This is what gets called when you call `next(reader_obj)` (which is what a for loop does internally)
{
PyObject *fields = NULL;
Py_UCS4 c;
Py_ssize_t pos, linelen;
unsigned int kind;
void *data;
PyObject *lineobj;
if (parse_reset(self) < 0)
return NULL;
do {
lineobj = PyIter_Next(self->input_iter); // Equivalent to calling `next(input_iter)`
if (lineobj == NULL) {
/* End of input OR exception */
if (!PyErr_Occurred() && (self->field_len != 0 ||
self->state == IN_QUOTED_FIELD)) {
if (self->dialect->strict)
PyErr_SetString(_csvstate_global->error_obj,
"unexpected end of data");
else if (parse_save_field(self) >= 0)
break;
}
return NULL;
}
如您所见,next(reader_object)
在内部调用next(file_object)
。所以你要逐行迭代,而不是将整个内容读入内存。
答案 1 :(得分:0)
我用过这个功能。 基本思想是使生成器生成文件中的数字。
def iter_loadtxt(filename, delimiter=',', skiprows=0, read_range=None, dtype=float):
'''
Read the file line by line and convert it to Numpy array.
:param delimiter: character
:param skiprows : int
:param read_range: [int, int] or None. set it to None and the function will read the whole file.
:param dtype: type
'''
def iter_func():
with open(filename, 'r') as infile:
for _ in range(skiprows):
next(infile)
if read_range is None:
for line in infile:
line = line.rstrip().split(delimiter)
for item in line:
yield dtype(item)
else:
counter = 0
for line in infile:
if counter < read_range[0]:
counter += 1
else:
counter += 1
for item in line:
yield dtype(item)
if counter >= read_range[1]:
break
iter_loadtxt.rowlength = len(line)
data = np.fromiter(iter_func(), dtype=dtype)
data = data.reshape((-1, iter_loadtxt.rowlength))
return data