Question

是他们使用chunkwise

从db获取数据的任何方法

我的数据库中有大约3000万个数据，如果不使用它，将导致大量内存使用我尝试使用pandas版本0.17.1

for sd in psql.read_sql(sql,myconn,chunksize=100):
    print sd

但是扔了

/usr/bin/python2.7 /home/subin/PythonIDE/workspace/python/pygram.py
Traceback (most recent call last):
  File "/home/subin/PythonIDE/workspace/python/pygram.py", line 20, in <module>
    for sd in psql.read_sql(sql,myconn,chunksize=100):
  File "/usr/lib/python2.7/dist-packages/pandas/io/sql.py", line 1565, in _query_iterator
    parse_dates=parse_dates)
  File "/usr/lib/python2.7/dist-packages/pandas/io/sql.py", line 137, in _wrap_result
    coerce_float=coerce_float)
  File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 969, in from_records
    coerce_float=coerce_float)
  File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 5279, in _to_arrays
    dtype=dtype)
  File "/usr/lib/python2.7/dist-packages/pandas/core/frame.py", line 5357, in _list_to_arrays
    content = list(lib.to_object_array_tuples(data).T)
TypeError: Argument 'rows' has incorrect type (expected list, got tuple)

请帮帮我

如何使用Python PETL或pygramETL或pandas通过chunk从数据库chunlby chunk中获取数据

0 个答案: