django加载数百万行

时间:2014-08-22 14:45:59

标签: python django

所以我需要使用django迭代数百万行。这不适用于django从数据库中获取它们的方式,因此我调查了server side cursors

我尝试了github建议的确切代码:

from djorm_core.postgresql import server_side_cursors

with server_side_cursors():
    for item in MyModel.objects.all():
        #do something

我收到以下错误:

(我无法显示错误开始的实际行,但它的开头相当于上面的for item in myModel.objects.all()

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in __iter__(self)
     94                - Responsible for turning the rows into model objects.
     95         """
---> 96         self._fetch_all()
     97         return iter(self._result_cache)
     98 

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in _fetch_all(self)
    852     def _fetch_all(self):
    853         if self._result_cache is None:
--> 854             self._result_cache = list(self.iterator())
    855         if self._prefetch_related_lookups and not self._prefetch_done:
    856             self._prefetch_related_objects()

/usr/local/lib/python2.7/dist-packages/django/db/models/query.pyc in iterator(self)
    941         names = extra_names + field_names + aggregate_names
    942 
--> 943         for row in self.query.get_compiler(self.db).results_iter():
    944             yield dict(zip(names, row))
    945 

/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.pyc in results_iter(self)
    707         fields = None
    708         has_aggregate_select = bool(self.query.aggregate_select)
--> 709         for rows in self.execute_sql(MULTI):
    710             for row in rows:
    711                 if has_aggregate_select:

/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.pyc in execute_sql(self, result_type)
    780 
    781         cursor = self.connection.cursor()
--> 782         cursor.execute(sql, params)
    783 
    784         if not result_type:

/usr/local/lib/python2.7/dist-packages/django/db/backends/util.pyc in execute(self, sql, params)
     67         start = time()
     68         try:
---> 69             return super(CursorDebugWrapper, self).execute(sql, params)
     70         finally:
     71             stop = time()

/usr/local/lib/python2.7/dist-packages/django/db/backends/util.pyc in execute(self, sql, params)
     51                 return self.cursor.execute(sql)
     52             else:
---> 53                 return self.cursor.execute(sql, params)
     54 
     55     def executemany(self, sql, param_list):

/usr/local/lib/python2.7/dist-packages/django/db/utils.pyc in __exit__(self, exc_type, exc_value, traceback)
     97                 if dj_exc_type not in (DataError, IntegrityError):
     98                     self.wrapper.errors_occurred = True
---> 99                 six.reraise(dj_exc_type, dj_exc_value, traceback)
    100 
    101     def __call__(self, func):

/usr/local/lib/python2.7/dist-packages/django/db/backends/util.pyc in execute(self, sql, params)
     51                 return self.cursor.execute(sql)
     52             else:
---> 53                 return self.cursor.execute(sql, params)
     54 
     55     def executemany(self, sql, param_list):

ProgrammingError: can't use a named cursor outside of transactions

还有另一种方法吗?有没有办法继续使用这个模块,还是只是打破了,因为这正是github帐户的建议?

0 个答案:

没有答案