Question

我正在使用Python的pickle模块保存一些大数据。更确切地说，我将两个对象转储到同一个pickle文件中，如下所示：

def save(sim):
    if sim.tstamp > 0:
        with open(filepath(sim.identifier), 'wb') as f:
            pickle.dump(sim.serialize(), f)
            pickle.dump(sim, f)
    else:
        raise ValueError("Simulation not yet simulated")

sim是一个自编写的类，它具有不同的属性和名为serialize()的方法。此方法使用以下内容将所有属性及其值转换为字典：

def serialize(self):
    keys = ['height', 'diameter', 'rpm', 'packaging']
    serialized = dict((key, getattr(self, key)) for key in keys)
    serialized['sectors'] = [sector.serialize() for sector in self.sectors]
    return serialized

所以基本上sim.serialize()返回一个字典，该字典作为第一个对象转储到pickle文件中。第二个对象是sim-object本身。这是由于每个sim对象的大尺寸。提供字典可以实现一种摘要，以防止在没有必要的情况下加载完整的模拟。

要从我使用的pickle文件加载数据：

def load(fname, params_only=False):
    simulation = find(fname) # find returns file name
    try:
        with open(simulation, 'r') as json_file:
            return json.load(json_file)
    except (UnicodeDecodeError, ValueError) as e:
        with open(simulation, 'rb') as pickle_file:
            params = pickle.load(pickle_file)
            if params_only:
                return params
            return pickle.load(pickle_file)

由于较旧的API能够使用JSON文件，因此我需要处理JSON文件和pickle文件。

根据需要转储对象（不会引发错误消息）。但是，在请求某些数据图时，API需要取消数据，这会引发以下错误：

[2016-08-07 14:39:34,637] ERROR in app: Exception on /api/simulations/20160807_123707581/heatflux_distribution.png [GET]
Traceback (most recent call last):
 File "/home/abkos/abkosproject/abkos/_tools.py", line 36, in load
   return json.load(json_file)
 File "/usr/lib/python3.4/json/__init__.py", line 265, in load
   return loads(fp.read(),
 File "/usr/lib/python3.4/codecs.py", line 319, in decode
   (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1988, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1641, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1544, in handle_user_exception
   reraise(exc_type, exc_value, tb)
 File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
   raise value
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1639, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1625, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "./app.py", line 118, in plot_figure
   img = plotter(identifier, figname, request.args.items())
 File "./app.py", line 122, in plotter
   sim = Simulation.from_file(identifier)
 File "/home/abkos/abkosproject/abkos/calculation.py", line 56, in from_file
   params_or_simulation = load(fname, params_only=new)
 File "/home/abkos/abkosproject/abkos/_tools.py", line 42, in load
   return pickle.load(pickle_file)
_pickle.UnpicklingError
10.0.2.2 - - [07/Aug/2016 14:39:34] "GET /api/simulations/20160807_123707581/heatflux_distribution.png HTTP/1.1" 500 -
[2016-08-07 14:39:38,691] ERROR in app: Exception on /api/simulations/20160807_123707581/heatmap_axial_temperatures.png [GET]
Traceback (most recent call last):
 File "/home/abkos/abkosproject/abkos/_tools.py", line 36, in load
   return json.load(json_file)
 File "/usr/lib/python3.4/json/__init__.py", line 265, in load
   return loads(fp.read(),
 File "/usr/lib/python3.4/codecs.py", line 319, in decode
   (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1988, in wsgi_app
   response = self.full_dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1641, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1544, in handle_user_exception
   reraise(exc_type, exc_value, tb)
 File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
   raise value
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1639, in full_dispatch_request
   rv = self.dispatch_request()
 File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1625, in dispatch_request
   return self.view_functions[rule.endpoint](**req.view_args)
 File "./app.py", line 118, in plot_figure
   img = plotter(identifier, figname, request.args.items())
 File "./app.py", line 122, in plotter
   sim = Simulation.from_file(identifier)
 File "/home/abkos/abkosproject/abkos/calculation.py", line 56, in from_file
   params_or_simulation = load(fname, params_only=new)
 File "/home/abkos/abkosproject/abkos/_tools.py", line 42, in load
   return pickle.load(pickle_file)
EOFError
10.0.2.2 - - [07/Aug/2016 14:39:38] "GET /api/simulations/20160807_123707581/heatmap_axial_temperatures.png HTTP/1.1" 500 -

使用Python 3.4.3，代码在MacOS和Linux Mint上运行良好。但是，它不适用于Ubuntu 14.04（Python 3.4.3）。

有关从哪里开始调查错误的任何想法？

Answer 1

由于我无法找到任何编码错误，我将三个系统相互比较，发现从软件的角度来看几乎完全相同。所有这些都使用Python 3.4.3 64位和pickle是Python的默认模块的一部分。

所以我得出的结论是，这可能是与硬件相关的问题。由于MacOS机器和Mint机器都是真实的＆＃39;机器和Ubuntu只是一个VM我只比较了一般的硬件规格，如CPU和RAM的数量。

真正的机器配备8 GB的RAM。薄荷糖12 GB。所以我将VM的RAM从大约1 GB增加到3 GB，一切正常。

因此即使没有MemoryError或类似的提升，EOFError也可能与记忆有关。

pickle在Ubuntu 14.04上引发了EOFError，但适用于Mac OS和Linux Mint

1 个答案: