在大文件上，pickle是否会随OSError随机失败？

Question

问题陈述

我正在使用python3并尝试挑选IntervalTrees字典，其重量为2到3 GB。这是我的控制台输出：

10:39:25 - project: INFO - Checking if motifs file was generated by pickle...
10:39:25 - project: INFO -   - Motifs file does not seem to have been generated by pickle, proceeding to parse...
10:39:38 - project: INFO -   - Parse complete, constructing IntervalTrees...
11:04:05 - project: INFO -   - IntervalTree construction complete, saving pickle file for next time.
Traceback (most recent call last):
  File "/Users/alex/Documents/project/src/project.py", line 522, in dict_of_IntervalTree_from_motifs_file
    save_as_pickled_object(motifs, output_dir + 'motifs_IntervalTree_dictionary.pickle')
  File "/Users/alex/Documents/project/src/project.py", line 269, in save_as_pickled_object
    def save_as_pickled_object(object, filepath): return pickle.dump(object, open(filepath, "wb"))
OSError: [Errno 22] Invalid argument

我尝试保存的行是

def save_as_pickled_object(object, filepath): return pickle.dump(object, open(filepath, "wb"))

错误发生在调用save_as_pickled_object后15分钟（在11:20）。

我尝试使用motifs文件的一个小得多的子部分，它工作得很好，所有完全相同的代码，所以它必须是一个规模问题。 在python 3.6中是否有任何已知的错误与你试图腌制的规模有关？是否有一般的酸洗大文件的已知错误？有没有任何已知的方法？

谢谢！

更新：此问题可能与Python 3 - Can pickle handle byte objects larger than 4GB?

重复

解决方案

这是我用的代码。

def save_as_pickled_object(obj, filepath):
    """
    This is a defensive way to write pickle.write, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    bytes_out = pickle.dumps(obj)
    n_bytes = sys.getsizeof(bytes_out)
    with open(filepath, 'wb') as f_out:
        for idx in range(0, n_bytes, max_bytes):
            f_out.write(bytes_out[idx:idx+max_bytes])


def try_to_load_as_pickled_object_or_None(filepath):
    """
    This is a defensive way to write pickle.load, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    try:
        input_size = os.path.getsize(filepath)
        bytes_in = bytearray(0)
        with open(filepath, 'rb') as f_in:
            for _ in range(0, input_size, max_bytes):
                bytes_in += f_in.read(max_bytes)
        obj = pickle.loads(bytes_in)
    except:
        return None
    return obj

Answer 1

Alex，如果我没有弄错的话，这个错误报告可以很好地描述你的问题。

http://bugs.python.org/issue24658

作为一种解决方法，我认为您可以pickle.dumps代替pickle.dump，然后以大小小于2 ** 31的大小写入您的文件。