我正在尝试使用Python分块读取和处理一个大文件。我遵循this blog，它提出了一种非常快速的方法来读取和处理散布在多个进程中的大量数据。我仅对现有代码进行了少许更新，即在stat(fin).st_size上使用os.path.getsize。在该示例中，我也没有实现多处理，因为该问题在单个过程中也很明显。这样可以更轻松地进行调试。
这段代码的问题是它返回断句。这是有道理的：指针不考虑行尾，而只返回给定的字节大小。实际上，人们会假定您可以通过在提取的行中省略最后一项来解决此问题，因为这很可能是折线。不幸的是，这也不可靠。
from os import stat
def chunkify(pfin, buf_size=1024):
file_end = stat(pfin).st_size
with open(pfin, 'rb') as f:
chunk_end = f.tell()
while True:
chunk_start = chunk_end
f.seek(buf_size, 1)
f.readline()
chunk_end = f.tell()
yield chunk_start, chunk_end - chunk_start
if chunk_end > file_end:
break
def process_batch(pfin, chunk_start, chunk_size):
with open(pfin, 'r', encoding='utf-8') as f:
f.seek(chunk_start)
batch = f.read(chunk_size).splitlines()
# changing this to batch[:-1] will result in 26 lines total
return batch
if __name__ == '__main__':
fin = r'data/tiny.txt'
lines_n = 0
for start, size in chunkify(fin):
lines = process_batch(fin, start, size)
# Uncomment to see broken lines
# for line in lines:
# print(line)
# print('\n')
lines_n += len(lines)
print(lines_n)
# 29
上面的代码将打印29作为已处理行的总数。当您不返回批次的最后一项时，天真地假设那是一条虚线，您将得到26。实际的行数是27。测试数据可以在下面找到。
She returned bearing mixed lessons from a society where the tools of democracy still worked.
If you think you can sense a "but" approaching, you are right.
Elsewhere, Germany take on Brazil and Argentina face Spain, possibly without Lionel Messi.
What sort of things do YOU remember best?'
Less than three weeks after taking over from Lotz at Wolfsburg.
The buildings include the Dr. John Micallef Memorial Library.
For women who do not have the genes, the risk drops to just 2% for ovarian cancer and 12% for breast cancer.
In one interview he claimed it was from the name of the Cornish language ("Kernewek").
8 Goldschmidt was out of office between 16 and 19 July 1970.
Last year a new law allowed police to shut any bar based on security concerns.
But, Frum explains: "Glenn Beck takes it into his head that this guy is bad news."
Carrying on the Romantic tradition of landscape painting.
This area has miles of undeveloped beach adjacent to the headlands.
The EAC was created in 2002 to help avoid a repeat of the disputed 2000 presidential election.
In May 1945, remnants of the German Army continue fight on in the Harz mountains, nicknamed "The Void" by American troops.
Dietler also said Abu El Haj was being opposed because she is of Palestinian descent.
The auction highlights AstraZeneca's current focus on boosting returns to shareholders as it heads into a wave of patent expiries on some of its biggest selling medicines including Nexium, for heartburn and stomach ulcers, and Seroquel for schizophrenia and bipolar disorder.
GAAP operating profit was $13.2 million and $7.1 million in the second quarter of 2008 and 2007, respectively.
Doc, Ira, and Rene are sent home as part of the seventh bond tour.
only I am sick of always hearing him called the Just.
Also there is Meghna River in the west of Brahmanbaria.
The explosives were the equivalent of more than three kilograms of dynamite - equal to 30 grenades," explained security advisor Markiyan Lubkivsky to reporters gathered for a news conference in Kyiv.
Her mother first took her daughter swimming at the age of three to help her with her cerebal palsy.
A U.S. aircraft carrier, the USS "Ticonderoga", was also stationed nearby.
Louis shocked fans when he unexpectedly confirmed he was expecting a child in summer 2015.
99, pp.
Sep 19: Eibar (h) WON 6-1
如果打印出创建的行，您会发现确实出现了断句。我觉得这很奇怪。 f.readline()是否必须确保在下一行之前都可以读取文件？在下面的输出中，空行将两个批次分开。这意味着您不能检查批处理中的下一行，如果它是子字符串，则不能将其删除-断句属于整个批处理之外的另一批处理。
...
This area has miles of undeveloped beach adjacent to the headlands.
The EAC was created in 2002 to help avoid a repeat of the disputed 2000 presidential election.
In May 1945, r
In May 1945, remnants of the German Army continue fight on in the Harz mountains, nicknamed "The Void" by American troops.
...
有没有办法消除这些断句而又不删除太多？
您可以下载更大的测试文件（100,000行）here。
经过大量挖掘，似乎实际上某些不可访问的缓冲区是导致搜寻行为不一致的原因，如here和here所述。我尝试了将iter(f.readline, '')与seek一起使用的建议解决方案，但这仍然给我带来不一致的结果。我已经更新了代码，以在每批1500行之后返回文件指针，但实际上批返回将重叠。
from os import stat
from functools import partial
def chunkify(pfin, max_lines=1500):
file_end = stat(pfin).st_size
with open(pfin, 'r', encoding='utf-8') as f:
chunk_end = f.tell()
for idx, l in enumerate(iter(f.readline, '')):
if idx % max_lines == 0:
chunk_start = chunk_end
chunk_end = f.tell()
# yield start position, size, and is_last
yield chunk_start, chunk_end - chunk_start
chunk_start = chunk_end
yield chunk_start, file_end
def process_batch(pfin, chunk_start, chunk_size):
with open(pfin, 'r', encoding='utf-8') as f:
f.seek(chunk_start)
chunk = f.read(chunk_size).splitlines()
batch = list(filter(None, chunk))
return batch
if __name__ == '__main__':
fin = r'data/100000-ep+gutenberg+news+wiki.txt'
process_func = partial(process_batch, fin)
lines_n = 0
prev_last = ''
for start, size in chunkify(fin):
lines = process_func(start, size)
if not lines:
continue
# print first and last ten sentences of batch
for line in lines[:10]:
print(line)
print('...')
for line in lines[-10:]:
print(line)
print('\n')
lines_n += len(lines)
print(lines_n)
下面是重叠批次的示例。最后一批的前两个半句子与之前一批的最后一个句子重复。我不知道如何解释或解决这个问题。
...
The EC ordered the SFA to conduct probes by June 30 and to have them confirmed by a certifying authority or it would deduct a part of the funding or the entire sum from upcoming EU subsidy payments.
Dinner for two, with wine, 250 lari.
It lies a few kilometres north of the slightly higher Weissmies and also close to the slightly lower Fletschhorn on the north.
For the rest we reached agreement and it was never by chance.
Chicago Blackhawks defeat Columbus Blue Jackets for 50th win
The only drawback in a personality that large is that no one els
For the rest we reached agreement and it was never by chance.
Chicago Blackhawks defeat Columbus Blue Jackets for 50th win
The only drawback in a personality that large is that no one else, whatever their insights or artistic pedigree, is quite as interesting.
Sajid Nadiadwala's reboot version of his cult classic "Judwaa", once again directed by David Dhawan titled "Judwaa 2" broke the dry spell running at the box office in 2017.
They warned that there will be a breaking point, although it is not clear what that would be.
...
除此之外，我还尝试过从原始代码中删除readline，并跟踪剩余的不完整块。然后将不完整的块传递到下一个块，并将其添加到其前面。我现在遇到的问题是，因为文本是按字节块读取的，所以可能发生块结尾而没有完全完成字符字节的情况。这将导致解码错误。
from os import stat
def chunkify(pfin, buf_size=1024):
file_end = stat(pfin).st_size
with open(pfin, 'rb') as f:
chunk_end = f.tell()
while True:
chunk_start = chunk_end
f.seek(buf_size, 1)
chunk_end = f.tell()
is_last = chunk_end >= file_end
# yield start position, size, and is_last
yield chunk_start, chunk_end - chunk_start, is_last
if is_last:
break
def process_batch(pfin, chunk_start, chunk_size, is_last, leftover):
with open(pfin, 'r', encoding='utf-8') as f:
f.seek(chunk_start)
chunk = f.read(chunk_size)
# Add previous leftover to current chunk
chunk = leftover + chunk
batch = chunk.splitlines()
batch = list(filter(None, batch))
# If this chunk is not the last one,
# pop the last item as that will be an incomplete sentence
# We return this leftover to use in the next chunk
if not is_last:
leftover = batch.pop(-1)
return batch, leftover
if __name__ == '__main__':
fin = r'ep+gutenberg+news+wiki.txt'
lines_n = 0
left = ''
for start, size, last in chunkify(fin):
lines, left = process_batch(fin, start, size, last, left)
if not lines:
continue
for line in lines:
print(line)
print('\n')
numberlines = len(lines)
lines_n += numberlines
print(lines_n)
运行上面的代码，不可避免地会产生UnicodeDecodeError。
Traceback (most recent call last):
File "chunk_tester.py", line 46, in <module>
lines, left = process_batch(fin, start, size, last, left)
File "chunk_tester.py", line 24, in process_batch
chunk = f.read(chunk_size)
File "lib\codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start byte
答案 0 :(得分：2)
您如此很亲密！对最终代码进行相对简单的更改（以bytes而不是str的形式读取数据）使所有工作（几乎）正常进行。
主要问题是因为从二进制文件读取的计数为 bytes ，但是从文本文件读取的计数为 text ，因此您第一次以字节为单位，第二个以 em> characters ，导致您对已经读取了哪些数据的假设是错误的。与内部隐藏缓冲区无关。
其他更改：
b'\n'上进行拆分，而不是使用bytes.splitlines()，并且仅在相关检测代码后 删除空白行。chunkify可以由功能相同的更简单，更快速的循环代替，而无需保持文件打开。这给出了最终代码：
from os import stat
def chunkify(pfin, buf_size=1024**2):
file_end = stat(pfin).st_size
i = -buf_size
for i in range(0, file_end - buf_size, buf_size):
yield i, buf_size, False
leftover = file_end % buf_size
if leftover == 0: # if the last section is buf_size in size
leftover = buf_size
yield i + buf_size, leftover, True
def process_batch(pfin, chunk_start, chunk_size, is_last, leftover):
with open(pfin, 'rb') as f:
f.seek(chunk_start)
chunk = f.read(chunk_size)
# Add previous leftover to current chunk
chunk = leftover + chunk
batch = chunk.split(b'\n')
# If this chunk is not the last one,
# pop the last item as that will be an incomplete sentence
# We return this leftover to use in the next chunk
if not is_last:
leftover = batch.pop(-1)
return [s.decode('utf-8') for s in filter(None, batch)], leftover
if __name__ == '__main__':
fin = r'ep+gutenberg+news+wiki.txt'
lines_n = 0
left = b''
for start, size, last in chunkify(fin):
lines, left = process_batch(fin, start, size, last, left)
if not lines:
continue
for line in lines:
print(line)
print('\n')
numberlines = len(lines)
lines_n += numberlines
print(lines_n)
答案 1 :(得分：2)
您在这里遇到了一个有趣的问题。您有n个进程，每个进程都指定了要处理的数据块的位置，但是由于要处理行和位置，因此无法提供这些块的 exact 位置以字节为单位。即使将文件分成几行以获取块的精确位置，您仍然遇到一些问题。
这是次优的解决方案（我假设您不想按顺序处理行：这很明显）：
B\nM\nA，其中B（之前）和A（之后）不包含任何换行符，但是M可以包含换行符； M中的行并将B\nA放在当前块索引的列表中； B\nA元素。这不是最佳选择，因为一旦处理完每个M，仍然必须处理所有B\nA，并且最后的工作必须等待其他过程完成。
代码如下：
def chunkify(file_end, buf_size=1024):
"""Yield chunks of `buf_size` bytes"""
for chunk_start in range(0, file_end, buf_size):
yield chunk_start, min(buf_size, file_end - chunk_start)
def process_batch(remainders, i, f, chunk_start, chunk_size):
"""Process a chunk"""
f.seek(chunk_start)
chunk = f.read(chunk_size)
chunk, remainders[i] = normalize(chunk)
# process chunk here if chunk is not None
return chunk
def normalize(chunk):
"""Return `M, B\\nA`
The chunk format is `B\\nM\\nA` where `B` (before) and `A` (after) do not contains any line feed,
but `M` may contain line feeds"""
i = chunk.find(b"\n")
j = chunk.rfind(b"\n")
if i == -1 or i == j:
return None, chunk
else:
return chunk[i+1:j], chunk[:i]+chunk[j:]
请注意，如果该块没有中间（M部分），那么我们将返回None作为块并将所有内容发送到remainders。
一些测试：
text = """She returned bearing mixed lessons from a society where the tools of democracy still worked.
If you think you can sense a "but" approaching, you are right.
Elsewhere, Germany take on Brazil and Argentina face Spain, possibly without Lionel Messi.
What sort of things do YOU remember best?'
Less than three weeks after taking over from Lotz at Wolfsburg.
The buildings include the Dr. John Micallef Memorial Library.
For women who do not have the genes, the risk drops to just 2% for ovarian cancer and 12% for breast cancer.
In one interview he claimed it was from the name of the Cornish language ("Kernewek").
8 Goldschmidt was out of office between 16 and 19 July 1970.
Last year a new law allowed police to shut any bar based on security concerns.
But, Frum explains: "Glenn Beck takes it into his head that this guy is bad news."
Carrying on the Romantic tradition of landscape painting.
This area has miles of undeveloped beach adjacent to the headlands.
The EAC was created in 2002 to help avoid a repeat of the disputed 2000 presidential election.
In May 1945, remnants of the German Army continue fight on in the Harz mountains, nicknamed "The Void" by American troops.
Dietler also said Abu El Haj was being opposed because she is of Palestinian descent.
The auction highlights AstraZeneca's current focus on boosting returns to shareholders as it heads into a wave of patent expiries on some of its biggest selling medicines including Nexium, for heartburn and stomach ulcers, and Seroquel for schizophrenia and bipolar disorder.
GAAP operating profit was $13.2 million and $7.1 million in the second quarter of 2008 and 2007, respectively.
Doc, Ira, and Rene are sent home as part of the seventh bond tour.
only I am sick of always hearing him called the Just.
Also there is Meghna River in the west of Brahmanbaria.
The explosives were the equivalent of more than three kilograms of dynamite - equal to 30 grenades," explained security advisor Markiyan Lubkivsky to reporters gathered for a news conference in Kyiv.
Her mother first took her daughter swimming at the age of three to help her with her cerebal palsy.
A U.S. aircraft carrier, the USS "Ticonderoga", was also stationed nearby.
Louis shocked fans when he unexpectedly confirmed he was expecting a child in summer 2015.
99, pp.
Sep 19: Eibar (h) WON 6-1"""
import io, os
def get_line_count(chunk):
return 0 if chunk is None else len(chunk.split(b"\n"))
def process(f, buf_size):
f.seek(0, os.SEEK_END)
file_end = f.tell()
remainders = [b""]*(file_end//buf_size + 1)
L = 0
for i, (start, n) in enumerate(chunkify(file_end, buf_size)):
chunk = process_batch(remainders, i, f, start, n)
L += get_line_count(chunk)
print("first pass: lines processed", L)
print("remainders", remainders)
last_chunk = b"".join(remainders)
print("size of last chunk {} bytes, {} lines".format(len(last_chunk), get_line_count(last_chunk)))
L += get_line_count(last_chunk)
print("second pass: lines processed", L)
process(io.BytesIO(bytes(text, "utf-8")), 256)
process(io.BytesIO(bytes(text, "utf-8")), 512)
with open("/home/jferard/prog/stackoverlfow/ep+gutenberg+news+wiki.txt", 'rb') as f:
process(f, 4096)
with open("/home/jferard/prog/stackoverlfow/ep+gutenberg+news+wiki.txt", 'rb') as f:
process(f, 16384)
输出：
first pass: lines processed 18
remainders [b'She returned bearing mixed lessons from a society where the tools of democracy still worked.\nWhat sort', b" of things do YOU remember best?'\nFor women who do not have the genes, the risk drops to just 2% for ovarian cancer and 12% for br", b'east cancer.\nBut, Frum explai', b'ns: "Glenn Beck takes it into his head that this guy is bad news."\nThe EAC was created in 2002 to help avoid a repeat of the dispu', b'ted 2000 presidential election.\nThe auction hig', b"hlights AstraZeneca's current focus on boosting returns to shareholders as it heads into a wave of patent expiries on some of its biggest selling medicines including Nexium, for heartburn and stomach ulcers, and Seroquel for schizophrenia and bipolar disor", b'der.\nAlso there is Meghn', b'a River in the west of Brahmanbaria.\nHer mother first to', b'ok her daughter swimming at the age of three to help her with her cerebal palsy.\nS', b'ep 19: Eibar (h) WON 6-1']
size of last chunk 880 bytes, 9 lines
second pass: lines processed 27
first pass: lines processed 21
remainders [b'She returned bearing mixed lessons from a society where the tools of democracy still worked.\nFor women who do not have the genes, the risk drops to just 2% for ovarian cancer and 12% for br', b'east cancer.\nThe EAC was created in 2002 to help avoid a repeat of the dispu', b"ted 2000 presidential election.\nThe auction highlights AstraZeneca's current focus on boosting returns to shareholders as it heads into a wave of patent expiries on some of its biggest selling medicines including Nexium, for heartburn and stomach ulcers, and Seroquel for schizophrenia and bipolar disor", b'der.\nHer mother first to', b'ok her daughter swimming at the age of three to help her with her cerebal palsy.\nSep 19: Eibar (h) WON 6-1']
size of last chunk 698 bytes, 6 lines
second pass: lines processed 27
first pass: lines processed 96963
remainders [b'She returned bearing mixed lessons from a society where the tools of democracy still worked, but where the native Dutch were often less than warm to her and her fellow exiles.\nOne of the Ffarquhar ', ..., b'the old device, Apple will give customers a gift card that can be applied toward the purchase of the new iPhone.']
size of last chunk 517905 bytes, 3037 lines
second pass: lines processed 100000
first pass: lines processed 99240
remainders [b'She returned bearing mixed lessons from a society where the tools of democracy still worked, but where the native Dutch were often less than warm to her and her fellow exiles.\nSoon Carroll was in push-up position walking her hands tow', b'ard the mirror at one side of the room while her feet were dragged along by the casual dinnerware.\nThe track "Getaway" was inspired by and allud', ..., b'the old device, Apple will give customers a gift card that can be applied toward the purchase of the new iPhone.']
size of last chunk 130259 bytes, 760 lines
second pass: lines processed 100000
最后一个示例显示您可以并行处理100,000条线中的99,240条，但是在所有处理完成之后，您必须处理最后760条线（130kio）。
关于并发性的注意事项：每个子进程都拥有remainders列表的固定单元，因此应该没有内存损坏。将每个剩余的对象存储在自己的流程对象中（围绕实际子流程的包装器），并在流程完成后将所有剩余的对象合并起来可能会更清洁。
答案 2 :(得分：1)
在文本模式下关闭文件时（第二个代码示例），read将size参数视为“字符数”（不是字节），但是seek和{ {1}}与“空缓冲区”中文件的当前位置有关，所以：
tell计算出块大小（供read使用）使用len(l)来计算最后一块的大小是不正确的（由于使用file_end = stat(pfin).st_size编码，非拉丁字母的字符数可能不等于已用字节数）
utf-8仍不能用于计算块大小，但可以为f.tell()提供正确的结果。我认为这与chunk_start的缓冲有关：TextIOWrapper提供有关缓冲区+解码器状态的信息，而不是有关文本流中实际位置的信息。您可以查看参考实现（def _read_chunk，def tell），发现它非常复杂，没有人应该相信通过不同的tell / tell调用计算出的增量（ "# Grab all the decoded text (we will rewind any extra bits later)."再次提示了“不正确”排名的原因）
搜索/讲述可以正常工作以进行“搜索”，但是不能用于计算seek -s之间的字符数（甚至字节数也不正确）。要获得正确的tell增量二进制非缓冲模式（byte），但在这种情况下，开发人员应确保所有读取均返回“全字符”（在“ utf-8”中为不同字符）具有不同的字节长度）
但是只需使用with open(path, 'rb', buffering=0) as f: ...就可以解决所有问题，因此您可以继续使用文本模式打开文件！您的代码的下一个修改版本似乎可以正常工作：
chunk_size + =len(l)