我一直在努力理解read和seek之间的权衡。对于小的“跳跃”，读取不需要的数据比使用seek跳过它更快。
在计时不同的读取/搜索块大小以找到临界点时，我遇到了一个奇怪的现象：read(1)比read(2)，read(3)等慢约20倍。对于不同的读取方法，效果是相同的，例如read()和readinto()。
为什么会这样？
搜索以下第2/3行的时间结果：
2 x buffered 1 byte readinto bytearray
环境：
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 5 2016, 11:45:57) [MSC v.1900 32 bit (Intel)]
计时结果：
Non-cachable binary data ingestion (file object blk_size = 8192):
- 2 x buffered 0 byte readinto bytearray:
robust mean: 6.01 µs +/- 377 ns
min: 3.59 µs
- Buffered 0 byte seek followed by 0 byte readinto:
robust mean: 9.31 µs +/- 506 ns
min: 6.16 µs
- 2 x buffered 4 byte readinto bytearray:
robust mean: 14.4 µs +/- 6.82 µs
min: 2.57 µs
- 2 x buffered 7 byte readinto bytearray:
robust mean: 14.5 µs +/- 6.76 µs
min: 3.08 µs
- 2 x buffered 2 byte readinto bytearray:
robust mean: 14.5 µs +/- 6.77 µs
min: 3.08 µs
- 2 x buffered 5 byte readinto bytearray:
robust mean: 14.5 µs +/- 6.76 µs
min: 3.08 µs
- 2 x buffered 3 byte readinto bytearray:
robust mean: 14.5 µs +/- 6.73 µs
min: 2.57 µs
- 2 x buffered 49 byte readinto bytearray:
robust mean: 14.5 µs +/- 6.72 µs
min: 2.57 µs
- 2 x buffered 6 byte readinto bytearray:
robust mean: 14.6 µs +/- 6.76 µs
min: 3.08 µs
- 2 x buffered 343 byte readinto bytearray:
robust mean: 15.3 µs +/- 6.43 µs
min: 3.08 µs
- 2 x buffered 2401 byte readinto bytearray:
robust mean: 138 µs +/- 247 µs
min: 4.11 µs
- Buffered 7 byte seek followed by 7 byte readinto:
robust mean: 278 µs +/- 333 µs
min: 15.4 µs
- Buffered 3 byte seek followed by 3 byte readinto:
robust mean: 279 µs +/- 333 µs
min: 14.9 µs
- Buffered 1 byte seek followed by 1 byte readinto:
robust mean: 279 µs +/- 334 µs
min: 15.4 µs
- Buffered 2 byte seek followed by 2 byte readinto:
robust mean: 279 µs +/- 334 µs
min: 15.4 µs
- Buffered 4 byte seek followed by 4 byte readinto:
robust mean: 279 µs +/- 334 µs
min: 15.4 µs
- Buffered 49 byte seek followed by 49 byte readinto:
robust mean: 281 µs +/- 336 µs
min: 14.9 µs
- Buffered 6 byte seek followed by 6 byte readinto:
robust mean: 281 µs +/- 337 µs
min: 15.4 µs
- 2 x buffered 1 byte readinto bytearray:
robust mean: 282 µs +/- 334 µs
min: 17.5 µs
- Buffered 5 byte seek followed by 5 byte readinto:
robust mean: 282 µs +/- 338 µs
min: 15.4 µs
- Buffered 343 byte seek followed by 343 byte readinto:
robust mean: 283 µs +/- 340 µs
min: 15.4 µs
- Buffered 2401 byte seek followed by 2401 byte readinto:
robust mean: 309 µs +/- 373 µs
min: 15.4 µs
- Buffered 16807 byte seek followed by 16807 byte readinto:
robust mean: 325 µs +/- 423 µs
min: 15.4 µs
- 2 x buffered 16807 byte readinto bytearray:
robust mean: 457 µs +/- 558 µs
min: 16.9 µs
- Buffered 117649 byte seek followed by 117649 byte readinto:
robust mean: 851 µs +/- 1.08 ms
min: 15.9 µs
- 2 x buffered 117649 byte readinto bytearray:
robust mean: 1.29 ms +/- 1.63 ms
min: 18 µs
基准代码：
from _utils import BenchmarkResults
from timeit import timeit, repeat
import gc
import os
from contextlib import suppress
from math import floor
from random import randint
### Configuration
FILE_NAME = 'test.bin'
r = 5000
n = 100
reps = 1
chunk_sizes = list(range(7)) + [7**x for x in range(1,7)]
results = BenchmarkResults(description = 'Non-cachable binary data ingestion')
### Setup
FILE_SIZE = int(100e6)
# remove left over test file
with suppress(FileNotFoundError):
os.unlink(FILE_NAME)
# determine how large a file needs to be to not fit in memory
gc.collect()
try:
while True:
data = bytearray(FILE_SIZE)
del data
FILE_SIZE *= 2
gc.collect()
except MemoryError:
FILE_SIZE *= 2
print('Using file with {} GB'.format(FILE_SIZE / 1024**3))
# check enough data in file
required_size = sum(chunk_sizes)*2*2*reps*r
print('File size used: {} GB'.format(required_size / 1024**3))
assert required_size <= FILE_SIZE
# create test file
with open(FILE_NAME, 'wb') as file:
buffer_size = int(10e6)
data = bytearray(buffer_size)
for i in range(int(FILE_SIZE / buffer_size)):
file.write(data)
# read file once to try to force it into system cache as much as possible
from io import DEFAULT_BUFFER_SIZE
buffer_size = 10*DEFAULT_BUFFER_SIZE
buffer = bytearray(buffer_size)
with open(FILE_NAME, 'rb') as file:
bytes_read = True
while bytes_read:
bytes_read = file.readinto(buffer)
blk_size = file.raw._blksize
results.description += ' (file object blk_size = {})'.format(blk_size)
file = open(FILE_NAME, 'rb')
### Benchmarks
setup = \
"""
# random seek to avoid advantageous starting position biasing results
file.seek(randint(0, file.raw._blksize), 1)
"""
read_read = \
"""
file.read(chunk_size)
file.read(chunk_size)
"""
seek_seek = \
"""
file.seek(buffer_size, 1)
file.seek(buffer_size, 1)
"""
seek_read = \
"""
file.seek(buffer_size, 1)
file.read(chunk_size)
"""
read_read_timings = {}
seek_seek_timings = {}
seek_read_timings = {}
for chunk_size in chunk_sizes:
read_read_timings[chunk_size] = []
seek_seek_timings[chunk_size] = []
seek_read_timings[chunk_size] = []
for j in range(r):
#file.seek(0)
for chunk_size in chunk_sizes:
buffer = bytearray(chunk_size)
read_read_timings[chunk_size].append(timeit(read_read, setup, number=reps, globals=globals()))
#seek_seek_timings[chunk_size].append(timeit(seek_seek, setup, number=reps, globals=globals()))
seek_read_timings[chunk_size].append(timeit(seek_read, setup, number=reps, globals=globals()))
for chunk_size in chunk_sizes:
results['2 x buffered {} byte readinto bytearray'.format(chunk_size)] = read_read_timings[chunk_size]
#results['2 x buffered {} byte seek'.format(chunk_size)] = seek_seek_timings[chunk_size]
results['Buffered {} byte seek followed by {} byte readinto'.format(chunk_size, chunk_size)] = seek_read_timings[chunk_size]
### Cleanup
file.close()
os.unlink(FILE_NAME)
results.show()
results.save()
答案 0 :(得分：2)
我能够用您的代码重现该问题。但是，我注意到以下情况：如果更换，您可以验证问题是否消失
file.seek(randint(0, file.raw._blksize), 1)
使用
file.seek(randint(0, file.raw._blksize), 0)
在setup中？我认为您可能在读取1个字节的某个时刻耗尽了数据。读取2个字节，3个字节等等将没有任何数据可读取，因此这就是它要快得多的原因。
答案 1 :(得分：1)
从文件句柄逐字节读取通常比分块读取慢。
通常，每个read（）调用都对应于Python中的C read（）调用。总结果涉及一个请求下一个字符的系统调用。对于2 kb的文件，这意味着对内核的2000次调用。每个都需要一个函数调用，然后向内核请求，然后等待响应，并将其传递给返回值。
这里最值得注意的是Foo，系统调用将一直阻塞，直到您的呼叫在队列中被确认为止，因此您必须等待。
呼叫越少越好，所以更多的字节会更快；这就是为什么缓冲io相当普遍的原因。
在python中，可以通过awaiting response或通过io.BufferedReader上的buffering关键字参数为文件提供缓冲
答案 2 :(得分：0)
在处理与EEPROM接口的arduino时，我也遇到过类似情况。基本上，为了从芯片或数据结构中进行写入或读取，您必须发送一个写入/读取启用命令，发送一个起始位置，然后抓取第一个字符。但是，如果抓取多个字节，大多数芯片将自动增加其目标地址寄存器。因此，开始读取/写入操作会有一些开销。两者之间的区别是
和
就机器指令而言，一次读取多个位/字节可节省大量开销。更糟糕的是，某些芯片要求您在发送读/写使能后让您闲置几个时钟周期，以使机械过程将晶体管物理地移动到适当的位置以进行读取或写入。