我正在尝试使用python模块gzip压缩文件,然后使用hashlib对gzip压缩文件进行哈希处理。我有以下代码:
import hashlib
import gzip
f_name = 'read_x.fastq'
for x in range(0,3):
file = open(f_name, 'rb')
myzip = gzip.open('test.gz', 'wb', compresslevel=1)
n = 100000000
try:
print 'zipping ' + str(x)
for chunk in iter(lambda: file.read(n), ''):
myzip.write(chunk)
finally:
file.close()
myzip.close()
md5 = hashlib.md5()
print 'hashing ' + str(x)
with open('test.gz', 'r') as f:
for chunk in iter(lambda: f.read(n), ''):
md5.update(chunk)
print md5.hexdigest()
print '\n'
我认为应该简单地压缩文件,哈希并连续三次显示相同的输出哈希值。但是,我得到的输出是:
zipping 0
hashing 0
7bd80798bce074c65928e0cf9d66cae4
zipping 1
hashing 1
a3bd4e126e0a156c5d86df75baffc294
zipping 2
hashing 2
85812a39f388c388cb25a35c4fac87bf
如果我省略了gzip步骤,并且连续三次散列相同的gzip压缩文件,我确实得到了相同的输出三次:
hashing 0
ccfddd10c8fd1140db0b218124e7e9d3
hashing 1
ccfddd10c8fd1140db0b218124e7e9d3
hashing 2
ccfddd10c8fd1140db0b218124e7e9d3
任何人都可以解释这里发生了什么吗?问题必须是每次gzip进程都不同。但据我所知,DEFLATE算法是Huffman编码,后跟LZ77(一种运行长度编码形式)或LZ77后跟Huffman,因此给定相同的输入应产生相同的输出。
答案 0 :(得分:5)
压缩完全相同的内容会产生不同的gzip输出有几个原因:
所以这里有一段代码,演示了从python gzip获取可重现输出的错误和正确方法:
import hashlib
import gzip
f_name = '/etc/passwd'
output_template = '/tmp/test{}.gz'
def digest(filename: str) -> str:
md5 = hashlib.md5()
with open(output_filename, 'rb') as f:
for chunk in iter(lambda: f.read(block_size), b''):
md5.update(chunk)
return md5.hexdigest()
print("The default way - non identical outputs")
for x in range(0,3):
input_handle = open(f_name, 'rb')
output_filename = output_template.format(x)
myzip = gzip.open(output_filename, 'wb')
block_size = 4096
try:
for chunk in iter(lambda: input_handle.read(block_size), b''):
myzip.write(chunk)
finally:
input_handle.close()
myzip.close()
print(digest(output_filename))
print("The right way to get identical outputs")
for x in range(3,6):
input_handle = open(f_name, 'rb')
output_filename = output_template.format(x)
myzip = gzip.GzipFile(
filename='', # do not emit filename into the output gzip file
mode='wb',
fileobj=open(output_filename, 'wb'),
mtime=0,
)
block_size = 4096
try:
for chunk in iter(lambda: input_handle.read(block_size), b''):
myzip.write(chunk)
finally:
input_handle.close()
myzip.close()
print(digest(output_filename))
答案 1 :(得分:2)
哦,但等等......显然gzip会将时间戳信息添加到gzip文件的标题中,因此哈希值会有所不同。