为什么使用官方gzip module中的open()与gzip.open()时文件模式有所不同？
Linux上的Python 2.7。
在已经打开的文件句柄上使用GzipFile时会发生同样的事情。
我认为它应该是透明的，为什么我会看到数字模式而不是rb / wb？
#!/usr/bin/env python
"""
Write one file to another, with optional gzip on both sides.
Usage:
gzipcat.py <input file> <output file>
Examples:
gzipcat.py /etc/passwd passwd.bak.gz
gzipcat.py passwd.bak.gz passwd.bak
"""
import sys
import gzip
if len(sys.argv) < 3:
sys.exit(__doc__)
ifn = sys.argv[1]
if ifn.endswith('.gz'):
ifd = gzip.open(ifn, 'rb')
else:
ifd = open(ifn, 'rb')
ofn = sys.argv[2]
if ofn.endswith('.gz'):
ofd = gzip.open(ofn, 'wb')
else:
ofd = open(ofn, 'wb')
ifm = getattr(ifd, 'mode', None)
ofm = getattr(ofd, 'mode', None)
print('input file mode: {}, output file mode: {}'.format(ifm, ofm))
for ifl in ifd:
ofd.write(ifl)
$ python gzipcat.py /etc/passwd passwd.bak
input file mode: rb, output file mode: wb
$ python gzipcat.py /etc/passwd passwd.bak.gz
input file mode: rb, output file mode: 2
$ python gzipcat.py passwd.bak.gz passwd.txt
input file mode: 1, output file mode: wb
$ python gzipcat.py passwd.bak.gz passwd.txt.gz
input file mode: 1, output file mode: 2
次要问题：这背后是否有任何正当理由，或者只是gzip模块中的遗漏/未处理案例？
我的实际用例是使用Google BigQuery加载程序，在将其用作数据源之前，该模式需要rb模式。追溯到下面。但我准备了上面的最小测试用例，以使这个问题更具可读性。
# python -c 'import etl; etl.job001()'
Starting job001.
Processing table: reviews.
Extracting reviews, time range [2018-04-07 17:01:38.172129+00:00, 2018-04-07 18:09:50.763283)
Extracted 24 rows to reviews.tmp.gz in 2 s (8 rows/s).
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "etl.py", line 920, in wf_dimension_tables
ts_end=ts_end)
File "etl.py", line 680, in map_table_delta
rewrite=True
File "etl.py", line 624, in bq_load_csv
job_config=job_config)
File "/usr/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 797, in load_table_from_file
_check_mode(file_obj)
File "/usr/lib/python2.7/site-packages/google/cloud/bigquery/client.py", line 1419, in _check_mode
"Cannot upload files opened in text mode: use "
ValueError: Cannot upload files opened in text mode: use open(filename, mode='rb') or open(filename, mode='r+b')
这是使用文件句柄的bigquery API调用：
def bq_load_csv(dataset_id, table_id, fileobj):
client = bigquery.Client()
dataset_ref = client.dataset(dataset_id)
table_ref = dataset_ref.table(table_id)
job_config = bigquery.LoadJobConfig()
job_config.source_format = 'text/csv'
job_config.field_delimiter = ','
job_config.skip_leading_rows = 0
job_config.allow_quoted_newlines = True
job_config.max_bad_records = 0
job = client.load_table_from_file(
fileobj,
table_ref,
job_config=job_config)
res = job.result() # Waits for job to complete
return res
此问题已在python bigquery客户端1.5.0中修复。 感谢@ a-queue提交了一份错误报告，感谢谷歌开发人员实际修复过它。
答案 0 :(得分：1)
解决此问题的正确方法是在Python和Google Cloud Client Library中针对Python各自的问题跟踪器提出问题。
您可以将_check_mode google.cloud.bigquery.client函数替换为接受1和2，如下所示。我已经尝试运行此代码并且它可以工作：
import gzip
from google.cloud import bigquery
def _check_mode(stream):
mode = getattr(stream, 'mode', None)
if mode is not None and mode not in ('rb', 'r+b', 'rb+', 1, 2):
raise ValueError(
"Cannot upload files opened in text mode: use "
"open(filename, mode='rb') or open(filename, mode='r+b')")
bigquery.client._check_mode = _check_mode
#...
def bq_load_csv(dataset_id, table_id, fileobj):
#...
跟踪显示失败的最后一个是来自google/cloud/bigquery/client.py的函数_check_mode：
if mode is not None and mode not in ('rb', 'r+b', 'rb+'):
raise ValueError(
"Cannot upload files opened in text mode: use "
"open(filename, mode='rb') or open(filename, mode='r+b')")
在类__init__的函数GzipFile中的gzip库中，您可以看到变量mode已传递给此函数，但 NOT 已分配给self.mode但用于分配interger：
READ, WRITE = 1, 2 #line 18
...
class GzipFile(_compression.BaseStream):
...
def __init__(self, filename=None, mode=None,
...
elif mode.startswith(('w', 'a', 'x')): #line 179
self.mode = WRITE
根据责备行18改变了21 years ago和第180行，self.mode = Write，20 years ago。