Question

我有gzipped JSON file我试图下载，但当我尝试使用请求库读取响应内容时，它会重置连接。

data = requests.request("GET", i, stream=True)
with gzip.open(i.rsplit("/")[-1], "wb") as fh:
  for chunk in data.iter_content(chunk_size=1024):
    fh.write(chunk)

以下是我尝试阅读内容时的内容：

Traceback (most recent call last):
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\packages\urllib3\response.py", line 228, in _error_catcher
    yield
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\packages\urllib3\response.py", line 501, in read_chunked
    chunk = self._handle_chunk(amt)
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\packages\urllib3\response.py", line 461, in _handle_chunk
    value = self._fp._safe_read(amt)
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python35\Lib\http\client.py", line 592, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python35\Lib\socket.py", line 575, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\models.py", line 664, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\packages\urllib3\response.py", line 349, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\packages\urllib3\response.py", line 526, in read_chunked
    self._original_response.close()
  File "C:\Users\Mike\AppData\Local\Programs\Python\Python35\Lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\packages\urllib3\response.py", line 246, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "H:/Programming/Python/warehouse/main.py", line 55, in <module>
    compile_auctions(slugs)
  File "H:/Programming/Python/warehouse/main.py", line 44, in compile_auctions
    for chunk in data.iter_content(chunk_size=1024):
  File "H:\Programming\Python\virtualenvs\warehouse\lib\site-packages\requests\models.py", line 667, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)", ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

我可以在浏览器和邮递员中阅读JSON，所以我不确定我做错了什么。我在Windows 10上使用Python 3.5.1。

这是保存gzip压缩JSON的正确方法吗？

Answer 1

请求已经解压缩数据。

您不需要使用gzip：

import requests

req = requests.get("http://auction-api-us.worldofwarcraft.com/auction-data/4923213e5eb3ec3b7e03d22b632bda36/auctions.json", stream=True)

with open("out.json", "wb") as f:
    for chunk in req.iter_content(chunk_size=4096):
        f.write(chunk)

Answer 2

所以问题实际上并非与我使用的API有关，它是Windows的请求库的错误或Windows 10上的Python 3.5.1的问题。

当我使用Python 3.5.1在Ubuntu 16.04上运行这个确切的代码时，它运行得很好。

保存gzip压缩JSON会导致连接重置

2 个答案: