Question

当我在Google或Stackoverflow上查找问题时，似乎已经解决了六起这样的情况，但是我似乎从未真正了解解决方案。

因此，我想通过与Anaconda一起启动的Jupyter Lab从服务器中删除.csv。该文件确实存在，单击几下即可下载。

现在，我尝试执行以下查询：

import pandas as pd
pd.read_csv("link")

它会产生以下错误：

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-37-aae59f2238c3> in <module>
----> 1 pd.read_csv("https://first-python-notebook.readthedocs.io/_static/committees.csv")
/Applications/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, delim_whitespace, low_memory, memory_map, float_precision)
    674         )
    675 
--> 676         return _read(filepath_or_buffer, kwds)
    677 
    678     parser_f.__name__ = name
/Applications/anaconda3/lib/python3.7/site-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds)
    429     # See https://github.com/python/mypy/issues/1297
    430     fp_or_buf, _, compression, should_close = get_filepath_or_buffer(
--> 431         filepath_or_buffer, encoding, compression
    432     )
    433     kwds["compression"] = compression
/Applications/anaconda3/lib/python3.7/site-packages/pandas/io/common.py in get_filepath_or_buffer(filepath_or_buffer, encoding, compression, mode)
    170 
    171     if isinstance(filepath_or_buffer, str) and is_url(filepath_or_buffer):
--> 172         req = urlopen(filepath_or_buffer)
    173         content_encoding = req.headers.get("Content-Encoding", None)
    174         if content_encoding == "gzip":
/Applications/anaconda3/lib/python3.7/site-packages/pandas/io/common.py in urlopen(*args, **kwargs)
    139     import urllib.request
    140 
--> 141     return urllib.request.urlopen(*args, **kwargs)
    142 
    143 
/Applications/anaconda3/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220     else:
    221         opener = _opener
--> 222     return opener.open(url, data, timeout)
    223 
    224 def install_opener(opener):
/Applications/anaconda3/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)
    529         for processor in self.process_response.get(protocol, []):
    530             meth = getattr(processor, meth_name)
--> 531             response = meth(req, response)
    532 
    533         return response
/Applications/anaconda3/lib/python3.7/urllib/request.py in http_response(self, request, response)
    639         if not (200 <= code < 300):
    640             response = self.parent.error(
--> 641                 'http', request, response, code, msg, hdrs)
    642 
    643         return response
/Applications/anaconda3/lib/python3.7/urllib/request.py in error(self, proto, *args)
    567         if http_err:
    568             args = (dict, 'default', 'http_error_default') + orig_args
--> 569             return self._call_chain(*args)
    570 
    571 # XXX probably also want an abstract factory that knows when it makes
/Applications/anaconda3/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    501         for handler in handlers:
    502             func = getattr(handler, meth_name)
--> 503             result = func(*args)
    504             if result is not None:
    505                 return result
/Applications/anaconda3/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650 
    651 class HTTPRedirectHandler(BaseHandler):
HTTPError: HTTP Error 403: Forbidden

虽然有效，是当我尝试这样做时：

f = requests.get(link)
print(f.text)

通过阅读其他资源，在我看来，问题可能是我的用户代理未正确定义，这导致目标服务器拒绝了我的请求。解决方案是添加正确或伪造的“标头”，其中包括我的user_agent：https://www.whatismybrowser.com/detect/what-is-my-user-agent

所以我尝试了这个：

import http.cookiejar
from urllib.request import urlopen
site= "link"
hdr = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
       'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
}
req = urllib2.Request(site, headers=hdr)
content = page.read()
print(content)

但首先，它返回

NameError: name 'urllib2' is not defined

...我找不到适合的解决方案。

当然，我的主要问题仍然没有解决。

我不太了解是否应该设置标题。您是否需要重新对网络上的每个文件执行类似的操作？没有更通用的解决方案吗？还是这甚至是我遇到的实际问题？

Answer 1

此脚本应与Python2 / Python3一起使用（Python3中的urllib2有所更改）：

import pandas as pd

try:
    from urllib.request import Request, urlopen  # Python 3
except ImportError:
    from urllib2 import Request, urlopen  # Python 2

req = Request('<YOUR URL WITH CSV>')
req.add_header('User-Agent', 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0')
content = urlopen(req)

df = pd.read_csv(content)
print(df)

pd.read_csv产生HTTPError：HTTP错误403：禁止

1 个答案: