Question

我尝试使用requests和BeautifulSoup模块从google.scholar.com下载页面。网址为https://scholar.google.com/citations?mauthors=computer+science&hl=en&view_op=search_authors。当我将网址复制到Chrome时，我可以查看该网页。但如果我尝试使用requests下载它，则会显示404 NOT FOUND error：

<!DOCTYPE html>
<html lang="en">
<head><meta charset="utf-8"/>
<meta content="initial-scale=1, minimum-scale=1, width=device-width" name="viewport"/>
<title>Error 404 (Not Found)!!1</title>
<style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
</head><body><a href="//www.google.com/"><span aria-label="Google" id="logo"></span></a>
<p><b>404.</b> <ins>That’s an error.</ins>
</p><p>The requested URL <code>/citations</code> was not found on this server.  <ins>That’s all we know.</ins>
</p></body></html>

下载页面的脚本除了标题（因为它太长了）

url = "https://scholar.google.com/citations?mauthors=computer+science&hl=en&view_op=search_authors"
for i in range(1200):
    r = requests.get(url, headers=headers)
    soup = BeautifulSoup(r.content, "lxml")
    print soup

有人可以帮我吗？

Answer 1

Google会阻止您的请求，因为它被视为不是来自浏览器。替代使用的是卷曲。如果要在Python脚本中使用它，可以使用以下代码。

import os

html_content = os.popen('curl https://scholar.google.com/citations?mauthors=computer%20science&hl=en&view_op=search_authors').read()

Answer 2

如果您正在使用Python，则可以使用BeautifulSoup和urllib2执行此类操作：

from urllib2 import Request, urlopen
from bs4 import BeautifulSoup as soup 

url = "https://scholar.google.com/citations?mauthors=computer%20science&hl=en&view_op=search_authors"

def load_url(url):
    request = Request(url)
    # Add your header here
    request.add_header('Referer', 'python.org')
    # Note here:
    # The charset used in your current website is: 'iso-8859-1'
    # data = urlopen(request).read().decode("iso-8859-1")
    data = urlopen(request).read()

    return soup(data, "lxml")


data = load_url(url)
m = data.findAll("h3", {"class": "gsc_1usr_name"})
for k in m:
    print k.get_text()

否则，如果您使用Python3，则可以使用BeautifulSoup和urllib.request执行此类操作：

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup as soup

url = "https://scholar.google.com/citations?mauthors=computer%20science&hl=en&view_op=search_authors"

def load_url(url):
    request = Request(url)
    # Add headers
    request.add_header('Referer', 'python.org')

    with urlopen(request) as f:
        # The charset used
        # charset = f.info().get_content_charset()
        # Debug
        # print("The current charset is:", charset)
        data = f.read()

    return soup(data, 'lxml')

data = load_url(url)
m = data.findAll("h3", {"class": "gsc_1usr_name"})
for k in m:
    print(k.get_text())

输出（使用Python和Python3的两个代码）：

Herbert Simon
Geoffrey Hinton
William H. Press
Jiawei Han
Stephen Boyd
anupam gupta
David S. Johnson
Scott Shenker
Jeffrey Ullman
Deborah Estrin

为什么我可以在浏览器中查看页面，但可以使用URL下载代码

2 个答案: