Question

我试图将Google PageRank作为我的域名列表，但我最终得到了这个错误：

Python3: raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden

我已经针对我的问题尝试了一些现有的解决方案，但没有一个解决了我的问题。这是我的代码：

#  Script for getting Google Page Rank of page
#  Google Toolbar 3.0.x/4.0.x Pagerank Checksum Algorithm
#
#  original from http://pagerank.gamesaga.net/
#  this version was adapted from http://www.djangosnippets.org/snippets/221/
#  by Corey Goldberg - 2010
#
#  Licensed under the MIT license: http://www.opensource.org/licenses/mit-license.php


from __future__ import print_function, division
import sys
import urllib.request as _urlib1  # py3 
import urllib.parse as _urlib2  # py 3




def get_pagerank(url):
    hsh = check_hash(hash_url(url))
    user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7'
    gurl = 'http://toolbarqueries.google.com/tbr?client=navclient-auto&features=Rank&ch=%s&q=info:%s' % (hsh, _urlib2.quote(url))
    headers={'User-Agent':user_agent,}
    request=_urlib1.Request(gurl,None,headers) #The assembled request
    u = _urlib1.urlopen(request)
    s = u.read().decode('utf-8')  # for py2, comment .decode() part
    #print(s)  # debug - response of server
    rank = s.strip()[9:]
    if rank == '':
        rank = 'None'
    if rank == 'None':
        rank = 'None'
    return rank


def  int_str(string, integer, factor):
    for i in range(len(string)) :
        integer *= factor
        integer &= 0xFFFFFFFF
        integer += ord(string[i])
    return integer


def hash_url(string):
    c1 = int_str(string, 0x1505, 0x21)
    c2 = int_str(string, 0, 0x1003F)

    c1 >>= 2
    c1 = ((c1 >> 4) & 0x3FFFFC0) | (c1 & 0x3F)
    c1 = ((c1 >> 4) & 0x3FFC00) | (c1 & 0x3FF)
    c1 = ((c1 >> 4) & 0x3C000) | (c1 & 0x3FFF)

    t1 = (c1 & 0x3C0) << 4
    t1 |= c1 & 0x3C
    t1 = (t1 << 2) | (c2 & 0xF0F)

    t2 = (c1 & 0xFFFFC000) << 4
    t2 |= c1 & 0x3C00
    t2 = (t2 << 0xA) | (c2 & 0xF0F0000)

    return (t1 | t2)


def check_hash(hash_int):
    hash_str = '%u' % (hash_int)
    flag = 0
    check_byte = 0

    i = len(hash_str) - 1
    while i >= 0:
        byte = int(hash_str[i])
        if 1 == (flag % 2):
            byte *= 2;
            byte = int(byte / 10) + byte % 10
        check_byte += byte
        flag += 1
        i -= 1

    check_byte %= 10
    if 0 != check_byte:
        check_byte = 10 - check_byte
        if 1 == flag % 2:
            if 1 == check_byte % 2:
                check_byte += 9
            check_byte >>= 1

    return '7' + str(check_byte) + hash_str

有人可以帮忙吗？

Answer 1

问题不在于阻止IP地址。我正在使用Python3并遇到同样的问题。我发现Google会阻止不会覆盖User-Agent和Accept-Encoding标头的urllib。

用于测试搜索的标题：

GET /search?q=f1+2015 HTTP/1.1
Accept-Encoding: identity
Connection: close
User-Agent: Python-urllib/3.4
Host: 127.0.0.1:8076

我将'Accept-Encoding''和'User-Agent'设置为'testing'，403错误停止。

Python3 - urllib.error.HTTPError：HTTP错误403：禁止

1 个答案: