修改

Question

但是当我运行我的代码时，它会挂起并且没有响应。这是我的代码。谢谢你的帮助。

import json, requests, time

url="http://www.indiegogo.com/private_api/campaigns/the-middleman-the-crowd-funded-franchise-resurrection/pledges"

r=requests.get(url)

print(r.text)

Answer 1

通过将User-Agent添加到标题

来解决

import requests

url = ("https://www.indiegogo.com/private_api/campaigns/"
       "the-middleman-the-crowd-funded-franchise-resurrection/pledges")
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) '
                         'AppleWebKit/537.36 (KHTML, like Gecko) '
                         'Chrome/39.0.2171.95 Safari/537.36'}
r = requests.get(url, headers=headers)
print(r.text)

给我们

<!DOCTYPE html>

<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
<meta http-equiv="cache-control" content="max-age=0" />
<meta http-equiv="cache-control" content="no-cache" />
<meta http-equiv="expires" content="0" />
<meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="refresh" content="10; url=/distil_r_blocked.html?requestId=c556d45a-825c-463b-ad6a-dace0f6ab5c9&httpReferrer=%2Fprivate_api%2Fcampaigns%2Fthe-middleman-the-crowd-funded-franchise-resurrection%2Fpledges" />
<script type="text/javascript">
    (function(window){
        try {
            if (typeof sessionStorage !== 'undefined'){
                sessionStorage.setItem('distil_referrer', document.referrer);
            }
        } catch (e){}
    })(window);
</script>
<script type="text/javascript" src="/gogodstlywuvxyesewzaxdveubcvzbbwtuvuxv.js" defer></script><style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#zxtwystcxcbebz{display:none!important}</style></head>
<body>
<div id="distilIdentificationBlock">&nbsp;</div>
</body>
</html>

或者它不是你想要的？

修改

看起来这个站点拒绝来自机器人的请求并创建无限循环。

可能的解决方法是将selenium包用于ChromeDriver：

import os
from contextlib import closing

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait

url = ("https://www.indiegogo.com/private_api/campaigns/"
       "the-middleman-the-crowd-funded-franchise-resurrection/pledges")

# here we are assuming that driver binary located in current directory
# (path to driver should be absolute),
# change if it differs
chrome_driver_path = os.path.abspath('chromedriver')

xpath = '/html/body/pre'
with closing(webdriver.Chrome(chrome_driver_path)) as web_driver:
    web_driver.get(url)
    web_driver_wait = WebDriverWait(web_driver, timeout=5)
    locator = (By.XPATH, xpath)
    web_driver_wait.until(expected_conditions
                          .presence_of_element_located(locator))
    text = web_driver.find_element_by_xpath(xpath).text
    print(text)

给我们

{"response": [{"pledger_display_name": "bobreturns",
               "pledger_profile_url": "/individuals/5355846",
               "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g1.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$20", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "Russell Auer",
               "pledger_profile_url": "/individuals/1167704",
               "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g2.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$15", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "Christopher Kindred",
               "pledger_profile_url": "/individuals/318158",
               "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g0.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$50", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "lukemjohnsonusd",
               "pledger_profile_url": null, "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g1.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$15", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "Kristian BjÃ¸rkelo",
               "pledger_profile_url": "/individuals/1330533",
               "time_ago": "over 3 years ago",
               "pledger_image_url": "https://c1.iggcdn.com/indiegogo-media-prod-cld/image/upload/c_fill,f_auto,h_50,w_50/v1432588431/yj6ruqvreu3hmbpqdvej.jpg",
               "display_amount": "$15", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "grandmoffdavid",
               "pledger_profile_url": null, "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g2.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$15", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "Anonymous",
               "pledger_profile_url": null, "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g0.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$30", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "Craig Shemin",
               "pledger_profile_url": "/individuals/4550201",
               "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g1.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$20", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "@Mech_Turtle",
               "pledger_profile_url": null, "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g2.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": null, "display_amount_iso_code": null,
               "pledge_details_url": null, "order_id": null},
              {"pledger_display_name": "venusdemolly",
               "pledger_profile_url": null, "time_ago": "over 3 years ago",
               "pledger_image_url": "https://g0.iggcdn.com/assets/individuals/missing/cubepeep-3c70b9b8bef12ca70a38d32e7c635576ae8a66bc7cfff10612ba6876fd7c68c7.png",
               "display_amount": "$50", "display_amount_iso_code": "USD",
               "pledge_details_url": null, "order_id": null}], "count": 10,
 "pagination": {"previous": null, "next": 2, "current": 1, "per_page": 10,
                "count": 1340, "pages": 134}}

警告

它不适用于PhantomJS驱动程序， ChromeDriver 使用--headless选项

实例化

我想使用python从API访问数据

1 个答案:

修改

警告