Question

我正在尝试从此网站“ https://www.forbes.com/fintech/2019/#ef5484d2b4c6”中删除表格

当我尝试查找主表时，它返回了以下内容而没有该表

Blockquote“” error message received

url = 'https://www.forbes.com/fintech/2019/#ef5484d2b4c6'
source = urlopen(url).read()
soup = BeautifulSoup(source)
div1 = soup.find('div', attrs={'id':'main-content'})
div1

quote：“”

这是我正在寻找的数据：

high level section of data

Data i would like to scrap

Answer 1

您必须先使用Selenium来使页面呈现，然后才能获取该html源并使用beautifulsoup进行解析。或者，您可以通过其api json响应访问该数据。

尽管您尝试获取的链接不在该json响应中，但它似乎遵循相同的格式/结构：

import requests

url = 'https://www.forbes.com/forbesapi/org/fintech/2019/position/true.json?limit=2000'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'}
data = requests.get(url, headers=headers).json()

for each in data['organizationList']['organizationsLists']:
    orgName = each['organizationName']
    link = 'https://www.forbes.com/companies/%s/?list=fintech' %each['uri']

    print ('%-20s %s' %(orgName, link ))

输出：

Acorns               https://www.forbes.com/companies/acorns/?list=fintech
Addepar              https://www.forbes.com/companies/addepar/?list=fintech
Affirm               https://www.forbes.com/companies/affirm/?list=fintech
Axoni                https://www.forbes.com/companies/axoni/?list=fintech
Ayasdi               https://www.forbes.com/companies/ayasdi/?list=fintech
Behavox              https://www.forbes.com/companies/behavox/?list=fintech
Betterment           https://www.forbes.com/companies/betterment/?list=fintech
Bitfury              https://www.forbes.com/companies/bitfury/?list=fintech
Blend                https://www.forbes.com/companies/blend/?list=fintech
Bolt                 https://www.forbes.com/companies/bolt/?list=fintech
Brex                 https://www.forbes.com/companies/brex/?list=fintech
Cadre                https://www.forbes.com/companies/cadre/?list=fintech
Carta                https://www.forbes.com/companies/carta/?list=fintech
Chime                https://www.forbes.com/companies/chime/?list=fintech
Circle               https://www.forbes.com/companies/circle/?list=fintech
Coinbase             https://www.forbes.com/companies/coinbase/?list=fintech
Credit Karma         https://www.forbes.com/companies/credit-karma/?list=fintech
Cross River          https://www.forbes.com/companies/cross-river/?list=fintech
Digital Reasoning    https://www.forbes.com/companies/digital-reasoning/?list=fintech
Earnin               https://www.forbes.com/companies/earnin/?list=fintech
Enigma               https://www.forbes.com/companies/enigma/?list=fintech
Even                 https://www.forbes.com/companies/even/?list=fintech
Flywire              https://www.forbes.com/companies/flywire/?list=fintech
Forter               https://www.forbes.com/companies/forter/?list=fintech
Fundrise             https://www.forbes.com/companies/fundrise/?list=fintech
Gemini               https://www.forbes.com/companies/gemini/?list=fintech
Guideline            https://www.forbes.com/companies/guideline/?list=fintech
iCapital Network     https://www.forbes.com/companies/icapital-network/?list=fintech
IEX Group            https://www.forbes.com/companies/iex-group/?list=fintech
Kabbage              https://www.forbes.com/companies/kabbage/?list=fintech
Lemonade             https://www.forbes.com/companies/lemonade/?list=fintech
LendingHome          https://www.forbes.com/companies/lendinghome/?list=fintech
Marqeta              https://www.forbes.com/companies/marqeta/?list=fintech
Nova Credit          https://www.forbes.com/companies/nova-credit/?list=fintech
Opendoor             https://www.forbes.com/companies/opendoor/?list=fintech
Personal Capital     https://www.forbes.com/companies/personal-capital/?list=fintech
Plaid                https://www.forbes.com/companies/plaid/?list=fintech
Poynt                https://www.forbes.com/companies/poynt/?list=fintech
Remitly              https://www.forbes.com/companies/remitly/?list=fintech
Ripple               https://www.forbes.com/companies/ripple/?list=fintech
Robinhood            https://www.forbes.com/companies/robinhood/?list=fintech
Roofstock            https://www.forbes.com/companies/roofstock/?list=fintech
Root Insurance       https://www.forbes.com/companies/root-insurance/?list=fintech
Stash                https://www.forbes.com/companies/stash/?list=fintech
Stripe               https://www.forbes.com/companies/stripe/?list=fintech
Symphony             https://www.forbes.com/companies/symphony/?list=fintech
Tala                 https://www.forbes.com/companies/tala/?list=fintech
Toast                https://www.forbes.com/companies/toast/?list=fintech
Tradeshift           https://www.forbes.com/companies/tradeshift/?list=fintech
TransferWise         https://www.forbes.com/companies/transferwise/?list=fintech

无法使用BeautifulSoup报废《福布斯》

1 个答案: