Question

所以我试图废弃psn商店。具体这个链接如下。我试图抓住游戏的数据和正在发售的价格。

https://store.playstation.com/#!/en-us/2-for-1/cid=STORE-MSF77008-PLAYCOLLMULTIBUY

r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")

我想要的数据是你右键点击网页然后点击检查。所以对于Firewatch来说，它看起来就像这样。

< h3 class="cellTitle">Firewatch</h3>  
< li class="buyPrice ">$19.99</li>

现在，当我打印出soup.prettify()时，我得到了这个

html,body,div,span,applet,object,iframe,h1,h2,h3,h4,h5,h6,p,blockquote,pre,a,abbr,acronym,address,big,cite,code,del,dfn,em,img,ins,kbd,q,s,samp,small,strike,strong,sub,sup,tt,var,b,u,i,center,dl,dt,dd,ol,ul,li,fieldset,form,label,legend,table,caption,

没有任何实际数据

我必须在这里用错误的方法做错事，但我正在阅读的指南和其他人的问题似乎都在做我自己的事情？

Answer 1

我查了一下这个网站有点难过。如果您使用浏览器检查链接。您将看到loading...文字。当您实际提出请求时，您只需获取此页面的一部分，而其他数据实际上未加载。它是由JavaScript加载的。也许你可以在这个网站上使用selenium类似的解决方案。

Answer 2

在phantomjs（http://phantomjs.org/download.html）和Selenium的帮助下你可以做到这一点

步骤：1。在终端或cmd上使用命令：pip install selenium 2.下载phantomjs＆amp;解压缩它而不是放入＆＃34; phantomjs.exe＆＃34;在python路径上，例如在Windows上，C：\ Python27

使用此代码，它将为您提供所需的结果：

from  selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By


url="https://store.playstation.com/#!/en-us/2-for-1/cid=STORE-MSF77008-PLAYCOLLMULTIBUY"

driver = webdriver.PhantomJS()
driver.get(url)

element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".cellTitle")))


gamenames=driver.find_elements(By.CLASS_NAME,'cellTitle')

prices= driver.find_elements(By.CLASS_NAME,'buyPrice ')

links= driver.find_elements(By.CLASS_NAME,'permalink')


time.sleep(2)

if len(gamenames) == len(prices):
    for i in range(len(prices)):
        print "The Name of Game is :" + gamenames[i].text + " The Price for Which is : "+ prices[i].text + " The url for it is: " + links[i].get_attribute('href')
else:
    print "Parsing fail as Some data is not parsed properlly, Try Again"
driver.quit()

它将打印：

The Name of Game is :Yu-Gi-Oh! Legacy of the Duelist The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/yu-gi-oh-legacy-of-the-duelist/cid=UP0101-CUSA02718_00-YGOLEGACYOFDUELB
The Name of Game is :Firewatch The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/firewatch/cid=UP0146-CUSA04107_00-FIREWATCH0000000
The Name of Game is :The Escapists The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/the-escapists/cid=UP4064-CUSA01880_00-THEESCAPISTS0000
The Name of Game is :Oxenfree The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/oxenfree/cid=UP0962-CUSA04950_00-OXENBASEENUS0000
The Name of Game is :Duke Nukem 3D: 20th Anniversary World Tour The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/duke-nukem-3d-20th-anniversary-world-tour/cid=UP0292-CUSA04899_00-PAGODA0000000000
The Name of Game is :Primal Carnage: Extinction The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/primal-carnage-extinction/cid=UP0505-CUSA03371_00-PRIMALCARNAGE000
The Name of Game is :The Bunker The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/the-bunker/cid=UP4459-CUSA06057_00-THEBUNKERGAMEPS4
The Name of Game is :Shantae and the Pirate's Curse The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/shantae-and-the-pirate's-curse/cid=UP2053-CUSA01609_00-SHANTAECURSENA01
The Name of Game is :Pure Pool The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/pure-pool/cid=UP2070-CUSA00328_00-UPUREPOOL0000001
The Name of Game is :Banner Saga 2 The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/banner-saga-2/cid=UP0134-CUSA04444_00-THEBANNERSAGA2VE
The Name of Game is :Armello™ The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/armello/cid=UP1120-CUSA03300_00-00ARMELLOONESCEA
The Name of Game is :Gone Home: Console Edition The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/gone-home-console-edition/cid=UP1012-CUSA01228_00-GONEHOME00000000
The Name of Game is :Amplitude The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/amplitude/cid=UP8802-CUSA02480_00-HMXAMPLITUDE2015
The Name of Game is :Dangerous Golf™ The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/dangerous-golf/cid=UP1898-CUSA05385_00-TFEDANGEROUSGOLF
The Name of Game is :Pure Hold'em World Poker Championship The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/pure-hold'em-world-poker-championship/cid=UP2070-CUSA01104_00-UPUREPOKER000001
The Name of Game is :Hard Reset Redux The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/hard-reset-redux/cid=UP1050-CUSA04041_00-HARDRESET0000000
The Name of Game is :Lifeless Planet: Premier Edition The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/lifeless-planet-premier-edition/cid=UP0604-CUSA05475_00-LIFELESSPLANETPS
The Name of Game is :The Escapists: The Walking Dead The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/the-escapists-the-walking-dead/cid=UP4064-CUSA04182_00-THEESCAPISTSWD00
The Name of Game is :100ft Robot Golf The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/100ft-robot-golf/cid=UP0476-CUSA04678_00-100FTGAMEPS4SIEA
The Name of Game is :Kholat The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/kholat/cid=UP1561-CUSA04464_00-KHOLATGAME000000
The Name of Game is :Pure Chess® Complete Bundle The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/pure-chess-complete-bundle/cid=UP2070-CUSA00240_00-B000000000000337
The Name of Game is :Rogue Stormers The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/rogue-stormers/cid=UP4402-CUSA06052_00-ROGUESTORMERS000
The Name of Game is :SNOW Beta The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/snow-beta/cid=UP2862-CUSA06096_00-0000000000000001
The Name of Game is :Assault Suit Leynos The Price for Which is : $19.99The url for it is: https://store.playstation.com/#!/en-us/games/assault-suit-leynos/cid=UP4034-CUSA04727_00-ASLEYNOS00000000

希望这就是你所期待的。

python web抓取请求和beautifulsoup

2 个答案: