按照我之前的question，我现在正试图抓取多个网页（所有页面都包含特定季节的游戏）。我也试图刮掉多个父母网址（季节）：
from selenium import webdriver
import pandas as pd
import time
url = ['http://www.oddsportal.com/hockey/austria/ebel-2014-2015/results/#/page/',
'http://www.oddsportal.com/hockey/austria/ebel-2013-2014/results/#/page/']
data = []
for i in url:
for j in range(1,8):
print i+str(j)
driver = webdriver.PhantomJS()
driver.implicitly_wait(10)
driver.get(i+str(j))
for match in driver.find_elements_by_css_selector("div#tournamentTable tr.deactivate"):
home, away = match.find_element_by_class_name("table-participant").text.split(" - ")
date = match.find_element_by_xpath(".//preceding::th[contains(@class, 'first2')][1]").text
if " - " in date:
date, event = date.split(" - ")
else:
event = "Not specified"
data.append({
"home": home.strip(),
"away": away.strip(),
"date": date.strip(),
"event": event.strip()
})
driver.close()
time.sleep(3)
print str(j)+" was ok"
df = pd.DataFrame(data)
print df
# ok for six results then socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host
# ok for two results, then infinite load
# added time.sleep(3)
# ok for first result, infinite load after that
# added implicitly wait
# no result, infinite load
首先，我尝试了两次代码，没有第14行的隐式等待或35的休眠。第一个结果给出了套接字错误。第二个结果在两个好的刮页之后没有错误就停止了。
然后添加了如上所述等待的时间，他们没有帮助。
由于结果不一致，我的猜测是在循环结束时重置连接。下一次运行。我想知道这是否是一个可能的解决方案以及如何实施。我检查了网站的robots.txt，看不到任何阻止设置间隔后刮擦的内容。
其次，说刮刀获得90％的页面，然后停止（无限等待）。有没有办法让它在x秒后重试该循环，以便保存你所拥有的并再次从停滞点重试？
答案 0 :(得分：7)
您需要做的是：
webdriver实例 - 不要在循环中初始化它实现：
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
urls = [
'http://www.oddsportal.com/hockey/austria/ebel-2014-2015/results/#/page/',
'http://www.oddsportal.com/hockey/austria/ebel-2013-2014/results/#/page/'
]
data = []
driver = webdriver.PhantomJS()
driver.implicitly_wait(10)
wait = WebDriverWait(driver, 10)
for url in urls:
for page in range(1, 8):
driver.get(url + str(page))
# wait for the page to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div#tournamentTable tr.deactivate")))
for match in driver.find_elements_by_css_selector("div#tournamentTable tr.deactivate"):
home, away = match.find_element_by_class_name("table-participant").text.split(" - ")
date = match.find_element_by_xpath(".//preceding::th[contains(@class, 'first2')][1]").text
if " - " in date:
date, event = date.split(" - ")
else:
event = "Not specified"
data.append({
"home": home.strip(),
"away": away.strip(),
"date": date.strip(),
"event": event.strip()
})
driver.close()
df = pd.DataFrame(data)
print(df)
打印：
away date event home
0 Salzburg 14 Apr 2015 Play Offs Vienna Capitals
1 Vienna Capitals 12 Apr 2015 Play Offs Salzburg
2 Salzburg 10 Apr 2015 Play Offs Vienna Capitals
3 Vienna Capitals 07 Apr 2015 Play Offs Salzburg
4 Vienna Capitals 31 Mar 2015 Play Offs Liwest Linz
5 Salzburg 29 Mar 2015 Play Offs Klagenfurt
6 Liwest Linz 29 Mar 2015 Play Offs Vienna Capitals
7 Klagenfurt 26 Mar 2015 Play Offs Salzburg
8 Vienna Capitals 26 Mar 2015 Play Offs Liwest Linz
9 Liwest Linz 24 Mar 2015 Play Offs Vienna Capitals
10 Salzburg 24 Mar 2015 Play Offs Klagenfurt
11 Klagenfurt 22 Mar 2015 Play Offs Salzburg
12 Vienna Capitals 22 Mar 2015 Play Offs Liwest Linz
13 Bolzano 20 Mar 2015 Play Offs Liwest Linz
14 Fehervar AV19 18 Mar 2015 Play Offs Vienna Capitals
15 Liwest Linz 17 Mar 2015 Play Offs Bolzano
16 Vienna Capitals 16 Mar 2015 Play Offs Fehervar AV19
17 Villach 15 Mar 2015 Play Offs Salzburg
18 Klagenfurt 15 Mar 2015 Play Offs Znojmo
19 Bolzano 15 Mar 2015 Play Offs Liwest Linz
20 Liwest Linz 13 Mar 2015 Play Offs Bolzano
21 Fehervar AV19 13 Mar 2015 Play Offs Vienna Capitals
22 Znojmo 13 Mar 2015 Play Offs Klagenfurt
23 Salzburg 13 Mar 2015 Play Offs Villach
24 Klagenfurt 10 Mar 2015 Play Offs Znojmo
25 Vienna Capitals 10 Mar 2015 Play Offs Fehervar AV19
26 Bolzano 10 Mar 2015 Play Offs Liwest Linz
27 Villach 10 Mar 2015 Play Offs Salzburg
28 Liwest Linz 08 Mar 2015 Play Offs Bolzano
29 Znojmo 08 Mar 2015 Play Offs Klagenfurt
.. ... ... ... ...
670 TWK Innsbruck 28 Sep 2013 Not specified Znojmo
671 Liwest Linz 27 Sep 2013 Not specified Dornbirn
672 Bolzano 27 Sep 2013 Not specified Graz 99ers
673 Klagenfurt 27 Sep 2013 Not specified Olimpija Ljubljana
674 Fehervar AV19 27 Sep 2013 Not specified Salzburg
675 TWK Innsbruck 27 Sep 2013 Not specified Vienna Capitals
676 Villach 27 Sep 2013 Not specified Znojmo
677 Salzburg 24 Sep 2013 Not specified Olimpija Ljubljana
678 Dornbirn 22 Sep 2013 Not specified TWK Innsbruck
679 Graz 99ers 22 Sep 2013 Not specified Klagenfurt
680 Vienna Capitals 22 Sep 2013 Not specified Villach
681 Fehervar AV19 21 Sep 2013 Not specified Bolzano
682 Dornbirn 20 Sep 2013 Not specified Bolzano
683 Villach 20 Sep 2013 Not specified Graz 99ers
684 Znojmo 20 Sep 2013 Not specified Klagenfurt
685 Olimpija Ljubljana 20 Sep 2013 Not specified Liwest Linz
686 Fehervar AV19 20 Sep 2013 Not specified TWK Innsbruck
687 Salzburg 20 Sep 2013 Not specified Vienna Capitals
688 Villach 15 Sep 2013 Not specified Klagenfurt
689 Liwest Linz 15 Sep 2013 Not specified Dornbirn
690 Vienna Capitals 15 Sep 2013 Not specified Fehervar AV19
691 TWK Innsbruck 15 Sep 2013 Not specified Salzburg
692 Graz 99ers 15 Sep 2013 Not specified Znojmo
693 Olimpija Ljubljana 14 Sep 2013 Not specified Dornbirn
694 Bolzano 14 Sep 2013 Not specified Fehervar AV19
695 Klagenfurt 13 Sep 2013 Not specified Graz 99ers
696 Znojmo 13 Sep 2013 Not specified Salzburg
697 Olimpija Ljubljana 13 Sep 2013 Not specified TWK Innsbruck
698 Bolzano 13 Sep 2013 Not specified Vienna Capitals
699 Liwest Linz 13 Sep 2013 Not specified Villach
[700 rows x 4 columns]