我已阅读Wait Until Page is Loaded,How to use Selenium Wait,Explicit Wait和其他文档,以等待页面加载然后开始抓取。等待成功通过,但是我仍然得到相同的一半/不完整的呈现HTML代码。
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
# prepare the option for the chrome driver
options = webdriver.ChromeOptions()
options.add_argument('headless')
# start chrome browser
browser = webdriver.Chrome(options=options,executable_path='C:/chromedriver_win32/chromedriver.exe')
browser.get('https://swappa.com/listing/view/LTNZ94446')
try:
WebDriverWait(browser, 30).until(EC.presence_of_element_located((By.ID, "wrap")))
print(browser.page_source)
except TimeoutException:
print("not found")
为此,我的输出从一半开始而不是从顶部的<html>
开始。
<div class="col-xs-6 col-sm-2 col-md-2">
<div class="img-container" style="margin-bottom: 15px;">
<a href="https://static.swappa.com/media/listing/LTNZ94446/mhhHypyw.jpg" class="lightbox">
<img class="img-responsive" src="https://static.swappa.com/images/cache/7b/67/7b679a1d89816bc341a802f19f661eac.jpg" alt="Listing Image" style="margin:0px 0px 0px 0px; ">
</a>
</div>
</div>
我不确定哪里出了问题。
<div id="wrap">
),因为它不会引发超时错误如果有其他方法可以使用其他库(例如BeautifulSoup / URLLib / URLlib2 / Scrapy),那么这些方法也会有所帮助
答案 0 :(得分:1)
您可以使用JavaScript检查页面是否已完全加载:
options = webdriver.ChromeOptions()
options.add_argument('headless')
# start chrome browser
browser = webdriver.Chrome(options=options)
browser.get('https://swappa.com/listing/view/LTNZ94446')
WebDriverWait(browser, 30).until(lambda d: d.execute_script(
'return ["complete", "interactive"].indexOf(document.readyState) != -1'))
# or use only complete
# WebDriverWait(browser, 30).until(lambda d: d.execute_script('return document.readyState == "complete"'))
print(browser.page_source)
答案 1 :(得分:0)
您可以使用python requests模块。
代码:
import requests
response=requests.get("https://swappa.com/listing/view/LTNZ94446")
if response.status_code==200:
print(response.text)