Question

我想从以下查询中加载所有结果：

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import time

page_link = 'https://www.tui.co.uk/destinations/packages?airports%5B%5D=&units%5B%5D=BGR%3ACOUNTRY&when=06-09-2019&until=&flexibility=true&flexibleDays=3&noOfAdults=2&noOfChildren=0&childrenAge=&duration=7114&searchRequestType=ins&searchType=search&sp=true&multiSelect=true&room=&isVilla=false'
options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
browser = webdriver.Chrome(chrome_options=options, executable_path=r'C:\temp\chromedriver.exe')
browser.get(page_link)

SCROLL_PAUSE_TIME = 0.5

# Get scroll height
last_height = browser.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = browser.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

soup = BeautifulSoup(browser.page_source, 'html.parser')
soup.findAll(class_='price-value')

但是，以上内容转到页面底部，而未加载新结果。我被卡住了。

谢谢

Answer 1

您可以检查直到找到具有xpath //p[@class="no-more-holidays"]的元素并且不能使用window.scrollTo()，因为它不会加载更多内容

browser.get(page_link)

SCROLL_PAUSE_TIME = 3 # slow down, faster has no effect

while True:
    browser.execute_script('''
        var loadingDiv = document.querySelector('div[class="results-pagination type-next"]');
        if(loadingDiv)
            loadingDiv.scrollIntoView();
    ''')
    time.sleep(SCROLL_PAUSE_TIME)
    no_more = browser.find_elements_by_xpath('//p[@class="no-more-holidays"]')
    if len(no_more) == 1:
        break

向下滚动页面，不显示任何按钮即可显示所有结果

1 个答案: