Question

我正在使用硒来搜索agoda并在页面中废弃所有酒店名称，但是输出仅返回2个名称。

然后我尝试添加一行以滚动到底部，现在输出显示了我的前2个名字和后2个名字（开头是前两个，底部是后两个）

我不明白这是什么问题，我为每个步骤添加了time.sleep（），因此整个页面应该已经完全加载。硒是否只能按页面查看限制它只能报废那些可见的元素？

我的下面的代码：

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

time.sleep(30)

def scrap():
    r = requests.get(current_page)

if r.status_code == requests.codes.ok:
    print('start scraping!')
    hotel = driver.find_elements_by_class_name('hotel-name')

    hotels = []

    for h in hotel:
        if hotel:
            hotels.append(h.text)

    print(hotels, file=open("output.txt", 'a', encoding="utf-8"))

scrap()

Here is the page i want to scrap

Answer 1

尝试使用以下脚本向下滚动页面，直到页面上不再显示结果，然后抓取所有可用名称：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Chrome()
driver.maximize_window()

driver.get('https://www.agoda.com/pages/agoda/default/DestinationSearchResult.aspx?asq=8wUBc629jr0%2B3O%2BxycijdcaVIGtokeWrEO7ShJumN8xsNvkFkEV9bUgNnbx6%2Bx22ncbzTLOPBjT84OgAAKXmu6quf8aEKRA%2FQH%2BGoyXgowLt%2BXyB8OpN1h2WP%2BnBM%2FwNPzD%2BpaeII93w%2Bs4dMWI4QPJNbZJ8DWvRiPsrPVVBJY7ilpMPlUermwV1UKIKfuyeis3BqRkJh9FzJOs0E98zXQ%3D%3D&city=9590&cid=-142&tick=636818018163&languageId=20&userId=3c2c4cb9-ba6d-4519-8ef4-c85dfd280b8f&sessionId=d4qzq2tgymjrwsf22lnadxpc&pageTypeId=1&origin=HK&locale=zh-TW&aid=130589&currencyCode=HKD&htmlLanguage=zh-tw&cultureInfoName=zh-TW&ckuid=3c2c4cb9-ba6d-4519-8ef4-c85dfd280b8f&prid=0&checkIn=2019-01-16&checkOut=2019-01-17&rooms=1&adults=2&children=0&priceCur=HKD&los=1&textToSearch=%E5%A4%A7%E9%98%AA&productType=-1&travellerType=1')
# Get initial list of names
hotels = wait(driver, 15).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'hotel-name')))

while True:
    # Scroll down to last name in list
    driver.execute_script('arguments[0].scrollIntoView();', hotels[-1])
    try:
        # Wait for more names to be loaded
        wait(driver, 15).until(lambda driver: len(wait(driver, 15).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'hotel-name')))) > len(hotels))
        # Update names list 
        hotels = wait(driver, 15).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'hotel-name')))
    except:
        # Break the loop in case no new names loaded after page scrolled down
        break

# Print names list
print([hotel.text for hotel in hotels])

硒不能获取页面的所有元素

1 个答案: