Question

我正在尝试抓取此页面"https://www.seloger.com/list_beta.htm?tri=initial&enterprise=0&idtypebien=2,1&idtt=2,5&naturebien=1,2,4&cp=75"

但是当我在搜索的第一个元素上使用.click()时，该页面已正确加载到浏览器中，但是我没有得到body及其所有子元素driver.find_element方法，而获取新加载页面的网址，让我找到它没有问题。 driver.current_url为我提供了第一页，就像.click()没有加载任何内容，而呈现器已成功加载到浏览器中一样。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.expected_conditions import visibility_of_element_located
from selenium.webdriver.support.ui import WebDriverWait
import pyautogui
import time


def cssconvert(tag):
    return '.'+tag.replace(' ', '.')


binary = r'C:\Program Files\Mozilla Firefox\firefox.exe'
options = Options()
options.set_headless(headless=False)
options.binary = binary
cap = DesiredCapabilities().FIREFOX
cap["marionette"] = True #optional
driver = webdriver.Firefox(firefox_options=options, capabilities=cap, executable_path="C:\\Users\\chrys\\Desktop\\DataScientist\\Selenium\\geckodriver_true\\geckodriver.exe")
driver.get("https://www.seloger.com/list_beta.htm?tri=initial&enterprise=0&idtypebien=2,1&idtt=2,5&naturebien=1,2,4&cp=75")

#time.sleep(2)
select = 'block__ShadowedBlock-sc-10w6hsj-0 ListContent__SmartClassifiedExtended-sc-1viyr2k-2 iddbNe classified__ClassifiedContainer-sc-1wmlctl-0 haLWMI Card__CardContainer-sc-7insep-7 jZkbME'
driver.find_element_by_css_selector(cssconvert(select)).click()
driver.find_element_by_id('js-descriptifBien')

    241             raise exception_class(message, screen, stacktrace, alert_text)
--> 242         raise exception_class(message, screen, stacktrace)
    243 
    244     def _value_or_default(self, obj, key, default):

NoSuchElementException: Message: Unable to locate element: [id="js-descriptifBien"]

如果现在我复制了网址：

driver.get('https://www.seloger.com/annonces/viagers/appartement/paris-11eme-75/belleville-saint-maur/145504325.htm?projects=2,5&types=2,1&natures=1,2,4&places=[{cp:75}]&qsVersion=1.0&bd=ListToDetail')
driver.find_element_by_id('js-descriptifBien').text

正在工作。因为我的目的是抓取结果研究的所有要素，所以我会知道如何处理。

Answer 1

您可以了解有关选择器here和here的信息。

问题是当您单击列表项时，它在新选项卡中将其打开。您必须先进行切换，然后再执行任何操作。下面是一些如何在窗口之间导航的示例：

current_window = driver.current_window_handle
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".iddbNe"))).click()
wait.until(EC.new_window_is_opened)
driver.switch_to.window(driver.window_handles[1])

使用Selenium，解决方案可以从列表中收集所有链接，然后导航到它们。为此，您可以使用.iddbNe a[name=classified-link] CSS选择器。从所有必需的页面收集所有链接后，您可以导航到它们并收集数据。
这是第一页的示例示例：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# ...

wait = WebDriverWait(driver, 20)

# you can put loop to go throw all pages you need and add to the list
links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".iddbNe a[name=classified-link]")))
for link in links:
    driver.get(link)
    # get you data

更好的解决方案是使用requests。下面的代码是一个简单的示例，您如何以json格式获取100个结果。要进行重新搜索，您可以找到结果总数，并将其用于循环并收集所需的所有信息：

import requests

headers = {
    'sec-fetch-mode': 'cors',
    'origin': 'https://www.seloger.com',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'ru,en-US;q=0.9,en;q=0.8,tr;q=0.7',
    'pragma': 'no-cache',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) '
                  'Chrome/77.0.3865.90 Safari/537.36',
    'content-type': 'application/json',
    'accept': 'application/json',
    'cache-control': 'no-cache',
    'authority': 'www.seloger.com',
    'referer': 'https://www.seloger.com/list_beta.htm?projects=2%2C5&types=2%2C1&natures=1%2C2%2C4&places=%5B%7Bcp'
               '%3A75%7D%5D&qsVersion=1.0&LISTING-LISTpg=2',
    'sec-fetch-site': 'same-origin',
    'dnt': '1',
}

params = (
    ('from', '0'),
    ('size', '100'),
    ('isSeo', 'false'),
)

data = '{"idPublication":null,"projects":[2,5],"types":[2,1],"natures":[1,2,4],"places":[{"label":"Paris",' \
       '"cities":null,"districts":null,"countries":null,"divisions":null,"subDivisions":["75"],"fakeCities":null}],' \
       '"searchAreas":null,"isochronePoints":null,"proximities":null,"withGeoloc":null,"price":null,' \
       '"groundSurface":null,"surface":null,"bedrooms":[],"rooms":[],"bedroom":null,"room":null,"sort":null,' \
       '"floor":null,"lastFloor":null,"hearth":null,"guardian":null,"view":null,"balcony":null,"pool":null,' \
       '"lift":null,"terrace":null,"cellar":null,"south":null,"parking":null,"box":null,"parquet":null,"locker":null,' \
       '"furnished":null,"disabledAccess":null,"alarm":null,"toilet":null,"bathtub":null,"shower":null,"hall":null,' \
       '"livingRoom":null,"diningRoom":null,"kitchen":null,"heating":null,"unobscured":null,"picture":null,' \
       '"exclusiveness":null,"priceChange":null,"privateSeller":null,"video":null,"vv":null,"enterprise":null,' \
       '"garden":null,"basement":null,"groundFloor":null,"houseboat":null} '

response = requests.post('https://www.seloger.com/list/api/externaldata', headers=headers, params=params, data=data)
print(response.text)

即使在浏览器中显示页面，也不会在.click（）之后加载页面

1 个答案: