How to click on Load More button within Google Trends and print all the titles through Selenium and Python

时间:2018-06-18 11:42:23

标签: python selenium selenium-webdriver xpath lazy-loading

this time I would like to click a button in order to load more real-time searches. Here is the link of the website:

https://trends.google.com/trends/trendingsearches/realtime?geo=AR&category=all

The button is located at the end of the page and it has the following code:

<div class="feed-load-more-button" ng-if="ctrl.shouldShowLoadingMoreItemsSpinner()" ng-click="ctrl.loadMoreFeedItems()" role="button" tabindex="0" style="">
Load more
</div>

Since theres some AngularJS involved I cant figure out how to do it... Any tips/help?

Thank you guys, Joan

4 个答案:

答案 0 :(得分:3)

要点击LOAD MORE按钮加载更多实时搜索,然后打印它们,您可以使用以下解决方案:

  • 代码块:

    # -*- coding: UTF-8 -*-
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.common.exceptions import TimeoutException
    
    options = webdriver.ChromeOptions() 
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
    driver.get("https://trends.google.com/trends/trendingsearches/realtime?geo=AR&category=all")
    myLength = len(WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='title']"))))
    
    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        try:
            WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='feed-load-more-button'][@ng-click=\"ctrl.loadMoreFeedItems()\"]"))).click()
            WebDriverWait(driver, 20).until(lambda driver: len(driver.find_elements_by_xpath("//div[@class='title']")) > myLength)
            titles = driver.find_elements_by_xpath("//div[@class='title']")
            myLength = len(titles)
        except TimeoutException:
            break
    
    for title in titles:
        print(title.text)
    driver.quit()
    
  • 控制台输出:

    Nikola Kalinic • 2018 World Cup • Zlatko Dalic
    Vicentico • Valeria Bertuccelli • Florián Fernandez Capello
    Mauricio Macri • Juan José Aranguren • Argentina • Jorge Lanata
    Goalkeeper • 2018 World Cup • Mohamed El-Shenawy • Ahmed El-Shenawy
    Argentina • Manuel Belgrano • María Josefa Ezcurra • Mercedes Tomasa de San Martín y Escalada
    South Korea national football team • Russia • Zlatan Ibrahimovic
    Italian nationality law • Mar del Plata
    Hirving Lozano • 2018 World Cup • Earthquake
    Central Bank of Argentina • Luis Caputo • Federico Sturzenegger • Nicolás Dujovne
    Porsche Mission E • Electric vehicle • Car • Electric car
    Intelligence • Intelligence quotient • Research
    Call for bids • National Flag Memorial • Monument • Rosario • Pablo Javkin
    Journalist • Club Atlético Belgrano • Córdoba • Manuel Belgrano
    France • Spain • Immigration
    Natti Natasha • Zum Zum • Daddy Yankee • R.K.M & Ken-Y • Francisco José Arcángel Ramos
    Mercosur • Pacific Alliance • Pact • Paraguay
    World Cup • England national football team • Romelu Lukaku • Dries Mertens
    House arrest • Detention • Robbery
    Argentina • Emerging markets • MSCI • Market • Finance • Morgan Stanley
    Eurozone • Budget • France • Bruno Le Maire • Germany • Finance
    Intelligence • Intelligence quotient • Research
    Brazil national football team • Tite • Philippe Coutinho • Rostov Arena
    2018 World Cup • Aleksandar Kolarov • Guillermo Ochoa
    2018 World Cup • Spain
    Sloth • Giant ground sloth • Fossil • San Pedro
    Government • Inflation • Alfredo Cornejo • Mendoza • Ministry of Finance
    Corrientes • Cannabis • Tractor • Argentine Naval Prefecture
    Argentine Chamber of Deputies • Chubut Province • Peronism • Front for Victory
    Jorge Rial • Father's Day • Intrusos en el espectáculo
    Debt • Debt relief • Refinancing • Chaco Province
    Polygraph
    Malaria • Paraguay • Americas
    General Confederation of Labour • Unemployment • Trade union
    Earthquake • Japan
    Angela Merkel • Europe • Germany • Emmanuel Macron • European Union
    Colombia • Ivan Duque • Gustavo Petro
    Lynx • Wildcat • Cat
    Rafael Márquez • 2018 World Cup • Antonio Carbajal
    Superior Council for Private Enterprise • Nicaraguan Social Security Institute
    2018 World Cup • William Kvist • Jefferson Farfán
    La Plata • Controlled-access highway • Buenos Aires Province • Ensenada
    Leonardo Mayer • Kevin Anderson • ATP World Tour 500 series • Association of Tennis Professionals
    Cristiano Ronaldo • 2018 World Cup • Spain
    Casino • Mendoza • Hyatt
    Mauricio Macri • Jorge Lanata • Argentina • Alejandro Wiebe
    Drug
    Buenos Aires Province • Judiciary
    2018 World Cup • Lionel Messi • Russia • Diego Maradona • Mario Kempes
    Santiago • Natural environment • Air pollution
    Chimbas Department
    Chaco Province • Judiciary • Unemployment
    Hailey Rhode Baldwin • Justin Bieber
    2018 World Cup • Russia
    Mauricio Macri • Luis Caputo • Politician • Argentine Chamber of Deputies
    Trade war • Agriculture • Donald Trump • China
    2018 World Cup • Dennis te Kloese • Juan Carlos Osorio
    Aaron Ramsey • 2018 World Cup • Arsenal F.C. • Football player • Gareth Bale
    2018 World Cup • Aleksandar Kolarov
    Iñaki Urdangarin • Cristina Federica, Infanta of Spain • Spain • Luis Roldán
    Argentina • Electric car
    San Lorenzo de Almagro • Liga Nacional de Básquet • Corrientes
    Christina Aguilera • Jimmy Fallon • Liberation • New York City
    Intelligence quotient • Intelligence • Flynn effect • Research
    María Eugenia Vidal • Republican Proposal • Horacio Rodríguez Larreta
    Natural environment • Pollution • Plastic pollution
    Funding • International Finance Corporation • World Bank • Córdoba
    Face • Zygomatic bone • Rosario
    Cristóbal López • Administración Federal de Ingresos Públicos • Ricardo Echegaray
    Club Atlético Belgrano • Lucas Bernardi • Superliga Argentina de Fútbol
    Juan José Aranguren • Mauricio Macri • Energy • YPF
    Greater Buenos Aires • Motor coach
    Stranger Things • MTV Movie & TV Awards
    Peru • Lima • Earthquake • 2018 World Cup • Geophysics Institute of Peru
    Juan Carlos Osorio • 2018 World Cup
    2018 World Cup • Player • Russia • Lionel Messi
    Raffaella Carrà
    America's Got Talent • Janis Joplin • Howie Mandel
    Homicide • Detention
    La Plata • Julio Garro • Víctor Manuel Fernández • María Eugenia Vidal
    Mariano Arcioni • Chubut Province • Payment
    Angela Merkel • Horst Seehofer • Germany • Government • Human migration
    Israel • Benjamin Netanyahu • Syria • Iran • Ali Khamenei • Vladimir Putin
    Yerba mate • Cannabis • Lomas de Zamora • Detention • Mate
    Senate • Argentine Chamber of Deputies
    2010 FIFA World Cup • South Africa
    Handball • Chile • Argentina national football team
    2018 World Cup • Russia • Terrorism • Attack
    Abortion • Conscientious objector
    Rosario • Trade
    Natalie Weber • Mauro Zárate • Pampita
    Water • Cipolletti • Pressure • Cleaning
    Chimbas
    NATO • Military exercise • Russia
    Federico Sturzenegger • Economist • Argentina • Arnaldo Bocco • Martín Redrado
    Nolle prosequi
    2018 World Cup • Russia • Paolo Guerrero • Christian Cueva
    Ricardo Darín • Valeria Bertuccelli • Érica Rivas
    Joachim Löw • 2018 World Cup
    Israel • Golan Heights • Syria • Donald Trump
    Meningitis • Bacteria • Salta • Streptococcus pneumoniae
    ANSES • Subsidy
    Argentine Chamber of Deputies • Radical Civic Union • Cambiemos •
    Martín Demichelis • 2018 World Cup • Russia
    Santiago de Compostela • Pilgrim • Galicia • Spain
    Refugee • Pope Francis • Human migration
    Stranger Things • Father's Day • Joe Keery
    Ivan Duque • President of Colombia • Latin America
    General Confederation of Labour • Unemployment • General strike • Buenos Aires Province
    Juan Carlos Osorio • Faustino Asprilla • 2018 World Cup
    Germán Burgos • Club Atlético River Plate • Diego Simeone • Atlético Madrid
    Nissan Navara • Pickup truck • Automotive industry
    Baobab • Research • Tree
    Susana Giménez • Alejandro Wiebe • Argentina • Telefe
    Wanda Nara • Maxi López • Mauro Icardi
    Damir Skomina • 2018 World Cup • Colombia • Referee • Russia • Mehdi Abid Charef
    Diabetes mellitus • Visual perception • Diabetic retinopathy
    Berisso • Threat • La Plata • Search and seizure • School
    Game of Thrones • HBO • San Diego Comic-Con • Spin-off • George R. R. Martin
    Argentina • Duet • Traveling Wilburys
    Blood donation • Uruguay • Maldonado
    2018 World Cup Group F • Mexico • Hirving Lozano
    Colombian presidential election, 2018 • Colombia • Juan Manuel Santos
    Colombia national football team
    Desertification • United Nations Convention to Combat Desertification
    Saski Baskonia • Liga ACB • Pablo Laso • Real Madrid C.F.
    Rosario de la Frontera • Salta • Spa town
    Neymar • 2018 World Cup • Philippe Coutinho • Tite • Russia
    Pocito Department
    Argentine peso • Depreciation • Central Bank of Argentina
    San Salvador de Jujuy • Buenos Aires International Book Fair • Fair •
    Córdoba • Shock
    Iceland • Immigration • Icelanders
    Mirtha Legrand
    Traffic collision • Wound
    2018 World Cup • Russia • Vikings
    Mauricio Macri • Businessperson • Argentina • Economic development
    Ricardo Darín • Valeria Bertuccelli • Érica Rivas • Vicentico
    María Eugenia Vidal • Martiniano Molina • Quilmes
    Argentina women's national field hockey team • Julieta Jankunas • Argentina
    Prince Harry • Catherine, Duchess of Cambridge • British royal family
    Sergej Milinkovic-Savic • SS Lazio • 2018 World Cup
    Organ donation • Organ • Organ transplantation
    Sex education • San Fernando del Valle de Catamarca • Argentine Chamber of Deputies
    Margarita Stolbizer • Sergio Massa • Peronism • Justicialist Party • Elisa Carrió
    Google Maps • Waze • Information • Radar
    Locomotive • Japan
    Spain • Human migration • Immigration • France • Valencia • Carmen Calvo Poyato
    Joaquín Sabina • Madrid
    Jorge Lorenzo • Marc Márquez • MotoGP • Dani Pedrosa
    Manuel Belgrano
    2018 World Cup • Russia • Mohamed Salah • Denis Cheryshev
    María Eugenia Vidal • Teacher
    Remand
    Magistrate • Judiciary • Research
    Laura Bush • Donald Trump • Immigration • George W. Bush • Melania Trump
    Sebastián Piñera • Chile • Michelle Bachelet
    School • Gender identity • Discrimination • National Institute Against Discrimination, Xenophobia and Racism
    Engineering • San Miguel de Tucumán
    School • Buenos Aires Province • Tariff
    Edgardo Bauza • Rosario Central • Marco Ruben • Rosario
    Bicameralism
    Russia • 2018 World Cup • Lionel Messi
    Rawson Department, San Juan
    El Litoral • Corrientes Province
    Abortion • PH: Podemos hablar • Andy Kusnetzoff • Charlotte Caniggia
    The Shining • Ewan McGregor • Doctor Sleep • Danny Torrance • Stephen King
    Paulo Ferrari • Rosario Central • Rosario • Superliga Argentina de Fútbol
    Carbon monoxide
    Ground frost • Fog • Rain and snow mixed • Cold • Posadas
    Argentine rock • Russia • Luis Alberto Spinetta • Gustavo Cerati • Charly García
    Incredibles 2 • Brad Bird • Pixar
    Carbohydrate • Dieting • Weight loss
    Uber • Mendoza Province • Government • System • Statute
    Paragliding
    Jorge Sampaoli • Pedro Pasculli • 2018 World Cup • Paulo Dybala • Russia
    Posadas • Cold
    Luis Miguel • Mexico
    Unidentified flying object • Russia • Phenomenon
    Lisandro Magallán • Boca Juniors • AFC Ajax • Wílmar Barrios
    Tandil • Fossil • Glyptodon
    Harry Kane • 2018 World Cup • Gareth Southgate
    Light welterweight • Boxing • Almirante Brown Partido • Mariano Cascallares
    New Jersey
    Small and medium-sized enterprises • Argentina • CAME - Argentina Confederation of Medium Enterprises
    Season • Ryan Murphy • American Horror Story: Murder House • Sarah Paulson
    Conflagration • Posadas
    Robot • Old age • China
    2018 World Cup • Russia • Terrorism • Islamic State of Iraq and the Levant
    Christian Cueva • 2018 World Cup
    Argentina national football team • Volleyball • Argentina
    National Electoral Institute • Candidate • National Action Party • Mexico
    Horoscope • Astrological sign
    Mauricio Macri • María Eugenia Vidal • Ensenada • Ambulance • Cambiemos
    Trade • Tax deduction • Debt • Macroeconomics
    Cristiano Ronaldo • Pelé • 2018 World Cup • Miroslav Klose • Uwe Seeler
    Martín Miguel de Güemes • Juan Manuel Urtubey • Salta Province • Argentina
    Compressed natural gas • La Pampa Province • General Pico • Camuzzi Gas Pampeana
    Santa Fe • Provincial Hospital Dr. José María Cullen • Baleada
    National University of La Plata • Vocational school • School • Vocational education
    

答案 1 :(得分:2)

向下滚动

可以使用以下代码:

driver.execute_script("window.scrollTo(0, Y)")  

其中Y:是您要滚动到的垂直位置。(根据您的要求,您可以将Y的值提供为100,200,300。)

完成向下滚动后,您可以使用以下代码:

load_more_button = WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH, "//div[@ng-click='ctrl.loadMoreFeedItems()']")))  
load_more_button.click()

答案 2 :(得分:2)

使用find_elements_by_css_selector找到它并单击它。我不认为你是否需要等待或滚动。

element = driver.find_elements_by_css_selector("div[ng-click=\"ctrl.loadMoreFeedItems()\"]")
element.click()

如果您想知道为什么我没有使用xPath

,请阅读此answer

答案 3 :(得分:1)

您可以使用:

wait = WebDriverWait(webdriver, 10)
load_button = wait.until(EC.presence_of_element_located((By.XPATH, "//div[@class = 'feed-load-more-button']")))
load_button.click()