如何从像Bloomberg这样的安全网站中提取数据

时间:2019-06-11 06:18:02

标签: python selenium scrapy bloomberg scrapy-splash

我正在尝试在此网址上抓取项目:

"https://www.bloomberg.com/news/articles/2019-05-30/tesla-dealt-another-blow-as-barclays-sees-it-as-niche-carmaker"

我只想让标题和日期发布, 您甚至可以给我的任何示例代码,等等...

到目前为止,我尝试过的是这个

 def parse(self, response):
   yield scrapy.Request('https://www.bloomberg.com/news/articles/2019-05-30/tesla-dealt-another-blow-as-barclays-sees-it-as-niche-carmaker -H x-crawlera-use-https:1',
        headers={'X-Crawlera-Session': create,
        'X-Crawlera-Timeout': 40000,
        'X-Crawlera-Max-Retries': 5,
        'X-Crawlera-Cookies': disable,
        'X-Crawlera-Session': self.session_id
        },
        callback=self.parse_sub,
        )

 def parse_sub(self, response):
    response.xpath("//h1[@class = 'lede-text-v2__hed']").extract_first()
    response.xpath("//meta[@property = 'og:title']/@content").extract_first()
    response.xpath("//time[@class = 'article-timestamp']/@datetime").extract_first()
   print(response.text)

我也在使用crawlera,但它不断将我视为机器人

1 个答案:

答案 0 :(得分:1)

仅使用来提取标题,即当巴克莱称其为“利基汽车制造商”时,特斯拉再次遭受打击,并且发布日期为< / em>即 2019年5月30日,格林尼治标准时间+5:30 ,您必须为visibility_of_element_located()诱导 WebDriverWait ,并且可以使用以下命令解决方案:

  • 代码块

    from selenium import webdriver
    
    driver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
    driver.get('https://www.bloomberg.com/news/articles/2019-05-30/tesla-dealt-another-blow-as-barclays-sees-it-as-niche-carmaker')
    print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='markets']//following:: h1[1]"))).get_attribute("innerHTML"))
    print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='markets']//following:: h1[1]//following::div[@class='lede-text-v2__times']/time[@itemprop='datePublished']"))).get_attribute("innerHTML"))
    driver.quit() 
    
  • 控制台输出:

    Tesla Dealt Another Blow When Barclays Calls It a ‘Niche Carmaker’
    May 30, 2019, 5:26 PM GMT+5:30
    
  • 注意:您必须添加以下导入:

    from selenium import webdriver
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC