Question

我正在尝试从右键单击并选择“查看页面源”时找到的网站上抓取代码。我的下面的代码从右键单击然后选择我认为的“检查”时找到的输出中抓取。我收到一个错误消息，提示“文件加载时使用了错误的编码：'UTF-8'，我正在基于原始页面源信息进行数据挖掘，但我不知道如何将其提取。

见下文

from bs4 import BeautifulSoup
import requests
import urllib.request
import urllib.error
import os, os.path, csv
import sys
from lxml import html
import requests

sys.stdout = open('scrapingoutput', 'a')
print(sys.stdout)


url= "https://www.geodatadirect.com/SearchResults/SuffolkSearchResults.aspx?state=NY&id=Suffolk&type=Sales"

urllib.request.urlopen("https://www.geodatadirect.com/SearchResults/SuffolkSearchResults.aspx?state=NY&id=Suffolk&type=Sales").read()

content = urllib.request.urlopen(url).read()

soup = BeautifulSoup(content)

print(soup.prettify())

Answer 1

import requests
import bs4
url = "https://www.geodatadirect.com/SearchResults/SuffolkSearchResults.aspx?state=NY&id=Suffolk&type=Sales"

page_content = requests.get(url).content
soup = bs4.BeautifulSoup(page_content, 'html.parser')
print(soup.prettify())

此打印

<!DOCTYPE html>
<html>
 <head>
  <title>
   Nationwide Property Data, Reports, Sales Comps
  </title>
  <meta charset="utf-8"/>
...

Answer 2

尝试selenium library到下载网页。硒库也有助于下载动态数据内容。

对于Chrome浏览器：

http://chromedriver.chromium.org/downloads

为Chrome浏览器安装Web驱动程序：

unzip ~/Downloads/chromedriver_linux64.zip -d ~/Downloads
chmod +x ~/Downloads/chromedriver
sudo mv -f ~/Downloads/chromedriver /usr/local/share/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/local/bin/chromedriver
sudo ln -s /usr/local/share/chromedriver /usr/bin/chromedriver

硒教程

https://selenium-python.readthedocs.io/

将代码替换为此。

from selenium import webdriver
from bs4 import BeautifulSoup
import time

driver = webdriver.Chrome('/usr/bin/chromedriver')
driver.get('https://www.geodatadirect.com/SearchResults/SuffolkSearchResults.aspx?state=NY&id=Suffolk&type=Sales')
time.sleep(3)

soup = BeautifulSoup(driver.page_source,'html.parser')

print(soup.prettify())

O / P：

<html>
 <head>
  <title>
   Nationwide Property Data, Reports, Sales Comps
  </title>
  <meta charset="utf-8"/>
  <meta content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no" name="viewport"/>
  <meta content="--XLC17oYuQE6UEhT-9_rC13L639t4C53w40_nWSbDM" name="google-site-verification"/>
  <meta content="GeoData Plus provides nationwide property reports, sales comparables, foreclosures, mortgages. Property data for residential and commercial real estate." name="description"/>
  <meta content="GeoData Plus" property="og:title"/>
  <meta content="https://www.geodataplus.com" property="og:url"/>
  <link href="https://www.geodataplus.com" rel="canonical"/>
  <meta content="website" property="og:type"/>
  <link href="/favicon.ico?v=Kx5JMIU84bo6i-lOxVlIH29IO5Qc9QPT6ENpVMaN-JE" rel="shortcut icon"/>
  <link href="/css/master.css?v=Liu7xdmA3BH167YXbnG76LfxA58TPHQR1J4L4ZzM5Qk" rel="stylesheet"/>
  <link href="/fonts/stylesheet.css?v=3NqqVyD10iq4848EK3FrA0HOaygo2MyDfL49n8ftRB0" rel="stylesheet"/>
  <link href="/css/Jquery-ui-auto.css?v=Nul8_ltyyt4O0iNe5la8BhlJ-Z84SOdeInfup2plryA" media="all" onload="if(media!='all')media='all'" rel="stylesheet"/>
  <noscript>
   <link href="/css/Jquery-ui-auto.css?v=Nul8_ltyyt4O0iNe5la8BhlJ-Z84SOdeInfup2plryA" rel="stylesheet"/>
  </noscript>
  <link href="theme/default/style.css" rel="stylesheet" type="text/css"/>
 </head>
 <body data-offset="200" data-spy="scroll" data-target=".navbar">
  <div class="" id="mainDiv">
   <div class="load-complete" id="site-loader">
      .........
      ..........
    </div>
  </div>
</body>
</html>

'/usr/bin/chromedriver' Chrome驱动程序路径在哪里。

查看页面源代码时发现的抓取代码

2 个答案: