Question

from selenium import webdriver
driver= webdriver.Chrome()
driver.get('https://www.realestate.com.au/sold/') #opens normally but anything I search for doesn't load
driver.get('https://www.realestate.com.au/sold/in-alice+springs+-+greater+region%2c+nt%3b/list-1')

网站打开，我没有任何验证码或阻止，但页面未加载

Answer 1

似乎有一些“ Flash”问题（您可以在“ chromedriver”窗口中读取错误：

Browser not compatible with given flash manifest

该网站使用Flash，而您的浏览器需要Flash Player。说您的浏览器是指硒浏览器，而不是您手动打开的Chrome。

要在Selenium chrome浏览器中安装“ Flash Player”，请运行脚本；它将打开浏览器，然后转到https://get.adobe.com/flashplayer/并安装Flash Player。

这应该可以解决问题

Answer 2

我想他们正在阻止硒的使用，请参阅相关的请求返回429。也许您可以找到一种方法来逃避硒的使用。

Answer 3

更多有关您的用例的信息将帮助我们以更好的方式调试问题。但是，使用Selenium可以访问网站并按以下方式检索页面源：

代码块：

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.realestate.com.au/sold/")
print(driver.page_source)

控制台输出：

<html class="js rui-burger rui-fancy-selects-on is-desktop app-loaded" lang="en"><head>
    <meta http-equiv="x-ua-compatible" content="IE=edge">
    <meta charset="utf-8">
    <meta name="viewport" content="width=1140">
    <meta name="description" content="Discover how much the latest properties have been sold for. Browse sold real estate listings to find auction results and sold prices from all over Australia.">
    .
    .
    .
<script type="text/javascript" async="" charset="utf-8" id="utag_rea-group.sp-only_27" src="//tags.tiqcdn.com/utag/rea-group/sp-only/prod/utag.27.js?utv=ut4.44.201710110106"></script><script type="text/javascript" async="" charset="utf-8" id="utag_rea-group.sp-only_64" src="//tags.tiqcdn.com/utag/rea-group/sp-only/prod/utag.64.js?utv=ut4.44.202005070432"></script><script type="text/javascript" src="https://odb.outbrain.com/utils/get?url=http%3A%2F%2Fwww.realestate.com.au%2Fsold&amp;settings=true&amp;recs=true&amp;widgetJSId=HPR&amp;key=NANOWDGT01&amp;idx=0&amp;version=20009&amp;apv=false&amp;sig=kLvLc8wa&amp;format=html&amp;va=true&amp;et=true&amp;rand=98241&amp;pdobuid=-1&amp;osLang=en-US&amp;winW=1366&amp;winH=657&amp;scrW=1366&amp;scrH=768&amp;adblck=false&amp;secured=true&amp;cmpStat=0&amp;ccpaStat=0&amp;ref=&amp;px=119&amp;py=1959&amp;vpd=1302" charset="UTF-8" async=""></script><script src="https://safeinator.reamedia.com.au/2073b42b71554175cf729737a838c1f1c6d7f862/lib/base.min.js" async="" type="text/javascript"></script></head>
  <!--[if IE 8]>         <body class="sold rui-preload ie ie8 lt-ie9 lt-ie10" data-rui-select-auto-init="false"> <![endif]-->
  <!--[if IE 9]>         <body class="sold rui-preload ie ie9 lt-ie10" data-rui-select-auto-init="false"> <![endif]-->
  <!--[if gt IE 9]><!-->

  <body class="sold" data-rui-select-auto-init="false"><iframe src="https://secure-au.imrworldwide.com/storageframe.html" id="LOCSTORAGE" scrolling="no" name="empty" hidden="true" style="width: 1px; height: 1px; position: absolute; top: -7px; left: -7px; border: 0px;"></iframe>
    <!--<![endif]-->
    <div class="hero">
      <ul class="rui-visuallyhidden rui-accessibility">
    <li>
      <a href="#rui-main-nav" title="Go directly to the website navigation">skip to Main Navigation</a>
    </li>
    <li>
      <a href="#rui-main-content" title="Go directly to main content">skip to Content</a>
    </li>
      </ul>
      <header class="rui-header rui-header-inverse rui-header-not-fixed rui-clearfix" data-auto-init="false">
    <div class="rui-grid rui-header-container">
      <div class="rui-clearboth"></div>
      <button class="rui-burger-toggle rui-icon rui-icon-navdeck rui-button-no-style rui-desktop">
        <span class="rui-visuallyhidden">Menu</span>
        <span class="hit-box"></span>
      </button>
      <nav class="rui-myrea-nav">
        <div id="webCollectionsSlot"></div>
        <div id="webInboxSlot"></div>
        <div id="webInboxOverlay"></div>
        <ul class="rui-user-menu logged-out" style="">
          <li>
        <a href="/my-real-estate/login" title="Sign in">Sign in</a>
          </li>
          <li>
        <a href="/my-real-estate/register" class="rui-button-brand join" title="Create an account">Join</a>
          </li>
        </ul>
        <ul class="rui-user-menu logged-in rui-toggle-wrapper" style="display:none;">
          <li>
        <a href="#" class="rui-person-guy rui-toggle-link rui-icon rui-icon-user-off"></a>
        <div class="rui-account-menu-container">
          <ul class="rui-account-menu rui-toggle-container">
            <li class="first">
              <a href="/saved-properties" title="View saved properties">
            <span class="rui-icon rui-icon-save"></span><span>Saved properties</span>
              </a>
            </li>
            <li>
              <a href="/property/my-property" title="Track your property">
            <span class="rui-icon rui-icon-home"></span><span>Track your property</span>
              </a>
            </li>
            <li>
              <a href="https://homeloans.realestate.com.au/my-finances" title="My finances">
            <span class="rui-icon rui-icon-commission"></span><span>My finances</span>
              </a>
            </li>
            <li><a href="/my-real-estate/saved-searches-and-email-alerts" title="View property alerts">Saved searches &amp; alerts</a></li>
            <li><a href="/my-real-estate/profile" title="View profile">My profile</a></li>
            <li><a href="/my-real-estate/account" title="Manage account settings">Account settings</a>
            </li>
            <li class="last"><a href="/logout.ds" title="Log out">Log out</a></li>
          </ul>
        </div>
          </li>
        </ul>
      </nav>
    </div>
    <div class="rui-grid rui-nav-container rui-burger-container">
      <a title="realestate.com.au homepage" href="/buy" class="logo">
        <img src="//s1.rui.au.reastatic.net/rui-static/img/rea-logo-thin-white-v3.png" alt="realestate.com.au Australia lives here">
      </a>
      <nav class="rui-main-nav">
        <ul>
          <li class="rui-nav-tab buy">
        <a href="/buy" title="Real estate and property for sale">
          <span>Buy</span>
        </a>
          </li>
          .
          .
          .
          <li class="rui-nav-tab commercial">
        <a href="http://www.realcommercial.com.au" target="_blank" title="Commercial real estate for sale and lease">
          <span>Commercial</span>
        </a>
          </li>
        </ul>
      </nav>
    </div>
      </header>


      <div class="hero-content">
    <div class="search-form-container">
      <form action="listsearchview.ds" class="search-form" name="search-form">
        <h1 id="search-form-title">Search sold properties</h1>
        <div class="search-container">
          <div class="search-inner-container" style="z-index: 1;">
        <nav class="search-channel-container">
          <a href="/buy" class="rui-search-tab">Buy</a>
          <a href="/rent/" class="rui-search-tab">Rent</a>
          <a href="/sold/" class="rui-search-tab rui-search-tab-selected">Sold</a>
          <a href="/property/" class="rui-search-tab property-value-tab">Property value</a>
          <a href="/find-agent" class="rui-search-tab find-agents-tab">Find agents</a>
        </nav>

        <div class="search-inner-container" style="z-index: 1;">


          <div class="rui-search-container search-input-container">
            <i class="rui-icon rui-icon-search search-icon"></i>

            <input name="where" id="where" tabindex="2" class="rui-input rui-location-box rui-auto-complete-input" data-auto-init="false" data-location-source="consumer-suggest" data-max-results="7" data-client-source="homepage" autocomplete="off" placeholder="Search by state, suburb or postcode" data-multi-select="true" data-suggest-types="suburb,precinct,region,state,postcode" data-auto-submit="false" autofocus=""><div class="clear-text-container"><a class="rui-icon rui-icon-cross" title="Clear text"> </a></div>
            <input name="theme" id="theme" type="hidden" value="rea.sold">
            <button class="rui-search-button" tabindex="10">
              <span class="rui-visually">Search</span>
            </button>
            <div class="focus-border" style="display: none;"></div>
          </div>
        </div>

          </div>
          <section class="search-refinements" id="search-refinements">


        <div class="condition property-select-holder">
          <div class="select-holder">
            <label>All property types</label>
            <select class="rui-input rui-select resize-select rui-select-rendered" id="rui-property-type-select-id" name="propertyType" multiple="multiple" style="display: none;">
              <option value="All" selected="selected">All property types</option>
              <option value="House">House</option>
              <option value="unit apartment">Apartment &amp; Unit</option>
              <option value="Townhouse">Townhouse</option>
              <option value="Villa">Villa</option>
              <option value="Land">Land</option>
              <option value="Acreage">Acreage</option>
              <option value="Rural">Rural</option>
              <option value="unitblock">Block of Units</option>
              <option value="retire">Retirement Living</option>
            </select>
        </div>
        .
        .
        .
</body></html>

更新

但是，在尝试调用网址https://www.realestate.com.au/sold/in-alice+springs+-+greater+region%2c+nt%3b/list-1'时，似乎是通过{strong>指纹

检测到Selenium驱动的浏览器代理

代码块：

driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.realestate.com.au/sold/in-alice+springs+-+greater+region%2c+nt%3b/list-1'")

控制台输出：

<html lang="en"><head>
    <meta charset="UTF-8">
    <link rel="shortcut icon" href="about:blank">
</head>
<body>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/j.js"></script>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/f.js"></script>
<script src="/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint/script/kpf.js?url=/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint&amp;token=34091693-e55f-5166-d979-67e9426e1030"></script>

</body></html>

您可以在Chrome browser initiated through ChromeDriver gets detected
中找到相关的详细讨论

硒未加载页面

3 个答案:

更新