Question

如何通过HtmlUnitDriver和HtmlUnit无头浏览器在amzscout的“选择类别（美国）”下创建项目列表？

使用 GeckoDriver / Firefox 和 ChromeDriver / Chrome 组合，我可以创建列表并打印如下：

代码试用：

System.setProperty("webdriver.gecko.driver", "C:/Utility/BrowserDrivers/geckodriver.exe");
WebDriver driver = new FirefoxDriver();
driver.get("https://amzscout.net/sales-estimator");
List<WebElement> elements = new WebDriverWait(driver, 10).until(ExpectedConditions.visibilityOfAllElementsLocatedBy(By.cssSelector("span.cat-pick_name-in")));
for (WebElement ele:elements)
    System.out.println(ele.getAttribute("innerHTML"));
driver.quit();

控制台输出：

Appliances
Arts, Crafts &amp; Sewing
Automotive
.
.
.

但是，使用 HtmlUnitDriver 和 HtmlUnit无头浏览器，似乎HTML呈现的方式如下：

完整的html位于pastebin

HTML的相关部分是：

<script type="application/ld+json">
  //<![CDATA[

  {
    "@context": "http://schema.org/",
    "@type": "Product",
    "name": "AMZScout Sales Estimator",
    "image": "",
    "brand": "AMZScout",
    "aggregateRating": {
      "@type": "AggregateRating",
      "ratingValue": "4.7",
      "bestRating": "5",
      "worstRating": "1",
      "ratingCount": "231"
    }
  }

  //]]>
</script>
<script type="text/javascript" src="/js/common.js">
</script>
<script type="text/javascript">
  //<![CDATA[

  const DATA = {
      COM: [
        ["Appliances", "s-cat-icon-appliances"],
        ["Arts, Crafts & Sewing", "s-cat-icon-craft"],
        ["Automotive", "s-cat-icon-automotive"],
        ["Baby", "s-cat-icon-baby"],
        ["Beauty & Personal Care", "s-cat-icon-beauty"],
        ["Books", "s-cat-icon-books"],
        ["Camera & Photo", "s-cat-icon-camera"],
        ["Cell Phones & Accessories", "s-cat-icon-phone"],
        ["Clothing, Shoes & Jewelry", "s-cat-icon-clothing"],
        ["Computers & Accessories", "s-cat-icon-computers"],
        ["Electronics", "s-cat-icon-electronics"],
        ["Grocery & Gourmet Food", "s-cat-icon-food"],
        ["Health & Household", "s-cat-icon-health"],
        ["Home and Garden", "s-cat-icon-home"],
        ["Home & Kitchen", "s-cat-icon-kitchen"],
        ["Industrial & Scientific", "s-cat-icon-gear"],
        ["Jewelry", "s-cat-icon-jewelry"],
        ["Kindle Store", "s-cat-icon-kindle"],
        ["Kitchen & Dining", "s-cat-icon-dining"],
        ["Musical Instruments", "s-cat-icon-musical-instruments"],
        ["Office Products", "s-cat-icon-office"],
        ["Patio, Lawn & Garden", "s-cat-icon-lawn"],
        ["Pet Supplies", "s-cat-icon-pet-food"],
        ["Shoes", "s-cat-icon-shoes"],
        ["Software", "s-cat-icon-software"],
        ["Sports & Outdoors", "s-cat-icon-sports"],
        ["Tools & Home Improvement", "s-cat-icon-repairs"],
        ["Toys & Games", "s-cat-icon-toys"],
        ["Watches", "s-cat-icon-watches"],
        ["Video Games", "s-cat-icon-joystick"]
      ],
      CO_UK: [
          ["Baby", "s-cat-icon-baby"],

其中引用了哪些：

$(function () { var rankInput = $('.cat-rank_input'); function toggleRank(e) { var cats = $('.cat-pick'); var rank = $('.cat-rank'); var list = rank.find('.cat-pick_list'); var $el = $(e.currentTarget).clone(); $el.on('click', toggleRank).css('cursor',
'pointer'); list.empty(); list.append($el); category = $el.find('.cat-pick_name-in').text(); rankInput.val(''); cats.toggle(); rank.toggle(); if ($(window).width() >= 768) { var catsHeight = cats.height(); rank.height(catsHeight); } if (rank.is(':visible'))
{ val.text('?'); setTimeout(function () {rankInput.focus()}, 0); } } function selectDomain(d) { const data = DATA[d]; const list = $('.cat-pick .cat-pick_list'); list.empty(); data.filter(function (d) {return d[1] != ''}).forEach(function (d) { var el
= $('
<div class="cat-pick_i"><span class="cat-pick_link"><span class="cat-pick_ico"><span></span></span><span class="cat-pick_name"><span class="cat-pick_name-in"></span></span>
  </span>
</div>'); el.find('.cat-pick_ico span').addClass(d[1]); el.find('.cat-pick_name-in').text(d[0]); el.on('click', toggleRank); list.append(el); }); domain = d; } rankInput.on('change', function () {rank = rankInput.val()}); rankInput.on('keyup', function(e) {e.keyCode
== 13 && (rank = rankInput.val()) && getEstSales()}); $('.cat-rank_another-link').on('click', toggleRank); $('#domain').on('change', function (e) {selectDomain(e.target.value);}); selectDomain(domain); });

有人可以帮我吗？

Answer 1

您已经知道，要查找的项目由javascript创建。这意味着您必须启用对HtmlUnit的javascript支持。

第二点是要以某种方式等待javascript完成。您正在使用“ visibilityOfAllElementsLocatedBy”，并且文档状态如下：

期望检查网页上与定位器匹配的所有所有元素是可见的。

如果没有符合您选择器的元素（或者不是所有元素，因为javascript还在创建新元素），则为true。因此，我将等待条件更改为真正等待直到创建元素为止。

我的最终资源看起来像这样，并精确地创建了您期望的列表：

String url = "https://amzscout.net/sales-estimator";                        

// true enables javascript support                                          
WebDriver driver = new HtmlUnitDriver(true);                                
try {                                                                       
    driver.get(url);                                                        

    // wait until the elements are created                                  
    List<WebElement> elements =
            new WebDriverWait(driver, 10)               
                .until(ExpectedConditions                                   
                    .numberOfElementsToBeMoreThan(                          
                            By.cssSelector("span.cat-pick_name-in"), 29));  

    System.out.println();                                                   
    for (WebElement ele : elements) {                                       
        System.out.println(ele.getAttribute("innerHTML"));
    }
} finally {                                                                 
    driver.quit();                                                          
}

希望有帮助。...

如何在“通过HtmlUnitDriver和HtmlUnit无头浏览器呈现不同显示的类别（美国）”下创建项目列表？

1 个答案: