获取所有可见的纯文本并找出每个文本所属的HTML标记或DOM元素

时间：2018-09-11 05:29:31

标签： puppeteer

我知道如何在页面上获得所有可见的纯文本：

const text = await page.$eval('*', el => el.innerText);

但是我还需要知道每个文本属于页面的哪个元素，而我找不到找到这种方法的方法。

2 个答案:

答案 0 :(得分：2)

在客户端，您可以使用TreeWalker保留订单的方式执行此操作。这是一个示例，其中包含Web Scraper Testing Ground中的示例内容：

const IGNORE = ["style", "script"];

const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT);

const pairs = [];

let node;

while ((node = walker.nextNode()) !== null) {
  const parent = node.parentNode.tagName;

  if (IGNORE.includes(parent)) {
    continue;
  }

  const value = node.nodeValue.trim();

  if (value.length === 0) {
    continue;
  }

  pairs.push([parent.toLowerCase(), value]);
}

console.log(pairs);
<div id="topbar"></div>
		<a href="/" style="text-decoration: none">
		    <div id="title">WEB SCRAPER TESTING GROUND</div>
		    <div id="logo"></div>
		</a>
		<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
	<ol>
		<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
		<li>Scrape discounted products only</li>
		<li>Scrape products with red prices only</li>
	</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
	</p><ul>
		<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
		</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
	</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
	<li><a href="/blocks?ver=1">Price list 1</a></li>
	<li><a href="/blocks?ver=2">Price list 2</a></li>
	<li><a href="/blocks?ver=3">Price list 3</a></li>
	<li><a href="/blocks?ver=4">Price list 4</a></li>
	<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>

<div id="case_blocks">

<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>

<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>

</div>
<br><br><br>
		</div>

根据Grant Miller的回答，使用evaluate在Puppeteer中称呼此人：

const pairs = await page.evaluate(() => {
  const IGNORE = ["style", "script"];
  const NONWHITESPACE_RE = /\S/;

  const result = document.evaluate(
    "//*[child::text()]",
    document,
    null,
    XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
    null
  );

  const pairs = [];

  for (let i = 0, j = result.snapshotLength; i < j; i++) {
    const element = result.snapshotItem(i);

    if (IGNORE.includes(element.tagName.toLowerCase())) {
      continue;
    }

    const nodes = [...element.childNodes];

    for (const node of nodes) {
      if (node.nodeType !== document.TEXT_NODE) {
        continue;
      }

      if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
        continue;
      }

      pairs.push({
        tag: element.tagName.toLowerCase(),
        text: node.nodeValue.trim()
      });
    }
  }

  return pairs;
});

console.log(pairs);

这是客户端功能的原始版本，该功能使用XPath，但始终将节点的直接子代放在其间接子代之前：

const IGNORE = ["style", "script"];
const NONWHITESPACE_RE = /\S/;

// get all text nodes in the document
const result = document.evaluate(
  // matches any node in the document that has at least one direct
  // text node child, including whitespace-only nodes
  "//*[child::text()]",
  document,
  null,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null
);

// the result doesn't use the JavaScript iterator protocol, so we have
// to manually iterate over the elements
const pairs = [];

for (let i = 0, j = result.snapshotLength; i < j; i++) {
  const element = result.snapshotItem(i);

  if (IGNORE.includes(element.tagName.toLowerCase())) {
    continue;
  }

  const nodes = [...element.childNodes];

  for (const node of nodes) {
    if (node.nodeType !== document.TEXT_NODE) {
      continue;
    }

    // filter out whitespace-only nodes
    if (node.nodeValue.search(NONWHITESPACE_RE) === -1) {
      continue;
    }

    pairs.push({
      tag: element.tagName.toLowerCase(),
      // remove the `.trim()` to preserve leading & trailing whitespace
      text: node.nodeValue.trim()
    });
  }
}

console.log(pairs);
        <div id="topbar"></div>
		<a href="/" style="text-decoration: none">
		    <div id="title">WEB SCRAPER TESTING GROUND</div>
		    <div id="logo"></div>
		</a>
		<div id="content">
<h1>BLOCKS: Price List </h1>
<div id="caseinfo">In this test, the web scraper needs to scrape a price list organized in a block layout. Specifically, it has to:
	<ol>
		<li>Extract all the products (their names, descriptions and prices), while skipping advertisements</li>
		<li>Scrape discounted products only</li>
		<li>Scrape products with red prices only</li>
	</ol>
<p>
</p><p>There is a <b>ver</b> parameter (which varies from 1 to 5) to show different table versions (with different product numbers, best price and advertisement positions).</p>
<p>Also there are two tables presented:
	</p><ul>
		<li><b>Case 1</b> (simple one, with products and prices placed into the same block)
		</li><li><b>Case 2</b> (complicated one, with products and prices placed into separate blocks)</li>
	</ul>
<p></p>
<p>For testing, you may use the following sample links. The scraper should sufficiently scrape all data from a certain case using the same project:
</p><ul>
	<li><a href="/blocks?ver=1">Price list 1</a></li>
	<li><a href="/blocks?ver=2">Price list 2</a></li>
	<li><a href="/blocks?ver=3">Price list 3</a></li>
	<li><a href="/blocks?ver=4">Price list 4</a></li>
	<li><a href="/blocks?ver=5">Price list 5</a></li>
</ul>
<p></p>
</div>

<div id="case_blocks">

<h2>Case 1</h2>
<div id="case1">
<div class="prod2"><span style="float: left"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</span><span style="float: right">$239.95</span></div><div class="prod1"><span style="float: left"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</span><span style="float: right" class="best">$249.00</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</span><span style="float: right">$1,099.99</span></div><div class="prod1"><span style="float: left"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</span><span style="float: right" class="best">$385.72</span><span style="float: right;margin-right:10px" class="best">BEST<br>PRICE!</span></div><div class="ads">ADVERTISEMENT</div><div class="prod2"><span style="float: left"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$549.99<div class="disc">discount 7%</div></span></div><div class="prod1"><span style="float: left"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</span><span style="float: right">$399.99</span></div></div>

<h2 style="margin-top: 50px">Case 2</h2>
<div id="case2">
<div class="left"><div class="prod2"><div class="name">Dell Latitude D610-1.73 Laptop Wireless Computer</div>2 GHz Intel Pentium M, 1 GB DDR2 SDRAM, 40 GB, Microsoft Windows XP Professional</div><div class="prod1"><div class="name">Samsung Chromebook (Wi-Fi, 11.6-Inch)</div>1.7 GHz, 2 GB DDR3 SDRAM, 16 GB, Chrome</div><div class="ads">ADVERTISEMENT</div><div class="prod2"><div class="name">Apple MacBook Pro MD101LL/A 13.3-Inch Laptop (NEWEST VERSION)</div>2.5 GHz Intel Core i5, 4 GB DDR3 SDRAM, 500 GB Serial ATA, Mac OS X v10.7 Lion</div><div class="prod1"><div class="name">Acer Aspire AS5750Z-4835 15.6-Inch Laptop (Black)</div>2 GHz Pentium B940, 4 GB SDRAM, 500 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$239.95</div><div class="price1 best">$249.00</div><div class="ads"></div><div class="price2">$1,099.99</div><div class="price1 best">$385.72</div></div><div class="ads" style="clear: both">ADVERTISEMENT</div><div class="left"><div class="prod2"><div class="name">HP Pavilion g7-2010nr 17.3-Inch Laptop (Black)</div>2.3 GHz Core i3-2350M, 6 GB SDRAM, 640 GB, Windows 7 Home Premium 64-bit</div><div class="prod1"><div class="name">ASUS A53Z-AS61 15.6-Inch Laptop (Mocha)</div>1.4 GHz A-Series Quad-Core A6-3420M, 4 GB DIMM, 750 GB, Windows 7 Home Premium 64-bit</div></div><div class="right"><div class="price2">$549.99<div class="disc">discount 7%</div></div><div class="price1">$399.99</div></div></div>

</div>
<br><br><br>
		</div>

答案 1 :(得分：1)

您可以使用以下解决方案获取包含标签名称及其关联文本的对象数组：

const example = await page.evaluate( () =>
{
    return Array.from( document.body.getElementsByTagName( '*' ), e => [...e.childNodes].filter( e =>
           e.nodeType === 3
        && !['SCRIPT','STYLE'].includes( e.parentNode.tagName )
        && e.textContent.trim().length
    )).flat().map( e => ({
        'tag_name'     : e.parentNode.tagName,
        'text_content' : e.textContent.trim()
    }));
});

console.log( example[0].tag_name );     // DIV
console.log( example[0].text_content ); // Hello, world!
相关问题
最新问题