Question

如何使用Puppeteer从多个网址中抓取内容？

我已经创建了一个循环，但是我只看到第一个URL的结果。

我怀疑这与声明结果变量的位置有关，但是我没有运气尝试，有人知道该怎么做吗？

const puppeteer = require('puppeteer');

function run() {
    return new Promise(async (resolve, reject) => {
        try {

            const browser = await puppeteer.launch();
            const page = await browser.newPage();
            const urls = ["https://www.marksandspencer.com/high-neck-long-sleeve-blouse/p/p60260040?image=SD_01_T43_5168_HD_X_EC_90&color=LIGHTDENIM&prevPage=plp", "https://www.marksandspencer.com/pure-cotton-printed-short-sleeve-t-shirt/p/p60263529?image=SD_01_T41_8030Z_Z4_X_EC_90&color=WHITEMIX&prevPage=plp"];
            

              for (let i = 0; i < urls.length; i++) {
                const url = urls[i];
                await page.goto(url);
                let products = await page.evaluate(() => {
                    let product = document.querySelector('h1[itemprop=name]').innerText;
                    let results = [];
                    let items = document.querySelectorAll('[data-ttip-id=sizeGridTooltip] tbody tr td label');
                    items.forEach((element) => {
                        let size = element.getAttribute('for');
                        let stockLevel = "";
                        let nearest_td = element.closest('td');
                        if (nearest_td.classList.contains('low-stock')) {
                            stockLevel = "Low stock"
                        } else if (nearest_td.classList.contains('out-of-stock')) {
                            stockLevel = "Out of stock"
                        } else {
                            stockLevel = "In stock"
                        }
                        results.push({
                            product: product,
                            size: size,
                            stock: stockLevel
                        })
                    });
                    return results
                })
                browser.close();
                return resolve(products);
            }
            
        } catch (e) {
            return reject(e);
        }
    })
}
run().then(console.log).catch(console.error);

Answer 1

这些行在您的for循环内：

                browser.close();
                return resolve(products);

因此，作为第一次迭代的一部分，请关闭浏览器并返回该函数。您应该将其移出for循环，并将products存储在这样的数组中：

              const urls = /* ... */;
              const productsList = [];

              for (let i = 0; i < urls.length; i++) {
                const url = urls[i];
                await page.goto(url);
                let products = await page.evaluate(/* ... */);
                productsList.push(products);
              }
              browser.close();
              return resolve(productsList); // resolve with an array containing the aggregated products

如果您正在寻找更优雅的解决方案（以并行方式抓取页面），则可能需要看一下puppeteer-cluster库（免责声明：我是作者）。

遍历Puppeteer中的一组网址

1 个答案: