Question

我正在尝试学习木偶，并且想在https://www.tapology.com/regions上抓取这个MMA网页。

我想抓取所有地区的标题，例如“美国中西部，美国东北，美国西南...等”

我尝试使用await page.waitForSelector等待选择器加载，但页面只是挂起。

我也尝试过使用innerHTML，但结果相同。

尝试使用page。$$ eval（selector，pageFunction [，... args]），但返回空数组。

尝试使用Google开发工具中的确切CSS选择器，但未成功。尝试了各种CSS选择组合，但是我仍然无法抓取h4的文本。

const puppeteer = require('puppeteer');
const REGIONS_URL = 'https://www.tapology.com/regions';

async function getRegionsNames(url) {
    const browser= await puppeteer.launch();
    const page= await browser.newPage();

  await (async ()=>{
  const MIDWEST_SELECTOR = "#content > div.regionIndex > h4:nth- child(1) > a";
  //await page.waitForSelector(MIDWEST_SELECTOR);  //hangs if used

  const mid = await page.evaluate((sel)=>{         //trying to grab 'MIDWEST'
    return document.querySelector(sel).innerText;
  },MIDWEST_SELECTOR);
 console.log(mid);      //throws error
 await browser.close();
}
getRegionsNames(REGIONS_URL);

（节点：23646）UnhandledPromiseRejectionWarning：错误：评估失败：TypeError：无法读取null的属性“ innerText”

Answer 1

尝试：

app.get('/testing',function(req,res){
    (async () => {

        const browser = await puppeteer.launch({
            headless: true
        });
        const page = await browser.newPage();
        await page.goto('https://www.tapology.com/regions',{waitUntil: 'domcontentloaded'});
        const example  = await page.$('.regionIndex');
        const scrapedData = await page.evaluate(() =>
        Array.from(document.querySelectorAll('h4 a'))
          .map(link => ({
            title: link.innerHTML,
            link: link.getAttribute('href')
          }))
      )
      console.log('scrapedData',scrapedData);
        await page.close();
        await browser.close();
        return res.send(scrapedData);

    })();
});

您将获得：

[
{
title: "US Midwest",
link: "/regions/us-midwest"
},
{
title: "US Northeast",
link: "/regions/us-northeast"
},
{
title: "US Southeast",
link: "/regions/us-southeast"
},
{
title: "US Southwest",
link: "/regions/us-southwest"
},
{
title: "US West",
link: "/regions/us-west"
},
{
title: "Asia Central",
link: "/regions/central-asia"
},
{
title: "Canada",
link: "/regions/canada"
},
{
title: "Europe Balkans",
link: "/regions/europe-balkans"
},
{
title: "Europe Eastern",
link: "/regions/europe-eastern"
},
{
title: "Europe Western",
link: "/regions/western-europe"
},
{
title: "Latin America",
link: "/regions/latin-america"
},
{
title: "Middle East",
link: "/regions/middle-east"
}
]

木偶：page.evaluate引发TypeError：无法读取null

1 个答案: