Question

在下面您可以找到发生的错误。

我正在尝试使用NodeJS和puppeteer抓取网站的内容。有时，代码因错误超时而停止。有没有一种方法可以使如果超过了页面加载的超时时间，我可以运行一个函数，该函数要么重新加载页面，要么让脚本等待几秒钟，然后重新加载页面，直到它正确获取数据为止而不会崩溃？如果是这样，我将如何实施呢？

谢谢。

(node:8300) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
    at Promise.then (C:\Users\danie\node_modules\puppeteer\lib\LifecycleWatcher.js:143:21)
  -- ASYNC --
    at Frame.<anonymous> (C:\Users\danie\node_modules\puppeteer\lib\helper.js:108:27)
    at Page.goto (C:\Users\danie\node_modules\puppeteer\lib\Page.js:656:49)
    at Page.<anonymous> (C:\Users\danie\node_modules\puppeteer\lib\helper.js:109:23)
    at scrape (C:\Users\danie\Documents\Node Projects\p-download.js:23:14)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:8300) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:8300) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

我的代码：

const puppeteer = require('puppeteer');

let scrape = async () => {
    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();

    await page.setRequestInterception(true);    
    page.on('request', (req) => {
        if(req.resourceType() == 'stylesheet' || req.resourceType() == 'script' || req.resourceType() == 'font' || req.resourceType() == 'media' || req.resourceType() == 'image'){
            req.abort();
        }
        else {
            req.continue();
        }
    }); //Disables loading CSS, images and scripts

    for(i=0; i<5000; i++){
        await page.goto('https://website.com/' + i);
        let result = await page.evaluate(() => {
            var result = '';
            for (i=1; i<=10; i++){
                result += document.getElementsByTagName('td')[i].innerText;
                result += ',';
            }
            result += '\n';
            return result;
        });
    }
}
scrape();

Answer 1

将您的代码放入try / catch块中，以避免崩溃……我会将循环代码移到另一个函数中

import com.vividsolutions.jts.geom.Point;

@Entity
@Table(name = "Points")
public class Points implements Serializable {

    private static final long serialVersionUID = 1L;

    @Id
    @GeneratedValue(strategy = GenerationType.SEQUENCE)
    private Long id;

    private String userId;

    private String pointName;

    private Point point;

/*GETTERS AND SETTERS*/

超过超时时间时如何停止NodeJS脚本崩溃

1 个答案: