使用Cheerio和Response for Node Web scraper,将响应函数结果传递给视图

时间:2016-01-08 20:42:08

标签: javascript node.js request handlebars.js cheerio

我正在使用本教程: https://www.smashingmagazine.com/2015/04/web-scraping-with-nodejs/

制作一个真正基本的节点web scraper。我在这里设置并运行了我的应用程序:

https://[redacted]/

我目前在我的节点应用程序幕后做的是使用两个模块cheerio并请求运行此功能(如下所示)。这个函数基本上是一个URL,发出请求并用数据变量抓取页面元素,擦除它的值,并将值(温度)记录到我的计算机终端的控制台。我需要一种方法将此值发送到我的视图并在页面上呈现它,而不是仅将其记录到我的计算机的控制台。

我遇到的问题是下面的请求函数的范围,我无法将任何返回值(温度)传递给我的视图。我知道我的设置有问题,因为我目前有router.get的请求函数INSIDE。如果我将请求函数放在router.get之外,我仍然无法将值传递给我的视图,但它将成功从Web URL获取数据并将其记录到终端的控制台。我希望我很清楚。请参阅res.render到我的视图,它包含正在进行网络抓取的请求功能..

router.get('/', function(req, res, next) {

    //target url we are scraping data from
    var url = "http://www.wunderground.com/cgi-bin/findweather/getForecast?&query=" + 02888;
    var temperature;
    // request function that uses the request module
    request(url, function (error, response, body) {

        if (!error) {
            // using cheerio module to load body of html req document and scrape the data
            var $ = cheerio.load(body),
                temperature = $("[data-variable='temperature'] .wx-value").html();
            // logs it to console (of my computer..)    
            console.log("It’s " + temperature + " degrees Fahrenheit.");
        } 

        else {
            console.log("We’ve encountered an error: " + error);
        }

        return temperature;
    });
    /* renders my view, this is my attempt to pass the values as variables to my handlebars template. Currently it is only passing the URL var as it exists before the request function call, and the empty var temperature (which gets its value during, and inside the request function call). How can i get at those values returned from the request function and pass to my view below? */
    res.render('index', {title: url, data: temperature } );  

});

1 个答案:

答案 0 :(得分:1)

request中的函数是异步执行的,因此,在设置温度之前调用render。您需要将render函数移动到异步函数中。

router.get('/', function(req, res, next) {

//target url we are scraping data from
var url = "http://www.wunderground.com/cgi-bin/findweather/getForecast?&query=" + 02888;
var temperature;
// request function that uses the request module
request(url, function (error, response, body) {

    if (!error) {
        // using cheerio module to load body of html req document and scrape the data
        var $ = cheerio.load(body),
            temperature = $("[data-variable='temperature'] .wx-value").html();
        // logs it to console (of my computer..)    
        console.log("It’s " + temperature + " degrees Fahrenheit.");
    } 

    else {
        console.log("We’ve encountered an error: " + error);
    }

    res.render('index', {title: url, data: temperature } );  
});

});