Question

*评论中的更多详细信息

我正在使用cheerio.js抓取数据并显示在3个HTML表中。它可以正常工作，除了在显示抓取数据之前在第二张和第三张表中添加了这些空白行（第一张表有25行，第二张表有25行空白行，然后是25行数据。第三张表有50行空白行，然后是25行数据）。

我相信这是由于通过一个请求刮取了所有数据，而不是每个表进行了1次刮取。

这是我第一次抓取，所以我想知道是否有办法更改我的代码以解决此问题，或者我是否必须在每个表中有1个抓取请求（但这似乎效率不高）。

如果您想查看我的猫鼬图式，请告诉我。

用于显示数据的HTML表（使用ejs）：

<table>
          <tr>
              <th>Environment</th>
              <th>Store Name</th>
              <th>Code Version</th>
          </tr>
      <% stores.forEach(function(store){ %>

          <tr>
              <td> <%= store.environment.QA1 %> </td>
              <td> <%= store.store.QA1 %> </td>
              <td> <%= store.version.QA1 %> </td>
          </tr>
      <% }); %>
  </table>

cheerio.js抓取信息：

router.get("/scrape", function(req, res){
    request("http://link-goes-here", function(error, response, html){
        var $ = cheerio.load(html);
        var QAStoreInfo = [];
        var QA1StoreInfo = [];
        
        $("body > table > tbody > tr > td:nth-child(1) > table").addClass("row");
        $('.row > tbody > tr:contains("QA")').each(function(i, element){
            var QAResult = {
                "environment.QA" : $(this).children().next(),
                "store.QA" : $(this).children().next().next(),
                "version.QA" : $(this).children().next().next().next()
            };
            QAStoreInfo.push(QAResult);
             Store.count({}, function(err, test){
                if (test === 0) {
                    var QAEntry = new Store(QAResult);
    
                    QAEntry.save(function(err, doc){
                        if(err){
                            console.log(err);
                        }
                    });
                }
             });
        });

        /* QA1 */
        $('.row > tbody > tr:contains("QA1")').each(function(i, element){
            var QA1Result = {
                "environment.QA1" : $(this).children().next(),
                "store.QA1" : $(this).children().next().next(),
                "version.QA1" : $(this).children().next().next().next()
            };

            QA1StoreInfo.push(QA1Result);
             Store.count({}, function(err, test){
                if (test === 0) {
                    var QA1Entry = new Store(QA1Result);
   
                    QA1Entry.save(function(err, doc){
                        if(err){
                            console.log(err);
                        }
                    });
                }
             });
        });

猫鼬模式：

var storeSchema = new mongoose.Schema({
  environment: {
    QA: String,
    QA1: String,
    QA2: String
  },
  store: {
    QA: String,
    QA1: String,
    QA2: String
  },
  version: {
    QA: String,
    QA1: String,
    QA2: String
  }
});

/ scrape调用后的

mongoDB集合：

{ "_id" : ObjectId("5f0610fee9b4c35d74a83600"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " Omega  QA " }, "version" : { "QA1" : " _2019.11_2019_11.2 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83601"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " Crescent Centre " }, "version" : { "QA1" : " _2020.06_2020_06.6 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83602"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " Cumberland " }, "version" : { "QA1" : " _2019.11_2019_11.2 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83603"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " East Denver " }, "version" : { "QA1" : " _2019.11_2019_11.2 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83604"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " Brookwood " }, "version" : { "QA1" : " _2019.11_2019_11.2 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83605"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " WheatRidge  " }, "version" : { "QA1" : " _2019.11_2019_11.2 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83606"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " Aurora Centerpoint  " }, "version" : { "QA1" : " _2020.06_2020_06.6 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83607"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " Castle Rock  " }, "version" : { "QA1" : " _2020.06_2020_06.6 " }, "__v" : 0 }
{ "_id" : ObjectId("5f0610fee9b4c35d74a83608"), "environment" : { "QA1" : " QA1 " }, "store" : { "QA1" : " Conyers  " }, "version" : { "QA1" : " _2020.06_2020_06.6 " }, "__v" : 0 }

屏幕截图：（第二个表格的显示方式）

Answer 1

Store集合包含具有以下架构的抓取文档：

{
    "environment" : { "QA":  <string> } ,  
    "store" : { "QA": <string> },
    "version" :  { "QA": <string> }
}

和

{
    "environment" : { "QA1":  <string> } ,  
    "store" : { "QA1": <string> },
    "version" :  { "QA1": <string> }
}

HTML表中的空行用于具有第一个架构的记录，其中QA1是undefined，environment和store字段中的version文档。

我建议将Store文档的架构与以下文档统一：

{
    "kind": "QA|QA1",
    "environment" : <string> ,  
    "store" : <string> ,
    "version" :  <string>
}

然后更新您的抓取脚本以存储文档，如下所示：

//...
            var QAResult = {
                "kind": "QA",
                "environment" : $(this).children().next(),
                "store" : $(this).children().next().next(),
                "version" : $(this).children().next().next().next()
            };
            QAStoreInfo.push(QAResult);
 
//...

            var QA1Result = {
                "kind": "QA1",
                "environment" : $(this).children().next(),
                "store" : $(this).children().next().next(),
                "version" : $(this).children().next().next().next()
            };

            QA1StoreInfo.push(QA1Result);

forEach模板中的ejs循环可以如下更新：

     <% stores.forEach(function(store){ %>

          <tr>
              <td> <%= store.environment %> </td>
              <td> <%= store.store %> </td>
              <td> <%= store.version %> </td>
          </tr>
      <% }); %>

将网页数据分割成多个HTML表格？

1 个答案: