空格问题

时间:2019-01-22 17:28:03

标签: web-crawler stormcrawler

使用Storm 1.13和Elastic Search 6.5.2。搜寻器工作正常,我在 / n / t 空格方面遇到问题。当我检查索引时,有很多 / n / t 。如何避免这些内容被抓取。

示例:

{
   "_index":"some-index",
   "_type":"doc",
   "_id":"12sfsdf3454r5fdsfdsfdgdf",
   "_score":1.0,
   "_source":{
      "content":"\n \n  \n    Skip to main content\n  \n\n    \n\n    \n        \n\t\t\t\n            \n                \n  \n    \n      \n        \n\n      \n\n      \n        \tDirectories | \n\t Home\n\n\n      \n    \n\n  \n                  \n\n            \n\n            \n\n            \n                \n                    \n                          \n    \n\n      \n  \thome ining Services\n\tContact Us\n\n\n\n  \n\n                    \n\n                    \n\n                \n\n            \n\n            \n\n            \n                \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\tToggle navigation\n                        \n                        \n                        \n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\n\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\n\n\t\t\t\t\t\t\t\n\n\t\t\t\t\t\t\n\n\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\tHours & Locations\n\tMenus\n\tStart an Order\n\tPolicies\n\tFAQs\n\tContact\n\tMy Cart\n\n\t\t\t\t\t\t\t\t\t\t\t\t\t\n\n                        \t\t\t\t\t\t\n\n\t\t\t\t\t\n\n\t\t\t\t\n\n                    \n\n\t\t\t\n\n\n\t\t\t\n\n        \n\n    \n    \n\n    \n\t\t\n        \n            \n            \n         \n\n        \n            \n\n\n                        \n            \t\t\t\t\t\t            \n                 \n\t\t\\t\tFeedback Form\n\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\n\t\t\t\t\n\t\t\t\t\n\t\t\t\t\n                                    \n                        \n                        \n                                                                                                                                                                           \n                          \n    \n\n      \n  \n    \n                  \n    This form is for general feedback about your . If you need to request a refund or cancel your order, please contact your order location directly, or consult our Refund and Cancellation policy.\n\n\n\n\n\n Full Name *\n\n\n\n\n\n E-mail Address *\n\n\n\n\n\n Your Message *\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSubmit\n\n\n\n    \n\n\n  \n\n                        \n                        \n                    \n\n                                                            \n\n            \n                        \n                                                \n\n            \n\t\t\t\t\n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\n\n\t\t\t\t\t\n\n\t\t\t\t\n\t\t\t\n\n            \n                \n                    \n\n                        \n                            \n                                \n                                    Contact Us\n                                \n\n                                \n                                    \n                                    \t\n                                        \t\n                                            \t\n                                        \t\n\n                                        \n                                        \n                                            \n                                                Email\n                                            \n\n                                            \n                                                                                            \n\n                                        \n\n                                        \n\n                                    \n\n                                    \n                                        \n                                            \n                                        \n\n                                        \n                                            \n                                                Phone\n                                            \n\n                                            \n                                            n                                            \n\n                                        \n\n                                        \n\n                                    \n\n                                    \n                                    \t\n                                        \t\n                                            \t\n                                        \t\n\n                                        \n                              ",
      "url":"https://someurl.test.edu/feedback-form",
      "domain":"test.edu",
      "title":[
         "Feedback Form | Organize",
         "Feedback Form | Organize"
      ]
   }
},

1 个答案:

答案 0 :(得分:0)

具有选项卡和换行符对使用Elasticsearch进行搜索没有影响,但是您当然可以编写自定义的ParseFilter来将它们重写为空格。