非法论证Logstash但不知道为什么

Question

目前我正在使用Logstash将大型CSV文件导入Elasticsearch。 50％的产品都插入数据库中，但另外50％的产品出现了错误：

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [OfferNL.releaseDate]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"\""}}}}}

这是产品的总日志：

[2018-02-21T19:10:15,390][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"books", :_type=>"bol_books", :_routing=>nil}, 2018-02-21T18:10:14.788Z DB "9200000030322473"|"9789043122849"|"Kids Play Blues"|"https://www.bol.com/nl/p/kids-play-blues/9200000030322473/"|"https://www.bol.com/be/p/kids-play-blues/9200000030322473/"|"https://s.s-bol.com/imgbase0/imagebase3/extralarge/FC/3/7/4/2/9200000030322473.jpg"|""|""|"Kids Play Blues"|""|"19.00"|"0.00"|"new"|"Y"|""|""|"17:00"|"1"|""|"P"|"19.00"|"0.00"|"new"|"Y"|""|""|"00:00"|"7"|""|"P"|"Textual/Printed/Reference Materials"|"Textual/Printed/Reference Materials"|"Boeken"|"Boeken"|"Boek"|"Lezen en Leren"|"Algemene Boeken"|"Nederlandse Boeken"|"Hobby Dutch Books"|"Sport & Hobby Muziek Dutch Books"|"Deze uitgave is geschikt voor gebruik naast de methode Horen, lezen & spelen, maar is ook een geweldig speelboek op zich. In Kids Play Blues staan dertien fantastische bluesstukken."|"Physical"|""|"Bladmuziek"|""|""|"Mailbox"|""|""|"nl"|""|""|""|"J.J.P. Kastelein"|""|"Hal Leonard"|""|""|""|""|""|""|""], :response=>{"index"=>{"_index"=>"books", "_type"=>"bol_books", "_id"=>"AWG5jwsm-_vwXsBg9Rgo", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [OfferNL.releaseDate]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"\""}}}}}

现在发布的是字符串格式。

这是我当前的logstash配置：

input {
        file {
                path => "/home/user/product-feed_general-books.csv"
                start_position => "beginning"
        }
}

filter {
        csv {
                separator => "|"
                columns => ["productId", "Ean", "Title", "productPageUrlNL", "productPageUrlBE", "imageUrl", "mpn", "brand", "familyName",
"subtitle",
"OfferNL.sellingPrice",
"OfferNL.shippingCost",
"OfferNL.condition",
"OfferNL.isDeliverable",
"OfferNL.releaseDate",
"OfferNL.listPrice",
"OfferNL.ultimateOrderTime",
"OfferNL.maximalDeliveryDays",
"OfferNL.fulfillmentLabel",
"OfferNL.sellerType",
"OfferBE.sellingPrice",
"OfferBE.shippingCost",
"OfferBE.condition",
"OfferBE.isDeliverable",
"OfferBE.releaseDate",
"OfferBE.listPrice",
"OfferBE.ultimateOrderTime",
"OfferBE.maximalDeliveryDays",
"OfferBE.fulfillmentLabel",
"OfferBE.sellerType",
"Gpc.segmentName",
"Gpc.familyName",
"Gpc.className",
"Gpc.brickName",
"Gpc.chunkName",
"Category.unit",
"Category.category",
"Category.productgroup",
"Category.subgroup",
"Category.subssubgroup",
"description",
"AppearanceName",
"Format",
"BindingCode",
"CoAuthor",
"Colour",
"DeliveryMethod",
"Gender",
"KindOfGame",
"Language",
"Material",
"Model",
"Actor",
"Author",
"Performer",
"Publisher",
"SecondaryActor",
"SecondaryPerformer",
"ShoeSize",
"SizeClothing",
"SuitableForTypeOfSeason",
"ThirdActor",
"ThirdPerformer",
"Category.unit",
"Category.category",
"Category.productgroup",
"Category.subgroup",
"Category.subssubgroup",
"description",
"AppearanceName",
"Format"]
}
mutate {convert => ["OfferNL.releaseDate", "string"]}
mutate {convert => ["OfferBE.releaseDate", "string"]}

}

output {
        elasticsearch {
                hosts => "localhost"
                index => "books"
                document_type => "bol_books"
        }
        stdout {}
}

在elastsearch中，类型保留在'Date'上：

Answer 1

您正试图将string插入date，这将被拒绝。使用match将字符串转换为正确的日期格式。