我有这个XML结构：
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE us-patent-application SYSTEM "us-patent-application-v44-2014-04-03.dtd" [ ]>
<us-patent-application lang="EN" dtd-version="v4.4 2014-04-03" file="US20180000001A1-20180104.XML" status="PRODUCTION" id="us-patent-application" country="US" date-produced="20171219" date-publ="20180104">
<us-bibliographic-data-application lang="EN" country="US">
<us-parties>
<inventors>
<inventor sequence="00" designation="us-only">
<addressbook>
<last-name>Evans</last-name>
<first-name>Mike</first-name>
<address>
<city>Emerald Park</city>
<country>CA</country>
</address>
</addressbook>
</inventor>
<inventor sequence="01" designation="us-only">
<addressbook>
<last-name>Lucas</last-name>
<first-name>Lisa</first-name>
<address>
<city>Regina</city>
<country>CA</country>
</address>
</addressbook>
</inventor>
<inventor sequence="02" designation="us-only">
<addressbook>
<last-name>Smith</last-name>
<first-name>John R.</first-name>
<address>
<city>Regina</city>
<country>CA</country>
</address>
</addressbook>
</inventor>
</inventors>
</us-parties>
</us-bibliographic-data-application>
</us-patent-application>
我希望Logstash输出以下结构：
{
"us-patent-application": {
"us-bibliographic-data-application": {
"us-parties": {
"inventors": [
"Mike Evans",
"Lisa Lucas",
"John R. Smith"
]
}
}
}
}
我试图在Logstash中将名称的这种“组合”解决为一个数组，但是找不到有效的解决方案。
到目前为止，我专注于在Logstash红宝石过滤器插件中使用红宝石脚本。我之所以使用这种方法，是因为我无法在Logstash XML过滤器中使用Xpath找到有效的解决方案。
这是Logstash'main.conf'配置文件：
input {
file {
path => [
"/opt/uspto/*.xml"
]
start_position => "beginning"
#use for testing
sincedb_path => "/dev/null"
# set this sincedb path when not testing
#sincedb_path => "/opt/logstash/tmp/sincedb"
exclude => "*.gz"
type => "xml"
codec => multiline {
#pattern => "<wo-ocr-published-application"
pattern => "<?xml version=\"1.0\" encoding=\"UTF-8\"\?>"
negate => "true"
what => "previous"
max_lines => 300000
}
}
}
filter {
if "multiline" in [tags] {
xml {
source => "message"
#store_xml => false # this limits the data indexed to only xpath and grok created fields
store_xml => true #saves ALL xml nodes if it can - can be VERY large
target => "xmldata" # only used with store_xml => true
}
ruby {
path => "/etc/logstash/rubyscripts/inventors.rb"
}
}
}
output {
file {
path => [ "/tmp/logstash_output_text_file" ]
codec => rubydebug
}
}
这是creator.rb脚本：
# the value of `params` is the value of the hash passed to `script_params`
# in the logstash configuration
def register(params)
@drop_percentage = params["percentage"]
end
def filter(event)
# get the number of inventors to loop over
# convert the array key string number 0 to an integer
n = event.get('[num_inventors][0]').to_i
# set a loop number to start with
i = 0
#create empty arrays to fill
firstname = []
lastname = []
# loop over inventors until n is reached
while (i < n) do
#get the inventors first name
fname = event.get('[event][us-patent-application][us-bibliographic-data-application][us-parties][inventors][inventor][addressbook][last-name]')
#puts"first name #{fname}"
# push the first name into firstname array
firstname.push(fname)
#get the inventors last name
lname = event.get('[event][us-patent-application][us-bibliographic-data-application][us-parties][inventors][inventor][addressbook][last-name]')
#puts"last name #{lname}"
# push the last name into firstname array
lastname.push(lname)
#increment n up 1
i += 1
end
#merge firstname and lastname arrays
names = firstname.zip(lastname)
# push the names array to the event
event.set('allnames', names)
return [event]
end
最后，这是Elasticsearch的输出：
{
"host" => "localhost.localdomain",
"allnames" => [],
"type" => "xml",
"@timestamp" => 2018-09-20T17:28:05.332Z,
"message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<!DOCTYPE us-patent-application SYSTEM \"us-patent-application-v44-2014-04-03.dtd\" [ ]>\r\n<us-patent-application lang=\"EN\" dtd-version=\"v4.4 2014-04-03\" file=\"US20180000001A1-20180104.XML\" status=\"PRODUCTION\" id=\"us-patent-application\" country=\"US\" date-produced=\"20171219\" date-publ=\"20180104\">\r\n\t<us-bibliographic-data-application lang=\"EN\" country=\"US\">\r\n\t\t<us-parties>\r\n\t\t\t<inventors>\r\n\t\t\t\t<inventor sequence=\"00\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Evans</last-name>\r\n\t\t\t\t\t\t<first-name>Mike</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Emerald Park</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t\t<inventor sequence=\"01\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Lucas</last-name>\r\n\t\t\t\t\t\t<first-name>Lisa</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Regina</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t\t<inventor sequence=\"02\" designation=\"us-only\">\r\n\t\t\t\t\t<addressbook>\r\n\t\t\t\t\t\t<last-name>Smith</last-name>\r\n\t\t\t\t\t\t<first-name>Scott R.</first-name>\r\n\t\t\t\t\t\t<address>\r\n\t\t\t\t\t\t\t<city>Regina</city>\r\n\t\t\t\t\t\t\t<country>CA</country>\r\n\t\t\t\t\t\t</address>\r\n\t\t\t\t\t</addressbook>\r\n\t\t\t\t</inventor>\r\n\t\t\t</inventors>\r\n\t\t</us-parties>\r\n\t</us-bibliographic-data-application>\r",
"@version" => "1",
"tags" => [
[0] "multiline",
[1] "_xmlparsefailure"
],
"path" => "/opt/uspto/test.xml"
}
我正在寻找具有“ allnames” => []数组的行为，如下所示：
"allnames" => ["Mike Evans",
"Lisa Lucas",
"John R. Smith"],
我无法弄清楚如何在我的ruby脚本中正确获取“名”和“名”节点。我正在努力寻找解决方案。任何想法都欢迎！
答案 0 :(得分：0)
请查看这是否满足您的要求。我真的希望可以通过xml + xpath解决此问题。但是我猜不支持所有的xpath函数。 :(
input {
file {
path => [
"/opt/uspto/*.xml"
]
start_position => "beginning"
#use for testing
sincedb_path => "/dev/null"
# set this sincedb path when not testing
#sincedb_path => "/opt/logstash/tmp/sincedb"
exclude => "*.gz"
type => "xml"
codec => multiline {
#pattern => "<wo-ocr-published-application"
pattern => "<?xml version=\"1.0\" encoding=\"UTF-8\"\?>"
negate => "true"
what => "previous"
max_lines => 300000
}
}
}
filter {
if "multiline" in [tags] {
xml {
source => "message"
store_xml => false
target => "xmldata" # only used with store_xml => true
force_array=> false
xpath => [ "//us-bibliographic-data-application/us-parties/inventors/inventor/addressbook/first-name/text()" , "[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]" ,
"//us-bibliographic-data-application/us-parties/inventors/inventor/addressbook/last-name/text()", "[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]" ]
}
ruby {
code => ' first_name = event.get("[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]")
last_name = event.get("[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]")
event.set("[xmldata][us-bibliographic-data-application][us-parties][inventors][names]", first_name.zip(last_name).map{ |a| a.join(" ") })
'
}
mutate {
remove_field => ["message", "host", "path", "[xmldata][us-bibliographic-data-application][us-parties][inventors][first_name]", "[xmldata][us-bibliographic-data-application][us-parties][inventors][last_name]" ]
}
}
}
output {
file {
path => [ "/tmp/logstash_output_text_file" ]
codec => rubydebug
}
}
输出为
{
"@timestamp": "2018-09-21T12:00:26.428Z",
"@version": "1",
"tags": [
"multiline"
],
"type": "xml",
"xmldata": {
"us-bibliographic-data-application": {
"us-parties": {
"inventors": {
"names": [
"Mike Evans",
"Lisa Lucas",
"John R. Smith"
]
}
}
}
}
}