Question

说我有100条推文在那些推文中，我需要提取：1）食品名称，以及2）饮料名称

推文示例：

“昨天我吃了可口可乐，还有一个热狗吃午饭，还有一些bana为沙漠分开。我喜欢可乐，但是香蕉分开的甜点中的香蕉已经成熟了”

我必须处理两个词典。一个有食物名称，一个有饮料名称。

食物名称词典中的例子：
“热狗”
“香蕉”
“香蕉分裂”

饮料名称词典中的示例：
“焦炭”
“可乐”
“可口可乐”

我应该提取的内容：

[[[“coca cola”，“饮料”]，[“热狗”，“食物”]，[“香蕉分开”，“食物”]]，
[[“可乐”，“饮料”]，[“香蕉”，“食物”]，[“香蕉分裂”，“食物”]]]

词典中的名称长度可以是1-5个字。如何使用我的词典从推文中提取n-gram？

Answer 1

这是一个简单的解决方案：

break

split_sentences函数取自此处：Splitting a sentence by ending characters

Answer 2

不确定到目前为止您尝试了什么，下面是使用ngrams中的nltk和dict()

的解决方案

from nltk import ngrams

tweet = "Yesterday I had a coca cola, and a hot dog for lunch, and some bana split for desert. I liked the coke, but the banana in the banana split dessert was ripe"

# Your lexicons
lexicon_food = ["hot dog", "banana", "banana split"]
lexicon_beverage = ["coke", "cola", "coca cola"]
lexicon_dict = {x: [x, 'Food'] for x in lexicon_food}
lexicon_dict.update({x: [x, 'Beverage'] for x in lexicon_beverage})

# Function to extract lexicon items
def extract(g, lex):
    if ' '.join(g) in lex.keys():
        return lex.get(' '.join(g))
    elif g[0] in lex.keys():
        return lex.get(g[0])
    else:
        pass

# Your task
out = [[extract(g, lexicon_dict) for g in ngrams(sentence.split(), 2) if extract(g, lexicon_dict)] 
        for sentence in tweet.replace(',', '').lower().split('.')]
print(out)

输出：

[[['coca cola', 'Beverage'], ['cola', 'Beverage'], ['hot dog', 'Food']], 
 [['coke', 'Beverage'], ['banana', 'Food'], ['banana split', 'Food']]]

方法2 （避免“可口可乐”和“可乐”）

def extract2(sentence, lex):
    extracted_words = []
    words = sentence.split()
    i = 0
    while i < len(words):
        if ' '.join(words[i:i+2]) in lex.keys():
            extracted_words.append(lex.get(' '.join(words[i:i+2])))
            i += 2
        elif words[i] in lex.keys():
            extracted_words.append(lex.get(words[i]))
            i += 1
        else:
            i += 1
    return extracted_words

out = [extract2(s, lexicon_dict) for s in tweet.replace(',', '').lower().split('.')]
print(out)

输出：

[[['coca cola', 'Beverage'], ['hot dog', 'Food']], 
 [['coke', 'Beverage'], ['banana', 'Food'], ['banana split', 'Food']]]

注意此处不需要nltk。

从python

2 个答案: