Question

我从一个需要完成这些事情的论坛中遇到了这个问题：您将获得一系列段落，并且必须过滤掉其文本（以空格分隔的单词序列）完全包含在其中一个或多个其他段落的子段落中的任何段落。

在比较遏制时，必须遵守某些规则：应忽略字母字符的情况应忽略前导和尾随空格任何其他连续空格块应视为单个空格应忽略非字母数字字符，应保留空格还必须过滤重复项 - 如果相对于上面列出的比较规则认为两个段落相等，则只保留最短的段落。如果它们的长度也相同，则应保留输入序列中较早的长度。保留的段落应以其原始形式输出（与输入段落相同），并以相同的顺序输出。

输入1：IBM认知计算| IBM“认知”计算是一场革命| ibm认知计算|'IBM认知计算'是一场革命？

输出1：IBM“认知”计算是一场革命

输入2：IBM认知计算| IBM“认知”计算是一场革命|认知计算是一场革命

输出2：IBM“认知”计算是一场革命|认知计算是一场革命

我在python中编写了以下代码，但它给了我一些其他输出而不是第一个测试用例：

f = open("input.txt",'r')
s = (f.read()).split('|')
str = ''
for a in s:
    for b in s:
        if(''.join(e for e in a.lower() if e.isalnum()))not in (''.join(e for e in b.lower() if e.isalnum())):
            str = a.translate(None, "'?")

print str

input.txt包含第一个测试用例输入。我得到的输出为： IBM认知计算是一场革命。有人可以请进来帮助我。感谢

Answer 1

我确实为你编写了这个，希望它很容易理解（我这样做不太理想）。如果你需要加快速度或者有任何问题，请告诉我们！

BTW，这是python 3，只需从print语句中删除括号，它就会复制并粘贴并运行python 2。

import re

def clean_input(text):
    #non-alphanumeric character should be ignored
    text = re.sub('[^a-zA-Z\s]', '', text)
    #Any other block of contiguous whitespace should be treated as a single space
    #white space should be retained
    text = re.sub(' +',' ',text)
    #Leading and trailing whitespace should be ignored
    text = text.strip(' \t\n\r')
    # You probably want this too
    text = text.lower()
    return text

def process(text):
    #If they are also the same length, the earlier one in the input sequence should be kept.
    # Using arrays (can use OrderedDict too, probably easier and nice, although below is clearer for you.
    original_parts = text.split('|')
    clean_parts = [clean_input(x) for x in original_parts]
    original_parts_to_check = []
    ignore_idx = []
    for idx, ele in enumerate(original_parts):
        if idx in ignore_idx:
            continue
        #The case of alphabetic characters should be ignored
        if len(ele) < 2:
            continue
        #Duplicates must also be filtered -if two passages are considered equal with respect to the comparison rules listed above, only the shortest should be retained.
        if clean_parts[idx] in clean_parts[:idx]+clean_parts[idx+1:]:
            indices = [i for i, x in enumerate(clean_parts) if x == clean_parts[idx]]
            use = indices[0]
            for i in indices[1:]:
                if len(original_parts[i]) < len(original_parts[use]):
                    use = i
            if idx == use:
                ignore_idx += indices
            else:
                ignore_idx += [x for x in indices if x != use]
                continue
        original_parts_to_check.append(idx)
    # Doing the text in text matching here. Depending on size and type of dataset,
    # Which you should test as it would affect this, you may want this as part
    # of the function above, or before, etc. If you're only doing 100 bits of
    # data, then it doesn't matter. Optimize accordingly.
    text_to_return = []
    clean_to_check = [clean_parts[x] for x in original_parts_to_check]
    for idx in original_parts_to_check:
        # This bit can be done better, but I have no more time to work on this.
        if any([(clean_parts[idx] in clean_text) for clean_text in [x for x in clean_to_check if x != clean_parts[idx]]]):
            continue
        text_to_return.append(original_parts[idx])
    #The retained passages should be output in their original form (identical to the input passage), and in the same order.
    return '|'.join(text_to_return)

assert(process('IBM cognitive computing|IBM "cognitive" computing is a revolution| ibm cognitive computing|\'IBM Cognitive Computing\' is a revolution?') ==
       'IBM "cognitive" computing is a revolution')
print(process('IBM cognitive computing|IBM "cognitive" computing is a revolution| ibm cognitive computing|\'IBM Cognitive Computing\' is a revolution?'))
assert(process('IBM cognitive computing|IBM "cognitive" computing is a revolution|the cognitive computing is a revolution') ==
       'IBM "cognitive" computing is a revolution|the cognitive computing is a revolution')
print(process('IBM cognitive computing|IBM "cognitive" computing is a revolution|the cognitive computing is a revolution'))

另外，如果这对你有所帮助，那么获得一些积分会很棒，所以接受会很好:)（我看到你是新来的）。

迭代通过字符串extrction的段落

1 个答案: