Python:如何跳过输入文件中的重复行?

时间:2017-09-24 22:37:04

标签: python python-2.7 python-3.x

我正在从一个文件中读取并且我想单独读取每一行,因为输出中的每个第3行必须是前两行的组合。这是一个小例子:

Input:

<www.example.com/apple> <Anything>
<www.example.com/banana> <Anything>

Output:

<www.example.com/apple> <Anything>
<www.example.com/banana> <Anything>
<Apple> <Banana>

如果任何行重复或者是空行,那么我不想处理它,我想每次只获得2行。

这是我真正意见的一部分:

<http://catalog.data.gov/bread> <http://dbpedia.org> 
<http://catalog.data.gov/bread> <http://dbpedia.org>
<http://catalog.data.gov/bread> <http://dbpedia.org> 
<http://catalog.data.gov/bread> <http://dbpedia.org> 

<http://catalog.data.gov/roll> <http://dbpedia.org>
<http://catalog.data.gov/roll> <http://dbpedia.org>

在这种情况下,我希望输出如下:

 <http://catalog.data.gov/bread> <http://dbpedia.org> 
 <http://catalog.data.gov/roll> <http://dbpedia.org>
 <bread> <roll>

这是我的代码:

file = open('rdfs.txt')
for id, line in enumerate(file):
        if id % 2 == 0:
            if line.isspace():
                continue
            line1 = line.split()
            sub_line1, rel_line1 = line1[0], line1[1]
            sub_line1 = sub_line1.lstrip("<").rstrip(">")
            print(sub_line1)

        else:
            if line.isspace():
                continue
            line2 = line.split()
            sub_line2, rel_line2 = line2[0], line2[1]
            sub_line2 = sub_line2.lstrip("<").rstrip(">")
            print(sub_line2)

它工作得很好,但我得到了所有的线,如果第二行等于之前的行,如何添加,然后跳过所有的行,直到找到一个新的不同的行。

我现在得到的输出:

http://catalog.data.gov/bread
http://catalog.data.gov/bread
http://catalog.data.gov/roll
http://catalog.data.gov/roll

谢谢!

1 个答案:

答案 0 :(得分:1)

您可以声明set()并将其命名为line_seen,其中包含所有已显示的行,并检查lines_seen中是否包含新行,并将其添加到您的支票中:

您的代码应如下所示:

file = open('rdfs.txt')
lines_seen = set() # holds lines already seen
for id, line in enumerate(file):
    if line not in lines_seen: # not a duplicate
        lines_seen.add(line)
        if id % 2 == 0:
            if line.isspace():
                continue
            line1 = line.split()
            sub_line1, rel_line1 = line1[0], line1[1]
            sub_line1 = sub_line1.lstrip("<").rstrip(">")
            print(sub_line1)

        else:
            if line.isspace():
                continue
            line2 = line.split()
            sub_line2, rel_line2 = line2[0], line2[1]
            sub_line2 = sub_line2.lstrip("<").rstrip(">")
            print(sub_line2)