Question

我想知道是否有人在java中有逻辑删除重复行，同时保持行顺序。

我不希望没有正则表达式解决方案。

Answer 1

public class UniqueLineReader extends BufferedReader {
    Set<String> lines = new HashSet<String>();

    public UniqueLineReader(Reader arg0) {
        super(arg0);
    }

    @Override
    public String readLine() throws IOException {
        String uniqueLine;
        if (lines.add(uniqueLine = super.readLine()))
            return uniqueLine;
        return "";
    }

  //for testing.. 

    public static void main(String args[]) {
        try {
            // Open the file that is the first
            // command line parameter
            FileInputStream fstream = new FileInputStream(
                    "test.txt");
            UniqueLineReader br = new UniqueLineReader(new InputStreamReader(fstream));
            String strLine;
            // Read File Line By Line
            while ((strLine = br.readLine()) != null) {
                // Print the content on the console
                if (strLine != "")
                    System.out.println(strLine);
            }
            // Close the input stream
            in.close();
        } catch (Exception e) {// Catch exception if any
            System.err.println("Error: " + e.getMessage());
        }
    }

}

修改版本：

public class UniqueLineReader extends BufferedReader {
    Set<String> lines = new HashSet<String>();

    public UniqueLineReader(Reader arg0) {
        super(arg0);
    }

    @Override
    public String readLine() throws IOException {
        String uniqueLine;
        while (lines.add(uniqueLine = super.readLine()) == false); //read until encountering a unique line
            return uniqueLine;
    }

    public static void main(String args[]) {
        try {
            // Open the file that is the first
            // command line parameter
            FileInputStream fstream = new FileInputStream(
                    "/home/emil/Desktop/ff.txt");
            UniqueLineReader br = new UniqueLineReader(new InputStreamReader(fstream));
            String strLine;
            // Read File Line By Line
            while ((strLine = br.readLine()) != null) {
                // Print the content on the console
                    System.out.println(strLine);
            }
            // Close the input stream
            in.close();
        } catch (Exception e) {// Catch exception if any
            System.err.println("Error: " + e.getMessage());
        }

    }
}

Answer 2

如果你将这些行输入LinkedHashSet，它会忽略重复的行，因为它是一个集合，但保留了顺序，因为它是链接的。如果你只是想知道你之前是否看过一个给定的行，请在继续时将它们提供给一个简单的Set，并忽略那些已经包含/包含的那些。

Answer 3

使用BufferedReader读取文本文件并将其存储在LinkedHashSet中。打印出来。

以下是一个例子：

public class DuplicateRemover {

    public String stripDuplicates(String aHunk) {
        StringBuilder result = new StringBuilder();
        Set<String> uniqueLines = new LinkedHashSet<String>();

        String[] chunks = aHunk.split("\n");
        uniqueLines.addAll(Arrays.asList(chunks));

        for (String chunk : uniqueLines) {
            result.append(chunk).append("\n");
        }

        return result.toString();
    }

}

这是一些要验证的单元测试（忽略我的邪恶复制粘贴;））：

import org.junit.Test;
import static org.junit.Assert.*;

public class DuplicateRemoverTest {

    @Test
    public void removesDuplicateLines() {
        String input = "a\nb\nc\nb\nd\n";
        String expected = "a\nb\nc\nd\n";

        DuplicateRemover remover = new DuplicateRemover();

        String actual = remover.stripDuplicates(input);
        assertEquals(expected, actual);
    }

    @Test
    public void removesDuplicateLinesUnalphabetized() {
        String input = "z\nb\nc\nb\nz\n";
        String expected = "z\nb\nc\n";

        DuplicateRemover remover = new DuplicateRemover();

        String actual = remover.stripDuplicates(input);
        assertEquals(expected, actual);
    }

}

Answer 4

这是另一种解决方案。我们只使用UNIX！

cat MyFile.java | uniq > MyFile.java

编辑：哦等等，我重新阅读了这个主题。这是一个合法的解决方案，因为我设法与语言无关吗？

Answer 5

使用新的java Stream API可以很容易地从文本或文件中删除重复的行。 Stream支持不同的聚合特性，如sort，distinct和使用不同java的现有数据结构及其方法。以下示例可用于使用Stream API删除文件中的内容重复或排序

package removeword;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.OpenOption;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.Scanner;
import java.util.stream.Stream;
import static java.nio.file.StandardOpenOption.*;
import static java.util.stream.Collectors.joining;

public class Java8UniqueWords {

public static void main(String[] args) throws IOException {        
    Path sourcePath = Paths.get("C:/Users/source.txt");
    Path changedPath = Paths.get("C:/Users/removedDouplicate_file.txt");
      try (final Stream<String> lines = Files.lines(sourcePath )
               // .map(line -> line.toLowerCase()) /*optional to use existing string methods*/
                .distinct()
               // .sorted())  /*aggregrate function to sort  disctincted line*/
       {
            final String uniqueWords = lines.collect(joining("\n"));
            System.out.println("Final Output:" + uniqueWords);
            Files.write(changedPath , uniqueWords.getBytes(),WRITE, TRUNCATE_EXISTING);
        }
}
}

Answer 6

要获得更好/最佳的性能，明智的做法是使用 Java 8的 API功能。带有 LinkedHashSet 的流和方法引用，如下所示：

import java.io.IOException;
import java.io.PrintWriter;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.LinkedHashSet;
import java.util.stream.Collectors;

public class UniqueOperation {

private static PrintWriter pw;  
enter code here
public static void main(String[] args) throws IOException {

    pw = new PrintWriter("abc.txt");

    for(String p : Files.newBufferedReader(Paths.get("C:/Users/as00465129/Desktop/FrontEndUdemyLinks.txt")).
                   lines().
                   collect(Collectors.toCollection(LinkedHashSet::new))) 
        pw.println(p);
    pw.flush();
    pw.close();

    System.out.println("File operation performed successfully");
}

Answer 7

这里我使用哈希集来存储看到的行

Scanner scan;//input
Set<String> lines = new HashSet<String>();
StringBuilder strb = new StringBuilder();
while(scan.hasNextLine()){
    String line = scan.nextLine();
    if(lines.add(line)) strb.append(line);
}

使用Java从文本中删除重复行

7 个答案: