Question

我正在尝试从停用词列表中删除txt文件中的停用词。有些停用词已被删除，有些则没有。

举例说明：“味道不错，不是吗？”应该有一个类似“ taste nice”的输出，但是我的代码输出是：“ taste nice没它”

我的停用词列表来自：https://www.ranks.nl/stopwords（长期停用词列表）。

这是我的代码：

    public static void main(String[] args) {

    ArrayList sw = new ArrayList<>();

    try{
        FileInputStream fis = new FileInputStream("/Users/Dan/Desktop/DATA/stopwords.txt");

        byte b[] = new byte[fis.available()];
        fis.read(b);
            fis.close();

            String data[] = new String(b).split("\n");

        for(int i = 0; i < data.length; i++)
        {
            sw.add(data[i].trim());
        }
         FileInputStream fis2 = new FileInputStream("/Users/Dan/Desktop/DATA/cleandata.txt");

        byte bb[] = new byte[fis2.available()];
        fis2.read(bb);
            fis2.close();

            String data2[] = new String(bb).split("\n");



            for(int i = 0; i < data2.length; i++)

        {
            String file = "";
            String s[] = data2[i].split("\\s");
            for(int j = 0; j < s.length; j++)
            {
                if(!(sw.contains(s[j].trim().toLowerCase())))
                {
                    file=file + s[j] + " ";
                }

            }
            file = file.replaceAll("[^a-zA-Z\\s+]", "");

            System.out.println(file.replaceAll("\\s+", " ").toLowerCase() + "\n");

        }

    } catch(Exception a){
        a.printStackTrace();
    }

}

我该怎么办？我认为我在打印时遇到问题

file = file.replaceAll("[^a-zA-Z\\s+]", "");

System.out.println(file.replaceAll("\\s+", " ").toLowerCase() + "\n");

Answer 1

使用了两种不同的引号字符。停用词文件包含doesn't，而您的输入包含doesn’t。

因为引号不同，所以单词不匹配。

编辑：这是一个稍微重构的解决方案，它会生成正确的输出（如果您在输入中未使用奇怪的引号）。

import java.util.Arrays;
import java.util.HashSet;
import java.util.Scanner;
import java.util.Set;
import java.util.stream.Collectors;

public class StopWordsApp {

    // the platform-specific end of line token
    private static final String EOL = String.format("%n");

    private final Set<String> stopWords = new HashSet<>(Arrays.asList(readLines("stopwords.txt")));

    public static void main(String[] args) {
        StopWordsApp stopWordsApp = new StopWordsApp();
        String[] lines = readLines("cleandata.txt");
        printLines(stopWordsApp.removeStopWords(lines));
    }

    private String[] removeStopWords(String[] inputLines) {
        return Arrays.stream(inputLines)
                // map the String array to a Line object
                .map(Line::new)
                // map the Line to a String without stop words
                .map(this::removeStopWords)
                // convert the stream to an array
                .toArray(String[]::new);
    }

    private String removeStopWords(Line line) {
        return line.words().stream()
                // map the word to its normalized version
                .map(Word::normalized)
                // remove stop words
                .filter(n -> !stopWords.contains(n))
                // join into a String separated by spaces
                .collect(Collectors.joining(" "));
    }

    private static String[] readLines(String fileName) {
        return readFile(fileName).split(EOL);
    }

    private static String readFile(String fileName) {
        return new Scanner(StopWordsApp.class.getResourceAsStream(fileName), "UTF-8").useDelimiter("\\A").next();
    }

    private static void printLines(String[] lines) {
        for (String line : lines) {
            System.out.println(line);
        }
    }
}

我为Line提取了单独的类：

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class Line {

    private final List<Word> words;

    public Line(String input) {
        String[] wordInputs = input.split("\\s+");
        words = Arrays.stream(wordInputs)
                // remove empty Strings
                .filter(v -> !v.isEmpty())
                // map String to a Word object
                .map(Word::new)
                // collect into a List
                .collect(Collectors.toList());
    }

    public List<Word> words() {
        return words;
    }

}

..和一个单词：

public class Word {

    private final String normalized;

    public Word(String input) {
        normalized = input
                // convert to lower case
                .toLowerCase()
                // remove everything that's not a lower case letter or a quote
                // (the stopwords file only contains lower case letters and quotes)
                .replaceAll("[^a-z']", "")
                // replace consecutive white space with a single space
                .replaceAll("\\s+", " ")
                // trim any white space at the edges
                .trim();
    }

    public String normalized() {
        return normalized;
    }

}

...和一个自定义（运行时）异常：

public class StopWordsException extends RuntimeException {
    public StopWordsException(Exception e) {
        super(e);
    }
}

我到处都使用Java 8流，并添加了注释以说明正在发生的事情。

使用输入：

it Taste nice, doesn't it?

输出为：

taste nice

P.S。文件'stopwords.txt'和'cleandata.txt'必须与StopWordsApp类位于同一软件包中。

如何删除带有非单词边界的停用词？

1 个答案: