Question

我们在Java中使用Lucene搜索文档并查明它们是否相关。我们以6种不同的方式进行搜索：

VSM与Porter词干和停用词的相似性
VSM与Porter stemmer的相似性，没有停用词
VSM与标准词干和停用词的相似性
BM25与Porter词干和停用词的相似性
BM25与Porter stemmer的相似性，没有停用词
BM25与标准词干和停用词的相似性

搜索配置3和6的结果相同，配置1,2,4和5的结果也相同。这表明只有改变分析仪（词干分析器）才能改变任何东西。

我们已经尝试调试它以检查对象是否符合我们的预期，但一切似乎都是有序的 - 只是对象的行为与我们希望的不同。我们还记得在索引和搜索时使用相同的相似性。

我们做错了什么？我们是否遗漏了一些代码来申请＆＃39;配置是否正确？

    public IndexWriterConfig index(List<DocumentInCollection> docs) throws IOException 
    {

        Analyzer analyz;
        IndexWriterConfig config;

        if (analyzer.equals("vsm") && stopwords && stemmer) 
        {
            //VSM cosine similarity with TFIDF + stopwords + stemmer
            CharArraySet stopWords = EnglishAnalyzer.getDefaultStopSet();
            analyz = new EnglishAnalyzer(stopWords);
            config = new IndexWriterConfig(analyz);
            config.setSimilarity(new ClassicSimilarity());
        } 
        else if (analyzer.equals("vsm") && !stopwords && stemmer) 
        {
            //VSM cosine similarity with TFIDF - stopwords + stemmer
            analyz = new EnglishAnalyzer(CharArraySet.EMPTY_SET);
            config = new IndexWriterConfig(analyz);
            config.setSimilarity(new ClassicSimilarity());
        } 
        else if (analyzer.equals("vsm") && stopwords && !stemmer) 
        {
            //VSM cosine similarity with TFIDF - stopwords - stemmer
            CharArraySet stopWords = StandardAnalyzer.STOP_WORDS_SET;
            analyz = new StandardAnalyzer(stopWords);
            config = new IndexWriterConfig(analyz);
            config.setSimilarity(new ClassicSimilarity());
        } 
        else if (analyzer.equals("bm25") && stopwords && stemmer) 
        {
            //Analyzer + stopwords + stemmer
            CharArraySet stopWords = EnglishAnalyzer.getDefaultStopSet();
            analyz = new EnglishAnalyzer(stopWords);
            config = new IndexWriterConfig(analyz);
            //BM25 ranking method
            config.setSimilarity(new BM25Similarity());
        } 
        else if (analyzer.equals("bm25") && !stopwords && stemmer) 
        {
            //Analyzer - stopwords + stemmer
            analyz = new EnglishAnalyzer(CharArraySet.EMPTY_SET);
            config = new IndexWriterConfig(analyz);
            //BM25 ranking method
            config.setSimilarity(new BM25Similarity());
        } 
        else if (analyzer.equals("bm25") && stopwords && !stemmer) 
        {
            //Analyzer + stopwords - stemmer
            CharArraySet stopWords = StandardAnalyzer.STOP_WORDS_SET;
            analyz = new StandardAnalyzer(stopWords);
            config = new IndexWriterConfig(analyz);
            //BM25 ranking method
            config.setSimilarity(new BM25Similarity());
        }
        else 
        {
            //some default
            analyz = new StandardAnalyzer();
            config = new IndexWriterConfig(analyz);
            config.setSimilarity(new ClassicSimilarity());
        }


        IndexWriter w = new IndexWriter(corpus, config);

        //total 153 documents with group 5
        for (DocumentInCollection doc1 : docs) {
            if (doc1.getSearchTaskNumber() == 5) {
                Document doc = new Document();
                doc.add(new TextField("title", doc1.getTitle(), Field.Store.YES));
                doc.add(new TextField("abstract_text", doc1.getAbstractText(), Field.Store.YES));
                doc.add(new TextField("relevance", Boolean.toString(doc1.isRelevant()), Field.Store.YES));
                w.addDocument(doc);
                totalDocs++;
                if (doc1.isRelevant()) relevantDocs++;
            }
        }

        w.close();

        return config;
    }

    public List<String> search(String searchQuery, IndexWriterConfig cf) throws IOException {

        printQuery(searchQuery);

        List<String> results = new LinkedList<String>();


        //Constructing QueryParser to stem search query
        QueryParser qp = new QueryParser("abstract_text", cf.getAnalyzer());
        Query stemmedQuery = null;
        try {
            stemmedQuery = qp.parse(searchQuery);
        } catch (ParseException e) {
            e.printStackTrace();
        }



        // opening directory for search
        IndexReader reader = DirectoryReader.open(corpus);
        // implementing search over IndexReader
        IndexSearcher searcher = new IndexSearcher(reader);

        searcher.setSimilarity(cf.getSimilarity());

        // finding top totalDocs documents qualifying the search
        TopDocs docs = searcher.search(stemmedQuery, totalDocs);

        // representing array of hits from TopDocs
        ScoreDoc[] scored = docs.scoreDocs;

        // adding matched doc titles to results
        for (ScoreDoc aDoc : scored) {
            Document d = searcher.doc(aDoc.doc);
            retrieved++;
            //relevance and score are printed out for debug purposes
            if (d.get("relevance").equals("true")) {
                relevantRetrieved++;
                results.add("+ " + d.get("title") + " | relevant: " + d.get("relevance") + " | score: " + aDoc.score);
            } else {
                results.add("- " + d.get("title") + " | relevant: " + d.get("relevance") + " | score: " + aDoc.score);
            }

        }


        return results;
    }

Answer 1

首先，您通常不会期望BM25和Classic Similarities返回不同的结果集，只是不同的分数（从而排序）。通常，相似性决定了如何计算已经找到与查询匹配的文档的分数。它们通常会返回相同的结果，但分数不同，因此顺序不同。

如果您使用bm25和vsm设置看到相同的分数，那么肯定会出现问题。但是，基于我的精简版，可运行的测试版本，您的代码看起来对我来说没问题：https://gist.github.com/anonymous/baf279806702edb54fab23db6d8d19b9

StopWord过滤器通常不是那么大的变化。它控制是否将停用词编入索引。停用词是“the”和“this”之类的词。使用停用词过滤器，它们不会被编入索引，也无法搜索。除非你正在寻找一个停用词，否则差异通常不会很明显。同样，这个似乎根据我的测试版本正常工作。

在Lucene中使用不同的相似性获得相同的结果

1 个答案: