我目前正在尝试计算lucene索引中术语的tf-idf矩阵。 我尝试使用以下函数执行此操作:
public Table<Integer, BytesRef, Double> tfidf(String field) throws IOException, ParseException{
//variables in complete context
int totalNoOfDocs = reader.numDocs(); //total no of docs
HashBasedTable<Integer, BytesRef, Double> tfidfPerDocAndTerm = HashBasedTable.create(); //tfidf value for each document(integer) and term(Byteref) pair.
//variables in loop context
BytesRef term; //term as BytesRef
int noOfDocs; //number of documents (a term occours in)
int tf; //term frequency (of a term in a doc)
double idf; //inverse document frequency (of a term in a doc)
double tfidf; //term frequency - inverse document frequency value (of a term in a doc)
Terms termVector; //all terms of current doc in current field
TermsEnum termsEnum; //iterator for terms
DocsEnum docsEnum; //iterator for documents (of current term)
List<Integer> docIds = getDocIds(totalNoOfDocs); //get internal documentIds of documents
try {
for(int doc : docIds){
termVector = reader.getTermVector(doc, field); //get termvector for document
termsEnum = termVector.iterator(null); //get iterator of termvector to iterate over terms
while((term = termsEnum.next()) != null){ //iterate of terms
noOfDocs = termsEnum.docFreq(); //add no of docs the term occurs in to list
docsEnum = termsEnum.docs(null, null); //get document iterator for this term (all documents the term occours in)
while((doc = docsEnum.nextDoc()) != DocIdSetIterator.NO_MORE_DOCS){ //iterate over documents - computation of all tf-idf values for this term
tf = docsEnum.freq(); //get termfrequency of current term in current doc
idf = Math.log((double)totalNoOfDocs / (double)noOfDocs); //calculate idf
tfidf = (double) tf * idf; //caculate tfidf
tfidfPerDocAndTerm.put(doc, term, tfidf); //add tf-idf value to matrix
}
}
}
} catch (IOException ex) {
Logger.getLogger(Index.class.getName()).log(Level.SEVERE, null, ex);
}
return tfidfPerDocAndTerm;
}
问题是:noOfDocs = termsEnum.docFreq();总是返回1.即使有明显的术语出现在多个文档中(通过打印“术语”手动检查)。
我还发现,docsEnum我检索的是:docsEnum = termsEnum.docs(null,null);总是只包含1个文档(doc 0)。
创建索引时,我使用带有停用词列表的标准分析器,因此所有术语都是小写的。
那么我的问题是什么? :/
感谢您的帮助!
答案 0 :(得分:0)
实际上,你的术语是BytesRef类型,是循环而不是你的termsenums,但不幸的是,BytesRef不支持一个名为freq()或docfreq()的方法
答案 1 :(得分:0)
实际上,枚举器总是返回1.但是你可以使用CollectionStatistics
得到正确的值:
DefaultSimilarity similarity = new DefaultSimilarity();
IndexReader reader = searcher.getIndexReader();
IndexReaderContext context = searcher.getTopReaderContext();
CollectionStatistics collectionStats = searcher.collectionStatistics(FIELD);
long totalDocCount = collectionStats.docCount();
Terms termVector = reader.getTermVector(docId, FIELD);
TermsEnum iterator = termVector.iterator(null);
while (true) {
BytesRef ref = iterator.next();
if (ref == null) {
break;
}
long termFreq = iterator.totalTermFreq();
float tf = similarity.tf(termFreq);
Term term = new Term(FIELD, ref);
TermContext termContext = TermContext.build(context, term);
TermStatistics termStats = searcher.termStatistics(term, termContext);
long docFreq = termStats.docFreq();
float idf = similarity.idf(docFreq, totalDocCount);
// do something with tf and idf
}
请注意,要使其工作,您需要在索引中存储术语向量。