我有一个程序接受输入查询并根据其TFIDF分数对类似文档进行排名。问题是，我想添加一些关键字并将它们视为＆＃34;输入＆＃34;同样。每个查询的关键字都不同。
例如，如果查询为"Logic Based Knowledge Representation"，则单词如下：
Level 0 keywords: [logic, base, knowledg, represent]
Level 1 keywords: [tempor, modal, logic, resolut, method, decis, problem,
reason, revis, hybrid, represent]
Level 2 keywords: [classif, queri, process, techniqu, candid, semant, data,
model, knowledg, base, commun, softwar, engin, subsumpt,
kl, undecid, classic, structur, object, field]
我想以不同的方式对待得分，例如，对于文档中与第0级中的单词相等的术语，我想将得分乘以1.对于文档中与术语1中的单词相等的术语，乘以得分为0.8。最后，对于文档中的术语，等于2级中的单词，将得分乘以0.64。
我的目的是扩展输入查询，同时确保包含0级以上关键字的文档更重要，包含1级和2级关键字的文档更少（即使输入已扩展）。 我没有把它包含在我的程序中。到目前为止，我的程序只计算查询中所有文档的TFIDF分数，并对结果进行排名：
public class Ranking{
private static int maxHits = 2000000;
public static void main(String[] args) throws Exception {
System.out.println("Enter your paper title: ");
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String paperTitle = null;
paperTitle = br.readLine();
// CitedKeywords ckeywords = new CitedKeywords();
// ckeywords.readDataBase(paperTitle);
String querystr = args.length > 0 ? args[0] :paperTitle;
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_42);
Query q = new QueryParser(Version.LUCENE_42, "title", analyzer)
.parse(querystr);
IndexReader reader = DirectoryReader.open(
FSDirectory.open(
new File("E:/Lucene/new_bigdataset_index")));
IndexSearcher searcher = new IndexSearcher(reader);
VSMSimilarity vsmSimiliarty = new VSMSimilarity();
searcher.setSimilarity(vsmSimiliarty);
TopDocs hits = searcher.search(q, maxHits);
ScoreDoc[] scoreDocs = hits.scoreDocs;
PrintWriter writer = new PrintWriter("E:/Lucene/result/1.txt", "UTF-8");
int counter = 0;
for (int n = 0; n < scoreDocs.length; ++n) {
ScoreDoc sd = scoreDocs[n];
float score = sd.score;
int docId = sd.doc;
Document d = searcher.doc(docId);
String fileName = d.get("title");
String year = d.get("pub_year");
String paperkey = d.get("paperkey");
System.out.printf("%s,%s,%s,%4.3f\n", paperkey, fileName, year, score);
writer.printf("%s,%s,%s,%4.3f\n", paperkey, fileName, year, score);
++counter;
}
writer.close();
}
}
-
import org.apache.lucene.index.FieldInvertState;
import org.apache.lucene.search.similarities.DefaultSimilarity;
public class VSMSimilarity extends DefaultSimilarity{
// Weighting codes
public boolean doBasic = true; // Basic tf-idf
public boolean doSublinear = false; // Sublinear tf-idf
public boolean doBoolean = false; // Boolean
//Scoring codes
public boolean doCosine = true;
public boolean doOverlap = false;
private static final long serialVersionUID = 4697609598242172599L;
// term frequency in document =
// measure of how often a term appears in the document
public float tf(int freq) {
// Sublinear tf weighting. Equation taken from [1], pg 127, eq 6.13.
if (doSublinear){
if (freq > 0){
return 1 + (float)Math.log(freq);
} else {
return 0;
}
} else if (doBoolean){
return 1;
}
// else: doBasic
// The default behaviour of Lucene is sqrt(freq),
// but we are implementing the basic VSM model
return freq;
}
// inverse document frequency =
// measure of how often the term appears across the index
public float idf(int docFreq, int numDocs) {
if (doBoolean || doOverlap){
return 1;
}
// The default behaviour of Lucene is
// 1 + log (numDocs/(docFreq+1)),
// which is what we want (default VSM model)
return super.idf(docFreq, numDocs);
}
// normalization factor so that queries can be compared
public float queryNorm(float sumOfSquaredWeights){
if (doOverlap){
return 1;
} else if (doCosine){
return super.queryNorm(sumOfSquaredWeights);
}
// else: can't get here
return super.queryNorm(sumOfSquaredWeights);
}
// number of terms in the query that were found in the document
public float coord(int overlap, int maxOverlap) {
if (doOverlap){
return 1;
} else if (doCosine){
return 1;
}
// else: can't get here
return super.coord(overlap, maxOverlap);
}
// Note: this happens an index time, which we don't take advantage of
// (too many indices!)
public float computeNorm(String fieldName, FieldInvertState state){
if (doOverlap){
return 1;
} else if (doCosine){
return super.computeNorm(state);
}
// else: can't get here
return super.computeNorm(state);
}
}
以下是我当前节目的示例输出（没有提升分数）：
3086,Logic Based Knowledge Representation.,1999,5.165
33586,A Logic for the Representation of Spatial Knowledge.,1991,4.663
328937,Logic Programming for Knowledge Representation.,2007,4.663
219720,Logic for Knowledge Representation.,1984,4.663
487587,Knowledge Representation with Logic Programs.,1997,4.663
806195,Logic Programming as a Representation of Knowledge.,1983,4.663
806833,The Role of Logic in Knowledge Representation.,1983,4.663
744914,Knowledge Representation and Logic Programming.,2002,4.663
1113802,Knowledge Representation in Fuzzy Logic.,1989,4.663
984276,Logic Programming and Knowledge Representation.,1994,4.663
任何人都可以让我知道如何为我上面提到的条件添加分数？ Lucene是否提供这种功能？我可以将它集成到VSMSimilarity类吗？
修改： 我在Lucene文档中找到了这个：
public void setBoost(float b)
将此查询子句的提升设置为b。符合此条款的文件（除正常权重外）的分数乘以b。
很遗憾，这似乎会增加文档级别的分数。我想将得分乘以一个学期水平，但我还没有找到实现这个目标的方法。因此，如果文档包含来自level0和level1的单词，则只有level1中的术语将乘以0.8，例如
答案 0 :(得分：1)
你可以使用Lucene术语提升。
增加您的查询（假设OR是默认运算符）
logic base knowledge representation temporal^0.8 modal^0.8 classification^0.64...
使用标准的模拟提供者之一。
PS：在您的示例中找到LUCENE_42。这个功能几乎存在于任何版本的Lucene中（我记得它在2.4.9中）。