Question

我需要使用Weka及其AttributeSelection算法LatentSemanticAnalysis来进行文本分类。我将我的数据集拆分为训练和测试集，我想在其上应用LSA。我已经阅读了一些关于LSA的帖子，但是我还没有找到如何使用它来分离数据集并使它们保持兼容。这是我到目前为止所用的但是内存不足...：

AttributeSelection selecter = new AttributeSelection();
weka.attributeSelection.LatentSemanticAnalysis lsa = new weka.attributeSelection.LatentSemanticAnalysis();
Ranker rank = new Ranker();

selecter.setEvaluator(lsa);
selecter.setSearch(rank);
selecter.setRanking(true);

selecter.SelectAttributes(input);
Instances outputData = selecter.reduceDimensionality(input);

EDIT1 在回复@Jose的回复时，我添加了一个新版本的源代码。这导致OutOfMemoryError：

AttributeSelection filter = new AttributeSelection(); // package weka.filters.supervised.attribute!
LatentSemanticAnalysis lsa = new LatentSemanticAnalysis();
Ranker rank = new Ranker();
filter.setEvaluator(lsa);
filter.setSearch(rank);
filter.setInputFormat(train);

train = Filter.useFilter(train, filter);
test = Filter.useFilter(test, filter);

EDIT2 我得到的错误：

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at weka.core.matrix.Matrix.getArrayCopy(Matrix.java:301)
at weka.core.matrix.SingularValueDecomposition.<init>(SingularValueDecomposition.java:76)
at weka.core.matrix.Matrix.svd(Matrix.java:913)
at weka.attributeSelection.LatentSemanticAnalysis.buildAttributeConstructor(LatentSemanticAnalysis.java:511)
at weka.attributeSelection.LatentSemanticAnalysis.buildEvaluator(LatentSemanticAnalysis.java:416)
at weka.attributeSelection.AttributeSelection.SelectAttributes(AttributeSelection.java:596)
at weka.filters.supervised.attribute.AttributeSelection.batchFinished(AttributeSelection.java:455)
at weka.filters.Filter.useFilter(Filter.java:682)
at test.main(test.java:44)

Answer 1

由于AttributeSelection是一个过滤器，您可以将其以批处理模式（-b选项）应用于培训＆amp;一次测试子集，从而根据训练集中定义的维度表示测试数据集。

您可以在Use Weka in your Java code - Filter - Batch filtering的程序中查看如何执行此操作。

使用WEKA API在火车和测试装置上执行LSA

1 个答案: