Question

假设我有以下两个属性的示例ARFF文件：

（1）情绪：阳性[1]或阴性[-1]

（2）推文：文字

@relation sentiment_analysis

@attribute sentiment {1, -1}
@attribute tweet string

@data
-1,'is upset that he can\'t update his Facebook by texting it... and might cry as a result  School today also. Blah!'
-1,'@Kenichan I dived many times for the ball. Managed to save 50\%  The rest go out of bounds'
-1,'my whole body feels itchy and like its on fire '
-1,'@nationwideclass no, it\'s not behaving at all. i\'m mad. why am i here? because I can\'t see you all over there. '
-1,'@Kwesidei not the whole crew '
-1,'Need a hug '
1,'@Cliff_Forster Yeah, that does work better than just waiting for it  In the end I just wonder if I have time to keep up a good blog.'
1,'Just woke up. Having no school is the best feeling ever '
1,'TheWDB.com - Very cool to hear old Walt interviews!  ? http://blip.fm/~8bmta'
1,'Are you ready for your MoJo Makeover? Ask me for details '
1,'Happy 38th Birthday to my boo of alll time!!! Tupac Amaru Shakur '
1,'happy #charitytuesday @theNSPCC @SparksCharity @SpeakingUpH4H '

我想将第二个属性的值转换为等效的TF-IDF值。

顺便说一下，我尝试了下面的代码，但是它的输出ARFF文件并没有包含各个实例的正（1）值的第一个属性。

// Set the tokenizer
NGramTokenizer tokenizer = new NGramTokenizer();
tokenizer.setNGramMinSize(1);
tokenizer.setNGramMaxSize(1);
tokenizer.setDelimiters("\\W");

// Set the filter
StringToWordVector filter = new StringToWordVector();
filter.setAttributeIndicesArray(new int[]{1});
filter.setOutputWordCounts(true);
filter.setTokenizer(tokenizer);
filter.setInputFormat(inputInstances);
filter.setWordsToKeep(1000000);
filter.setDoNotOperateOnPerClassBasis(true);
filter.setLowerCaseTokens(true);
filter.setTFTransform(true);
filter.setIDFTransform(true);

// Filter the input instances into the output ones
outputInstances = Filter.useFilter(inputInstances, filter);

示例输出ARFF文件：

@data
{0 -1,320 1,367 1,374 1,397 1,482 1,537 1,553 1,681 1,831 1,1002 1,1033 1,1112 1,1119 1,1291 1,1582 1,1618 1,1787 1,1810 1,1816 1,1855 1,1939 1,1941 1}
{0 -1,72 1,194 1,436 1,502 1,740 1,891 1,935 1,1075 1,1256 1,1260 1,1388 1,1415 1,1579 1,1611 1,1818 2,1849 1,1853 1}
{0 -1,374 1,491 1,854 1,873 1,1120 1,1121 1,1197 1,1337 1,1399 1,2019 1}
{0 -1,240 1,359 2,369 1,407 1,447 1,454 1,553 1,1019 1,1075 3,1119 1,1240 1,1244 1,1373 1,1379 1,1417 1,1599 1,1628 1,1787 1,1824 1,2021 1,2075 1}
{0 -1,198 1,677 1,1379 1,1818 1,2019 1}
{0 -1,320 1,1070 1,1353 1}
{0 -1,210 1,320 2,477 2,867 1,1020 1,1067 1,1075 1,1212 1,1213 1,1240 1,1373 1,1404 1,1542 1,1599 1,1628 1,1815 1,1847 1,2067 1,2075 1}
{179 1,1815 1}
{298 1,504 1,662 1,713 1,752 1,1163 1,1275 1,1488 1,1787 1,2011 1,2075 1}
{144 1,785 1,1274 1}
{19 1,256 1,390 1,808 1,1314 1,1350 1,1442 1,1464 1,1532 1,1786 1,1823 1,1864 1,1908 1,1924 1}
{84 1,186 1,320 1,459 1,564 1,636 1,673 1,810 1,811 1,966 1,997 1,1094 1,1163 1,1207 1,1592 1,1593 1,1714 1,1836 1,1853 1,1964 1,1984 1,1997 2,2058 1}
{9 1,1173 1,1768 1,1818 1}
{86 1,935 1,1112 1,1337 1,1348 1,1482 1,1549 1,1783 1,1853 1}

正如您所看到的，前几个实例是可以的（因为它们包含-1类以及其他功能），但最后剩下的实例不包含正类属性（1）。

我的意思是，在输出ARFF文件的最后一个实例中应该有{0 1，...}作为第一个属性，但是它丢失了。

Answer 1

您必须在java程序中明确指定哪个类属性，因为当您应用StringToWordVector过滤器时，您的输入将在指定的n-gram之间进行划分。因此，一旦StringToWordVector矢量化输入，类属性位置就会改变。您可以使用重新排序文件管理器，它最终将类属性放在最后位置，Weka将选择最后一个属性作为类属性。

有关Rekadering in Weka的更多信息，请访问http://weka.sourceforge.net/doc.stable-3-8/weka/filters/unsupervised/attribute/Reorder.html。另外，http://www.programcreek.com/java-api-examples/index.php?api=weka.filters.unsupervised.attribute.Reorder处的示例5可以帮助您进行重新排序。

希望它有所帮助。

Answer 2

您获取TF-IDF的过程似乎是正确的。

根据我的实验，如果您有n个类别，Weka将显示每个n-1个类别的记录的信息标签，并暗示第n ^th个类别的记录。

在您的情况下，您有2个类别-1和1，因此weka在具有类别标签-1的记录中显示标签，并且隐含具有标签1的记录。

如何使用Java中的Weka将文本转换为TF-IDF格式？

2 个答案: