Question

我正在制作一份关于作者身份归属的小型NLP项目：我有一些来自两位作者的文章，我想说是谁写的。

我有一些预处理文本（标记化，pos标记等），我想将其加载到sciki-learn中。

文件有这样的形状：

Testo   -   SPN Testo   testare+v+indic+pres+nil+1+sing testo+n+m+sing  O
:   -   XPS colon   colon+punc  O
"   -   XPO "   quotation_mark+punc O
Buongiorno  -   I   buongiorno  buongiorno+inter buongiorno+n+m+_   O
a   -   E   a   a+prep  O
tutti   -   PP  tutto   tutto+adj+m+plur+pst+ind tutto+pron+_+m+_+plur+ind  O
.   <eos>   XPS full_stop   full_stop+punc  O
Ci  -   PP  pro loc+pron+loc+_+3+_+clit pro+pron+accdat+_+1+plur+clit   O
sarebbe -   VI  essere  essere+v+cond+pres+nil+2+sing   O
molto   -   B   molto   molto+adj+m+sing+pst+ind

因此它是一个6个列的选项卡分隔文本文件（单词，句末标记，词性，引理，形态信息和命名实体识别标记）。

每个文件代表一个要分类的文档。

为scikit学习塑造它们的最佳方法是什么？

Answer 1

这里描述了他们在scikit-learn示例http://scikit-learn.org/stable/auto_examples/document_classification_20newsgroups.html中使用的结构 http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_files.html

替换此

# Load some categories from the training set
if opts.all_categories:
    categories = None
else:
    categories = [
        'alt.atheism',
        'talk.religion.misc',
        'comp.graphics',
        'sci.space',
    ]

if opts.filtered:
    remove = ('headers', 'footers', 'quotes')
else:
    remove = ()

print("Loading 20 newsgroups dataset for categories:")
print(categories if categories else "all")

data_train = fetch_20newsgroups(subset='train', categories=categories,
                                shuffle=True, random_state=42,
                                remove=remove)

data_test = fetch_20newsgroups(subset='test', categories=categories,
                               shuffle=True, random_state=42,
                               remove=remove)

使用您的数据加载语句，例如：

# Load some categories from the training set
categories = [
        'high',
        'low',
]

print("loading dataset for categories:")
print(categories if categories else "all")

train_path='c:/Users/username/Documents/SciKit/train'
data_train = load_files(train_path, encoding='latin1')

test_path='c:/Users/username/Documents/SciKit/test'
data_test = load_files(test_path, encoding='latin1')

并在每个列车和测试目录中创建＆＃34;高＆＃34;和＆＃34;低＆＃34;您的类别文件的子目录。

为scikit-learn准备数据

1 个答案: