我对机器学习非常陌生，我想知道是否有人可以带我理解这段代码，以及为什么它不起作用。这是我自己的scikit-learn教程的变体，可以在http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html中找到，这基本上就是我想要做的。我需要使用带标签的训练集训练模型，以便在使用测试集时可以预测测试集的标签。如果有人可以向我展示如何保存和加载模型，那也将非常有用。非常感谢你。这是我到目前为止的内容：
import codecs
import os
import numpy as np
import pandas as pd
from Text_Pre_Processing import Pre_Processing
filenames = os.listdir(
"...scikit-machine-learning/training_set")
files = []
array_data = []
array_label = []
for file in filenames:
with codecs.open("...scikit-machine-learning/training_set/" + file, "r",
encoding='utf-8', errors='ignore') as file_data:
open_file = file_data.read()
open_file = Pre_Processing.lower_case(open_file)
open_file = Pre_Processing.remove_punctuation(open_file)
open_file = Pre_Processing.clean_text(open_file)
files.append(open_file)
# ----------------------------------------------------
# PUTTING LABELS INTO LIST
for file in files:
if 'inheritance' in file:
array_data.append(file)
array_label.append('Inheritance (object-oriented programming)')
elif 'pagerank' in file:
array_data.append(file)
array_label.append('PageRank')
elif 'vector space model' in file:
array_data.append(file)
array_label.append('Vector Space Model')
elif 'bayes' in file:
array_data.append(file)
array_label.append('Bayes' + "'" + ' Theorem')
else:
array_data.append(file)
array_label.append('Dynamic programming')
#----------------------------------------------------------
csv_array = []
for i in range(0, len(array_data)):
csv_array.append([array_data[i], array_label[i]])
# format of array [[string, label], [string, label], [string, label]]
import csv
with open('data.csv', 'w') as target:
writer = csv.writer(target)
writer.writerows(zip(test_array))
data = pd.read_csv('data.csv')
numpy_array = data.as_matrix()
X = numpy_array[:, 0]
Y = numpy_array[:, 1]
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.4, random_state=42)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
text_clf = Pipeline(['vect', CountVectorizer(stop_words='english'), 'tfidf', TfidfTransformer(),
'clf', MultinomialNB()])
text_clf = text_clf.fit(X_train, Y_train)
predicted = text_clf.predict(X_test)
np.mean(predicted == Y_test)
我看到在线人们使用csv文件输入数据，所以我也尝试过这样做，我可能不需要它，所以如果这不正确，我深表歉意。
显示错误：
C:.../scikit-machine-learning/train.py:63: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
numpy_array = data.as_matrix()
Traceback (most recent call last):
File "C:/...scikit-machine-learning/train.py", line 66, in <module>
Y = numpy_array[:,1]
IndexError: index 1 is out of bounds for axis 1 with size 1
非常感谢您的帮助，如果您需要进一步的说明，请告诉我。
csv中两个条目的示例：
"['dynamic programming is an algorithmic technique used to solve certain optimization problems where the object is to find the best solution from a number of possibilities it uses a so called bottomup approach meaning that the problem is solved as a set of subproblems which in turn are made up of subsubproblemssubproblems are then selected and used to solve the overall problem these subproblems are only solved once and the solutions are saved so that they will not need to be recalculated again whilst calculated individually they may also overlap when any subproblem is met again it can be found and reused to solve another problem since it searches all possibilities it is also very accurate this method is far more efficient than recalculating and therefore considerably reduces computation it is widely used in computer science and can be applied for example to compress data in high density bar codes dynamic programming is most effective and therefore most often used on objects that are ordered from left to right and whose order cannot be rearranged this means it works well on character chains for example ', 'Dynamic programming']"
"['inheritance is one of the basic concepts of object oriented programming its objective is to add more detail to preexisting classes whilst still allowing the methods and variables of these classes to be reused the easiest way to look at inheritance is as an is a kind of relationship for example a guitar is a kind of string instrument electric acoustic and steel stringed are all types of guitar the further down an inheritance tree you get the more specific the classes become an example here would be books books generally fall into two categories fiction and nonfiction each of these can then be subdivided into more groups fiction for example can be split into fantasy horror romance and many more nonfiction splits the same way into other topics such as history geography cooking etc history of course can be subdivided into time periods like the romans the elizabethans the world wars and so on', 'Inheritance (object-oriented programming)']"
答案 0 :(得分：2)
您需要从csv中删除字符['和']，因为read_csv将它们视为字符串（一列）而不是两列数据帧。 在text_clf = Pipeline行上也存在拼写错误，因此我也进行了修复。祝你好运！
data = pd.read_csv('data.csv', header=None)
numpy_array = data.as_matrix()
strarr = numpy_array[:, 0]
X=[strarr[i].split(",")[0].replace("[",'').replace("'",'') for i in range(len(strarr))]
Y=[strarr[i].split(",")[1].replace("]",'').replace("'",'') for i in range(len(strarr))]
from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.4, random_state=42)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
text_clf = Pipeline([('vect', CountVectorizer(stop_words='english')), ('tfidf', TfidfTransformer()), ('clf', MultinomialNB())])
text_clf = text_clf.fit(X_train, Y_train)
predicted = text_clf.predict(X_test)
np.mean(predicted == Y_test)