我正在制作一个程序，可以根据文本中的数据预测相应的业务部门。我已经建立了一个词汇表来查找文本中与某个单元相对应的单词出现，但是我不确定如何将这些数据与机器学习模型一起用于预测。
它可以预测四个单位，分别是：MicrosoftTech，JavaTech，Pythoneers和JavascriptRoots。在词汇表中，我输入了表示某些单位的单词。例如JavaTech：Java，Spring，Android； MicrosoftTech：.Net，csharp；等等。现在，我将单词袋模型与自定义词汇结合使用，以查找这些单词出现的频率。
这是我获取字数数据的代码：
def bagOfWords(description, vocabulary):
bag = np.zeros(len(vocabulary)).astype(int)
for sw in description:
for i,word in enumerate(vocabulary):
if word == sw:
bag[i] += 1
print("Bag: ", bag)
return bag
因此，可以说词汇为：[java, spring, .net, csharp, python, numpy, nodejs, javascript]。
其描述为："Company X is looking for a Java Developer. Requirements: Has worked with Java. 3+ years experience with Java, Maven and Spring."
运行代码将输出以下内容：Bag: [3,1,0,0,0,0,0,0]
如何将这些数据用于ML算法的预测？
到目前为止，我的代码：
import pandas as pd
import numpy as np
import warnings
import tkinter as tk
from tkinter import filedialog
from nltk.tokenize import TweetTokenizer
warnings.filterwarnings("ignore", category=FutureWarning)
root= tk.Tk()
canvas1 = tk.Canvas(root, width = 300, height = 300, bg = 'lightsteelblue')
canvas1.pack()
def getExcel ():
global df
vocabularysheet = pd.read_excel (r'Filepath\filename.xlsx')
vocabularydf = pd.DataFrame(vocabularysheet, columns = ['Word'])
vocabulary = vocabularydf.values.tolist()
unitlabelsdf = pd.DataFrame(vocabularysheet, columns = ['Unit'])
unitlabels = unitlabelsdf.values.tolist()
for voc in vocabulary:
index = vocabulary.index(voc)
voc = vocabulary[index][0]
vocabulary[index] = voc
for label in unitlabels:
index = unitlabels.index(label)
label = unitlabels[index][0]
unitlabels[index] = label
import_file_path = filedialog.askopenfilename()
testdatasheet = pd.read_excel (import_file_path)
descriptiondf = pd.DataFrame(testdatasheet, columns = ['Description'])
descriptiondf = descriptiondf.replace('\n',' ', regex=True).replace('\xa0',' ', regex=True).replace('•', ' ', regex=True).replace('u200b', ' ', regex=True)
description = descriptiondf.values.tolist()
tokenized_description = tokanize(description)
for x in tokenized_description:
index = tokenized_description.index(x)
tokenized_description[index] = bagOfWords(x, vocabulary)
def tokanize(description):
for d in description:
index = description.index(d)
tknzr = TweetTokenizer()
tokenized_description = list(tknzr.tokenize((str(d).lower())))
description[index] = tokenized_description
return description
def wordFilter(tokenized_description):
bad_chars = [';', ':', '!', "*", ']', '[', '.', ',', "'", '"']
if(tokenized_description in bad_chars):
return False
else:
return True
def bagOfWords(description, vocabulary):
bag = np.zeros(len(vocabulary)).astype(int)
for sw in description:
for i,word in enumerate(vocabulary):
if word == sw:
bag[i] += 1
print("Bag: ", bag)
return bag
browseButton_Excel = tk.Button(text='Import Excel File', command=getExcel, bg='green', fg='white', font=('helvetica', 12, 'bold'))
predictionButton = tk.Button(text='Button', command=getExcel, bg='green', fg='white', font=('helvetica', 12, 'bold'))
canvas1.create_window(150, 150, window=browseButton_Excel)
root.mainloop()
答案 0 :(得分：0)
您已经知道如何准备训练数据集。
这是我要解释的一个例子：
voca = ["java", "spring", "net", "csharp", "python", "numpy", "nodejs", "javascript"]
units = ["MicrosoftTech", "JavaTech", "Pythoneers", "JavascriptRoots"]
desc1 = "Company X is looking for a Java Developer. Requirements: Has worked with Java. 3+ years experience with Java, Maven and Spring."
desc2 = "Company Y is looking for a csharp Developer. Requirements: Has wored with csharp. 5+ years experience with csharp, Net."
x_train = []
y_train = []
x_train.append(bagOfWords(desc1, voca))
y_train.append(units.index("JavaTech"))
x_train.append(bagOfWords(desc2, voca))
y_train.append(units.index("MicrosoftTech"))
而且，我们获得了2个训练数据集：
[array([3, 1, 0, 0, 0, 0, 0, 0]), array([0, 0, 1, 3, 0, 0, 0, 0])] [1, 0]
array([3, 1, 0, 0, 0, 0, 0, 0]) => 1 (It means JavaTech)
array([0, 0, 1, 3, 0, 0, 0, 0]) => 0 (It means MicrosoftTech)
而且，模型需要以您定义的4个单位来预测一个单位。 因此，我们需要一个分类机器学习模型。 分类机器学习模型需要“ softmax”作为输出层的激活函数。 并且，需要“交叉熵”损失函数。 这是由Tensorflow的keras api编写的非常简单的深度学习模型。
import tensorflow as tf
import numpy as np
units = ["MicrosoftTech", "JavaTech", "Pythoneers", "JavascriptRoots"]
x_train = np.array([[3, 1, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 1, 1, 0, 0, 0, 0],
[0, 0, 2, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 2, 1, 0, 0],
[0, 0, 0, 0, 1, 2, 0, 0],
[0, 0, 0, 0, 0, 0, 1, 1],
[0, 0, 0, 0, 0, 0, 1, 0]])
y_train = np.array([0, 0, 1, 1, 2, 2, 3, 3])
而且，它是由一个包含256个单元的隐藏层和一个包含4个单元的输出层组成的模型。
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(256, activation=tf.nn.relu),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(len(units), activation=tf.nn.softmax)])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
我将纪元设置为50。在学习时，您需要查看损失和acc。实际上，十个是不够的。 我将开始学习。
model.fit(x_train, y_train, epochs=50)
而且，这是预测的一部分。 newSample只是我所做的样品。
newSample = np.array([[2, 2, 0, 0, 0, 0, 0, 0]])
prediction = model.predict(newSample)
print (prediction)
print (units[np.argmax(prediction)])
最后，我得到如下结果：
[[0.96280855 0.00981709 0.0102595 0.01711495]]
MicrosoftTech
表示每个单元的可能性。可能性最大的是MicrosoftTech。
MicrosoftTech : 0.96280855
JavaTech : 0.00981709
....
而且，这是学习步骤的结果。您可以看到损失正在不断减少。因此，我增加了纪元的数量。
Epoch 1/50
8/8 [==============================] - 0s 48ms/step - loss: 1.3978 - acc: 0.0000e+00
Epoch 2/50
8/8 [==============================] - 0s 356us/step - loss: 1.3618 - acc: 0.1250
Epoch 3/50
8/8 [==============================] - 0s 201us/step - loss: 1.3313 - acc: 0.3750
Epoch 4/50
8/8 [==============================] - 0s 167us/step - loss: 1.2965 - acc: 0.7500
Epoch 5/50
8/8 [==============================] - 0s 139us/step - loss: 1.2643 - acc: 0.8750
........
........
Epoch 45/50
8/8 [==============================] - 0s 122us/step - loss: 0.3500 - acc: 1.0000
Epoch 46/50
8/8 [==============================] - 0s 140us/step - loss: 0.3376 - acc: 1.0000
Epoch 47/50
8/8 [==============================] - 0s 134us/step - loss: 0.3257 - acc: 1.0000
Epoch 48/50
8/8 [==============================] - 0s 137us/step - loss: 0.3143 - acc: 1.0000
Epoch 49/50
8/8 [==============================] - 0s 141us/step - loss: 0.3032 - acc: 1.0000
Epoch 50/50
8/8 [==============================] - 0s 177us/step - loss: 0.2925 - acc: 1.0000