在改变数据表示“LabelBinarizer”之后尺寸不匹配

时间:2017-04-04 15:42:47

标签: python-3.x scikit-learn deep-learning keras image-preprocessing

我有66个字符类。我训练了一个多层感知器给它提供图像,每个图像的类别是一个字符

  

Aa..Zz 0-9,;:! èàâ€

我应用LabelBinarizer将每个字符转换为66个类的向量(例如[0 0 0 0 ....1 ...... 0]),因为模型不接受非数字数据。 我很惊讶地得到y_test(错误)的维度61 y_train(66正确)。 我的代码出了什么问题?

from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(data_pixels, classes_dataset, test_size=0.3)
print(x_train.shape)
print(y_train.shape)
print(x_test.shape)
print(y_test.shape)
print(x_test)
print("my y_train")
print(y_train[0:100])
print("my y_test ")
print(y_test[0:100])

**

(1708, 3072) # x_train shape
(1708,) # y_train shape
(732, 3072) #x_test shape
(732,) y_train shape

**

x_test

[[ 1.  1.  1. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 ..., 
 [ 1.  1.  1. ...,  1.  1.  1.]
 [ 1.  1.  1. ...,  1.  1.  1.]
 [ 1.  1.  1. ...,  1.  1.  1.]]

y_train

 ['5' 'O' '9' '0' 'E' 'D' '9' ',' 'R' 'H' 'T' 'i' '\xc3\xa9' 'T' 'o' 'u' 'R'
     '2' '0' 'L' 'o' '0' '6' 'q' 'P' '6' '2' 'T' '2' '0' 'i' '0' 'f' 'A' 'r'
     'n' 'T' 'O' '8' 'B' 'T' 'd' '0' 'V' 'X' '9' '.' '6' 'J' 'S' 'E' 'O' 'T'
     '4' 'E' '0' 'I' '3' 'o' 'E' '6' 'R' 'M' '0' 'E' '1' 'R' 'T' 'E' '.' '0'
     'G' 'R' 'E' 'E' '0' '9' 'd' '1' '7' 'A' 'B' 'L' '4' 'l' 'O' '1' 'v' '3'
     '%' 'd' '0' 'T' 's' 'A' '6' 'w' 'slash' '2' '9']

y_test

['N' '0' 'F' '4' 'U' 'C' 'u' 'e' '0' ',' 'G' 'C' 'T' '%' '-' 'V' '5' 'P'
 'N' 'S' '8' '4' ',' 'm' '5' '3' 'e' 'I' 'i' 'M' 'I' '3' 'C' 'F' 'e' 'a'
 '6' 'R' 'V' '4' '0' 'f' '9' 'E' '2' '0' 'E' 'N' 'I' '5' '0' 'A' '%' '-'
 'G' '0' ',' 'O' 'Y' '\xc3\x89' 'R' 's' ',' 'A' 'I' '3' 'S' '2' 'P' '.' 'I'
 ',' 'r' 'I' 'i' '5' '5' 'R' 'C' 'e' '2' 'q' '.' 'R' 'O' 'n' 'S' '6' 'G'
 '0' 'R' 'i' 't' 'i' '9' 'I' 'D' 'slash' '0' 'A']

当我申请LabelBinazer时,我会获得y_test(732,61)而不是(732,66) 66表示班级数:

from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer()
y_train = encoder.fit_transform(y_train)
y_test= encoder.fit_transform(y_test)

print(y_train.shape)

print(y_test.shape)

(1708,66)

(732,61)我怎么得到61而不是66

   **Edit**

按照建议进行更改后

    y_test= encoder.transform(y_test)
l got the following error :
ValueError                                Traceback (most recent call last)
<ipython-input-109-b554d33049ab> in <module>()
      1 y_said=encoder.fit(y_train)
      2 y_said
----> 3 y_test= encoder.transform(y_test)

/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/label.pyc in transform(self, y)
    333                               pos_label=self.pos_label,
    334                               neg_label=self.neg_label,
--> 335                               sparse_output=self.sparse_output)
    336 
    337     def inverse_transform(self, Y, threshold=None):

/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/label.pyc in label_binarize(y, classes, neg_label, pos_label, sparse_output)
    492     if (y_type == "multilabel-indicator" and classes.size != y.shape[1]):
    493         raise ValueError("classes {0} missmatch with the labels {1}"
--> 494                          "found in the data".format(classes, unique_labels(y)))
    495 
    496     if y_type in ("binary", "multiclass"):

ValueError: classes [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65] missmatch with the labels [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
 50 51 52 53 54 55 56 57 58 59 60]found in the data

EDIT2:

encoder = LabelBinarizer()
encoder.fit(y_train + y_test)
y_train= encoder.transform(y_train)  
y_test= encoder.transform(y_test)

我收到以下错误:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-125-45c4512ecc1f> in <module>()
      1 from sklearn.preprocessing import LabelBinarizer
      2 encoder = LabelBinarizer()
----> 3 encoder.fit(y_train + y_test)
      4 y_train= encoder.transform(y_train)
      5 y_test= encoder.transform(y_test)

TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('S5') dtype('S5') dtype('S5')

EDIT3: 我的代码来自jupyter:

from __future__ import print_function
from keras.models import Sequential
from keras.layers import Dense
import keras
import numpy as  np
# fix random seed for reproducibility
numpy.random.seed(7)
batch_size = 128
num_classes = 66
epochs = 12

data_pixels=np.genfromtxt("pixels_dataset.csv", delimiter=',')

classes_dataset=np.genfromtxt("labels.csv",dtype=np.str , delimiter='\t')

from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(data_pixels, classes_dataset, test_size=0.3)

from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer()
encoder.fit(y_train + y_test)
y_train= encoder.transform(y_train)  
y_test= encoder.transform(y_test)

1 个答案:

答案 0 :(得分:2)

您需要transform使用y_test使用encoder列入您的列车数据。因此,您需要将代码更改为:

 y_test= encoder.transform(y_test)

否则您的encoder将被重新安装,导致问题。

修改

您也可以尝试:

encoder.fit(y_train + y_test) # in case of lists
encoder.fit(numpy.append(y_train, y_test)) # in case of numpy arrays
y_train= encoder.transform(y_train)  
y_test= encoder.transform(y_test)

编辑2:

最终成功的是:

encoder.fit(numpy.append(y_train, y_test)) # in case of numpy arrays
y_train= encoder.fit_transform(y_train)  
y_test= encoder.transform(y_test)