Question

将Keras DL库与Tensorflow后端一起使用，我正在尝试使用内置的IMDB数据集实现批处理和验证生成器以进行情感分析。

该数据集包含25000个训练样本和25000个测试样本。因为设置每个样本的单词数量的截止值会产生相当低的准确度，所以我试图批量训练和测试样本，以便内存负载不会很糟糕。

当前代码：

/***************************************************************************
**  PROGRAM: MACRO.DS2LIST.SAS
**
**  UTILITY PROGRAM THAT DETECTS RETURNS A LIST OF FIELD VALUES FROM A 
**  DATASET IN DELIMITED FORMAT.
**
**  PARAMETERS:
**  iDs       : THE LIBNAME.DATASET NAME THAT YOU WANT TO CHECK.
**  iField    : THE FIELD THAT CONTAINS THE VALUES YOU WANT RETURNED IN A 
**              DELIMITED FORMAT.
**  iDelimiter: DEFAULT IS A COMMA. THE DELIMITER TO USE FOR THE RETURNED LIST.
**  iDsOptions: ANY STANDARD DATASET OPTIONS THAT YOU WOULD LIKE TO APPLY SUCH 
**              AS A WHERE STATEMENT.
**  iQuote    : (0=NO,1=YES). DEFAULT=0/NO. DETERMINES WHETHER THE RETURNED 
**              LIST IS QUOTED OR NOT.
**  iQuoteChar: (SINGLE,DOUBLE) DEFAULT=SINGLE. SPECIFIES WHETHER SINGLE
**              OR DOUBLE QUOTES ARE USED WHEN QUOTING THE RETURNED LIST
**
*****************************************************************************/

%macro ds2list(iDs=, iField=, iDsOptions=, iDelimiter=%str(,), iQuote=0, iQuoteChar=single);
  %local dsid pos rc result cnt quotechar value;

  %let result=;
  %let cnt=0;

  %if &iQuote %then %do;
    %if "%upcase(&iQuoteChar)" eq "DOUBLE" %then %do;
      %let quotechar = %nrstr(%");
    %end;
    %else %if "%upcase(&iQuoteChar)" eq "SINGLE" %then %do;
      %let quotechar = %nrstr(%');
    %end;
    %else %do;
      %let quotechar = %nrstr(%");
      %put WARNING: MACRO.DS2LIST.SAS: PARAMETER IQUOTECHAR INCORRECT. DEFAULTED TO DOUBLE;
    %end;
  %end;
  %else %do;
    %let quotechar = ;
  %end;

  /*
  ** ENSURE ALL THE REQUIRED PARAMETERS WERE PASSED IN.
  */
  %if "&iDs" ne "" and "&iField" ne "" %then %do;

    %let dsid=%sysfunc(open(&iDs(&iDsOptions),i));
    %if &dsid %then %do;

      %let pos=%sysfunc(varnum(&dsid,&iField));
      %if &pos %then %do;

        %let rc=%sysfunc(fetch(&dsid));
        %do %while (&rc eq 0);

          %if "%sysfunc(vartype(&dsid,&pos))" = "C" %then %do;
            %let value = %qsysfunc(getvarc(&dsid,&pos));
            %if "%trim(&value)" ne "" %then %do;
              %let value = %qtrim(&value);
            %end;
          %end;
          %else %do;
            %let value = %sysfunc(getvarn(&dsid,&pos));
          %end;

          /* WHITESPACE/CARRIAGE RETURNS REMOVED IN THE BELOW LINE */
          /* TO ENSURE NO WHITESPACE IS RETURNED IN THE OUTPUT.    */
          %if &cnt ne 0 %then %do;%unquote(&iDelimiter)%end;%unquote(&quotechar&value&quotechar.)

          %let cnt = %eval(&cnt + 1);
          %let rc  = %sysfunc(fetch(&dsid));
        %end;

        %if &rc ne -1 %then %do;
          %put WARNING: MACRO.DS2LIST.SAS: %sysfunc(sysmsg());
        %end;

      %end;
      %else %do;
        %put ERROR: MACRO.DS2LIST.SAS: FIELD &iField NOT FOUND IN DATASET %upcase(&iDs).;
      %end;
    %end;
    %else %do;
      %put ERROR: MACRO.DS2LIST.SAS: DATASET %upcase(&iDs) COULD NOT BE OPENED.;
    %end;

    %let rc=%sysfunc(close(&dsid));

  %end;
  %else %do;
    %put ERROR: MACRO.DS2LIST.SAS: YOU MUST SPECIFY BOTH THE IDS AND IFIELD PARAMETERS TO CALL THIS MACRO.;
  %end;

%mend;

但是，在尝试运行当前代码时，Keras会抛出以下错误：

from __future__ import print_function
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding, Dropout
from keras.layers import LSTM, TimeDistributed
from keras.datasets import imdb
from keras.callbacks import EarlyStopping, ModelCheckpoint
import numpy as np


max_features = 20000

def generate_batch(batchsize):
'''

'''
(x_train, y_train), (_,_) = imdb.load_data()
for i in range(0, len(x_train), batchsize):
    x_batch = x_train[i:(i+batchsize)]
    y_batch = y_train[i:(i+batchsize)]
    x_batch = sequence.pad_sequences(x_train, maxlen=None)
    yield(x_batch, y_batch)

def generate_val(valsize):
'''
'''
(_,_), (x_test, y_test) = imdb.load_data()    
for i in range(0, len(x_test), valsize):
    x_val = x_test[i:(i+valsize)]
    y_val = y_test[i:(i+valsize)]
    x_val = sequence.pad_sequences(x_test, maxlen=None)
    yield(x_val, y_val)

print('Build model...')
primary_model = Sequential()
primary_model.add(Embedding(input_dim = max_features,
                    output_dim = max_features,
                    trainable=False, 
                    weights=[(np.eye(max_features,max_features))], 
                    mask_zero=True))
primary_model.add(TimeDistributed(Dense(150, use_bias=False)))
primary_model.add(LSTM(128))
primary_model.add(Dense(2, activation='softmax'))
primary_model.summary()
primary_model.compile(loss='sparse_categorical_crossentropy', 
              optimizer='adam',
              metrics=['accuracy'])

print('Train...')
filepath = "primeweights-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath,
                            verbose=1,
                            save_best_only=True)
early_stopping_monitor = EarlyStopping(patience=2)

primary_model.fit_generator(generate_batch(25),
                            steps_per_epoch = 1000,
                            epochs = 1, 
                            callbacks=[early_stopping_monitor],
                            validation_data=generate_val(25),
                            validation_steps=1000)


score, acc = primary_model.evaluate(x_test, y_test,
                            batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

primary_model.save('primary_model_imdb.h5')

Answer 1

代码中存在多个错误：

正如@Y的评论中指出的那样。罗：

x_batch = sequence.pad_sequences(x_train, maxlen=None) # gives 25000 samples

x_batch = sequence.pad_sequences(x_batch, maxlen=None) # gives batch_size

加载imdb数据集时，您必须提供num_words=max_features，否则您的嵌入层会将输入max_words视为max_features，但最终会使word_ids大于此值。

(x_train, y_train), (_,_) = imdb.load_data(num_words=max_features)

建议在使用填充时提供maxlen，否则maxlen批次将被使用，这可能会因批次而异;

x_batch = sequence.pad_sequences(x_batch, maxlen=maxlen, padding='post')

您正在使用嵌入层而不对其进行训练，并保持输入和输出维度相同。对我来说没有意义，所以我改变了。

primary_model.add(Embedding(input_dim = max_features,
                    output_dim = embedding_dim,
                    trainable=True, 
                    weights=[(np.eye(max_features,embedding_dim))], 
                    mask_zero=True))

要在测试数据上评估您的模型，您必须先加载它，然后将其转换为填充序列

(_,_), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen, padding='post')
score, acc = primary_model.evaluate(x_test, y_test, batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

在使用Generator进行多个时期的训练时，我们必须确保生成器继续产生值。为此，一旦数据集结束，我们需要再次从0开始屈服。

def generate_batch(batchsize):

    (x_train, y_train), (_,_) = imdb.load_data(num_words=max_features)
    print("train_size", x_train.shape)
    while True:
        for i in range(0, len(x_train), batchsize):
            x_batch = x_train[i:(i+batchsize)]
            y_batch = y_train[i:(i+batchsize)]
            x_batch = sequence.pad_sequences(x_batch, maxlen=maxlen, padding='post')
            yield(x_batch, y_batch)

链接完整的工作代码（使用针对第6点的修复更新）here

Keras TF：ValueError：输入数组应具有与目标数组相同的样本数

1 个答案: