Question

我有一个运行良好的生成器功能。我有一堆.txt文件，其中每个文件也很长。现在的任务是编写一个生成器函数，该函数需要：

一批文件
然后从一个文件中取出一批大小为128的

现在我的代码：

data_files_generator <- function(train_set) {

  files <- train_set
  next_file <- 0

  function() {

    # move to the next file (note the <<- assignment operator)
    next_file <<- next_file + 1

    # if we've exhausted all of the files then start again at the
    # beginning of the list (keras generators need to yield
    # data infinitely -- termination is controlled by the epochs
    # and steps_per_epoch arguments to fit_generator())
    if (next_file > length(files))
    {next_file <<- 1}

    # determine the file name
    file <- files[[next_file]]

    text <- read_lines(paste(data_dir, file, sep = "" )) %>%
      str_to_lower() %>%
      str_c(collapse = "\n") %>%
      removeNumbers() %>%
      tokenize_characters(strip_non_alphanum = FALSE, simplify = TRUE)

    text <- text[text %in% chars]

    dataset <- map(
      seq(1, length(text) - maxlen - 1, by = 3), 
      ~list(sentece = text[.x:(.x + maxlen - 1)], next_char = text[.x + maxlen])
    )

    dataset <- transpose(dataset)

    # Vectorization
    x <- array(0, dim = c(length(dataset$sentece), maxlen, length(chars)))
    y <- array(0, dim = c(length(dataset$sentece), length(chars)))

    for(i in 1:length(dataset$sentece)){

      x[i,,] <- sapply(chars, function(x){
        as.integer(x == dataset$sentece[[i]])
      })

      y[i,] <- as.integer(chars == dataset$next_char[[i]])

    }
    rounded_dim <- floor(dim(x)[1]/mini_batch_size)
    match_size_to_batch <- 128 * rounded_dim

    x <- x[1:match_size_to_batch, 1:maxlen, 1:length(chars)]
    y <- y_val[1:match_size_to_batch, 1:length(chars)]

    return(list(x, y))

  }
}

因此，即将到来的是一个文本文件，该文件被转换为较小的文本（长度为maxlen），然后被热编码为0和1矩阵。

问题是从我的代码中输出的是一个大小为maxlen x lenght(chars) x samples的数据多维数据集，其中样本数量非常大，为什么我希望生成器函数始终输出大小为{{1 }}，然后输出下一个maxlen x lenght(chars) x samples(128)大小的批处理，直到读取了整个文本文件，然后转到下一个文本文件...

现在的输出是错误：

maxlen x lenght(chars) x samples

希望我已经解释得足够好理解。我想我必须输入某种for循环来迭代样本长度，但是我不知道如何将其包含到gen中。功能。

Answer 1

根据该错误，您试图输入形状为(112512, 40, 43)的对象，但您的LSTM层期望使用形状为(128, 40, 43)的对象。似乎缺少一些代码，但是在定义输入层时，是否要固定批处理大小？我很幸运地将输入层定义为：

l_input = Input(shape = (None, num_features), name = 'input_layer')

我怀疑错误是由于以下几行代码引起的：

rounded_dim <- floor(dim(x)[1]/mini_batch_size)
match_size_to_batch <- 128 * rounded_dim

这使您的批次大小比128大得多。从Keras documentation开始，输入形状应为(batch_size, timesteps, input_dim)。整个史诗中的批处理大小不必相同，但是对于一个批处理，它们都需要具有相同数量的timesteps（看起来就像您用maxlen处理的一样）。

Answer 2

我实现了一个for循环，该循环现在返回批量为128的批次：

更改代码：

data_files_generator <- function(train_set) {

  files <- train_set
  next_file <- 0

  function() {

    # move to the next file (note the <<- assignment operator)
    next_file <<- next_file + 1

    # if we've exhausted all of the files then start again at the
    # beginning of the list (keras generators need to yield
    # data infinitely -- termination is controlled by the epochs
    # and steps_per_epoch arguments to fit_generator())
    if (next_file > length(files))
    {next_file <<- 1}

    # determine the file name
    file <- files[[next_file]]

    text <- read_lines(paste(data_dir, file, sep = "" )) %>%
      str_to_lower() %>%
      str_c(collapse = "\n") %>%
      removeNumbers() %>%
      tokenize_characters(strip_non_alphanum = FALSE, simplify = TRUE)

    text <- text[text %in% chars]

    dataset <- map(
      seq(1, length(text) - maxlen - 1, by = 3), 
      ~list(sentece = text[.x:(.x + maxlen - 1)], next_char = text[.x + maxlen])
    )

    dataset <- transpose(dataset)

    # Vectorization
    x <- array(0, dim = c(length(dataset$sentece), maxlen, length(chars)))
    y <- array(0, dim = c(length(dataset$sentece), length(chars)))

    for(i in 1:length(dataset$sentece)){

      x[i,,] <- sapply(chars, function(x){
        as.integer(x == dataset$sentece[[i]])
      })

      y[i,] <- as.integer(chars == dataset$next_char[[i]])

    }
    rounded_dim <- floor(dim(x)[1]/mini_batch_size)
    match_size_to_batch <- 128 * rounded_dim

    x <- x[1:match_size_to_batch, 1:maxlen, 1:length(chars)]
    y <- y_val[1:match_size_to_batch, 1:length(chars)]

    #Edit:
    span_start <-1
    for (iter in 1:rounded_dim){
     i <- iter * 128
     span_end <- iter * 128
     x <- x[span_start:span_end, 1:maxlen, 1:length(chars)]
     y <- y[span_start:span_end, 1:length(chars)]
     span_start <- i
     return(list(x, y))
    }
  }
}

LSTM Keras中的生成器功能可输出一个文件的小批量

2 个答案: