使用张量流解析用于回归的大数据集

时间:2016-10-30 11:53:33

标签: python numpy tensorflow

我目前正在尝试为我想要解决的回归任务设置张量流。

我不确定如何输入我的输入数据,因为它以某种方式存储。

我的输入是一个范围为0.5秒的样本音频文件。每个音频文件都附加到列表中,每个帧都存储为numpy.array。

这样的事情

#One audio set
[array([[frame],[frame],...,[frame]],dtype=float32)]

输出以这种方式存储

[array([[  4.53255900e+01,  -9.82247700e+00,  -1.22920200e+00,
         -6.81728800e+00,  -8.23808400e+00,  -1.42367900e+01,
         -4.64904100e+00,  -1.49673100e+01,  -4.75058700e+00,
         -7.54071600e+00,   5.11365500e+00,   9.80618400e+00,
         -6.03554700e-01],....,[...]])]

我试图实现一些我认为可以工作的东西..但是得到这个错误信息: 代码:

os.chdir(dnn_train)

train_name,train_data = generate_list_of_names_data(files_train_path)
train_data, train_output_data = load_sound_files(files_train_path,train_data)


number_of_examples = len(train_data)/2
train_set_data = train_data[:number_of_examples]
train_set_output = train_output_data[:number_of_examples]

test_set_data = train_data[number_of_examples:]
test_set_output = train_output_data[number_of_examples:]
############################# Training setup ##################################
# Parameters
learning_rate = 0.01
training_epochs = 1000
display_step = 50

# tf Graph Input
X = tf.placeholder("float")
Y = tf.placeholder("float")
n_samples = train_set_data[0].shape[0]

# Set model weights
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")


# Construct a linear model
pred = tf.add(tf.mul(X, W), b)


# Mean squared error
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)


# Initializing the variables
init = tf.initialize_all_variables()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_set_data, train_set_output):
            sess.run(optimizer, feed_dict={X: x, Y: y})

        #Display logs per epoch step
        if (epoch+1) % display_step == 0:
            c = sess.run(cost, feed_dict={X: train_set_data, Y:train_set_output})
            print "Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c), \
                "W=", sess.run(W), "b=", sess.run(b)

    print "Optimization Finished!"
    training_cost = sess.run(cost, feed_dict={X: train_set_data, Y: train_set_output})
    print "Training cost=", training_cost, "W=", sess.run(W), "b=", sess.run(b), '\n'

    #Graphic display
    plt.plot(train_set_data, train_set_output, 'ro', label='Original data')
    plt.plot(train_set_data, sess.run(W) * train_set_data + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

错误:

File "tensorflow_datapreprocess_mfcc_extraction_rnn.py", line 191, in <module>
    sess.run(optimizer, feed_dict={X: x, Y: y})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 340, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 564, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 637, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 659, in _do_call
    e.code)
tensorflow.python.framework.errors.InvalidArgumentError: Incompatible shapes: [2048,121] vs. [288,13]
     [[Node: gradients/sub_grad/BroadcastGradientArgs = BroadcastGradientArgs[_device="/job:localhost/replica:0/task:0/cpu:0"](gradients/sub_grad/Shape, gradients/sub_grad/Shape_1)]]
Caused by op u'gradients/sub_grad/BroadcastGradientArgs', defined at:
  File "tensorflow_datapreprocess_mfcc_extraction_rnn.py", line 178, in <module>
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 190, in minimize
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/optimizer.py", line 241, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients.py", line 481, in gradients
    in_grads = _AsList(grad_fn(op, *out_grads))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_grad.py", line 386, in _SubGrad
    rx, ry = gen_array_ops._broadcast_gradient_args(sx, sy)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 140, in _broadcast_gradient_args
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2154, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1154, in __init__
    self._traceback = _extract_stack()

...which was originally created as op u'sub', defined at:
  File "tensorflow_datapreprocess_mfcc_extraction_rnn.py", line 176, in <module>
    cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py", line 518, in binary_op_wrapper
    return func(x, y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 1683, in sub
    return _op_def_lib.apply_op("Sub", x=x, y=y, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2154, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1154, in __init__
    self._traceback = _extract_stack()

如何将这些信息提供给张量流?

1 个答案:

答案 0 :(得分:0)

tf.mul(X,W)只是标量乘法(W是代码中的标量),因此得到的形状与X的形状相同,看起来就像错误的来源(损失函数中的减法可以起作用的唯一方法是X和Y具有相同的形状)。

X和Y在示例中是否具有一致的形状?如果是这样,您可能希望将两者展平为具有tf.reshape(X,[1,-1])和tf.reshape(Y,[1,-1])的行向量。我将调用这些向量X_LENGTH和Y_LENGTH的长度。然后我会用一个tf.get_variable(name =&#34; W&#34;,shape = [X_LENGTH,Y_LENGTH])使W成为矩阵,然后用tf.matmul(X,W)代替tf.mul。 matmul()的输出将具有形状[1,Y_LENGTH],这将解决您在损失函数中遇到的形状问题。

当你在它的时候,你可能想要制作一个长度为Y_LENGTH的向量来获得每个输出的偏差。