Question

我想在下面（在jupyter笔记本中）重现与Geron书第637页示例相对应的学习曲线。

尤其是这是我使用的代码：

%pylab inline
%matplotlib inline

import tensorflow as tf
from tf import keras
import matplotlib
import numpy as np
import matplotlib.pyplot as plt
from collections import deque
import gym

env = gym.make("CartPole-v1")
input_shape = [4]
n_outputs = 2
replay_buffer = deque(maxlen = 1000)
batch_size = 32
discount_factor = 0.95

model = keras.models.sequential([
    keras.layers.Dense(32, activation="elu", input_shape = input_shape),
    keras.layers.Dense(32, activation="elu"),
    keras.layers.Dense(n_outputs)
])

def epsilon_greedy_policy(state, epsilon=0):
    if np.random.rand() < epsilon:
        return np.random.randint(2)
    else:
        Q_values = model.predict(state[np.newaxis])
        return np.argmax(Q_values[0])

def sample_experiences(batch_size):
    indices = np.random.randint(len(replay_buffer), size = batch_size)
    batch = [replay_buffer[index] for index in indices]
    states, actions, rewards, next_states, dones = [
        np.array([experience[field_index] for experience in batch])
        for field_index in range(5)]
    return states, actions, rewards, next_states, dones

def play_one_step(env, state, epsilon):
    action = epsilon_greedy_policy(state, epsilon)
    next_state, reward, done, info = env.step(action)
    replay_buffer.append((state, actions, reward, next_state, done))
    return next_state, reward, done, info

optimizer = keras.optimizer.Adam(lr=1e-3)
loss_fn = keras.losses.mean_squared_error

def training_step(batch_size):
    experiences = sample_experiences(batch_size)
    states, actions, rewards, next_states, dones = experiences
    next_Q_values = model.predict(next_states)
    max_next_Q_values = np.max(next_Q_values, axis=1)
    target_Q_values = (rewards + (1-dones) * discount_factor 8 max_next_Q_values)
    mask = tf.one_hot(actions, n_outputs)
    with tf.GradientTape as tape:
        all_Q_values = model(states)
        Q_values = tf.reduce_mean(all_Q_values * mask, axis = 1, keepdims = True)
        loss = tf.reduce_mean(loss_fn(target_Q_values, Q_values))
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.appy_gradients(zip(grads, model.trainable_variables))

ep = []
rew = []
    
for episode in range(600):
    obs = env.reset()
    for step in range(200):
        epsilon = max(1 - episode / 500, 0.01)
        obs, reward, done, info = play_one_step(env, obs, epsilon)
        ep.append(epsilon)
        rew.append(reward)
        if done:
            break
    if episode > 50:
        training_step(batch_size)

plt.plot(ep, rew)
plt.show()

除以下几点外，代码与本书中的代码完全相同。

1-我添加了

%pylab inline
%matplotlib inline

2-我在训练循环中添加了ep和rew列表，以便以后绘制它们的内容。

但是，当我执行上述操作时，什么都没有发生。有人可以指导我吗？

Answer 1

您正尝试运行此脚本600x200 = 120,000次迭代。我不知道您正在使用哪台机器，但是除非您拥有强大的硬件-否则它可能要花费很长时间（strongstrong）。我想“什么也没有发生”只是您的计算机仍在运行，试图运行120,000次迭代...

此外，您还将所有这些迭代收集到列表中。您期望一个pythonic列表将包含120,000个值。我不太确定是否有可能。尝试改为使用numpy数组。与matplotlib相同，我不确定是否知道可以绘制120,000个值。

您在上方看到的图表是600集，而不是每个集中所有步骤的所有120,000值。

总结：从小处开始。播放10集，每集执行10个步骤或类似的操作。看看是否可行，然后从那里开始。

matplotlib图未显示在jupyter笔记本中

1 个答案: