Question

这是从头开始进行LSTM反向传播的实现，我不确定dfhs（hs状态的衍生物）是否使用正确的方法？请纠正我在我尝试了很多次的代码中错了的地方，但LSTM的预测仍然可怕，训练LSTM所需的时间最长吗？关于反向传播的任何建议都非常有用，因为我不知道出了什么问题-提前谢谢

我尝试将迭代次数从5,000增加到10,000，这最初是并尝试降低学习率并增加批次大小

def backward(xint,hprev,pcs,vy,nw):
    xs , ys , cs , hs , fg , ig , cg , og = {},{},{},{},{},{},{},{}
    hs[-1] = hprev
    cs[-1] = pcs
    intx = xint
#Forward propagation to store all the state necessary for back prop
    for i in range(nw):
        xp = np.zeros(xl)
        xp[intx] = 1

        x = np.hstack((hs[i-1],xp))
        xs[i] = x
        fg[i] = sigmoid(np.dot(x,wf))
        ig[i] = sigmoid(np.dot(x,wi))
        cg[i] = tangent(np.dot(x,wc))
        csc = (cs[i-1] * sigmoid(np.dot(x,wf))) + (sigmoid(np.dot(x,wi)) * sigmoid(np.dot(x,wc)))
        cs[i] = (cs[i-1] * sigmoid(np.dot(x,wf))) + (sigmoid(np.dot(x,wi)) * sigmoid(np.dot(x,wc)))
        og[i] = sigmoid(np.dot(x,wo))
        hs[i] = sigmoid(np.dot(x,wo)) * tangent(csc)
        hsc = sigmoid(np.dot(x,wo)) * tangent(csc)
        ys[i] = sigmoid(np.dot(hsc,wy))
        intx = np.argmax(vy[i-1])

    dwy = np.zeros((yl,d))
    dwf = np.zeros((xl+yl,yl))
    dwi =  np.zeros((xl+yl,yl))
    dwc =  np.zeros((xl+yl,yl))
    dwo =  np.zeros((xl+yl,yl))

    dfhs = np.zeros(yl)
    dfcs = np.zeros(yl)
    totalError = 0
#Back Propagation
    for i in reversed(range(nw)):
        merror = ys[i] - vy[i]
        dwy += np.dot(np.atleast_2d(hs[i]).T,np.atleast_2d((merror*dsigmoid(ys[i]))))
        error = np.dot(merror,wy.T)
        totalError += np.sum(error)
        e = np.clip(error+dfhs,-6,6)
        dho = tangent(cs[i]) * e
        dho = dsigmoid(og[i]) * dho
        dwo += np.dot(np.atleast_2d(xs[i]).T,np.atleast_2d(dho))
        dc = og[i] * e * dtangent(cs[i])
        dc = np.clip(dc + dfcs,-6,6)
        dhf = cs[i-1] * dc
        dhf = dsigmoid(fg[i]) * dhf
        dwf += np.dot(np.atleast_2d(xs[i]).T,np.atleast_2d(dhf))
        dhi = cg[i] * dc
        dhi = dsigmoid(ig[i]) * dhi 
        dwi += np.dot(np.atleast_2d(xs[i]).T,np.atleast_2d(dhi))
        dhc = ig[i] * dc
        dhc = dsigmoid(cg[i]) * dhi 
        dwc += np.dot(np.atleast_2d(xs[i]).T,np.atleast_2d(dhc))

        dfhs = np.dot(dho,wo.T)[:yl]+np.dot(dhc,wc.T)[:yl]+np.dot(dhi,wi.T)[:yl]+np.dot(dhf,wf.T)[:yl]
        dfcs = fg[i] * dc
    return totalError , hs[nw-1] , cs[nw-1] , dwf , dwi , dwc , dwo , dwy

预期的年：

[59, 28, 60, 4, 16, 41, 59, 23, 59, 49, 61, 60, 17, 1, 28, 59, 60, 66, 41, 28, 49, 16, 41, 60, 26, 22, 4, 12, 22, 60, 17, 16, 18, 28, 60, 9, 41, 16, 4, 60, 63, 41, 16, 34, 51, 79, 28, 65, 60, 65, 41, 28, 22, 4, 12, 61, 60, 1, 28, 60, 9, 16, 34, 59, 65, 60, 1, 23, 4, 12, 28, 79, 9, 60, 63, 41, 22, 59, 12, 9, 16, 41, 4, 28, 65, 60, 23, 59, 60, 1, 23, 12, 60, 51, 28, 65, 60, 23, 59, 63, 16, 60, 22, 60, 1, 16, 41, 41, 23, 51, 79, 28, 60, 46, 28, 41, 4, 23, 59, 47, 60, 45, 28, 60, 79, 22, 5, 60, 16, 59, 60, 1, 23, 12, 60, 22, 41, 4, 16, 34, 41, 14, 79, 23, 18, 28, 60, 51, 22, 52, 18, 61, 60, 22, 59, 65, 60, 23, 9, 60, 1, 28, 60, 79, 23, 9, 63, 28, 65, 60, 1, 23, 12, 60, 1, 28, 22, 65, 60, 22, 60, 79, 23, 63, 63, 79, 28, 60, 1, 28, 60, 52, 16, 34, 79, 65, 60, 12, 28, 28, 60, 1, 23, 12, 60, 51, 41, 16, 17, 59, 60, 51, 28, 79, 79, 5, 61, 60, 12, 79, 23,........]

早晨，当格雷戈尔·萨姆萨（Gregor Samsa）从动荡的梦中醒来时，他发现自己躺在床上变成了可怕的害虫。他躺在盔甲一样的背上，如果稍稍抬起头，他可以看到棕色的腹部，略微呈圆顶状，被拱形部分分成僵硬的部分。床上用品几乎无法覆盖它，似乎随时都可以滑下来。与他其余部分的大小相比，他的多条腿可怜的稀薄，他无助地挥了挥手。

“我怎么了？”他想。这不是梦。他的房间虽然很小，但却是一个合适的人类房间，它和平地位于四面熟悉的墙壁之间。桌子上散布着一系列纺织品样品-萨姆萨是一位旅行推销员-上面放着一张他最近从一本插图杂志上剪下来的照片，并装在一个漂亮的镀金框架中。它显示一位女士穿着直立的毛皮帽子和毛皮围巾，坐着，抬起沉重的皮草罩，覆盖了她的整个下臂，向着观众。

格雷戈尔然后转身向沉闷的天气望向窗外。雨滴……。

Actual Output:
[79, 66, 41, 16, 67, 49, 66, 73, 5, 9, 18, 76, 35, 22, 69, 75, 28, 55, 60, 6, 28, 6, 75, 62, 44, 62, 6, 62, 23, 23, 20, 35, 20, 20, 23, 55, 35, 35, 6, 53, 35, 63, 63, 63, 63, 63, 53, 63, 27, 53, 53, 23, 6, 26, 55, 35, 16, 23, 55, 38, 23, 38, 53, 53, 53, 74, 79, 79, 70, 61,........]
/*lGro"gG2yfk9;aq8e: ReR8xMxRxii1;11i:;;R);ttttt)t*))iRS:;oi:ziz)))0llB,b0oo)4***0ooooooooooooooooooooo00:oOzoo0:oooczzzznoooo9oomWaBob,1oFm"5ferr499hAd1""rrfBII..AAAAhkCWk...%%;qzz7/V@@PLx:P@o0PIPPJJk%M96*b@,BoTi ,/mU)UM*0YBa;oOjouu':ahhBBhQçdVgfPtP-L0JFuuJOwwsY6E/5XdzzgBB9z05LuVxx7xy;LLU6@eee..B5QuL@'ilPg66:@@@4@e@,4,k.../v?

对“ LSTM向后传播”的怀疑及其受训练的指数时间长

0 个答案: