Average Pool in Time

Reduce t-dimention in RNN with average pooling.

Forward

  • Let $s$ be the stride.
  • $0\le t\le T$
  • $S^{(l,-1)}=S^{(l,T+1)}=0$
  • $S^{(l,\bar t)}=z^{(l,\bar t)}={1\over3}\sum_{i=-1}^1S^{(l-1,2\bar t+i)}$

Backward

  • ${\delta E\over\delta z^{(l,\bar t)}}={\delta S^{(l,\bar t)}\over\delta z^{(l,\bar t)}}{\delta E\over\delta S^{(l,\bar t)}}
    ={\delta E\over\delta S^{(l,\bar t)}}$
  • ${\delta E\over\delta S^{(l-1,t)}}
    =\left\{\begin{array}{c}
    \sum_{i=0}^1{\delta z^{(l-1,\left\lfloor{t\over 2}\right\rfloor+i)}\over\delta S^{(l-1,t)}}{\delta E\over\delta z^{(l-1,\left\lfloor{t\over 2}\right\rfloor+i)}}; \space t\space odd\\
    {\delta z^{(l-1,\left\lfloor{t\over 2}\right\rfloor)}\over\delta S^{(l-1,t)}}{\delta E\over\delta z^{(l-1,\left\lfloor{t\over 2}\right\rfloor)}}
    ; \space t\space even\\
    \end{array}\right.
    =\left\{\begin{array}{c}
    \sum_{i=0}^1{1\over 3}{\delta E\over\delta z^{(l-1,\left\lfloor{t\over 2}\right\rfloor+i)}}; \space t\space odd\\
    {1\over 3}{\delta E\over\delta z^{(l-1,\left\lfloor{t\over 2}\right\rfloor)}}
    ; \space t\space even\\
    \end{array}\right.
    $
    $t$ odd, then $(t-1)\over 2$ and ${(t-1)\over 2}+1$
    $t$ even, then $t\over 2$