Activations

Activation units and derivatives.

  • $Tanh(x)=1-{2 \over e^{2x}+1}={e^x-e^{-x} \over e^x+e^{-x}}$
  • ${\delta Tanh \over \delta x}=({2 \over e^x+e^{-x}})^2=1-Tanh^2(x)$
  • $Sig(x)={1\over{1+e^{-x}}}$
  • ${\delta Sig\over\delta x}=Sig(x)(1-Sig(x))
    $