Activations

Activation units and derivatives.

• $Tanh(x)=1-{2 \over e^{2x}+1}={e^x-e^{-x} \over e^x+e^{-x}}$
• ${\delta Tanh \over \delta x}=({2 \over e^x+e^{-x}})^2=1-Tanh^2(x)$
• $Sig(x)={1\over{1+e^{-x}}}$
• ${\delta Sig\over\delta x}=Sig(x)(1-Sig(x))$