BP-Network

Defination

desired output $\boldsymbol{d}\in \mathbb{R}^{1\times 1}$

1st laywer input $\boldsymbol{x}^{(1)}\in \mathbb{R}^{n\times 1}$

1st laywer weight $\boldsymbol{W}^{(1)}\in \mathbb{R}^{m\times n}$

1st laywer bias $\boldsymbol{b}^{(1)}\in \mathbb{R}^{m\times 1}$

1st laywer output $\boldsymbol{z}^{(1)}\in \mathbb{R}^{m\times 1}$

2st laywer input $\boldsymbol{x}^{(2)}\in \mathbb{R}^{m\times 1}$

2nd laywer weight $\boldsymbol{W}^{(2)}\in \mathbb{R}^{1\times m}$

2nd laywer bias $\boldsymbol{b}^{(2)}\in \mathbb{R}^{1\times 1}$

2st laywer output $\boldsymbol{z}^{(2)}\in \mathbb{R}^{1\times 1}$

output $\boldsymbol{y} \in \mathbb{R}^{1\times 1}$

Activation function:

1. sigmod

   $$f(x) = \frac{1}{1+e^{-x}}\\ f'(x) = f(x)(1-f(x))$$

2. RELU

   $$f(x) =\left\{ \begin{matrix} x \ x>0\\ 0 \ x\leq0 \end{matrix}\right.\\ f'(x)= \left\{ \begin{matrix} 1 \ x>0\\ 0 \ x\leq0 \end{matrix}\right.$$

forward propagation

$$\left\{ \begin{aligned} &\boldsymbol{z}^{(1)} = \boldsymbol{W}^{(1)}\boldsymbol{x}^{(1)}+\boldsymbol{b}^{(1)}\\ &\boldsymbol{x}^{(2)} = f(\boldsymbol{z}^{(1)})\\ &\boldsymbol{z}^{(2)} = \boldsymbol{W}^{(2)}\boldsymbol{x}^{(2)}+\boldsymbol{b}^{(2)}\\ &\boldsymbol{y}=f(\boldsymbol{z}^{(2)}) \end{aligned} \right.$$

Define Loss function$J(\boldsymbol{\cdot})$

$$J(\boldsymbol{W}^{(1)},\boldsymbol{W}^{(2)},\boldsymbol{b}^{(1)},\boldsymbol{b}^{(2)}) =\frac{1}{2}\boldsymbol{e}^2$$ $$\boldsymbol{e}^2=\Vert \boldsymbol{y}-\boldsymbol{d} \Vert_2^2=(\boldsymbol{y}-\boldsymbol{d})^T(\boldsymbol{y}-\boldsymbol{d})$$

Take the derivative of the loss function

$$\left\{ \begin{aligned} \Delta J_{\boldsymbol{b}^{(2)}}&=-ey(1-y)\\ \Delta J_{\boldsymbol{W}^{(2)}}&=-ey(1-y){\boldsymbol{x}^{(2)}}^T\\ \Delta J_{\boldsymbol{b}^{(1)}}&=-ey(1-y){\boldsymbol{W}^{(2)}}^T\cdot\boldsymbol{x}^{(2)}\cdot(1-\boldsymbol{x}^{(2)})\\ \Delta J_{\boldsymbol{W}^{(1)}}&=-ey(1-y){\boldsymbol{W}^{(2)}}\boldsymbol{x}^{(2)}\cdot(1-\boldsymbol{x}^{(2)})\boldsymbol{x}^{(1)}\\ \end{aligned} \right.$$

Back propagation

Learning step $\mu$

$$\left\{ \begin{aligned} \boldsymbol{b}^{(2)}_{new} &= \boldsymbol{b}^{(2)}_{old} + \mu\Delta J_{\boldsymbol{b}^{(2)}}\\ \boldsymbol{W}^{(2)}_{new} &= \boldsymbol{W}^{(2)}_{old} + \mu\Delta J_{\boldsymbol{W}^{(2)}}\\ \boldsymbol{b}^{(1)}_{new} &= \boldsymbol{b}^{(1)}_{old} + \mu\Delta J_{\boldsymbol{b}^{(1)}}\\ \boldsymbol{W}^{(1)}_{new} &= \boldsymbol{W}^{(1)}_{old} + \mu\Delta J_{\boldsymbol{W}^{(1)}} \end{aligned} \right.$$

Simulation