单层前馈神经网络

符号解释

1. $X^k=\{x_1^k,x_2^k,\cdots ,x_d^k\}$$X^{k} = \{x_1 ^{k},x_2 ^{k},\cdots,x_d ^{k} \}$：输入的第$k$$k$个数据，共$d$$d$维
2. $V_{d\times q}$$V_{d \times q}$：输入层到隐藏层的权重，$q$$q$为隐藏层结点的个数
3. ${\gamma }_h,(1\le h\le q)$$\gamma_h,(1 \le h \le q)$：输入层到隐藏层的阈值（偏置）
4. ${\alpha }_h,(1\le h\le q)$$\alpha_h,(1 \le h \le q)$：隐藏层的输入
5. $b_h,(1\le h\le q)$$b_h,(1 \le h \le q)$：隐藏层的输出
6. $W_{q\times l}$$W_{q \times l}$：隐藏层到输出层的权重，$l$$l$为输出层结点的个数
7. ${\theta }_j,(1\le j\le l)$$\theta_j,(1 \le j \le l)$：隐藏层到输出层的阈值（偏置）
8. ${\beta }_j,(1\le j\le l)$$\beta_j,(1 \le j \le l)$，输出层的输入
9. ${\hat{y}}_j^k,(1\le j\le l)$$\hat{y}_j ^{k}, (1 \le j \le l)$：输出层的输出
10. $y_j^k,(1\le j\le l)$$y_j ^{k}, (1 \le j \le l)$：输出层的label
11. $f$$f$：sigmoid函数，其导数特点为$f^{\prime }(x)=f(x)(1-f(x))$$f'(x) = f(x)(1-f(x))$
12. $E_K$$E_K$：损失函数

前馈计算

1. 输入层到隐藏层

   $$\begin{gathered} \begin{array}{}\text{(1)}& \begin{array}{rl}{\alpha }_h& =\sum _{i=1}^d{v}_{ih}{x}_i^k\\ b_h& =f({\alpha }_h-{\gamma }_h)\end{array}\end{array} \\ \text{其}\text{中}，1\le h\le q \end{gathered}$$

2. 隐藏层到输出层

   $$\begin{gathered} \begin{array}{}\text{(2)}& {\beta }_j=\sum _{h=1}^q{w}_{hj}{b}_h \\ {\hat{y}}_j^k=f({\beta }_j-{\theta }_j) \\ \text{其}\text{中}，1\le j\le l\end{array} \end{gathered}$$

3. 损失函数

反向传播

1. $w_{hj}$$w_{hj}$

   

   因此，

   $$\begin{array}{}\text{(4)}& \frac{\partial E_k}{\partial w_{hj}}=\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial w_{hj}}\end{array}$$

   分开计算：

   $$\begin{array}{}\text{(5)}& \begin{array}{rl}\frac{\partial E_k}{\partial {\hat{y}}_j^k}& =\frac{\partial \left[\frac{1}{2}\sum _{j=1}^l{({\hat{y}}_j^k-y_j^k)}^2\right]}{\partial {\hat{y}}_j^k}\\ & =\frac{1}{2}\times 2\times ({\hat{y}}_j^k-y_j^k)\times 1\\ & ={\hat{y}}_j^k-y_j^k\\ \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}& =\frac{\partial \left[f({\beta }_j-{\theta }_j)\right]}{\partial {\beta }_j}\\ & =f^{\prime }({\beta }_j-{\theta }_j)\times 1\\ & =f({\beta }_j-{\theta }_j)\times [1-f({\beta }_j-{\theta }_j)]\\ & ={\hat{y}}_j^k(1-{\hat{y}}_j^k)\\ \frac{\partial {\beta }_j}{\partial w_{hj}}& =\frac{\partial \left(\sum _{h=1}^q{w}_{hj}{b}_h\right)}{\partial w_{hj}}\\ & =b_h\end{array}\end{array}$$

   令：

   $$\begin{array}{}\text{(6)}& g_j=-\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}=-({\hat{y}}_j^k-y_j^k)\cdot {\hat{y}}_j^k(1-{\hat{y}}_j^k)={\hat{y}}_j^k(1-{\hat{y}}_j^k)(y_j^k-{\hat{y}}_j^k)\end{array}$$

   所以：

   $$\begin{array}{}\text{(7)}& \begin{array}{rl}\mathrm{\Delta }{w}_{hj}& =-\eta \frac{\partial E_k}{\partial w_{hj}}\\ & =-\eta \frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial w_{hj}}\\ & =\eta g_j{b}_h\end{array}\end{array}$$

2. ${\theta }_j$$\theta_j$

   

   因此，

   $$\begin{array}{}\text{(8)}& \begin{array}{rl}\frac{\partial E_k}{\partial {\theta }_j}& =\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\theta }_j}\\ & =({\hat{y}}_j^k-y_j^k)\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\theta }_j}\\ & =({\hat{y}}_j^k-y_j^k)\cdot \frac{\partial \left[f({\beta }_j-{\theta }_j)\right]}{\partial {\theta }_j}\\ & =({\hat{y}}_j^k-y_j^k)\cdot f^{\prime }({\beta }_j-{\theta }_j)\times -1\\ & =(y_j^k-{\hat{y}}_j^k)\cdot f^{\prime }({\beta }_j-{\theta }_j)\\ & =(y_j^k-{\hat{y}}_j^k){\hat{y}}_j^k(1-{\hat{y}}_j^k)\end{array}\end{array}$$

   所以：

   $$\begin{array}{}\text{(9)}& \begin{array}{rl}\mathrm{\Delta }{\theta }_j& =-\eta \frac{\partial E_k}{\partial {\theta }_j}\\ & =-\eta (y_j^k-{\hat{y}}_j^k){\hat{y}}_j^k(1-{\hat{y}}_j^k)\\ & =-\eta g_j\end{array}\end{array}$$

3. $v_{ih}$$v_{ih}$

   

   因此，

   $$\begin{array}{}\text{(10)}& \frac{\partial E_k}{\partial v_{ih}}=\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\alpha }_h}\cdot \frac{\partial {\alpha }_h}{\partial v_{ih}}\end{array}$$

   分开计算，

   $$\begin{array}{}\text{(11)}& \begin{array}{rl}\frac{\partial {\beta }_j}{\partial b_h}& =\frac{\partial \left(\sum _{h=1}^q{w}_{hj}{b}_h\right)}{\partial b_h}\\ & =w_{hj}\\ \frac{\partial b_h}{\partial {\alpha }_h}& =\frac{\partial \left[f({\alpha }_h-{\gamma }_h)\right]}{\partial {\alpha }_h}\\ & =f^{\prime }({\alpha }_h-{\gamma }_h)\times 1\\ & =f({\alpha }_h-{\gamma }_h)\times [1-f({\alpha }_h-{\gamma }_h)]\\ & =b_h(1-b_h)\\ \frac{\partial {\alpha }_h}{\partial v_{ih}}& =\frac{\partial \left(\sum _{i=1}^d{v}_{ih}{x}_i\right)}{\partial v_{ih}}\\ & =x_i\end{array}\end{array}$$

   令：

   $$\begin{array}{}\text{(12)}& e_h=-\frac{\partial E_k}{\partial {\alpha }_h}=-\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\alpha }_h}=b_h(1-b_h)\sum _{j=1}^l{w}_{hj}{g}_j\end{array}$$

   所以，

   $$\begin{array}{}\text{(13)}& \begin{array}{rl}\mathrm{\Delta }{v}_{ih}& =-\eta \frac{\partial E_k}{\partial v_{ih}}\\ & =-\eta \sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\alpha }_h}\cdot \frac{\partial {\alpha }_h}{\partial v_{ih}}\\ & =\eta e_h{x}_i\end{array}\end{array}$$

4. ${\gamma }_h$$\gamma_h$

因此，

所以：