Study Notes of a Simple Neuro Network

 Thanks for Stanford University's online course of Loss Functions and Optimisation and its accompanying web demo, now I finally have a concrete simple example of several Loss Function implementations, which I can delve into the maths.

The Architecture

In this architecture, W ∈  ^2×3, x ∈  ^1×2, b ∈  ^1×3, s ∈  ^1×3.

s = f(x, W) = Wx + b

The value of W is initialised to: ${\displaystyle {\begin{bmatrix}1&9&-13\\20&5&-6\end{bmatrix}}}$

The initial value of b is set to: ${\displaystyle {\begin{bmatrix}1&9&-13\\20&5&-6\end{bmatrix}}}$

Calculating the Scores

There are 9 sets of input values for X, hence, X_i, i=[0..8]. Given the dataset, the s is calculated:

x0	x1	y	s0 = w0,0*x0 + w1,0*x1 + b0	s1 = w0,1*x0 + w1,1*x1 + b1	s2 = w0,2*x0 + w1,2*x1 + b2
0.5	0.4	0	1*0.5 + 2*0.4 + 0 = 1.3	2*0.5 + (-4)*0.4 + 0.5 =  -0.1	3*0.5 + (-1)*0.4 + (-0.5) =  0.6
0.8	0.3	0	1.4	0.9	1.6
0.3	0.8	0	1.9	-2.1	-0.4
-0.4	0.3	1	0.2	-1.5	-2
-0.3	0.7	1	1.1	-2.9	-2.1
-0.7	0.2	1	-0.3	-1.7	-2.8
0.7	-0.4	2	-0.1	3.5	2
0.5	-0.6	2	-0.7	3.9	1.6
-0.4	-0.5	2	-1.4	1.7	-1.2

Multiclass SVM Loss Functions

Weston Watkins 1999

$$L_i=\sum _{j\ne y_i}^{}max(0,s_j–s_{y_i}+\; 1)$$

where i=[0, number of input samples), j=[0, number of output classes)

since  s_j = W_jx_i + b_j and s_yi= W_yix_i + b_yi  

we have 

$$\frac{\partial L_i}{\partial s_j}=1\text{ , }\frac{\partial L_i}{\partial W_j}=x_i\text{ and }\frac{\partial L_i}{\partial b_j}=1$$ $$\frac{\partial L_i}{\partial s_{y_i}}=\mathrm{–n}\text{ , }\frac{\partial L_i}{\partial W_{y_i}}={\mathrm{–x}}_{i}n\text{ and }\frac{\partial L_i}{\partial b_{y_i}}=–1n$$

where n is the number of times that s_yi appeared in L_i (since L_i is a sum of many terms).
n should be in the range of [0, number of output classes - 1).

y	s0	s1	s2	Li	Note
0	1.3	-0.1	0.6	max(0, s1-s0+1) + max(0, s2-s0+1) = max(0, -0.1-1.3+1) + max(0.6-1.3+1)= 0+0.3= 0.3	Syi=S0
0	1.4	0.9	1.6	max(0, 0.9-1.4+1) + max(1.6-1.4+1) = 0.5 + 1.2 = 1.7
0	1.9	-2.1	-0.4	max(0, -2.1-1.9+1) + max(0, -0.4-1.9+1) = max(0, -3) + max(0, -1.3) = 0+0 = 0
1	0.2	-1.5	-2	max(0, s0-s1+1) + max(0, s2-s1+1) = max(0, 0.2+1.5+1) + max(0, -2+1.5+1) = 2.7+0.5 = 3.2	Syi=S1
1	1.1	-2.9	-2.1	max(0, 1.1+2.9+1) + max(-2.1+2.9+1) = 5+1.8 = 6.8
1	-0.3	-1.7	-2.8	max(0, -0.3+1.7+1) + max(-2.8+1.7+1) = 2.4+0 = 2.4
2	-0.1	3.5	2	max(0, s0-s2+1) + max(0, s1-s2+1) = max(0, -0.1-2+1) + max(0, 3.5-2+1) = 0 + 2.5 = 2.5	Syi=S2
2	-0.7	3.9	1.6	max(-0.7-1.6+1) + max(0, 3.9-1.6+1) = 0 + 3.3 = 3.3
2	-1.4	1.7	-1.2	max(0, -1.4+1.2+1)+max(0, 1.7+1.2+1) = 0.8 + 3.9 = 4.7
				L = 1/N ⋅∑Li = 2.766666667

One vs. All

$$L_i=max(0,{\mathrm{–s}}_{y_i}+1)+\sum _{j\ne y_i}^{}max(0,s_j+1)$$

The partial derivatives are the same as the Weston Watkins 1999 formula, with n=1 because there is only one S_yi in the formula.

y	s0	s1	s2	Li	Note
0	1.3	-0.1	0.6	max(0, 1-s0) + max(0, 1+s1) + max(0, 1+s2) = max(0, 1-1.3) + max(0, 1-0.1) + max(0, 1+0.6) = 0 + 0.9 + 1.6 = 2.5	Syi=S0
0	1.4	0.9	1.6	max(0, 1-1.4) + max(0, 1+0.9) + max(0, 1+1.6) = 0 + 1.9 + 2.6 = 4.5
0	1.9	-2.1	-0.4	0.6
1	0.2	-1.5	-2	max(0, 1+s0) + max(0, 1-s1) + max(0, 1+s2) = max(0, 1+0.2) + max(0, 1+1.5) + max(0, 1-2) = 1.2 + 2.5 + 0 = 3.7	Syi=S1
1	1.1	-2.9	-2.1	max(0, 1+1.1) + max(0, 1+2.9) + max(0, 1-2.1) = 2.1 + 3.9 + 0 = 6
1	-0.3	-1.7	-2.8	3.4
2	-0.1	3.5	2	max(0, 1+s0) + max(0, 1+s1) + max(0, 1-s2) = max(0, 1-0.1) + max(0, 1+3.5) + max(0, 1-2) = 0.9 + 4.5 + 0 = 5.4	Syi=S2
2	-0.7	3.9	1.6	5.2
2	-1.4	1.7	-1.2	4.9
				L = 1/N ⋅∑Li = 4.022222222

Structured SVM

L_i= max(0,  max( s_j ) – s_yi+ 1)    , where j ≠ y_i

The partial derivatives are the same as the One vs. All formula.

y	s0	s1	s2	Li	Note
0	1.3	-0.1	0.6	max(0, max(s1, s2)-s0+1) = max(0, max(-0.1, 0.6)-1.3+1) = max(0, 0.6-1.3+1) = 0.3	Syi=S0
0	1.4	0.9	1.6	1.2
0	1.9	-2.1	-0.4	max(0, max(-2.1, -0.4)-1.9+1) = max(0, -0.4-1.9+1) = 0
1	0.2	-1.5	-2	max(0, max(s0, s2)-s1+1) = max(0, max(0.2, -2)+1.5+1) = max(0, 0.2+1.5+1) = 2.7	Syi=S1
1	1.1	-2.9	-2.1	5
1	-0.3	-1.7	-2.8	2.4
2	-0.1	3.5	2	2.5	Syi=S2
2	-0.7	3.9	1.6	3.3
2	-1.4	1.7	-1.2	3.9
				L = 1/N ⋅∑Li = 2.366666667

Softmax

$$L_i=–ln(\mathrm{P(}Y=y_i|X=x_i))=–ln(\frac{e^{s_{y_i}}}{\sum _j^{}{e}^{s_j}})$$

Finding partial derivatives using chain rule:

$\text{let }\frac{\partial L_i}{\partial W_j}=\frac{\partial L_i}{\partial u}\times \frac{\partial u}{\partial v}\times \frac{\partial v}{\partial W_j}$ $\text{ where }u=e^{s_j}\text{, }v=s_j=W_j\cdot x_{}+b_j$

solve for each term:

∂v/∂W_j = ∂(W_j⋅x+b_j)/∂W_j = x

∂u/∂v = ∂(e^v)/∂v = e^v = e^s_j

∂L_i/∂u = ∂[–ln( e^s_yi / ∑_ju )]/∂u =  ∂[–ln( e^s_yi ) + ln(∑_ju )]/∂u

    =  ∂[–s_yi + ln(∑_ju )]/∂u = –0 + ∂[ln(∑_ju )]/∂u

   =  1 /  ∑_ju = 1 /  ∑_je^s_j     

(chain rule steps omitted showing ∂[ln(A+B+C)]/∂A = 1/(A+B+C) )

therefore, $$\frac{\partial L_i}{\partial W_j}=P_j\cdot x_i$$  

Solve for  ∂L_i/∂b_j = ∂L_i/∂u ⋅ ∂u/∂v ⋅ ∂v/∂b_j 

the only new term is ∂v/∂b_j = ∂(W_j⋅x+b_j)/∂b_j = 1

therefore,  $$\frac{\partial L_i}{\partial b_j}=P_j$$

Solve for ∂L_i/∂W_yi 

let ∂L_i/∂W_yi = ∂L_i/∂u ⋅ ∂u/∂v ⋅ ∂v/∂W_yi   where u=e^s_yi  , v=s_yi = W_yi⋅x_i+b_yi

solve for each term:

∂v/∂W_yi = ∂(W_yi⋅x_i+b_yi)/∂W_yi = x_i

∂u/∂v = ∂(e^v)/∂v = e^v = e^s_yi

∂L_i/∂u = ∂[–ln( u / ∑_ju )]/∂u =  ∂[–ln( u ) + ln(∑_ju )]/∂u 

    = ∂[–ln( u )]/∂u + ∂[ln(∑_ju )]/∂u

   =  –1/u + 1 /  ∑_ju  = –1/e^s_yi  + 1 /  ∑_je^s_j    

therefore,  $$\frac{\partial L_i}{\partial W_{y_i}}=(P_{y_i}–1)\cdot x_i$$

Solve for ∂L_i/∂b_yi =  ∂L_i/∂u ⋅ ∂u/∂v ⋅ ∂v/∂b_yi 

the only different term here is 

∂v/∂b_yi = ∂(W_yi⋅x_i+b_yi)/∂b_yi  = 1

therefore,  $$\frac{\partial L_i}{\partial b_{y_i}}=P_{y_i}–1$$  

y	s0	s1	s2	Li (using ln)	Li  (log10)	Note
0	1.3	-0.1	0.6	-ln( exp(s0) / (exp(s0)+exp(s1)+exp(s2) )= 0.3	2.5	Syi=S0
0	1.4	0.9	1.6	1.7	4.5
0	1.9	-2.1	-0.4	0	0.6
1	0.2	-1.5	-2	-ln( exp(s1) / (exp(s0)+exp(s1)+exp(s2) )= 3.2	3.7	Syi=S1
1	1.1	-2.9	-2.1	6.8	6
1	-0.3	-1.7	-2.8	2.4	3.4
2	-0.1	3.5	2	-ln( exp(s2) / (exp(s0)+exp(s1)+exp(s2) )= 2.5	5.4	Syi=S2
2	-0.7	3.9	1.6	3.3	5.2
2	-1.4	1.7	-1.2	4.7	4.9
				L = 1/N ⋅∑Li = 1.836640394	0.7976427883