SELU integrals

Here we compute the integrals for the first and second moment moment of the next layer.

Simplify[Integrate[λ(α*Exp[z]-α)*PDF[NormalDistribution[μ*ω,Sqrt[ν* τ]],z],{z,-Infinity,0}]+Integrate[λ*z*PDF[NormalDistribution[μ*ω,Sqrt[ν* τ]],z],{z,0,Infinity}],Assumptions -> ν∈ Reals&& ν>0&&τ∈ Reals&&τ>0]

Simplify[Integrate[(λ(α*Exp[z]-α))^2*PDF[NormalDistribution[μ*ω,Sqrt[ν* τ]],z],{z,-Infinity,0}]+Integrate[(λ*z)^2*PDF[NormalDistribution[μ*ω,Sqrt[ν* τ]],z],{z,0,Infinity}],Assumptions -> ν∈ Reals&& ν>0&&τ∈ Reals&&τ>0]

Mean, second moment,  variance

We define the mean, second moment and the variance in the next layer.

firstMoment[μ_, ω_, ν_, τ_,λ_,α_ ]:=(1/2) λ ((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) (Sqrt[ν τ])+μ ω+μ ω Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]-α Erfc[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+(E^(((ν τ)/2)+μ ω)) α Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))])
secondMoment[μ_, ω_, ν_, τ_,λ_,α_ ]:=(1/2) (λ^2) (ν τ+(E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) μ (Sqrt[ν τ]) ω+(μ^2) (ω^2)+(ν τ+(μ^2) (ω^2)) Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+(α^2) Erfc[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]-2 (E^(((ν τ)/2)+μ ω)) (α^2) Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+(E^(2 ν τ+2 μ ω)) (α^2) Erfc[((2 ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))])

variance[μ_, ω_, ν_, τ_,λ_,α_ ]:=secondMoment[μ, ω, ν, τ,λ,α ]-firstMoment[μ, ω, ν, τ,λ,α] ^2

Solve fixed point equations for alpha and lambda

Here, we solve the fixed point equations with respect to alpha and lambda.

Solve[firstMoment[0, 0, 1, 1,λ,α]==0,α]

α01:=-Sqrt[2/Pi]/(Exp[1/2]*Erfc[1/Sqrt[2]]-1)
N[α01]

Solve[variance[0, 0, 1, 1,λ,α01]==1,λ]

λ01:=((1-Sqrt[Exp[1]]*Erfc[1/Sqrt[2]])*Sqrt[2*Pi/(Exp[1]*Pi*Erfc[1/Sqrt[2]]^2-2*Sqrt[Exp[1]](2+Pi)Erfc[1/Sqrt[2]]+2*Exp[1]^2Erfc[Sqrt[2]]+Pi+2)])

Simplify[λ01]

N[λ01]

Fixed point for normalized weights

This is just a check whether the obtained parameters λ01 and α01 lead to the fixed point (0,1) for normalized weights.

Simplify[firstMoment[0, 0, 1, 1,λ01,α01]]

Simplify[variance[0, 0, 1, 1,λ01,α01 ]]

Definition of Jacobian entries

We now define the entries in the (2x2) Jacobian matrix of the mapping g: new mean derived w.r.t. old mean, new mean derived w.r.t. old variance, new variance derived w.r.t.o old mean, new variance derived w.r.t.o old variance.

We derive the mean (first moment) with respect to mu to obtain the entry H11 of the Jacobian:

Simplify[D[firstMoment[μ, ω, ν, τ,λ,α ],μ]]

H11[μ_, ω_, ν_, τ_,λ_,α_ ]:=(1/2) λ ω (1+Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+(E^(((ν τ)/2)+μ ω)) α Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))])

We derive the mean (first moment) with respect to nu to obtain the entry H12 of the Jacobian:

Simplify[D[firstMoment[μ, ω, ν, τ,λ,α ],ν]]

H12[μ_, ω_, ν_, τ_,λ_,α_ ]:=((E^(-(((μ^2) (ω^2))/(2 ν τ)))) λ τ (-(Sqrt[2]) (-1+α)+(E^((((ν τ+μ ω))^2)/(2 ν τ))) (Sqrt[π]) α (Sqrt[ν τ]) Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]))/(4 (Sqrt[π]) (Sqrt[ν τ]))

We derive the variance with respect to mu to obtain the entry H21 of the Jacobian:

D[variance[μ, ω, ν, τ,λ,α ],μ]

H21[μ_, ω_, ν_, τ_,λ_,α_ ]:=-(1/2) (λ^2) ((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) (Sqrt[ν τ])+μ ω+μ ω Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]-α Erfc[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+(E^(((ν τ)/2)+μ ω)) α Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]) (ω+(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) α ω)/(Sqrt[ν τ]))-(((E^(((ν τ)/2)+μ ω-((((ν τ+μ ω))^2)/(2 ν τ)))) (Sqrt[(2/π)]) α ω)/(Sqrt[ν τ]))+(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) μ (ω^2))/(Sqrt[ν τ]))-(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) μ (Sqrt[ν τ]) (ω^2))/(ν τ))+ω Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+(E^(((ν τ)/2)+μ ω)) α ω Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))])+(1/2) (λ^2) (-(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) (α^2) ω)/(Sqrt[ν τ]))+((2 (E^(((ν τ)/2)+μ ω-((((ν τ+μ ω))^2)/(2 ν τ)))) (Sqrt[(2/π)]) (α^2) ω)/(Sqrt[ν τ]))-(((E^(2 ν τ+2 μ ω-((((2 ν τ+μ ω))^2)/(2 ν τ)))) (Sqrt[(2/π)]) (α^2) ω)/(Sqrt[ν τ]))+(E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) (Sqrt[ν τ]) ω+2 μ (ω^2)-(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) (μ^2) (Sqrt[ν τ]) (ω^3))/(ν τ))+(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) ω (ν τ+(μ^2) (ω^2)))/(Sqrt[ν τ]))+2 μ (ω^2) Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]-2 (E^(((ν τ)/2)+μ ω)) (α^2) ω Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+2 (E^(2 ν τ+2 μ ω)) (α^2) ω Erfc[((2 ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))])

We derive the variance with respect to nu to obtain the entry H22 of the Jacobian:

D[variance[μ, ω, ν, τ,λ,α ],ν]

H22[μ_, ω_, ν_, τ_,λ_,α_ ]:=-(1/2) (λ^2) ((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (Sqrt[(2/π)]) (Sqrt[ν τ])+μ ω+μ ω Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]-α Erfc[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+(E^(((ν τ)/2)+μ ω)) α Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]) ((((E^(-(((μ^2) (ω^2))/(2 ν τ)))) τ)/((Sqrt[2 π]) (Sqrt[ν τ])))-(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) α μ τ ω)/((Sqrt[2 π]) (((ν τ))^(3/2))))-(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (μ^2) τ (ω^2))/((Sqrt[2 π]) (((ν τ))^(3/2))))+(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (μ^2) (Sqrt[ν τ]) (ω^2))/((Sqrt[2 π]) (ν^2) τ))-(1/(Sqrt[π]))2 (E^(((ν τ)/2)+μ ω-((((ν τ+μ ω))^2)/(2 ν τ)))) α ((τ/((Sqrt[2]) (Sqrt[ν τ])))-((τ (ν τ+μ ω))/(2 (Sqrt[2]) (((ν τ))^(3/2)))))+(1/2) (E^(((ν τ)/2)+μ ω)) α τ Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))])+(1/2) (λ^2) (τ+(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (α^2) μ τ ω)/((Sqrt[2 π]) (((ν τ))^(3/2))))+(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) μ τ ω)/((Sqrt[2 π]) (Sqrt[ν τ])))+(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) (μ^3) (Sqrt[ν τ]) (ω^3))/((Sqrt[2 π]) (ν^2) τ))-(((E^(-(((μ^2) (ω^2))/(2 ν τ)))) μ τ ω (ν τ+(μ^2) (ω^2)))/((Sqrt[2 π]) (((ν τ))^(3/2))))+(1/(Sqrt[π]))4 (E^(((ν τ)/2)+μ ω-((((ν τ+μ ω))^2)/(2 ν τ)))) (α^2) ((τ/((Sqrt[2]) (Sqrt[ν τ])))-((τ (ν τ+μ ω))/(2 (Sqrt[2]) (((ν τ))^(3/2)))))-(1/(Sqrt[π]))2 (E^(2 ν τ+2 μ ω-((((2 ν τ+μ ω))^2)/(2 ν τ)))) (α^2) ((((Sqrt[2]) τ)/(Sqrt[ν τ]))-((τ (2 ν τ+μ ω))/(2 (Sqrt[2]) (((ν τ))^(3/2)))))+τ Erf[((μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]-(E^(((ν τ)/2)+μ ω)) (α^2) τ Erfc[((ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))]+2 (E^(2 ν τ+2 μ ω)) (α^2) τ Erfc[((2 ν τ+μ ω)/((Sqrt[2]) (Sqrt[ν τ])))])

Jacobian entries for normalized weights

Here are the values of the Jacobian at the fixed point (0,1) for normalized weights.

N[H11[0, 0, 1, 1,λ01,α01 ]]

N[H21[0, 0, 1, 1,λ01,α01 ]]

N[H12[0, 0, 1, 1,λ01,α01 ]]

N[H22[0, 0, 1, 1,λ01,α01 ]]

{{N[H11[0, 0, 1, 1,λ01,α01 ]],N[H12[0, 0, 1, 1,λ01,α01 ]]},{N[H21[0, 0, 1, 1,λ01,α01 ]],N[H22[0, 0, 1, 1,λ01,α01 ]]}} //MatrixForm

Singular value

We have an explicit formula for the largest singular value of a (2x2) matrix, i.e. the following:

S[μ_, ω_, ν_, τ_,λ_,α_ ]:=0.5*Sqrt[(H11[μ, ω, ν, τ,λ,α]+H22[μ, ω, ν, τ,λ,α])^2 +(H21[μ, ω, ν, τ,λ,α]-H12[μ, ω, ν, τ,λ,α])^2 ]+0.5*Sqrt[(H11[μ, ω, ν, τ,λ,α]-H22[μ, ω, ν, τ,λ,α])^2 +(H21[μ, ω, ν, τ,λ,α]+H12[μ, ω, ν, τ,λ,α])^2 ]

We now check, whether the largest singular value of the Jacobian fixed point (0,1) is smaller then one. If it is smaller than 1, the fixed point (0,1) is a stable fixed point.

N[S[0, 0, 1, 1,λ01,α01  ]]

Numeric checks for Theorem 1: Singular value < 1; Mapping to domain

For non-normalized weights, the singular value is still below one in certain intervals for mean, variance of activations and mean and variance of weights. We numerically check if the statements of Theorem 1 are true.
First, we check whether the singular value stays below 1.

Maximize[{S[μ, ω, ν, τ,λ01,α01],-0.1≤μ≤0.1 && -0.1 ≤ω≤0.1 && 0.8≤ν≤ 1.5 &&  0.95≤τ≤1.1},{μ, ω, ν, τ}]

The maximal value of the singular value is below 1, therefore the numeric check is ok.

We now check whether the mapping stays in the pre-defined intervals.

Minimize[{firstMoment[μ, ω, ν, τ,λ01,α01],-0.1≤μ≤0.1 && -0.1 ≤ω≤0.1 && 0.8≤ν≤ 1.5 &&  0.95≤τ≤1.1},{μ, ω, ν, τ}]

Maximize[{firstMoment[μ, ω, ν, τ,λ01,α01],-0.1≤μ≤0.1 && -0.1 ≤ω≤0.1 && 0.8≤ν≤ 1.5 &&  0.95≤τ≤1.1},{μ, ω, ν, τ}]

The new mean is still in the interval [-0.1,0.1], therefore the numeric check is ok.

Minimize[{variance[μ, ω, ν, τ,λ01,α01],-0.1≤μ≤0.1 && -0.1 ≤ω≤0.1 && 0.8≤ν≤ 1.5 &&  0.95≤τ≤1.1},{μ, ω, ν, τ}]

Maximize[{variance[μ, ω, ν, τ,λ01,α01],-0.1≤μ≤0.1 && -0.1 ≤ω≤0.1 && 0.8≤ν≤ 1.5 &&  0.95≤τ≤1.1},{μ, ω, ν, τ}]

The new variance is still in the interval [0.8,1.5], therefore the numeric check is ok.

Numeric checks for Theorem 2: new variance smaller than original variance

We can also numerically check theorem 2. We define a new function that represents the difference of the new and the old variance:

varianceDifference[μ_, ω_, ν_, τ_,λ_,α_]:=variance[μ, ω, ν, τ,λ,α]-ν

If this difference is always below zero in the domain with high variance, then the new variance is smaller than the old variance, which is the statement of the theorem:

Maximize[{varianceDifference[μ, ω, ν, τ,λ01,α01],-1≤μ≤1 && -0.1 ≤ω≤0.1 && 3≤ν≤ 16 &&  0.8≤τ≤1.25},{μ, ω, ν, τ}]

The maximal value of the function is smaller than zero, therefore the numeric check of the statement is ok.

Numeric checks for Theorem 3: new variance larger than original variance

Similarly, we now check the variance difference function in the domains of low variance. If this difference is always above zero in these domains, then the new variance is larger than the old variance, which is the statement of the theorem:

Minimize[{varianceDifference[μ, ω, ν, τ,λ01,α01],-0.1≤μ≤0.1 && -0.1 ≤ω≤0.1 && 0.02≤ν≤ 0.16 &&  0.8≤τ≤1.25},{μ, ω, ν, τ}]

Minimize[{varianceDifference[μ, ω, ν, τ,λ01,α01],-0.1≤μ≤0.1 && -0.1 ≤ω≤0.1 && 0.02≤ν≤ 0.24 &&  0.9≤τ≤1.25},{μ, ω, ν, τ}]

The minimal value of these functions is larger than zero, therefore the numeric check of the statement is ok.