Retargeting and Optimization

And solving the linear equation system when the system matrix is a triangular matrix is very efficient!

# Normal equations:

(J^TJ)x = J^Ty

# This is exactly our Ax = b where:

A = J^TJ # symmetric positive definite!

b = J^Ty

# Solve using Cholesky:

A = LL^T # Cholesky decomposition

Ly = b # Forward substitution

L^Tx = y # Backward substitution

 

# For n×n system:

General matrix: O(n³) # Using Gaussian elimination

Triangular matrix: O(n²) # Using forward/backward substitution

 

So why is it a bad idea to do Gaussian Newton to optimize a neural network instead of gradient descent? When should and should not I use Gaussian Newton?