L-BGFS in Spark

package org.apache.spark.mllib.optimization

LBFGS

Study materials:

http://aria42.com/blog/2014/12/understanding-lbfgs/

Mathematics:

1. https://en.wikipedia.org/wiki/Gradient
2. https://en.wikipedia.org/wiki/Hessian_matrix
3. https://en.wikipedia.org/wiki/Taylor_series
4. https://en.wikipedia.org/wiki/Positive-definite_matrix
5. https://en.wikipedia.org/wiki/Quasi-Newton_method

Paper

• http://papers.nips.cc/paper/5333-large-scale-l-bfgs-using-mapreduce.pdf

Aja Notes

Jacobian Matrix -> Hessian Matrix -> BGFS -> L-BGFS

In class LBFGS CachedDiffFunction is used, 

CachedDiffFunction vs DiffFunction

• DiffFunctions per se don't have any caching whereas CachedDiffFunction is a simple wrapper around any DiffFunction that caches the last input and its result.
• In case the CachedDiffFunction gets called with the same input, it returns the result of the previous evaluation of the inner DiffFunction instead of re-evaluating.
• Especially in the various Minimizer implementations, DiffFunctions can get called multiple times with the same input, so in that case CachedDiffFunctions help a lot
• if the evaluation of your goal function is expensive.