Descenso de gradiente estocástico (Spanish Wikipedia)

Finkel, Jenny Rose; Kleeman, Alex; Manning, Christopher D. (2008). «Efficient, Feature-based, Conditional Random Field Parsing». Proc. Annual Meeting of the ACL.

archive.org

Bottou, Léon (1998). «Online Algorithms and Stochastic Approximations». Online Learning and Neural Networks. Cambridge University Press. ISBN 978-0-521-65263-6.
Ferguson, Thomas S. (1982). «An inconsistent maximum likelihood estimate». Journal of the American Statistical Association. JSTOR 2287314 77 (380): 831-834. doi:10.1080/01621459.1982.10477894.
Robbins, Herbert; Siegmund, David O. (1971). «A convergence theorem for non negative almost supermartingales and some applications». En Rustagi, Jagdish S., ed. Optimizing Methods in Statistics. Academic Press. ISBN 0-12-604550-X.
Spall, J. C. (2003). Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control. Hoboken, NJ: Wiley. Sections 4.4, 6.6, and 7.5. ISBN 0-471-33052-3.
Bhatnagar, S.; Prasad, H. L.; Prashanth, L. A. (2013). Stochastic Recursive Algorithms for Optimization: Simultaneous Perturbation Methods. London: Springer. ISBN 978-1-4471-4284-3.
Ruppert, D. (1985). «A Newton-Raphson Version of the Multivariate Robbins-Monro Procedure». Annals of Statistics 13 (1): 236-245. doi:10.1214/aos/1176346589.

arxiv.org

Toulis, Panos; Airoldi, Edoardo (2017). «Asymptotic and finite-sample properties of estimators based on stochastic gradients». Annals of Statistics 45 (4): 1694-1727. doi:10.1214/16-AOS1506.
Zeiler (2012). «ADADELTA: An adaptive learning rate method». arXiv:1212.5701 [cs.LG].
Diederik (2014). «Adam: A Method for Stochastic Optimization». arXiv:1412.6980 [cs.LG].
Hu, Yuzheng; Lin, Licong; Tang, Shange (20 de diciembre de 2019). Second-order Information in First-order Optimization Methods.
Reddi; Kumar (2018). «On the Convergence of Adam and Beyond». arXiv:1904.09237.
Loshchilov, Ilya; Hutter, Frank (4 de enero de 2019). Decoupled Weight Decay Regularization.
Byrd, R. H.; Hansen, S. L.; Nocedal, J.; Singer, Y. (2016). «A Stochastic Quasi-Newton method for Large-Scale Optimization». SIAM Journal on Optimization 26 (2): 1008-1031. doi:10.1137/140954362.

books.google.com

Bottou, Léon; Bousquet, Olivier (2012). «The Tradeoffs of Large Scale Learning». En Sra, Suvrit; Nowozin, Sebastian, Wright, Stephen J., eds. Optimization for Machine Learning. Cambridge: MIT Press. pp. 351-368. ISBN 978-0-262-01646-9.

boostedml.com

«Gradient Descent and Momentum: The Heavy Ball Method». 13 de julio de 2020.

bottou.org

leon.bottou.org

Bottou, Léon; Bousquet, Olivier (2008). «The Tradeoffs of Large Scale Learning». Advances in Neural Information Processing Systems 20: 161-168.

deeplearningbook.org

Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep Learning. MIT Press. p. 291. ISBN 978-0262035613.

doi.org

dx.doi.org

Ferguson, Thomas S. (1982). «An inconsistent maximum likelihood estimate». Journal of the American Statistical Association. JSTOR 2287314 77 (380): 831-834. doi:10.1080/01621459.1982.10477894.
Bilmes, Jeff; Asanovic, Krste; Chin, Chee-Whye; Demmel, James (Abril de 1997). «Using PHiPAC to speed error back-propagation learning». ICASSP (Munich, Alemania) 5: 4153-4156. doi:10.1109/ICASSP.1997.604861. «1997 IEEE International Conference on Acoustics, Speech, and Signal Processing».
Kiwiel, Krzysztof C. (2001). «Convergence and efficiency of subgradient methods for quasiconvex minimization». Mathematical Programming, Series A (Berlin, Heidelberg: Springer) 90 (1): 1-25. ISSN 0025-5610. doi:10.1007/PL00011414.
Darken, Christian; Moody, John (1990). «Fast adaptive k-means clustering: some empirical results». Int'l Joint Conf. on Neural Networks (IJCNN). doi:10.1109/IJCNN.1990.137720.
Toulis, Panos; Airoldi, Edoardo (2017). «Asymptotic and finite-sample properties of estimators based on stochastic gradients». Annals of Statistics 45 (4): 1694-1727. doi:10.1214/16-AOS1506.
Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (8 de octubre de 1986). «Learning representations by back-propagating errors». Nature 323 (6088): 533-536. doi:10.1038/323533a0.
Borysenko, Oleksandr; Byshkin, Maksym (2021). «CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing». Scientific Reports 11 (1): 10705. PMID 34021212. doi:10.1038/s41598-021-90144-3.
Polyak, Boris T.; Juditsky, Anatoli B. (1992). «Acceleration of stochastic approximation by averaging». SIAM J. Control Optim. 30 (4): 838-855. doi:10.1137/0330046. Archivado desde el original el 12 de enero de 2016. Consultado el 12 de julio de 2023.
Naveen, Philip (9 de agosto de 2022). FASFA: A Novel Next-Generation Backpropagation Optimizer. doi:10.36227/techrxiv.20427852.v1. Consultado el 19 de noviembre de 2022.
Byrd, R. H.; Hansen, S. L.; Nocedal, J.; Singer, Y. (2016). «A Stochastic Quasi-Newton method for Large-Scale Optimization». SIAM Journal on Optimization 26 (2): 1008-1031. doi:10.1137/140954362.
Spall, J. C. (2000). «Adaptive Stochastic Approximation by the Simultaneous Perturbation Method». IEEE Transactions on Automatic Control 45 (10): 1839−1853. doi:10.1109/TAC.2000.880982.
Spall, J. C. (2009). «Feedback and Weighting Mechanisms for Improving Jacobian Estimates in the Adaptive Simultaneous Perturbation Algorithm». IEEE Transactions on Automatic Control 54 (6): 1216-1229. doi:10.1109/TAC.2009.2019793.
Ruppert, D. (1985). «A Newton-Raphson Version of the Multivariate Robbins-Monro Procedure». Annals of Statistics 13 (1): 236-245. doi:10.1214/aos/1176346589.
Amari, S. (1998). «Natural gradient works efficiently in learning». Neural Computation 10 (2): 251-276. doi:10.1162/089976698300017746.

harvard.edu

seas.harvard.edu

Pfeffer, Avi. «CS181 Lecture 5 — Perceptrons». Harvard University.

ui.adsabs.harvard.edu

Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (8 de octubre de 1986). «Learning representations by back-propagating errors». Nature 323 (6088): 533-536. doi:10.1038/323533a0.

ieee.org

ieeexplore.ieee.org

Bilmes, Jeff; Asanovic, Krste; Chin, Chee-Whye; Demmel, James (Abril de 1997). «Using PHiPAC to speed error back-propagation learning». ICASSP (Munich, Alemania) 5: 4153-4156. doi:10.1109/ICASSP.1997.604861. «1997 IEEE International Conference on Acoustics, Speech, and Signal Processing».

issn.org

portal.issn.org

Kiwiel, Krzysztof C. (2001). «Convergence and efficiency of subgradient methods for quasiconvex minimization». Mathematical Programming, Series A (Berlin, Heidelberg: Springer) 90 (1): 1-25. ISSN 0025-5610. doi:10.1007/PL00011414.

jmlr.org

Duchi, John; Hazan, Elad; Singer, Yoram (2011). «Adaptive subgradient methods for online learning and stochastic optimization». Journal of Machine Learning Research 12: 2121-2159.
Gupta, Maya R.; Bengio, Samy; Weston, Jason (2014). «Training highly multiclass classifiers». JMLR 15 (1): 1461-1492.

lecun.com

yann.lecun.com

Lecun, Yann A. et al. (2012). «Efficient backprop." Neural networks: Tricks of the trade.». Springer Berlin Heidelberg.

mlr.press

proceedings.mlr.press

«SignSGD: Compressed Optimisation for Non-Convex Problems». 3 de julio de 2018. pp. 560-569.

nih.gov

ncbi.nlm.nih.gov

Borysenko, Oleksandr; Byshkin, Maksym (2021). «CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics with Simulated Annealing». Scientific Reports 11 (1): 10705. PMID 34021212. doi:10.1038/s41598-021-90144-3.

openreview.net

Balles, Lukas, Hennig, Philipp (15 de febrero de 2018). «Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients».

oreilly.com

«4. Beyond Gradient Descent - Fundamentals of Deep Learning [Book]».

paperswithcode.com

«Papers with Code - Nesterov Accelerated Gradient Explained».

probml.github.io

Murphy, Kevin (2021). Probabilistic Machine Learning: An Introduction. MIT Press. Probabilistic Machine Learning: An Introduction. Consultado el 10 de abril de 2021.

ruder.io

«An overview of gradient descent optimization algorithms». 19 de enero de 2016.

seg.org

library.seg.org

Krebs, Jerome R.; Anderson, John E.; Neelamani, Ramesh; Lee, Sunwoong; Baumstein, Anatoly; Lacasse, Martin-Daniel. «Fast full-wavefield seismic inversion using encoded sources». Geophysics 74.

semanticscholar.org

Dozat, T. (2016). «Incorporating Nesterov Momentum into Adam». S2CID 70293087.

toronto.edu

cs.toronto.edu

Hinton, Geoffrey. «Lecture 6e rmsprop: Divide the gradient by a running average of its recent magnitude». p. 26. Consultado el 19 de marzo de 2020.

towardsdatascience.com

«Understanding RMSprop — faster neural network learning». Consultado el 2 de setiembre de 2018.

ufl.edu

meyn.ece.ufl.edu

Polyak, Boris T.; Juditsky, Anatoli B. (1992). «Acceleration of stochastic approximation by averaging». SIAM J. Control Optim. 30 (4): 838-855. doi:10.1137/0330046. Archivado desde el original el 12 de enero de 2016. Consultado el 12 de julio de 2023.

utoronto.ca

cs.utoronto.ca

Sutskever, Ilya; Martens, James; Dahl, George; Hinton, Geoffrey E. (Junio de 2013). «On the importance of initialization and momentum in deep learning». En Dasgupta, Sanjoy, Mcallester, David, ed. In Proceedings of the 30th international conference on machine learning (ICML-13) (Atlanta, GA) 28 (1139–1147). Consultado el 14 de enero de 2016.
Sutskever, Ilya (2013). «Training recurrent neural networks (Ph. D.)». University of Toronto: 74.

web.archive.org

Polyak, Boris T.; Juditsky, Anatoli B. (1992). «Acceleration of stochastic approximation by averaging». SIAM J. Control Optim. 30 (4): 838-855. doi:10.1137/0330046. Archivado desde el original el 12 de enero de 2016. Consultado el 12 de julio de 2023.

worldcat.org

Whye, Schwarz, Jonathan Jayakumar, Siddhant M. Pascanu, Razvan Latham, Peter E. Teh, Yee (2021). Powerpropagation: A sparsity inducing weight reparameterisation.