Is deep learning theory fundamental?
I heard several times that one of the fundamental/open problems of deep learning is the lack of "general theory" on it, because, actually, we don't know why deep learning works so well. Even the Wikipedia page on deep learning has similar comments. Are such statements credible and representative of the state of the field?
It probably depends on what one means by "fundamental theory", but there is no lack of rigorous quantitative theory in deep learning theory, some of which is very general, despite claims to the contrary. One good example is the work around energy-based methods for learning. See e.g. Neal & Hinton's work on variational inference and free energy: http://www.cs.toronto.edu/~fritz/absps/emk.pdf Also this guide to energy minimization as a "common theoretical framework for many learning models" by Yann LeCun and colleagues: http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf
And a general framework for energy-based models by Scellier and Bengio: https://arxiv.org/pdf/1602.05179.pdf There is also Hinton & Sejnowski earlier work which shows analytically that a particular Hopfield-inspired network + unsupervised learning algorithm can approximate Bayes-optimal inference: https://papers.cnl.salk.edu/PDFs/Optimal Perceptual Inference 1983-646.pdf There are many papers linking deep learning with theoretical neuroscience as well, such as the following, which shows that the effects of backpropagation can be achieved in biologically plausible neural architectures: https://arxiv.org/pdf/1411.0247.pdf Of course there are many open questions and no single, uncontroversial unified theory, but the same could be said of almost any field.