
Innovation
It is Not About Deep Learning, But Learning How to Represent
10/10/2017
It seems appropriate to reflect and think about what Deep Learning has actually brought to the Machine Learning community, and separate the wheat from the chaff.
Recently, we setup a workgroup dedicated to Deep Learning (DL). Workgroups offer opportunities to share internally ideas, concepts, resources, code, etc. Additionally, they are meant to promote the use of Machine Learning at BBVA. I remember vividly how José Antonio Rodríguez, one of the impellers of this workgroup, told us back then: “We should call it the workgroup on representation learning, rather than on deep learning”. Today, I am convinced his perspective was correct: a crucial aspect of DL is that it helps make better representations (abstractions) of nature -and business, for that matter.
So, given all the hype that is surrounding DL these days, I thought it would be timely to reflect and think about what DL has actually brought to the Machine Learning community, and separate the wheat from the chaff. In this post, I will try to highlight the importance of representation learning for the development of Artificial Intelligence, and describe the role of DL as a representation learning framework.
Representing Objects and Concepts
One of the fundamental problems in Machine Learning is how a machine should represent an object/concept. Machines only understand numbers, so we need to find a way to condensate all the information about an object in a set of numbers (i.e. a vector). Roughly speaking, the vector would be the representation -I’ll be more precise later on-. Such representation can later be used to perform different tasks; whether we want to classify the object, generate new ones or do anything else, it’s not important for our discussion. In any case, we will need a proper representation. Let me explain what a representation is with a simple example: an apple. The machine needs to find a set of numbers (a vector) that properly represents what makes an apple an apple: its texture, color, taste, smell, shape, etc. Moreover, the representation should reflect different contexts where the concept apple appears: it shouldn’t appear in the space (not often, hopefully), although it is common to observe an apple hanging from a tree. This generic representation of the apple can later be used to create new apples with different brightness, shadows and so on -each particular apple will be a realization of the abstract representation of the concept apple-. Or to paint a still life that contains a beautiful and shiny apple at the centre. As we said above, the task doesn’t matter for this discussion; what is important is that the machine needs a proper representation of the concept apple. As some of you might be wondering at this point, can all apples be represented by a single vector? Certainly not. We need a complete set of vectors, that will be most likely similar -for some definition of similarity, that I leave for another post-. The (Riemannian) space where these apple vectors live in is a mathematical structure known as a manifold (see subsection 5.11.3 in the Deep Learning book and the excellent post of Christopher Olah). Thus, the task of any representation learning technique is to approximate such manifolds.Deep Neural Networks



Next steps
Coming back to the definition of DL, are the hierarchical and compositional aspects the whole story? Well, not really. One thing that, in my opinion, has made DNNs so successful is the incorporation of symmetries. Symmetries have allowed DNNs to overcome the curse of dimensionality and, in turn, being able to select a small family of functions from the infinite set of possible ones. In other words, they have allowed computers to learn representations from data efficiently and effectively . But this is another -fascinating, though- story. I will leave the discussion on the curse of dimensionality, and the importance of incorporating symmetries while learning, for another post. While we work on it, you might want to have a look at the work of Stephane Mallat and Joan Bruna on Scattering Convolutional Networks, that provides a mathematical framework for understanding Deep Convolutional Nets, and the recent work on the connection between Deep Learning and Physics by Lin, Tegmark and Rolnick. Hopefully after this reading, you will get a clearer view of what Representation Learning means, and how Deep Learning fits into this discipline.Thanks to
Iskra Velitchkova, Leonardo Baldassini, Fabien Girardin, José Antonio Rodríguez, Roberto Maestre, Juan Ramón Duque, Marco Creatura and Antonio Moneo.